(1)
f(x)=a·b
=cos(2x-π/3)+(cosx+sinx)(cosx-sinx)
=cos2xcosπ/3+sin2xsinπ/3+(cosx)^2-(sinx)^2
=1/2cos2x+√3/2sin2x+cos2x
=√3/2sin2x+3/2cos2x
=√3(1/2sin2x+√3/2cos2x)
=√3sin(2x+π/3)
由2kπ-π/2≤2x+π/3≤2kπ+π/2,k∈Z
得 kπ-5π/12≤x≤kπ+π/12,k∈Z
f(x)递增区间为 [kπ-5π/12,kπ+π/12],k∈Z
(2)
f(A)=√3sin(2A+π/3)=√3/2
∴sin(2A+π/3)=1/2
又π/3