fx=ab-cos2x
=cos²x-3sin²x+2√3sinxcosx-cos2x
=cos²x-sin²x-cos2x+√3sin2x+1-2sin²x-1
=cos2x-cos2x+√3sin2x+cos2x-1
=√3sin2x+cos2x-1
=2(√3/2*sin2x+1/2*cos2x)-1
=2sin(2x+π/6)-1
最大值=2-1=1
最小值=-2-1=-3
∴值域是[-3,1]
(2)
f(C)=2sin(2C+π/6)-1=1/5
sin(2C+π/6)=3/5
∵C∈(π/6,π/3)
∴2C∈(π/3,2π/3)
2C+π/6∈(π/2,5π/6)
∴cos(2C+π/6)