f(x)=sin(2x+π/6)+sin²x
=sin2xcosπ/6+cos2xsinπ/6+(1-cos2x)/2
=(√3/2)sin2x+(1/2)cos2x+1/2-(1/2)cos2x
=(√3/2)sin2x+1/2,
(1)f(x)的最小正周期为π,
由π/2+2kπ≤2x≤3π/2+2kπ,得π/4+kπ≤x≤3π/4+kπ(k∈Z),
f(x)的单调减区间是[π/4+kπ,3π/4+kπ] (k∈Z).
(2)∵f(B/2)=(√3/2)sinB+1/2=(√2+1)/2,∴sinB=√2/√3,
∴cosB=±√(1-2/3)=±1/√3,
由余弦定理b²=a²+c²-2accosB,得
当cosB=1/√3时,5=a²+3-2a,即a²-2a-2=0,解得a=1+√3(负值舍去)；
当cosB=-1/√3时,5=a²+3+2a,即a²+2a-2=0,解得a=-1+√3(负值舍去),
故a=1+√3或a=-1+√3.