4a^2b^2-(a^2+b^2)^2
=(2ab)^2-(a^2+b^2)^2
=(2ab-a^2-b^2)(2ab+a^2+b^2)
=-(a-b)^2(a+b)^2.
a^5b-8a^3b^3+16ab^5
=ab(a^4-8a^2b^2+16b^4)
=ab(a^2-4b^2)^2
=ab(a-2b)^2(a+2b)^2.
4a^2b^2-(a^2+b^2)^2
=(2ab)^2-(a^2+b^2)^2
=(2ab-a^2-b^2)(2ab+a^2+b^2)
=-(a-b)^2(a+b)^2.
a^5b-8a^3b^3+16ab^5
=ab(a^4-8a^2b^2+16b^4)
=ab(a^2-4b^2)^2
=ab(a-2b)^2(a+2b)^2.