∵X-Y=3,
∴X^2-2XY+Y^2=9,
又X^2+Y^2=13,
∴-2XY=-4,
XY=2
x³y-8x²y²+xy³
=XY(X^2-8XY+Y^2)
=XY[(X^2+Y^2)-8XY]
=2×[13-16]
=-6.