a,b,x,y应该有个范围吧.
否则例如a = b = 2,S = 1,当x从左侧趋近-2时,y趋向-∞,xy没有最大值.
考虑最简单的情况:a,b,x,y > 0,为使这样的x,y存在,易见有S > ab.
此时根据均值不等式,有S = ab+2ay+2bx+4xy ≥ ab+4√(abxy)+4xy = (√(ab)+2√(xy))².
开方得√(ab)+2√(xy) ≤ √S,故√(xy) ≤ (√S-√(ab))/2,有xy ≤ (√S-√(ab))²/4.
当且仅当ay = bx时等号成立,代回解得x = (√(abS)-ab)/(2b),y = (√(abS)-ab)/(2a).