You don't have to come back.意思是：你没有必要一定回来.
或者：I didn't insist on your coming back.意思是：我没有坚持要求你回来.
希望楼主满意!