Many will gather with their dad this weekend but Chris hates the day for a reason that makes TJ and Terry cry