Setting boundaries is the most self-respectful thing you can do for yourself. It's only difficult the first time.
