Not all calories are created equal. The thermic effect of food is one of the ways our bodies use energy throughout the day.
The effect of food varies from person to person. If you eat whole, nutritious foods, you should be just fine. Processed and packaged foods are where efforts get torpedoed.