Yea that's my hunch. Just like they used to say in women's dieting books that feeling about eating something you shouldn't is worse than the eating itself.
I think it's the same here. I get so stressed feeling there's nothing I can or should eat when I go out. And then I get worried or feeling bad I did eat something. Not good.