I think it depends. I work and get legitimately dirty. I wash off, and sometimes need to scrub hard. It's better to do that than to leave all the crud on my skin.

But I've also been more choosey about the soap I use, and I think that alone is going to be better for your skin.