Arguably there are social downsides in the short term. Case in point being COVID. If you refuse to take the vaccine, there were social downsides.
Other than that, I can't really see any downsides. Definitely sleep better at night.
Discussion
But even the social downside bit is subjective, I guess, depending on your views.
Yeah, the social consequences when the group is participating in a lie can be excommunication.
And in some cases, excommunication is death. And that can be ok if people who also want to defect see it and choose the same eventually to turn the herd. You'd have to make that choice wisely.