It's only bad when the few community members rally together, and begin thinking they're the majority, and start trying to control everything.
That being said, I bought a middle of the road priced "cabin" 10 years ago, for $240k and now they are selling for $800k. I've considered leaving, but it's my dream home, and I'd rather keep it in my family and rent it out, if needed in the future.
Rentals are one thing they are dying to control, but without rule changing and community votes, that's a dead initiative.