Anyone will always be able to add you to groups. It's similar to anyone will always be able to send you a notification. If they can't spam you via DM, they will spam you via notifications. There is no way to block people from citing your key. But we can hide them.