Imo its only annoying if the note isn't really about the tag. But many tags can be fine.

This issue could be mitigated if there was a way to minimize a note without unfollowing or blocking anything. Maybe there is and I just don't know...