Notes don't matter if nobody sees them. Relays can't take away your notes but they can certainly curtail your reach if they're large enough, and I fear that specialized relays would augment this issue.
You see it on other Fediverse tech, where certain large instances control the most thriving communities for various topics. I think specialized relays would risk moving us backwards in that direction.