The trolls can't come without bringing legitimate users, and they lose power when legitimate users move here.
I wouldn't say "they won't touch nostr with a 10 foot pole" but maybe a 10 foot pole is the only way they will touch nostr while it's this small.
They will coordinate a massive attack when it has the right timing to put a setback on a critical mass snowball of growth. They do not want to accidentally trigger that snowball themselves.