They would need the secret key that only the bridge has, but yes. If someone broke into the server they could impersonate ActivityPub users. That's no different from any site though. Except that Nostr would make it hard or impossible to recover from a breach like that since compromising a key is permanent.