That seems like an easy attack vector... Someone could make a bunch of connections with someone else's pub key to get them rate-limited