"Who runs the load balancer" could be a problem at some point but since relays can't really do anything malicious I don't think I guess it isn't a terrible idea.
Discussion
I'm on that same line, contributors to the load balancing should be able to migrate to other load balancers quick when things turn tyrannical.