It's really simple to just say, Men aren't allowed in Womens' restrooms and vice versa.

If you're jumping through hoops to avoid saying that it means someone controls you.