I think you’re blowing it out of proportion. Historically transgender people have been victimized by trans phobic straight men. In fact, a friend of mine murdered a transgender man who wanted to have sex with him on the beach. My friend got confused, lost his temper and .killed the transgender man. Of course he went to prison.
Don’t let your feelings control you - there are all types of people in the world.