When I came out to my close group of friends, one of them was telling his wife - which, by the way, was funny: he braced her for this "very serious discussion," and then of course not only was his wife not in the slightest shocked by the whole thing, but she was totally happy for me.
Anyhow, while telling her, he admitted to her that the part that was bothering him the most was, what if I decided at some point that I liked men, and what if I was attracted to him. His wife, point blank, said, "Are you seriously egotistical enough to think that if shidoshi ends up liking guys that out of every possible guy out there he's going to go after you?"
So, yeah, that's where a lot of the fear comes from in regards to men, on both the gay and trans sides: men think way, way too highly of themselves, and assume that everybody is going to be hot for them. *laughs*