The idea of Gerudo town being a "safe haven" from men in ridiculous.
1. First of all, It's just a carry over from Ocarina of Time where the gender segregation was merely a justification for allowing them to have a stealth segment of the game. In BoTW it manifests itself as forcing you to do one additional quest before entering the town (and also putting on a silly outfit every time after when you want to enter).
2. Second of all, it reflects poorly on the Gerudo themselves. The idea that they could be in any way threatened by men that are half their size is crazy. And it really shows their prejudice when, even after calming Vah Naboris, they still can't allow the man who brought peace to their region. Throughout Hyurle, I can't recall a single Gerudo that was "afraid" of men, just bewildered and confused by them, which is point number three.
3. Gender segregation is counter-productive and harmful to the Gerudo. Everywhere you go you find Gerudo that are desperately trying to find a man, and many of them are hopeless and lonely because of it. Yet, there are men just outside of Gerudo Town wishing they could hook up with the Gerudo women. Why separate them? It only takes a little bit of talking to see how ineptly prepared many of the Gerudo are for dealing with men. It's because they're being trained by women who are also unmarried, and away from men... Blind leading the blind. It's like a man learning to court women entirely on 4chan.
So what do I read from the Gerudo civilization? Segregate yourselves at your own peril. It comes off more as a cautionary tale on safe spaces. Don't get me wrong, their culture is interesting and fun, but I view their women-only rule as a fault, not something to admire.