"[women] should lower their gaze and guard their modesty; they should not display their beauty and ornaments except what (must ordinarily) appear thereof; they should draw their khimār over their breasts and not display their beauty"
why doesn't it apply to men though?
I think you're just minimizing the issue (outright discrimination), reducing it to a matter of traditional attires.
It's apply to men as well, it's the verse just before:
"Tell the believing men to lower their gaze and guard their modesty. That is purer for them. Indeed, Allah is Acquainted with what they do."
There is a general ruling in the interpretation of the Quran that anything that apply to women apply to men, except if there is a specific mention that it doesn't.
Like in this verse, the punishment also go for the one who accuse a man of adultery without the four witnesses, but the verse only speak about women:
"And those who accuse chaste women (of adultery) and then do not produce four witnesses — lash them with eighty lashes and do not accept from them testimony ever after. And those are the defiantly disobedient. Except for those who repent thereafter and reform, for indeed Allah is Forgiving and Merciful." [Quran 24: 2-5]"
(You can understand why you hardly find any report of punishment for adultery in islamic history)
It's of course not an accident that one verse speak about women and another about men, since women would generally more likely to be accused of such things and men will get free pass.
Also the modesty of women in the public sphere is emphasized because, we could say, Islam is against that a society equate the value of a woman with her attractiveness. It was the case in pre-Islamic arabia and we could argue that it is the case nowadays in the West, just look about makeup or clothing budget gender wise.
Clothing-wise, all traditional wearing for men is as covering as the women one. With westernization, women tend to maintain traditional clothing, especially if it have a religious base. If you go in Bolivia, it's the same. A lot of women is wearing traditional garment and the men are not.
In any case, it's prohibited for a man to wear anything revealing, even the shape of the thighs.
It's not like in the West we don't make a difference between men and women in clothing, in many countries you can go shirtless in the street if you're a man but not as a woman, why? Without mentioning the topless situation in many western beach. I believe it's widely forbidden in USA, no ?