I don't know how more men don't get offended by "boys will be boys." Doesn't it imply that they have low standards and that their attraction towards others haven't matured since they hit puberty? Not to mention reinforcing sexist stereotypes that creates bias that men MUST find tits and asses attractive or else there's something wrong with them.
Straight, white guys are the most privileged group of people on the planet. We will always be judged on our own merits and the actions of others will never stick to us, thus any stereotypes that may be said about us can just be ignored because "they're not talking about me". I do find it sexist, stupid, and problematic, specifically with regards to excusing creepy and sexist behavior toward women, however I don't find it personally offensive because ultimately it doesn't affect me.
I think many men lack empathy and haven't evaluated their own privilege, so they think it works this way for everyone else and the stereotypes they face. I think this is why many seem to be enraged at the very notion of being offended, it's something they've never experienced.