I've WANTED to wear them under swim shorts (not trunks, as if I'd wear trunks), but never had as if anyone found out I'd look like the biggest loser ever.
The reason is the boner thing some people have said. I've been in a situation (which I've mentioned on GAF before in a completely different topic) where a few of my girlspacefriends decided it'd be really funny to made me hard in a pool. Needless to say, they had no problem. Then most of them left knowing I'd have to get out with a boner obvious if I wanted to go with them, but luckily one took pity on me and stayed until it went away.
However, that took bloody ages. There were kids in that pool! No way in hell could I have got out like that; forget ridicule and humiliation, it's the fear of being accused of being a paedophile that made me stay in there. Ever since then, I've been tempted by the underwear thing.
Of course, that was a couple years ago and I doubt I've been swimming more than twice since then, so it's not the biggest problem.