Well, you guys must have far more faith in your fellow man to restrain themselves than I do.
I don't trust anyone in a position of power to not abuse that power when given sufficient motivation to do so. And something as base as sexual desire being that motivation makes anyone who subscribes to it extremely circumspect in my view.
It also doesn't help that it's a very narrow view of what women are like, which is itself a form of sexism. Being 'cool' is not, for example, suddenly going to make him physically attractive, which is a problem he has from the off. I know that sounds terrible, but that let's be honest - there's not many people who are going to be interested in him that way just because of that aspect of his character. Which, naturally, leads me to the understanding that he may, at some point, become desperate enough to pull some seriously questionable bullshit down the line.
So yeah, not giving him the benefit of the doubt. Doubly so for how he's been portrayed in the series thus far.