Probably because most only appeal to a niche market? Dungeon crawlers like Ys and Visual Novels/Whodunnits like 999, Danganronpa aren't exactly in high demand in the West, add that to the fact most exist on handhelds now..
I highly doubt it's because people think they are sexist. Games like Mortal Kombat do pretty well despite the fact such assertions could be made.
This might be one of the biggest reasons - many of the best Anime-styled games are currently on handhelds or in genres not popular with too many people. On the other hand, games like Ys, 999 and so on don't even pop up in general public discussion about Anime games. Even here on NeoGaf you have topics on a regular basis asking where the good Japanese games have gone and people assuming the only things coming from Japan nowadays are Final Lightning Fantasy and Metal Gear (both not really fitting the bill of "classic Anime look") and Nintendo, which somehow rarely counts in discussion about quality games from Japan because of "international appeal such as Pixar"...
People are just tsundere for anime.
"I-i-it's not like I s-s-secretly watch anime or something. D-don't get the wrong idea, OK!? I was just passing by, that's all!"
I paraphrase that and interpret it as something - probably completely - different but I also get the feeling a huge part of today's gaming audience is very self-conscious about what they play and what others might think they are playing. As is, Anime is not a "cool" thing to like these days, games are though. Of course, there are people who just dislike the overall most cliché stylistics of Anime in general (which is not only the character designs, but also very often the colour palettes used, the way it's animated, the timing of interaction/gags). Of course, there's Anime which does not even look much like the Anime cliché would suggest. But: I like Anime style, but e.g. I don't like military styled FPS/TPS games. There are probably 100 different sorts of military styled FPS/TPS games and even some who are not at all cliché but I wouldn't know since I don't invest my time in research of a topic not interesting to me at all (I got pointed towards Spec Ops, though). So I get not wanting to invest time in something you're not inherently interested in anyways.
And then there's the negative cliché of "half naked girls, stupid companion monsters, colorful bullshit storylines". Since these negative clichés are somehow often picked up upon in general discussion (it is sadly part of our meme culture to point at Japanese stuff and say "LOL Japan so perverted so crazy"), people who are not as informed about the topic might just think "OK, this is embarrasing and I should not appear playing stuff like this." That would be like Justin Bieber suddenly producing a CD full of genuinly good songs of the quality usually ascribed to Pink Floyd or David Bowie (for me personally that is) - yet I'd still think "I can't buy this CD, it's Justin Bieber!". And then there's the crowd who actually likes a certain Anime stylistic, mainly the 80s/90s action Anime of lore (Akira, Bubblegum Crisis, Ghost in the Shell and so on) - and I would mostly compare that style to what we get from companies such as Platinum and to a certain degree Metal Gear Solid.
To wrap it up, I think there are people who don't dislike the looks of Anime generally and would probably dig a good JRPG or Anime-style Action game but refrain from buying/playing it because of the negative clichés of the artform. Playing an Anime game in the dorm or in front of a group of friends might be seen as potentially embarrasing. This has nothing to do with the neccessary reality of things - there are embarrasing Anime productions, there are profound Anime productions, there are Anime productions which stand among the best entertainment options available today - but with the perception of the reality by the mainstream audience.
I kind of feel like I have a love/hate relationship with it sometimes. There's a lot of anime junk that puts me off, but there's some great stuff out there and I generally like the basic style, even from said junk that puts me off otherwise.
There's the aspect that even fans of the genre (and I think it is difficult to speak of Anime as a game genre but let's go with it) are conflicted. I feel very similiar to you in that regard and I probably have a very thick skin. I even enjoy the occasional fanservice (I don't mean only the sexual kind... but also exhibition of many details and backgrounds in a game about giant robots or crossreferences to other japanese products) in my games if the game itself is still quality. In general, as some others have pointed out, it would be best to look at the quality of the game first and the style second. But these are intervened and style (also the technical aspect of it) is part of the perceived quality for most people, myself included.
To answer the question of the thread, I think Anime-styled games should be taken equally serious as any other game out there. The general misconception of Anime in gaming culture is similiar to the misconception of "violence in video games" in mainstream media. If someone thinks of it in such broad terms, maybe he/she should look a bit deeper.
