This is a solid hypothesis, but it still needs to differentiate itself from other "deal-breaker" preferences (physical, mental, and emotional) that manifest themselves on a wide scale. Overweight women, skinny men, tall women, short men, pimples, nerdy voice, nerdy hobbies, too much makeup, shrilly voice, laughter with a snort, leathery skin, old people, ears poke straight out, nose points up showing nostrils, outdated clothing style, strong body-odor, unkempt facial hair, baldness, buck teeth, teeth with large gaps, etc.
There are many, many, many preferences that people consider "deal breakers" on a mass scale. Some of those I listed are pretty easy to differentiate (e.g. pimples indicate infection, whereas race does not indicate a lack of health), but others are more difficult to differentiate (e.g. ears poking straight out, laughter with a snort, and a shrilly voice have little to do with survival, yet are widely seen by individuals as undesirable, and are potentially deal-breakers for many).
I'm not suggesting that they can't be differentiated, though. Or perhaps you could say that there has also been systematic discrimination against individuals with shrilly voices, but we don't have a word for that -ism like we do with racism. But if we go that route, that's a whole lot of -isms. And considering how, historically, societal preferences abandoned have been replaced by different societal preferences (e.g. preference for overweight women being replaced by the preference for skinny women), labelling those all as discriminatory -isms seems to suggest an impossible task of eliminating all societal non-survival-related preferences.
Last point: Our ancestors who discriminated their mates based on skin color were naturally selected over those who did not (i.e. preference for black skin helped their offspring survive the intense sun exposure around the equator; preference for white skin helped their offspring survive the lack of sun-derived vitamin D in the northern areas like Europe). While this is not applicable today, and it is probably not genetically coded in us to prefer a black skin tone or a white skin tone, there IS a strong probability that it's genetically coded in us to view skin tone generally as a major factor for consideration in mate selection, and how we view that factor is probably largely societally-influenced (e.g. In the USA white-controlled media, "fair skin" was preferred, but then later replaced by a preference for "tan skin" rather than fair skin, which then was pejoratively termed "pasty").
TL;DR: Societally-influenced sexual preferences do not necessarily indicate a discriminatory -ism is afoot, although they could. But to assert the presence of one, we need a strong showing of evidence. (That evidence of discrimination likely exists with regard to a sexual preference against black skin. Mumei's probably got a study he can drop in here.)
