People of color can and are racist towards other people of color all the goddamned time. See; virtually every black cop. Blacks, asians, etc. absorb white supremacist ideology the same way white people do. But when a person of color is shitty to a white person, it's not racism, it's just shittyness. That's because people of color have no access to a power structure like white supremacy that can amplify the harm they do to a white person:
https://www.youtube.com/watch?v=6CmzT4OV-w0
It seems that at this point, the argument is simply about what is the correct definition of racism. So, it's semantics. If you define racism as white supremacy and nothing else, you're right. You can define it any way you want. Here's the thing, though...
Words don't have "correct" definitions. They only have mutually accepted definitions that are currently used.
The word 'racism', by itself, is just a set of vocals and consonants. What it means is determined by what people understand it to mean. As it stands, most people understand the word 'racism' to mean discrimination and/or prejudice based on race. So a black person can be racist toward and Asian person, and a Hispanic can be racist towards a white person.
You can argue that your definition is the "correct" one till you're blue in the face, it won't make a lick of difference. You can't mount an argument for what the definition of a word should be, since all definitions are arbitrary.
So you can redefine it as long as everyone else understands the definition you're using, but I question your motivation for doing so. What's the benefit of using the word 'racism' only when it's about whites being shitty toward people of colour, and using something like 'racial prejudice' when it involves other groups? What's the significance?