You are missing the point.
I have an Asian friend that dislikes all black people because he was mugged three times in his life, all by black people. I can say that he is racist while still understanding that why he feels that way. Yes, I do think he's racist. Yes, I do wish he didn't feel that way.
On the culture/context note, my grandma, and many older gen Taiwanese/Asian people, dislike the Japanese because of what happened in the war. It is understandable, but it ends up still being racist (they dislike Japanese people because they are Japanese and it is assumed that "they" somehow being Japanese contributes to the likelihood of a personality willing to do that.)
So, it isn't that I think it's unfounded to feel that all nonblack people must use the n word in a racist/demeaning manner, because I know they used to and some still do now, it doesn't change the fact that to assume ALL people of x color must be like y because of their skin color...is racist.
You are right. I guess it depends on if you think that it is okay to do it, or not.
Just as a note, some women don't think that it's okay to do that, and it HAS been argued that calling each other bitch, even in a good way, does make men think it is okay to use it against women. I don't want to turn this into a gender topic, but I would like to point out that there is also different viewpoints on whether or not this is 'okay.'