Right. A better way of explaining it might be that "racism" has several negative connotations tied to it. So if someone says "affirmative action is racism," typically they aren't actually making any kind of interesting point--sure, by definition it's racism, so what?--but rather trying to bring in the negative connotations without actually having to argue for them. Which is why saying that affirmative action is racism generally gets pushback.
This is sort of a losing battle because it's really difficult to get people to ignore the connotations and argue what they're actually trying to argue, but yes, it would be nice if we could have discussions where we use phrases like "sure this is racism, but it's the good and consequentially useful kind of racism!" Unfortunately that one is pretty difficult to say without sounding weird or possibly insane.
Yeah. I kind of explained it above, but I think the dictionary definition of racism that most people adhere to (i.e. discrimination based on race) existing with the cultural notion that "racism" is always wrong is actually what's creating the friction, though. Usually, saying we're "arguing semantics" is a way to show how pointless the argument is, but "racism"--especially as it applies to affirmative action--is a rare case, I think, where semantics are hugely important.
Applying an analogy, to clarify how I see it damaging the conversation and creating a schism: A school cafeteria sign says "No butting in line. It's mean and wrong, period." The kids are taught that same thing, every day, and are told to recite it. One day, a parent with her son with a peanut allergy shows up and takes him in front of all the kids so he can get lunch and leave before the school opens up the peanut cans. "Hey, he butted!" the kids all shout. They feel offended. He was mean to them, because butting is always mean--it's self-evident. The mother and son know that, too; they hate butting just as much as anyone else. But they also know that what they did was perfectly okay. It can't be okay AND be butting, so if it was okay, it wasn't butting. The mom says, "We didn't butt; he has a peanut allergy." The kids shout, "Butting is cutting ahead of people who were waiting in line. That's what you did. So you butted! Butting is wrong, and you butted!" "No, it wasn't butting, because we were justified!"
Thus, instead of actually talking about whether the action is justified or not, the argument turns into fights where each side thinks the other is CLEARLY wrong and just doesn't get it. But if they could all acknowledge that butting is sometimes okay, the mom could just say, "Yeah, sorry for butting, but he has a peanut allergy and has to go first," then the kids can understand why they have to wait 15 seconds longer than they otherwise would have.
I seriously think the "racism is ALWAYS wrong" notion is creating a huge portion of the friction regarding affirmative action. Racism can't be right, both sides think. Those supporting it say, affirmative action is right, so it can't be racism. Those opposing it say, affirmative action is racism, so it can't be right.
We as a society seriously need to reconcile with ourselves that, in rare cases, racism can be right. The debate is going to be gravely hampered until we do.