As a Republican you can take this kind of bullshit and shove it straight up your ass.
Democrats love to toss the word Racism around as the ultimate copout. I remember in 2008 democrats saying this country as a whole was too racist to even elect a black man president. That was their copout if indeed Obama failed to get elected. Then when polls shifted in Romney's favor it was because the USA is too racists to re-elect a black man. That same dumb ass line has been used for EVERY SINGLE POLITICAL SITUATION ever since he has taken office. If he is unable to get his way, it only happened because whomever was against his ideas were racists. It is tiresome, and the ultimate copout. I really do not think there has been a debate since Obama was elected where race did not become a part of the conversation.
I can see why some people, who aren't racist, but are conservative, would find the claim of racism as the end-all-be-all excuse and dislike it being used in place of conversation. I totally get that.
However, any person with two eyes can see the certain fringe of the party that is undoubtedly racist, and despite seeing that, conservative representatives have thrown their lot in with that same group. And it's easy to then see how people could see that 2+2=racist.
What's worse, is it ISN'T much of a stretch to see why people would think that the GOP is racist, or has some underlying racism when it comes to it's decision making. The GOP of the last 5 years has been one that shits on the poor, the disenfranchised, women, gay people, immigrants, anyone that isn't christian, etc. They're practically the poster children of intolerance, so it's not hard to come to the conclusion that the same group pushing the the same examples of intolerance I mentioned might also be racist. Especially when you see pictures of people waving a confederate flag in front of the white house, and no one in those groups telling the those people to move the fuck on. They don't say "We do not want to be associated with you." They don't say "We dislike the president for X, Y, and Z, but because he's black isn't one of them." Instead, those people all come together in some weird, vitriolic hatred of a man that, for the last 5 years, has been the target of some of the most hateful speech I've ever seen. Outright lies that republicans refuse to condemn, and instead go along with. John McCain was the only republican in the republican party with enough of a sac to tell a woman that Obama wasn't a Muslim.
The GOP hasn't done anything to shed the badge of racism, nor do they seem to want to. So while I understand why some conservatives might get pissy about being labeled a racist, you are who you associate with. My mother taught me that. When I was 6.
So, I"m sorry it offends you that you might get called a racist, but throwing your hat in with people that seem to be racist will tend to get you labeled as one.