All bullshit aside, I agree with the notion being a pedo and liking cute anime characters are not the same thing. They're not even close. In fact, I almost find it a little disturbing that somebody would make this big if a leap between the two.
At the same time, people are going to judge you regardless. It comes with the territory. Doesn't make it right, but people are people. We're all prone to ignorance in one way or another. Explaining it may help in some cases and it may not. You shouldn't sweat it. Cajunator has the right attitude. He likes the stuff, he's honest (hiding the fact that you like cute anime character is fine too) and willing to explain it, but he doesn't let people not "getting it" get to him either. Unless someone starts to consistently bully you about your hobbies (in which case, they're just an asshole that should be ignored anyway), you shouldn't worry about it.
I think it's one of my redeeming qualities.