I mean the hate really isn't going to affect others, let's be honest about that. It's a symptom not the disease.
If you honestly believe the hat might cause someone danger then feel free to do something about it. I wouldn't question your action at that point, only your judgement.
Wear a hood of the KKK around some black people, and let me know how safe they feel. You could be the most peaceful person in the world, and of course a garment isn't going to hurt anyone. But it sure as hell is a harsh fucking reminder to the targets of that campaign that they're in the presence of somewhat that hates. fears, and otherwise wants them branded, rounded up, or kicked out entirely.