It would be like telling a crude death joke about a parent to a party where there is someone you know just had a parent die. Sure, it's totally within your right to tell the joke. And while the joke isn't aimed at this person, you are still an asshole (insensitive) for telling it since you knew about him and his histroy. OKAY maybe that's not the best example.
But point is, things have a history and meaning. It MEANS something to people. You have the right to do it, but ignoring how it effects other people is really messed up (and at least makes you a jerk). One issue that really bugs me about racism is this:
I get it, you are from a new generation. You have been raised your entire life to NOT be racist, to not see color etc. You live in a place where this isn't that much racism, and diversity is standard. These kind of people then sometimes don't understand the depth of racism. And then these same people can start to get resentment, and feel like people are holding to the past too much. They also feel like they are being persecuted since THEY didn't cause slavery, and THEY didn't have relatives that were apart of that.
The problem with that is, well I. Racism still happens all the time today. Even if you aren't the one perpetrating it, or you don't experience it, it still happens. II. Even if in an ideal world racism was for the most part over (it isn't), do you really think that the lingering effects of racism still aren't present today? An entire people were basically forced to live in specific areas, and limited to the jobs they could get in the market. Just imagine how even white immigrants, it took their family decades to get into the middle class. You magically expect an entire people that were deprived of this, to magically have the same opportunities as your family/history when racism was still a major issue up to the 60s? Please.
History matters, as it still impacts things today. It's non-sense when you ignore that.