That was actually kind of my point and I actually agree that you can get away with a racey or offensive part of a joke if it is part of something larger and funnier. I actually mentioned Jim Jeffries has an entire bit in one of his stand up routines about getting caught saying one really offensive thing taken out of his bit even if all the stuff around it makes sense, I believe he used,"God hates fags" as his example.
That all said, I find a lot of arguments end up being,"There is context for what he says, leave him alone!" and this is true. Some of the things he's being called out for are being taken out of context but the issue is that many of them are still in incredibly poor taste, offensive, and not very funny in the first place. The social and political climate today is a hot one and people are not prone to taking even stupid racist jokes rather seriously considering all the fucked up shit that minorities of all kinds have to put up with on a daily basis.
Something having context doesn't mean its suddenly immune or free from criticism and having context doesn't mean its all fine and dandy either.