Let's scale this down a sec:
Say you like to visit a specific forum on the internet. Maybe it's a subreddit, maybe it's a facebook group, maybe it's an actual, old-fashioned-ass messageboard.
That group is valued by you; for to the number of contributors, the level of discourse provided, and the quality of those contributions. It's an entertaining place with insightful exchange of ideas happening fairly frequently. It's got a good signal-to-noise ratio.
Part of the reason it has that ratio is because people with bad faith arguments, shitty trolling practices, and just flat out repulsive/destructive ideals are discovered and then removed from the discussion group. They still have their voice, they just have to use it somewhere else. Maybe they start up their own little spinoff forum where they and 15 other like-minded individuals sit around and circlejerk their sad, poisoned seed onto laminated photos of the "elites" who shunned them for their outlandish/stupid behavior. They still have the freedome to do whatever it is they do over in whatever corner of the internet they choose to gather and yawp at each other unhindered.
Now, if you've ever cheered a ban based on blatant misogyny, racism, homophobia, transphobia, and harassment; if you've applauded that ban, nodded sagely at it, or even gone so far as to notify a moderator of vile shit you've seen of that type - you more or less agree with the notion that some ideas no longer deserve the benefit of the doubt, and are not welcome on the platform you value, because you recognize how worthless those contributions are, and how worthless indulging their acting out really is, and how damaging to the community the normalization of that behavior can be.
Further, if you've ever suggested/applauded the notion of someone getting dinged for trying to come back to this platform they've already proven they don't deserve, a platform you know they're looking at as a means of exploitation for personal gain (of whatever value they've erroneously assigned in that case)—whether it be via sockpuppet, or alt-account, however you wanna term it—If you've ever thought that pre-emptive denial of access to this platform based on prior, easily observed behaviors, was a positive action that led to better discussion/signal-to-noise ratio...
...then how could you possibly give into the idea that denying Milo Yiannopolous access to Bill Maher's show is somehow detrimental to public discourse.
He's a troll trying to join a popular messageboard in the hopes he can normalize his bullshit for a percentage of the lurkers scrolling through, and then use his inevitable conflict/ejection as fuel for when he eventually returns to his homebase of already-lost assholes.
Essentially: How many of you, if you heard Richard Spencer wanted to join NeoGAF to talk about Star Wars and Zelda, (oh, and also the Lugenpresse and peaceful ethnic cleansing) would then suggest that the only proper course of action would be to let him do so?
And if so: Why?