the point shouldn't be to shelter people from hate speech. all hate speech does is delegitimize the people using it
there was an opportunity to show the viewers how abhorrent milo/bannon/breitbart are, and even a mediocre debater could have taken advantage of that
You can attack their ideas and their words without ever having them on the show, without ever giving them the platform. Colbert does that shit 5 days a week, Sam Bee does it weekly. You can attack their ideas and words without having them on, we see people do it all the time. The second you invite them on you've lost.
Also, as much as we'd like to think using hate speech delegitimizes those using it, it doesn't. It just doesn't. I wish it did, society would be better for it, but that's not how it works. If you want to make hate speech taboo you can't go around giving those who spew it a platform, you can't elevate the level of their hate to that of your ideals by engaging in debate.
You can't debate hate because it's irrational. It's not based in reality.