When I was at University the Campus Republicans had some kind of panel about 9/11 or Muslims or something, I can't even remember the supposed topic, because it was four absolute fucking nutjobs saying extremely ridiculous things about all kinds of subjects and social groups. I think it was right to let the Campus Republicans embarrass themselves in that fashion, and in general, I think it's right to expose students to complete fucklords for the purpose of 1) showing them what some people actually believe and 2) letting them argue against it, either in their heads or out loud.
However, a speaker specifically targeting and harassing a student is so far beyond crossing the line that said speaker should be forcefully thrown out, immediately. The problem is knowing ahead of time whether a speaker would do such a thing. With Milo, it seems like students could make a strong case that he's likely to do something that shitty. But in general, I think it would be hard to predict this kind of thing ahead of time.
So in conclusion, I don't know.