People act like it is incompetence but to me this just seems like the inherent flaw in drawing conclusions from statistics and models. What happened in this campaign is the same thing which undermines so many conclusions based on economic, social, and even scientific models based on statistics.
Due to scarce resources in polling and modeling, you distill infinitely complex systems like how people are going to vote on election day to a few categories you can manageably address. You make assumptions about other factors in your model based upon historical precedent and trends. Basically you focus on things that have a wide range of possible outcomes and ignore the things that have a very predictable outcome. You say for example, we know with 95% confidence that black people will turn out at a rate of 60%+/- 4% and vote 95% for the democrat +/-1 5% so we are not going to do much to influence that group. We know that rural evangelicals turn out at 40% +/- 15% and vote 40% democratic +/- 10%. That is a high degree of variability so we are going to focus on that group.
Now you have tens if not hundreds of assumptions you make due to the fact you can only poll so many people a day, and you have even more assumptions you make that you don't even realize you are making. (A huge snowstorm doesn't shut down Philly turnout.) Even if results only have a 1% chance of being wrong, you have 100s of these so some of them are going to be wrong. What typically happens is they are not all wrong in the same direction, so you never even notice. But every now in then you get results outside of the model. And by the way, the field of statistics is all about quantifying the probability of these type of results. It is just people inherently interpret something that has a 99% certainty to meaning it has 100% certainty ignoring the fact that when you have 1000 things of 99% uncertainty that a handful of them are going to be out of range all the time.
The solution I see advocated is that you throw out what statistics and modeling tells you in favor of anecdotes. A lot but not all the critiques of the campaign seem based on this. They should have recognized these anecdotal examples as being indicative of flawed assumptions. Yet to me that seems worse, you would have a very reactionary campaign based on nothing. What basis do you have on trusting some anecdotes over others. Volunteer X says a 20 year democrat is going to vote for Trump, the sky is falling. Volunteer Y says a 20 year Republican is voting for Hillary, we are doing it guys!
TL/DR statistics and models based on historical trends cannot predict outcomes with 100% certainty because of limited historical data and limited budgets for polling, but it is better than running a campaign reacting to anecdotes.
The ignoring of primary results criticism I can see. But it is still a different situation. There is a huge difference between voting for Bernie and voting for Trump. But probably the real choice was voting for Hillary or not voting at all. What they probably should have done is put turnout numbers at the lower bound of possible outcomes, if not even lower to be conservative and run that as a worse case model. They probably thought Trump was so bad though that turnout couldn't possibly be as low as it ended up being.