Electoral-Vote.com - FiveThirtyEight: An Assessment

Status
Not open for further replies.
So why should getting it right build trust?

by itself, it shouldn't. Over time with enough of them it should make it a better indication of quality, but we also need to look at the failures and see why it failed over a much longer period of time.
 
But even a Trump victory isn't evidence of a model being wrong. Let's say we play Russian Roulette and I give you a six-chamber gun and tell you there's an 83% chance that you'll survive. If then you die, that isn't evidence that my projection was wrong. I was also telling you you had a 17% chance of dying. Unlikely, but totally possible. Unlikely things happen.

Exactly. That's why we have uncertainty attached to these things.

The models always have some reasonable chance of being wrong, and it's impossible to fully estimate confidence. This is because we don't know what we don't know. The models are just based on past data and things we've observed, so at best we can say that given our history, X is likely.
 
An animated graph of all the major polling trackers. Silver's frequently diverges from the pack, especially in the last week.

https://twitter.com/jshkatz/status/793553664724140032
And to respond to the person I'm the other thread saying it was the nowcast in this graph, it wasn't. The nowcast is even more volitile, him thinking the polls only was a nowcast actually shows how it's a problem how volitile Silver's model is. A model should not swing from 50% chance to 80% chance in a relatively short span of time unless something major happens, which is not the case.


I think I figured out the problem. You want Nate to build a model that tells the future, Nate built a model that takes polls and calculates the chances of who's going to win.

Nate isn't in the business of telling the future, only what the current data says.
Then why not just have the nowcast? The whole point of polls only and polls plus is to predict November, which thr model clearly does no do. A model of the election should not be this volitile. Not in a race like this.

I guess I don't see instability as a sign of model weakness. Speaking subjectively, this has been a pretty unusual election! Emails, leaks, Russia, taxes, sexual assault, you name it. It's hard to forecast something as unusual as this year when there isn't really precedent for these kinds of things. So, I don't necessarily dismiss model uncertainty as bad math.

There are lots of other measures that we can use. i.e. historically high unfavorable numbers for both candidates, that would suggest uncertainty as well.That's not to say that I believe Nate's numbers to be gospel but I don't think being lower than the rest invalidates his findings.
Actually, it's been a remarkably stable year. Clibton has always led, by about a 4%, though the margin has changed from 2% to 8% over time. Every election since the 2000's has been far more stable than those before, because of hyper-partisanship.
 
But even a Trump victory isn't evidence of a model being wrong. Let's say we play Russian Roulette and I give you a six-chamber gun and tell you there's an 83% chance that you'll survive. If then you die, that isn't evidence that my projection was wrong. I was also telling you you had a 17% chance of dying. Unlikely, but totally possible. Unlikely things happen.
Then that means these aggregates can never be wrong. You can literally have any % chance for Hillary to win other than 0/100% and it would be correct.

How do you judge how good a poll aggregate is then?
 
The haters cover Nate Silver like he covers Trump. They follow everything he does with hyperbole and name drop his brand for clicks. Allow the data to stand on its own and let the man make a living off the masses.
 
Then that means these aggregates can never be wrong. You can literally have any % chance for Hillary to win other than 0/100% and it would be correct.

How do you judge how good a poll aggregate is then?

obviously if the prediction is only "Hillary wins" or "Trump wins" it is hard to measure them given there is only one experiment to judge from, ie: the actual election

but all these polls aggregators predict results state per state, and not every model gets every state right

once you have aggregators giving you the same results on every or almost every state, it becomes harder to differentiate and I think that is indeed a problem
 
Then that means these aggregates can never be wrong. You can literally have any % chance for Hillary to win other than 0/100% and it would be correct.

How do you judge how good a poll aggregate is then?

It's a hard problem!

The first thing you need to do is let go of "correct" as any kind of metric when it comes to forecasting. "Correct" isn't a thing.

But yeah. I've already said this but you can't evaluate a forecaster on a single prediction. You can first check the methodology and throw out everyone who's doing nonsense things. But after that, you gotta judge a forecaster on their track record. On a long horizon, unlikely events will wash out. It's easy to imagine if we keep the numbers constant: let's say I predict a 10% of rain ten times. If I'm a good forecaster, one of those times will see rain. But really, I should get ~10 in 100 or ~100 in 1000 before we declare me good.

People designing models will use this intuition. You could, say, build a model based on data from every other election, then use your model to forecast the remainder and check how often you're right. If your data set of election results is big enough, you can get a feel for how accurate your model is. But even this is tricky because the time horizon is so big here. A model that works well in the '50s might not work so well today as more and different data become available and underlying facts change. (For example, massive fundamentals changes like Women's Suffrage or Civil Rights could require totally different assumptions be built into a model.) And we just haven't had that many Presidential elections. You might not want to biforcate your data to make a testing set if doing by chance means train without many no-incumbent races, for example.

The big secret of "data science" is that the science is easy but there's a ton of art to doing it well.
 
his site seems to be very hyperbolic lately but man does he not get me worried.

if clinton was still strong heading into this stretch why did her campaign suddenly shift all resources back to the states she MUST win?
 
his site seems to be very hyperbolic lately but man does he not get me worried.

if clinton was still strong heading into this stretch why did her campaign suddenly shift all resources back to the states she MUST win?

She got a huge boost in campaign funds since this FBI debacle
 
Brexit taught me that polls are interesting but the worst thing you can do is start thinking any of them are infallible. No matter how scientific the method, at the end of the day it's still using a few thousand peoples stated voting intention to project that of 120+ million actual voters.

Best thing you can do is accept that you don't know who will win, no one does. All you can do is vote (and campaign if that's your thing) and let the results fall where they will.
 
his site seems to be very hyperbolic lately but man does he not get me worried.

if clinton was still strong heading into this stretch why did her campaign suddenly shift all resources back to the states she MUST win?
Because she has the resources to do so.
It would be stupid to be complacent, when you can both spend time and money in states you need to keep, and the ones you might be able to flip.
She also has a lot of heavy hitters that can target both kind of places, from Potus and Flotus, to Sanders, VP and Warren.
 
his site seems to be very hyperbolic lately but man does he not get me worried.

if clinton was still strong heading into this stretch why did her campaign suddenly shift all resources back to the states she MUST win?

She has a ton of campaign money, and getting more dems in offices benefits everyone including her.
 
his site seems to be very hyperbolic lately but man does he not get me worried.

if clinton was still strong heading into this stretch why did her campaign suddenly shift all resources back to the states she MUST win?


Clinton and Kaine have both held events in Arizona in the last 2 days, a state that hasn't gone blue since 20 years ago. It's been considered her most-realistic long-shot flip, but still a lofty goal. I think the latest tightening took off the table the truly far-reaching states like Georgia/Utah/Missouri/Indiana, but there's no sign of panic from the Clinton team at all. They aren't going back to Virginia for example as far as I know, which was a surprising flip in 2008 and still a major battleground in 2012.

The final week both camps have been focusing on Pennsylvania/Ohio/Iowa/Florida/North Carolina/Nevada/New Hampshire, and in no way whatsoever does Hillary need to win ALL of them. Trump basically does though.

The recent attention from the Clinton team on Michigan/Wisconsin is a liiitle bit eyebrow-raising, but Dems always fret about those states and then do fine (and polls seem...fine). And I think once the "what totally crazy states can we target?" idea becomes not worth it you might as well invest a little back in a better-safe-than-sorry thing with some of your "weaker" states. But I really don't see that as a retreat or a worrisome move at all.


I've been waffling between nervousness and confidence ever since the FBI/tightening stuff last week, but as far as the vibe the campaign itself is giving off I see no signs of alarm. And they have more accurate polling info than we see.
 
Brexit taught me that polls are interesting but the worst thing you can do is start thinking any of them are infallible. No matter how scientific the method, at the end of the day it's still using a few thousand peoples stated voting intention to project that of 120+ million actual voters.

Best thing you can do is accept that you don't know who will win, no one does. All you can do is vote (and campaign if that's your thing) and let the results fall where they will.

Polls showed the two outcomes for Brexit would flip-flop with each one leading at various times.

Clinton's pretty much been in the lead for for the entire election (it's by how much that's changed and it's only a few outliers that have Trump in the lead).

So they aren't really comparable.
 
here in Portugal news keep saying that polls are neck and neck lately, and you guys keep saying Clinton is safe. So which is it?
 
here in Portugal news keep saying that polls are neck and neck lately, and you guys keep saying Clinton is safe. So which is it?

Clinton is pretty safe. Early voting has been good for Hillary in states like Florida and North Carolina. She also has poll leads in Wisconsin, Michigan, and Pennsylvania. If she captures four of these states, the race is practically over. If she captures all of them, you can expect the senate to flip democrat and there will probably be a closer house of reps.

PollyVote, which has predicted the past three elections with incredible accuracy, their margin of error has been less then 1%, has Clinton winning the popular vote 53 to 47.

Here is their last three election predictions

Obama 2012:
Actual - 52.0%
Prediction - 51.3%

Obama 2008:
Actual - 53.7
Prediction - 53.9

Bush 2004:
Actual - 51.2
Prediction - 51.5

http://charts.pollyvote.com
 
Brexit taught me that polls are interesting but the worst thing you can do is start thinking any of them are infallible. No matter how scientific the method, at the end of the day it's still using a few thousand peoples stated voting intention to project that of 120+ million actual voters.

Best thing you can do is accept that you don't know who will win, no one does. All you can do is vote (and campaign if that's your thing) and let the results fall where they will.
Please stop bringing up Brexit. If you don't know the reasons why you should look up what the Brexit polls actually looked like, and compare them to what Hillary's poll's look like. There a huge differece. Britain leaving the EU is shocking like Trump winning the nomination was, in the sense that the polls said one thing but our guts another. That's absolutely 1000000000000000% NOT the case here.
 
Brexit taught me that polls are interesting but the worst thing you can do is start thinking any of them are infallible. No matter how scientific the method, at the end of the day it's still using a few thousand peoples stated voting intention to project that of 120+ million actual voters.

Best thing you can do is accept that you don't know who will win, no one does. All you can do is vote (and campaign if that's your thing) and let the results fall where they will.

I agree that I have some nervousness about polling error, but I reject the reasoning for your argument. Although sample sizes in individual polls may lead to somewhat large margins of error, this is exactly what poll aggregation is designed to combat. The aggregate sample sizes are more than sufficient to make accurate estimates for the population.

The area where there is room for doubt is if there is a systemic error in likely voter models being used by the polls. For instance if turnout among some demographic group is greatly different than expected by the bulk of pollsters. This sort of thing can be partially validated by early voting data, but it only goes so far since early voters are not exactly a representative sample.
 
his site seems to be very hyperbolic lately but man does he not get me worried.

if clinton was still strong heading into this stretch why did her campaign suddenly shift all resources back to the states she MUST win?

She has more money for her campaign than she could ever hope to spend.
 
whats interesting is that most major polling place projects a 85% to 99% probability that Clintoin wins. But 538 is at 60% right now? I wonder why 538 is on a such a outlier.
 
Clinton is pretty safe. Early voting has been good for Hillary in states like Florida and North Carolina. She also has poll leads in Wisconsin, Michigan, and Pennsylvania. If she captures four of these states, the race is practically over. If she captures all of them, you can expect the senate to flip democrat and there will probably be a closer house of reps.

PollyVote, which has predicted the past three elections with incredible accuracy, their margin of error has been less then 1%, has Clinton winning the popular vote 53 to 47.

Here is their last three election predictions

Obama 2012:
Actual - 52.0%
Prediction - 51.3%

Obama 2008:
Actual - 53.7
Prediction - 53.9

Bush 2004:
Actual - 51.2
Prediction - 51.5

http://charts.pollyvote.com

Wow, that site has Florida clearly in the blue. While I certainly hope Florida ends up going to Clinton, putting it as squarely Democrat right now seems a bit too optimistic.
 
http://projects.fivethirtyeight.com/2016-election-forecast/florida/

Florida is close, and there's a pretty insane spread in the recent polls. So that leads to a more uncertain position even off of a similar average lead.

http://elections.huffingtonpost.com...residential-general-election-trump-vs-clinton

She's led all year in the polling averages. The EV indicates that she might be out performing them because a large proportion of it is from new or lapsed women and Hispanic voters, who don't make it through likely voter screens. 55% of the EV is from women.

I think the spread we're seeing in the polls reflects the increase in polls conducted as election day nears, rather than increased uncertainty.
 
whats interesting is that most major polling place projects a 85% to 99% probability that Clintoin wins. But 538 is at 60% right now? I wonder why 538 is on a such a outlier.

Nate Silver has talked about it on their podcast. 538's model accounts for a larger potential systemic polling error than other models. That is the main reason. It may also be more aggressive about extrapolating perceived trends than others.
 
http://elections.huffingtonpost.com...residential-general-election-trump-vs-clinton

She's led all year in the polling averages. The EV indicates that she might be out performing them because a large proportion of it is from new or lapsed women and Hispanic voters, who don't make it through likely voter screens. 55% of the EV is from women.

I think the spread we're seeing in the polls reflects the increase in polls conducted as election day nears, rather than increased uncertainty.

I think the big dilemma for this year is disagreement on the composition of the electorate. Poll variance is generally quite a bit higher this year than last cycle, which implies that the results depend more greatly on assumptions than usual.

The flipside is that the spread may have been artificially low previously, due to herding.

Nate Silver has talked about it on their podcast. 538's model accounts for a larger potential systemic polling error than other models. That is the main reason. It may also be more aggressive about extrapolating perceived trends than others.

Yeah. Correlations between states and state polls and national polls is a really tough topic to model. I think part of Wang's overestimation of odds his simulations, and his mid-year change in expected volatility based on this year's polling volatility. I don't think that assumption is justified.
 
I think the big dilemma for this year is disagreement on the composition of the electorate. Poll variance is generally quite a bit higher this year than last cycle, which implies that the results depend more greatly on assumptions than usual.

The flipside is that the spread may have been artificially low previously, due to herding.

Yeah. Correlations between states and state polls and national polls is a really tough topic to model. I think part of Wang's overestimation of odds his simulations, and his mid-year change in expected volatility based on this year's polling volatility. I don't think that assumption is justified.

Wang doesn't use National polls in his model to estimate state totals. A better direct comparison to 538 would probably be The Upshot, anyways. It's probably part of the reason why PEC has remained at 99 percent for a while now.
 
Just a heads up but Hillary has been trending upwards on 538 all morning. It appears that whatever was going on with their model might be correcting itself.
 
Just a heads up but Hillary has been trending upwards on 538 all morning. It appears that whatever was going on with their model might be correcting itself.

More polls. There's been a shortage of good quality polls the last week, which put more emphasis on their trend adjustments.

Also, PPP's batch of state polls for states that have been under polled helped a lot? Maybe? Wisconsin data has been sparse.
 
More polls. There's been a shortage of good quality polls the last week, which put more emphasis on their trend adjustments.

Yeah I noticed that as well. They're still incredibly conservative compared to most polls, but they're correcting in states where they clearly appeared to have a weird trend.

e: This is why I find the polling data infinitely more entertaining than the election itself (much how I love advanced stats in baseball more than watching games). The way things get calculated and the models and tools used to adjust and forecast is just so interesting to me. When did the PPP polls come out? E: Quick tweet with the hard numbers. This probably has a lot to do with it.

New @ppppolls dump of battleground state polling:
NH: Clinton 48-43
NV: Clinton 48-45
WI: Clinton 48-41
PA: Clinton 48-44
NC: Clinton 49-47
 
Yeah I noticed that as well. They're still incredibly conservative compared to most polls, but they're correcting in states where they clearly appeared to have a weird trend.

e: This is why I find the polling data infinitely more entertaining than the election itself (much how I love advanced stats in baseball more than watching games). The way things get calculated and the models and tools used to adjust and forecast is just so interesting to me. When did the PPP polls come out? E: Quick tweet with the hard numbers. This probably has a lot to do with it.

New @ppppolls dump of battleground state polling:
NH: Clinton 48-43
NV: Clinton 48-45
WI: Clinton 48-41
PA: Clinton 48-44
NC: Clinton 49-47

Like an hour ago. Trump recently had a great round of state polling in those states. PPP has softened some of those fears, despite them not being a great firm.
 
http://projects.fivethirtyeight.com/2016-election-forecast/florida/

Florida is close, and there's a pretty insane spread in the recent polls. So that leads to a more uncertain position even off of a similar average lead.

If any of the early voting polls are true - then Republicans are actually voting Clinton.

Every report I've seen thus far shows women at much higher participation %'s in this election than past ones.

It is not a stretch to assume that Republican-registered women are going out and voting Clinton.

In fact it would quite well explain that 28% of Republicans voting for Clinton thing we saw earlier this week in FL.
 
Yeah I noticed that as well. They're still incredibly conservative compared to most polls, but they're correcting in states where they clearly appeared to have a weird trend.

e: This is why I find the polling data infinitely more entertaining than the election itself (much how I love advanced stats in baseball more than watching games). The way things get calculated and the models and tools used to adjust and forecast is just so interesting to me. When did the PPP polls come out? E: Quick tweet with the hard numbers. This probably has a lot to do with it.

New @ppppolls dump of battleground state polling:
NH: Clinton 48-43
NV: Clinton 48-45
WI: Clinton 48-41
PA: Clinton 48-44
NC: Clinton 49-47

If Hillary wins North Carolina, it'll be an early Thanksgiving miracle.

I just don't believe it. They will stop at nothing to rig it for Donald. They already have.
 
Yeah I noticed that as well. They're still incredibly conservative compared to most polls, but they're correcting in states where they clearly appeared to have a weird trend.

e: This is why I find the polling data infinitely more entertaining than the election itself (much how I love advanced stats in baseball more than watching games). The way things get calculated and the models and tools used to adjust and forecast is just so interesting to me. When did the PPP polls come out? E: Quick tweet with the hard numbers. This probably has a lot to do with it.

New @ppppolls dump of battleground state polling:
NH: Clinton 48-43
NV: Clinton 48-45
WI: Clinton 48-41
PA: Clinton 48-44
NC: Clinton 49-47

Those polls are not yet included in their model. You can see the explicit adjustments here:
http://projects.fivethirtyeight.com/2016-election-forecast/updates/

By the way, I wouldn't rack my brain looking at that page and trying to figure out why each update moved the percentage slightly one way or another. It is a stochastic model, so you can probably expect random movement of +/- 1% even if you were adding a null poll.
 
Clinton is pretty safe. Early voting has been good for Hillary in states like Florida and North Carolina. She also has poll leads in Wisconsin, Michigan, and Pennsylvania. If she captures four of these states, the race is practically over. If she captures all of them, you can expect the senate to flip democrat and there will probably be a closer house of reps.

PollyVote, which has predicted the past three elections with incredible accuracy, their margin of error has been less then 1%, has Clinton winning the popular vote 53 to 47.

Here is their last three election predictions

Obama 2012:
Actual - 52.0%
Prediction - 51.3%

Obama 2008:
Actual - 53.7
Prediction - 53.9

Bush 2004:
Actual - 51.2
Prediction - 51.5

http://charts.pollyvote.com

Wasn't aware of this site, thanks for sharing.

Things are still way too close for my comfort, though.
 
If Hillary wins North Carolina, it'll be an early Thanksgiving miracle.

I just don't believe it. They will stop at nothing to rig it for Donald. They already have.
Voter supression isn't that effective. It can make a huge difference in local and statewide elections, but not enough for a national elections where Hillary leads by the ammount she is.

I'd give it a 75% chance NC goes blue on Tuesday.
 
Voter supression isn't that effective. It can make a huge difference in local and statewide elections, but not enough for a national elections where Hillary leads by the ammount she is.

I'd give it a 75% chance NC goes blue on Tuesday.

Which sucks, because the statewide elections are the ones that the Dems really need to win in NC (Ross and Cooper).
 
I think the big dilemma for this year is disagreement on the composition of the electorate. Poll variance is generally quite a bit higher this year than last cycle, which implies that the results depend more greatly on assumptions than usual.

The flipside is that the spread may have been artificially low previously, due to herding.
Agreed. I'm looking forward to the post-mortems on the effectiveness of different LV screens. My hypothesis is they are missing the new and reactivated voters we're seeing in places like FL, but we'll see.
 
I'm glad you guys make it sound positive, but locally, things sound neck and neck.
I'm also super disappointed that Ohio is proving itself to be majority bigot.
 
Agreed. I'm looking forward to the post-mortems on the effectiveness of different LV screens. My hypothesis is they are missing the new and reactivated voters we're seeing in places like FL, but we'll see.

I completely agree with that.

However, electorate selection is beyond the scope of poll aggregation and modeling, so aggregation sites will have a systematic miss if it is true, unless they're biasing their own results or heavily judging and weighting polling firms for reasons beyond their prior performance and contact methodologies.. The campaigns themselves are probably doing extensive modeling that hasn't been made public.
 
Just a heads up but Hillary has been trending upwards on 538 all morning. It appears that whatever was going on with their model might be correcting itself.

Nope. At 64.2% now, continuing her downward trend.

FL and NC both red, at R+0.3

The current map on 538 is disgusting. I can't believe that much of the country wants Donald Trump to be fucking president.
 
Nope. At 64.2% now, continuing her downward trend.

FL and NC both red, at R+0.3

The current map on 538 is disgusting. I can't believe that much of the country wants Donald Trump to be fucking president.

I think I'm gonna go vote tomorrow, and then turn off the news until Tuesday.
 
Status
Not open for further replies.
Top Bottom