Support NeoGAF

[brianmcdoogle]

There may also be a second major issue with FiveThirtyEight, which we will describe as one of economy. We start this part of the discussion by noting that every one of us who is writing about politics this year benefits from a horse race. "Things are the same as they were yesterday" is not a story. "Clinton extends her lead" and "Trump makes up ground on Clinton" are. Similarly, we also benefit from finding things that are new and different to talk about. Sen. Bernie Sanders (I-VT) and his rallies tended to get relatively little media coverage; not because of any particular bias against him, but because they were all the same. You can only write, "10,000 young, mostly white people show up to cheer Sanders" so many times. Hillary Clinton, evenhanded and cautious as she is, also tends to give us relatively little to talk about much of the time. With Donald Trump, on the other hand, it's several new and outrageous and previously unheard of things almost every day. Hence his dominance of the headlines.

Point is, all the political sites have a certain bias towards "dog bites man." However, there is reason to believe the bias is unusually strong for Silver and his crew. Many political sites and prognosticators—NBC News, the Wall Street Journal, Fox News, Bloomberg Politics—are part of organizations for whom political coverage is part of their core mission. Others—Sabato's Crystal Ball, the Harvard Political Review—are part of (and are supported by) universities. Still others—HuffPo, Breitbart, Politico, The Hill—are already stable, self-sustaining businesses. And a few—this site, Sam Wang's Princeton Election Consortium—are side projects of academics who already have day jobs. The point is that while we all like page views and clicks, none of these sites is—as far as we know—facing an immediate existential crisis. Page views could go up or down by 50%, and most or all of the above would keep on trucking.

An interesting critique of 538 and Silver this campaign cycle from one of the first people to do this type of analysis, or as Sam Wang at Princeton Election calls, "a pioneer!"

I figured I'd be nice and just bump this instead of making a new thread:

I Think Nate Silver Is Broken, Maybe?

On its own, this wouldn’t be anything much; going by every way of looking at it we have, this race is tight, and anyone invested in reporting true facts to readers will say as much. The thing is that Silver has been describing the state of the race as unknown and unknowable for some time now, and, further, offering thinly-sourced explanations for why this is so. Take this bad tweet ...

It then gives a series of really stupid, bad punditry tweets...

Here’s my theory: I think Nate Silver is broken. I think the primaries, in which he took Ls just about hourly on his (reasonable and justified!) certainty that Trump was about to flame out, broke him; the lesson he took away from the beating seems to have been Nothing is knowable. I think he is Nate Bronze now.

Or, as happened today, Nate tweeted this:

Nate Silver: Good news for Democrats: Really hard to see how that debate helped Trump.
Bad news: If it somehow did help Trump, maybe no stopping him?

To which Princeton Election Consortium's Sam Wang, a pioneer in this field responded:
please clap

 

[Man God]

It's a business first.

 

[brianmcdoogle]

Bumping since people are freaking out about 538 again today.

 

[Fenderputty]

I think most people who follow this kinda thing assumed this would be the case when ESPN bought them out. Sam Wang or Nate Cohn are where it's at now.

 

[perfectchaos007]

So are they saying since 538 benefits from polls changing, and its in their best personal interests that 538 skews their numbers to favor massive poll swings to get more clicks? Sounds very conspiracy like. 538 seems to be very math driven and not politically biased. I would be surprised if 538 is making up numbers to get more clicks

 

[Slayven]

So are they saying since 538 benefits from polls changing, and its in their best personal interests that 538 skews their numbers to favor massive poll swings to get more clicks? Sounds very conspiracy like. 538 seems to be very math driven and not politically biased. I would be surprised if 538 is making up numbers to get more clicks

You just described modern media

 

[Zukkoyaki]

So are they saying since 538 benefits from polls changing, and its in their best personal interests that 538 skews their numbers to favor massive poll swings to get more clicks? Sounds very conspiracy like. 538 seems to be very math driven and not politically biased. I would be surprised if 538 is making up numbers to get more clicks

They aren't making numbers up but they do have a model intentionally designed to "overreact" to shifts in polls. We've seen single polls move their model as much as 3% in both directions. They say it allows the model to detect potential trends. It isn't really wrong or anything but it's peculiar and has drawn criticism from many pundits and other data analysts.

 

[Eidan]

Honest question: Does anyone remember Silver's model being so volatile in 08 or 12? His write ups have obviously delved into full-on punditry ("The Electoral College can't save Clinton!"...give me a break), but it's the swings in his model that are really surprising me this cycle.

 

[IrishNinja]

538 was about the most reliable source for the last few elections, were they not? i know nate took shit for his commentary on trump's odds last year, but if there's a better place for analysis, i'm all ears

 

[bachikarn]

So are they saying since 538 benefits from polls changing, and its in their best personal interests that 538 skews their numbers to favor massive poll swings to get more clicks? Sounds very conspiracy like. 538 seems to be very math driven and not politically biased. I would be surprised if 538 is making up numbers to get more clicks

The other math driven sites don't show as big as swings as he does.

 

[Trojita]

Nate poisoned his own well.

 

[LiveFromKyoto]

As someone who's been reading 538 throughout this campaign...I'm not seeing it. Their podcasts and articles are constantly commenting on people reading too much into poll swings. They've said Hillary has had this firmly in hand pretty much the whole time. They've been very up front about why swings were showing up in their numbers, why certain things are weighted the way they are, and why they do or do not matter at a given time.

 

[Hellsing321]

538 was about the most reliable source for the last few elections, were they not? i know nate took shit for his commentary on trump's odds last year, but if there's a better place for analysis, i'm all ears

Princeton Election Consortium is the real gold standard.

 

[Blader]

If nothing else, I still enjoy 538 for their slack chats.

 

[Timeless]

I've gained a lot more statistics knowledge since the 2012 election. Back then, I had taken introductory statistics. Now I've taken 4 additional courses, and although the math behind PEC is probably beyond me, I feel a lot more comfortable with the kind of work PEC is doing than what 538 has done in the last few years. Silver may know what he's doing statistically (at least better than almost anyone outside the field of statistics) but a lot of his staff (outside of the political posts) make shaky assumptions and Silver himself thought Trump wouldn't win the nomination. With the latest 538 news, Clinton down to 60ish % from the 90s just prior, it makes me all the more skeptical of 538.

I'm sticking with PEC this election and hope that they make the right prediction when it matters (election night).

 

[kirblar]

538 was about the most reliable source for the last few elections, were they not? i know nate took shit for his commentary on trump's odds last year, but if there's a better place for analysis, i'm all ears

Yup.

The moment it became a business and not his blog, everything changed. Existential crisis + page click economy incentives = Silver became what he hates, a Clickbait Pundit.

 

[eBay Huckster]

538 was about the most reliable source for the last few elections, were they not? i know nate took shit for his commentary on trump's odds last year, but if there's a better place for analysis, i'm all ears

PEC is basically 538 without punditry (and a much simpler model)

 

[FreezeSSC]

Princeton Election Consortium is the real gold standard.

Weren't they really off in 2014?

 

[IrishNinja]

Princeton Election Consortium is the real gold standard.

Yup.

The moment it became a business and not his blog, everything changed. Existential crisis + page click economy incentives = Silver became what he hates, a Clickbait Pundit.

PEC is basically 538 without punditry (and a much simpler model)

bookmarking now, thanks guys

 

[brianmcdoogle]

I figured I'd be nice and just bump this instead of making a new thread:

I Think Nate Silver Is Broken, Maybe?

On its own, this wouldn’t be anything much; going by every way of looking at it we have, this race is tight, and anyone invested in reporting true facts to readers will say as much. The thing is that Silver has been describing the state of the race as unknown and unknowable for some time now, and, further, offering thinly-sourced explanations for why this is so. Take this bad tweet ...

It then gives a series of really stupid, bad punditry tweets...

Here’s my theory: I think Nate Silver is broken. I think the primaries, in which he took Ls just about hourly on his (reasonable and justified!) certainty that Trump was about to flame out, broke him; the lesson he took away from the beating seems to have been Nothing is knowable. I think he is Nate Bronze now.

Or, as happened today, Nate tweeted this:

Nate Silver: Good news for Democrats: Really hard to see how that debate helped Trump.
Bad news: If it somehow did help Trump, maybe no stopping him?

To which Princeton Election Consortium's Sam Wang, a pioneer in this field responded:
please clap

 

[Basileus777]

Nate Silver's punditry has descended into self-parody. His tweets are barely above the level of a troll nowadays.

 

[Sub_Level]

*Spoiler alert*

SPOILER

Spoiler

Hillary clinton wins the election

END SPOILERS

 

[Kangi]

Yeah, Nate's really been going off the deep end. His tweets the other day about a "dead heat" were... kind of hilarious. His model's been acting weird this time around, too. A single national poll brought Hillary's chance to win down seven points. That is not the sign of a stable forecasting model.

Regardless, his response to any such criticism is basically, "I'm smarter than you." Which is also hilarious coming from somebody who just earlier this year got burned badly for his primary punditry.

 

[RustyNails]

Punished Nate

A hero denied his statistics

 

[Maengun1]

Nate Silver: Good news for Democrats: Really hard to see how that debate helped Trump.
Bad news: If it somehow did help Trump, maybe no stopping him?

To which Princeton Election Consortium's Sam Wang, a pioneer in this field responded:
please clap

I honestly unfollowed Nate after that tweet, I couldn't take it anymore. He's been tweeting (and writing articles) that say stuff like "~If~ Trump keeps gaining, he ~could~ eventually surpass Clinton ... ... ... AND WIN AHHHHH!!!!" for the last 6 weeks or so. Yeah it's technically true, but 1) duh, and 2) he used to make fun of the people who do that stuff.

As for his model, I dunno. He has a good track record, but he's more secretive about what makes his tick than others and it really does seem wack this year. Clinton was all over the place yesterday, she went from like 60% down to 50% then back up to 55% in the span of a couple hours. That seems like...not a great predictor?

Obviously it's annoying to someone like me who finds Trump having even a 1% chance of becoming president horrifying, but in any case the swingy model combined with his incredibly grating punditry lately just comes across like he's desperate to be seen as a genius who was "right" again this year but is actually lost in the weeds. He had another article that was like "Trump might narrowly win, OR Clinton might narrowly win, OR Clinton might win in a landslide!!" He can cite is prediction literally no matter what happens lol

 

[Toxi]

Yeah, Nate's really been going off the deep end. His tweets the other day about a "dead heat" were... kind of hilarious. His model's been acting weird this time around, too. A single national poll brought Hillary's chance to win down seven points. That is not the sign of a stable forecasting model.

Regardless, his response to any such criticism is basically, "I'm smarter than you." Which is also hilarious coming from somebody who just earlier this year got burned badly for his primary punditry.

As for his model, I dunno. He has a good track record, but he's more secretive about what makes his tick than others and it really does seem wack this year. Clinton was all over the place yesterday, she went from like 60% down to 50% then back up to 55% in the span of a couple hours. That seems like...not a great predictor? Obviously it's annoying to someone like me who finds Trump having even a 1% chance of becoming president horrifying, but in any case the swingy model combined with his incredibly grating punditry lately just comes across like he's desperate to be seen as a genius who was "right" again this year but is actually lost in the weeds.

I didn't hear about this. What the hell happened?

 

[brochiller]

I figured I'd be nice and just bump this instead of making a new thread:

I Think Nate Silver Is Broken, Maybe?



It then gives a series of really stupid, bad punditry tweets...



Or, as happened today, Nate tweeted this:

Nate Silver: Good news for Democrats: Really hard to see how that debate helped Trump.
Bad news: If it somehow did help Trump, maybe no stopping him?

To which Princeton Election Consortium's Sam Wang, a pioneer in this field responded:
please clap

Don't write that Bad news tweet off.

If somehow getting shalacked by Hillary in the debate doesn't help her, or actually helps him, I don't know what else she can do. She'll need a Trump scandal or something to hope to win if she can't do it in a debate.

 

[Basileus777]

Don't write that Bad news tweet off.

If somehow getting shalacked by Hillary in the debate doesn't help her, or actually helps him, I don't know what else she can do. She'll need a Trump scandal or something to hope to win if she can't do it in a debate.

Lol. She was winning before the debate, Trump is the one who needed to do something to change the game.

And this is all terrible punditry that Nate would have mocked years ago when he was still coherent.

 

[brianmcdoogle]

I didn't hear about this. What the hell happened?

Here's the thing, he bills his erratic model as... a good thing...

And, they have constantly been the outlier this race. He has to resort to comparing himself to Betfair, a betting site; betting sites aren't statistical! The NYT Upshot and PEC have been close to one another this entire race without wild fluctuations while 538 can move 10 points in a day.

 

[Kangi]

I didn't hear about this. What the hell happened?

538 REALLY likes Selzer & Co.

Basically, they released a poll that had Trump up by 2 nationally. As Nate Cohn put it, that's noise. As Nate Silver put it, "TRENDLINES??"

His model instantly tanked Clinton to near-50%. Then Silver tweeted about DEAD HEAT. Then a bunch of other polls came out that showed that the sky was, in fact, not falling. So it adjusted her right back up several points.

Very stable model. The best.

 

[EatinOlives]

Nate has been reduced to a bumbling Diablosing mess. He really, really, REALLY wants to not go down in history as someone who was wrong about the 2016 presidential election. He was already wrong about the primary, but the general is something else entirely.

To me it seems like Nate is, rather than talking to his Twitter followers, talking to himself. This entire time he keeps giving himself an out, nearly every statement is him covering his own ass. Maybe the debate helped! Maybe it didn't! If it helped, then it was meant to be! If it wasn't, then it wasn't! And then he takes it out on what he calls "Democrats in denial" for people going "dude, what the fuck are you talking about".

 

[Eidan]

Lol. She was winning before the debate, Trump is the one who needed to do something to change the game.

And this is all terrible punditry that Nate would have mocked years ago when he was still coherent.

Precisely. What was so refreshing about Silver was that he reveled in cutting through Beltway punditry bullshit, and avoided horserace nonsense. The guy is just a smugger Chris Cillizza now.

 

[Wafflecakes]

Glad to see I am not the only one who follows the electoral-vote blog.

Visiting the site has been a daily ritual for me during election season since I was a teenager (I am now almost 29). I think I started following it around the 2004 election.

538 has nothing on this blog.

 

[Peru]

I'm not a polling expert, but have found the massively swinging 'likelihood of winning' numbers completely useless. If Hilary's chances of winning are 85% one week, 49% a few weeks later (for no obvious reason), then the likelihood never was 85%, or there should at least be mechanics in place to consider that swings like this happen and the implausibly high numbers are limited.

 

[exmachina64]

Nate has been reduced to a bumbling Diablosing mess. He really, really, REALLY wants to not go down in history as someone who was wrong about the 2016 presidential election. He was already wrong about the primary, but the general is something else entirely.

To me it seems like Nate is, rather than talking to his Twitter followers, talking to himself. This entire time he keeps giving himself an out, nearly every statement is him covering his own ass. Maybe the debate helped! Maybe it didn't! If it helped, then it was meant to be! If it wasn't, then it wasn't! And then he takes it out on what he calls "Democrats in denial" for people going "dude, what the fuck are you talking about".

I'll take that a step further:

Nate's reputation and his entire self-image was predicated on being right most of the time and bringing "science" to politics and forecasting. After years of praise and feeling like he's the smartest person in the room, his ego won't let him be wrong about something this important.

 

[Eidan]

I'm not a polling expert, but have found the massively swinging 'likelihood of winning' numbers completely useless. If Hilary's chances of winning are 85% one week, 49% a few weeks later (for no obvious reason), then the likelihood never was 85%, or there should at least be mechanics in place to consider that swings like this happen and the implausibly high numbers are limited.

I feel like Silver's models were much more stable in 12 than they've been this year. Whatever changes he made, they've made his models worthless, for the very reasons you outlined.

 

[Shalashaska]

I feel like Silver's models were much more stable in 12 than they've been this year. Whatever changes he made, they've made his models worthless, for the very reasons you outlined.

We can't exactly gauge any model worthless until there is a result to compare it to.

 

[BanGy.nz]

I feel like Silver's models were much more stable in 12 than they've been this year. Whatever changes he made, they've made his models worthless, for the very reasons you outlined.

Higher undecided and third party vote this years has his model shook.

 

[brianmcdoogle]

I'll take that a step further:

Nate's reputation and his entire self-image was predicated on being right most of the time and bringing "science" to politics and forecasting. After years of praise and feeling like he's the smartest person in the room, his ego won't let him be wrong about something this important.

And yet, he or Harry Enten (I honestly forget which) was claiming just weeks before Trump became the undisputed leader of the Republican primary, a position he would never relinquish, that Trump had a better chance of playing in the NBA finals or starring in another Home Alone sequel than being the nominee...

And when people on this board say "ALL THE POLLSTERS GOT IT WRONG ABOUT TRUMP" it's just 538. Other people saw and read the polls correctly. FiveThirtyEight didn't.

And utter embarrassment where Nate's "punditry" continued to be laughable.

There's a reason why I haven't read an article or visited their site in many months. Want to know their model? Follow it on twitter or visit the New York Times which includes their model in their comparison.

I feel like Silver's models were much more stable in 12 than they've been this year. Whatever changes he made, they've made his models worthless, for the very reasons you outlined.

A fledgling ESPN sinking ship and a desire to get page views tanked his model. Or, his hubris and incompetence finally shinning through to all, unvarnished.

 

[Wafflecakes]

We can't exactly gauge any model worthless until there is a result to compare it to.

When do we compare it to the result though? Do we only compare the model the day before the election? If so, why even bother having it before that?

The fact of the matter is that the chances of a specific outcome occurring do not vary dramatically on a day to day basis in real life. Therefor any model that dramatically changes its predicted outcomes often is not a worthy model.

 

[Miletius]

538 was still dead on about the midterms when, as I remember, Wang and Co were pretty off. So, while I understand their point, I think that Nate still has a better record than most when it comes to proven forecasting.

I think people underestimate Nate's model at their own peril. Just because I'd like Hillary to have a more comfortable lead doesn't make it so.

 

[TheGraykid]

When do we compare it to the result though? Do we only compare the model the day before the election? If so, why even bother having it before that?

The fact of the matter is that the chances of a specific outcome occurring do not vary dramatically on a day to day basis in real life. Therefor any model that dramatically changes its predicted outcomes often is not a worthy model.

But they do.

The odds of me giving a homeless man some change on the 26th October 2016 can change drastically based on what is going on in my life. If I get fired from my job in the next couple of weeks doesn't that change the odds of the outcome?

 

[Rentahamster]

I'm not a polling expert, but have found the massively swinging 'likelihood of winning' numbers completely useless. If Hilary's chances of winning are 85% one week, 49% a few weeks later (for no obvious reason), then the likelihood never was 85%, or there should at least be mechanics in place to consider that swings like this happen and the implausibly high numbers are limited.

Isn't that particular one more like "if the election were held today, this is probably the winner" type of thing?

 

[Wafflecakes]

But they do.

The odds of me giving a homeless man some change on the 26th October 2016 can change drastically based on what is going on in my life. If I get fired from my job in the next couple of weeks doesn't that change the odds of the outcome?

Absurd comparison.

We are talking about an election where untold millions vote, not some anecdote of one guy.

On a large scale the chances do not shift significantly on a day to day basis.

 

[exmachina64]

Isn't that particular one more like "if the election were held today, this is probably the winner" type of thing?

Yes.

 

[Shalashaska]

When do we compare it to the result though? Do we only compare the model the day before the election? If so, why even bother having it before that?

The fact of the matter is that the chances of a specific outcome occurring do not vary dramatically on a day to day basis in real life. Therefor any model that dramatically changes its predicted outcomes often is not a worthy model.

That is a good question but if that's the case, why do we bother with any of these at all until the end then?

The best analogy I can think of is the win probability for in-game sports, based at a specific point in time, the factors are analyzed and give a certain win percentage. When those factors change, the model changes, sometimes dramatically.

I guess for me, Silver has a solid track record with Presidential forecasting so I'll give him the benefit of the doubt.

 

[Maledict]

That is a good question but if that's the case, why do we bother with any of these at all until the end then?

The best analogy I can think of is the win probability for in-game sports, based at a specific point in time, the factors are analyzed and give a certain win percentage. When those factors change, the model changes, sometimes dramatically.

I guess for me, Silver has a solid track record with Presidential forecasting so I'll give him the benefit of the doubt.

The thing is, so do the other people. Nate got all the credit but numerous statisticians predicted 2008 and 2012 spot on - Sam Wang being one of them. Those are the people who are now critisicing Nate for descending into desperate punditry, and for a model that is *clearly* going a bit wonky.

 

[titiklabingapat]

Nate has been reduced to a bumbling Diablosing mess. He really, really, REALLY wants to not go down in history as someone who was wrong about the 2016 presidential election. He was already wrong about the primary, but the general is something else entirely.

Just a friendly PSA and it's not really a slight against you specifically, but Diablos has mentioned that he/she doesn't like his/her name used a prerogative verb anymore.

 

[Kathian]

The election is just closer because no one has an incumbent factor.

 

[TheGraykid]

Absurd comparison.

We are talking about an election where untold millions vote, not some anecdote of one guy.

On a large scale the chances do not shift significantly on a day to day basis.

Day to day maybe not drastically, but week to week probably yes. I feel like you're questioning the wrong thing. Shouldn't the question be towards polling data and why the percentage points are so volatile this year?