I agree, I think this will happen if Nate is wrong.
A random Newspaper or TV pundit who makes confident claims is not held to this standard, obviously. They make wrong predictions all the time -- and not even probabilistic ones, like Silver makes, but confident ones without probabilistic language -- and nothing happens to them. Often, their ratings or readership go up. By contrast, if Nate Silver makes one wrong prediction, he's toast.
Science-y/math-y people are simply held to a different standard. Implicitly, we know that they have to follow different rules than typical bloviators and pundits who can say anything they want with impunity. These additional rules are necessary because math-y science-y types actually are able to predict the future in a serious way, and as such they are held to much tighter standards. We may not openly admit that Joe Scarborough or Ed Sullivan are entertainers, we may not even consciously admit it, but the way in which we immediately discard their (frequently wrong) predictions suggests that we know it, deep down.
Just as we all know deep down that English majors are less important than Biochemistry majors, even if we don't admit it, we know that Joe Scarborough isn't the same as Nate Silver and thus we don't hold Scaroborough to the sort of standards we hold a statistician to.