I think you're looking at it too extremely. I think the general idea is that Labor Day weekend is a long weekend off from politics for most, and generally speaking, that tends to be true from my experience.
It's just I keep hearing the media and a lot of surrogates saying how people don't start seriously looking at the candidates until after Labor Day. Which I find ridiculous.