Of all the recent scandals, I think the "rigged" thing is going to hurt Trump the most. Not because it's super absurd (although it is).
It hurts him for a few reasons. One, Americans don't like whiners. We just don't. It undercuts his message as a strong man who wins, wins, wins. Now, he's admitting he's going to lose because FRAUD! It's not a good look. Secondly, it cements a narrative. It's unpresidential, yes, but it also reinforces the idea that he is losing. Bigly. Why would a person who is really winning be bitching about fraud and shit? Thirdly if people actually want to think about "Is this election being rigged?" it draws us back to the only actual attempts at swaying the election--the Russian hacks. Finally, it's damaging because it has the potential to decrease Trump's turnout. There's no ground game to harass people into voting.