I honestly think the biggest problem is that there have been a LOT of mini-scandals over the years that Harper's been in office. Each time you'd see the Liberals & NDP sounding the alarm that this is the biggest deal ever, people should resign, etc etc (it's their job as opposition, but still), it makes front page news + first spot on evening news shows, but each time it was something... not that big.
It came off as making a mountain out of a molehill. Over time now, I think people have just become numb to it. Oh another scandal that's the biggest thing ever? OK :\
Duffy seemed to be the first one to actually stick.