They feel that way because Daigo's performance in international tournaments, particularly US majors, hit a huge wall in 2011/2012 and he never really recovered his dominance over here. Infiltration meanwhile dominated in 2012 and 2013, and Momochi carried 2014.
To go from the unequivocal best to "could place top-8 or could place 256th at EVO" is a big step down on what people wanted or expected of him, even if his actual performance was better than it seemed.
But it didn't in 2011 or 2012. In 2011 he got 3rd at SBO, 4th at EVO, and won a few tournaments. In 2012, he had a streak of like 20+ games without dropping a game at the Japan nationals and 25th anniversary global finals which got him 1st and 2nd. Fuudo was the 3rd best player that year winning Topanga League over Daigo and yet Daigo was making him look free at the end of 2012. He got 5th at EVO, among other performances. He was the 2nd best player in 2012 and if not for Infiltration, people would be talking about how great Daigo was in 2012. In my eyes, that just makes Infiltration's 2012 even more impressive.
Only 2013 was a down year for Daigo and that was the year he promoted his book and the arcade tour. That was also the year he did the Gunslinger Stratos shit. However, 2013 was also the year Daigo made Infil and Xian look free.
So apart from 2013, it is easily arguable that Daigo was top2-3 in the year.
In my eyes this is how I viewed the years of SF4 (no order) through looking at results of arcade rankings early on, sbo, evo, topanga, global finals, etc...
2009: Daigo, Mago, Momochi,
2010: Daigo, Sako, Mago, Kindevu?,
2011: Daigo, Poongko, Fuudo, Kazunoko,
2012: Daigo, Infiltration, Fuudo,
2013: Xian, Infiltration, Daigo, Bonchan, Sako
2014: Momochi, Luffy, Bonchan, Daigo,
2015: Momochi, Daigo, Kazunoko
SFV
2016: Infiltration, Tokido, Fuudo, Momochi, Xiaohai,
Then again, I've always held the Topanga League and Gods Garden tournaments in high regard. Not only because of the prize money, but also due to the format and competition. It tests you how great you are at both short and long sets.