It's not an awful game but it doesn't deserve to be discussed like the peak of the series. People take that as an article of faith but it doesn't feel earned.
Mario 64 is the peak of the series IMO... but World is right alongside 3 as the best of 2D.
It's funny, because I view most of your criticisms as some of the many things I like most about the game! Specifically because of the way it differentiates it from -- and IMO improves on -- Mario 3. The cape and Yoshi are such an improvement in terms of actual platforming gameplay, much better than anything in 3 for me... 3 powerups I view as very gimmicky and/or boring. The controls of the cape, and the versatility and actual fun of it are unlike anything in 3. Yoshi, as not just an "ability" but an "easy mode" that doesn't make things TOO easy, is fantastic. Again, completely changes gameplay, is still fun, and again has variation via the Yoshi colors.
Those are the main reasons, but there are a lot of others... I think World feels much more vertical, which the cape & yoshi tie in to very well. The ghost houses, branching paths, and keys I don't think of as "anti-fun" at all but they in practice encourage a different kind of Mario level playing, which benefits the game and genre in general. Quality of life stuff is just not even close -- the save system, the backup item system, the "open world" element of going back to any previous level, the coins completionism/replayability addition. And of course Star Road.
One thing I do agree with that you're suggesting is in terms of the design, 3 is more groundbreaking than World... World is obviously using a lot of what 3 did in terms of how it improved on 2. And in terms of difficulty, 3 is definitely harder. Which is a credit to it considering the genre.
So it's always been close for me, I love them both, but I've always been a World guy... the actual gameplay is just streets ahead!