Can't speak for everybody, but personally I consider it a spinoff because of its radical deviation when compared to previous entries (it's not even that it became a different kind of platformer, it's a whole other genre altogether), and because, again for me, it reaally gives me that vibe in its presentation and tone. Gruntilda even quips that "after some consideration, I'll treat this ruse as a free vacation!". Spinoff is not an inherently derogatory term so I don't think it's used out of bitterness, but rather to more quickly summarize the feeling it gives off to longtime fans.
I'll also say that I didn't play this back in the day, had my expectations in check when I tried it, and so far it just isn't working for me. Will power through to see if my mind changes.