Casual consumers tiring of the Nintendo "brand" franchise extensions (Mario, Zelda).
Gamecube has only 15,000,000 consoles sold and only one and a half years left; as opposed to 30,000,000 overall for N64. Casual consumers have even less interest in Nintendo.
And soundwave05 said it best:
"People are not willing to buy another console just for Nintendo games. Most people who bought the N64 thought it would end up being like the SNES/NES before it (lots of games/variety). So Nintendo has not done a good enough job in surrounding their core franchises with enough support."
The third party support is atrocious, and the vast majority of casual consumers don't want to put their weight behind Nintendo franchise extensions if they don't have alternatives.