As for what happened between the N64 and Gamecube, there were several factors.
1) Microsoft's entry into making consoles was huge. Microsoft's most important release was Halo, and with that they won over shooter fans, both those previously PC-only and those who had loved Goldeneye. Nintendo didn't, or couldn't, match it. MS also pushed the mostly PC-focused Western development base to also support their console, and publishers, starting to struggle because PC-only sales weren't keeping up with the rising costs of development, listened. Over the course of the '00s this badly damaged the US PC game development base, but was a big boost to console development. Nintendo ended up mostly missing out on this, as a lot of games either were for Xbox, or were on the PS360 and not Wii.
2) The two leaders of NoA from the '90s, Howard Lincoln and Minoru Arakawa, retired, and Satoru Iwata decided to take over NoA himself after they left. Instead of trying to hold on to the N64's hard-won success with core Western gamers, Iwata made a doomed effort to match Sony in Japan, and gave up on core Western audiences in favor of Japanese partnerships. That was great, but Nintendo needed both, not one or the other. He made some bad decisions, most notably to drop Rare, and failed to come up with ideas to counter Halo, letting MS take the Western 'core' audience away from Nintendo.
Iwata also separated from their three second-party studios (Rare, Left Field, and Silicon Knights). I've already said how bad a decision selling off Rare was, both here and
here. The two American first party studios Lincoln started, NST and Retro, did survive, though both reduced in size eventually -- NST's console team was gotten rid of in the early Wii years leaving only its handheld and other staff (who did Virtual Console work among other things, I believe?), and Retro reduced to only one game at a time, and dropped some staff as a result, early in its life. Of course Metroid Prime is absolutely amazing, but Nintendo needed Rare too!
3) As a result of #1, mostly, and maybe a bit of #2, the GC failed to sell in the US as well as the N64 had. Third parties started out supporting the system reasonably well, but as sales failed to match expectations, by 2003 most Western third parties dramatically cut back on GC support. From that point on the system only got more family-friendly games and the occasional major title, with very few major exclusives or ports. Nintendo's response to this was the less-powerful Wii, and we all know how that went for Nintendo and third parties -- the GC-era losses became permanent, and Nintendo now has entirely lost the core Western base, both developers and fans, at a time when they are absolutely vital for success. And on top of that, as casuals switched from the DS or Wii over to smartphones Nintendo lost a lot of sales.