Yeah, the N64 had very good Western third party support, that is true. Sure, the quantity wasn't equal to other platforms, but the quality very much is! While many of the system's best games are first party, the N64 third party library is great, and includes some all-time classics, with Rush 2049 at the top of that list for me, of course.
But yeah, in the US the N64 wasn't just a 'kiddy system'. Yes, Nintendo never entirely got rid of that label, but between Goldeneye, Turok, wrestling and sports games, and more, the N64 obviously wasn't "kiddy" either in its library or its audience, even if some Sony fans liked to claim so at times. The N64 was popular with children, but it was also successful with a lot of 'core' gamers in a way Nintendo has not been again since.
I could get into specifics about how much I like a lot of third-party N64 games, but... I did that already:
http://www.neogaf.com/forum/showthread.php?t=473262&highlight=64
A few points though:
- San Francisco Rush 2049 is the best racing game ever made, and the N64 version is my favorite version.
- Other good third-party N64 racing games include Wipeout 64, all four Top Gear games, Extreme-G 2, Hydro Thunder, Stunt Racer 64, World Driver Championship, and more. Racing games are one of the platform's best genres.
- Rayman 2 isn't best on the N64, but the game, one of the best 3d platformers ever, was first developed for the N64, and the N64 version is great, one of the genre's best. Another all-time great is Goemon's Great Adventure, my favorite 2.5d platformer ever. Some more good third-party platformers include Mystical Ninja Starring Goemon, Rocket: Robot on Wheels, Space Station Silicon Valley, and more.
-For FPSes, the Turok series is pretty good. Unlike most people Turok 3 is actually my favorite one, but all four games are quite good, ambitious titles. Of course Doom 64 is also very good. Duke Nukem: Zero Hour is also a good game, among others. There are no third-party FPSes quite on par with Perfect Dark, but the best ones are pretty good, even if there are only a relative handful in numbers.
Etc.
As for what happened between the N64 and Gamecube, there were several factors.
1) Microsoft's entry into making consoles was huge. Microsoft's most important release was Halo, and with that they won over shooter fans, both those previously PC-only and those who had loved Goldeneye. Nintendo didn't, or couldn't, match it. MS also pushed the mostly PC-focused Western development base to also support their console, and publishers, starting to struggle because PC-only sales weren't keeping up with the rising costs of development, listened. Over the course of the '00s this badly damaged the US PC game development base, but was a big boost to console development. Nintendo ended up mostly missing out on this, as a lot of games either were for Xbox, or were on the PS360 and not Wii.
2) The two leaders of NoA from the '90s, Howard Lincoln and Minoru Arakawa, retired, and Satoru Iwata decided to take over NoA himself after they left. Instead of trying to hold on to the N64's hard-won success with core Western gamers, Iwata made a doomed effort to match Sony in Japan, and gave up on core Western audiences in favor of Japanese partnerships. That was great, but Nintendo needed both, not one or the other. He made some bad decisions, most notably to drop Rare, and failed to come up with ideas to counter Halo, letting MS take the Western 'core' audience away from Nintendo.
Iwata also separated from their three second-party studios (Rare, Left Field, and Silicon Knights). I've already said how bad a decision selling off Rare was, both here and
here. The two American first party studios Lincoln started, NST and Retro, did survive, though both reduced in size eventually -- NST's console team was gotten rid of in the early Wii years leaving only its handheld and other staff (who did Virtual Console work among other things, I believe?), and Retro reduced to only one game at a time, and dropped some staff as a result, early in its life. Of course Metroid Prime is absolutely amazing, but Nintendo needed Rare too!
3) As a result of #1, mostly, and maybe a bit of #2, the GC failed to sell in the US as well as the N64 had. Third parties started out supporting the system reasonably well, but as sales failed to match expectations, by 2003 most Western third parties dramatically cut back on GC support. From that point on the system only got more family-friendly games and the occasional major title, with very few major exclusives or ports. Nintendo's response to this was the less-powerful Wii, and we all know how that went for Nintendo and third parties -- the GC-era losses became permanent, and Nintendo now has entirely lost the core Western base, both developers and fans, at a time when they are absolutely vital for success. And on top of that, as casuals switched from the DS or Wii over to smartphones Nintendo lost a lot of sales.