N64 software sales were actually very profitable for Nintendo. The console also sold 3x more than the Saturn (30m) worldwide so it was not that big of a failure either.
The important thing isn't that the N64 sold three times as much as the Saturn. It's the "why" and the "how."
We know the N64 is the successor to an excellent console, the SNES, however, the N64 has its own merits we can't pretend that SM64, Wave Race, and Golden Eye are ordinary games. The N64 had mature 3D, anyone who lived through that era knows that Nintendo positioned the N64 as half a generation ahead of the PS1 (Nintendo even avoided PS1 ports whenever possible to convey that impression to consumers).
N64 facing off against the PS1 and its endless marketing dollars to brainwash consumers. But none of this stopped the N64 from being a success.
Pilotwings 64, Cruis'n USA, SM64, Wave Race 64, Star Wars Shadow Empire, and many other games were hits; the only flop was KI Gold.
In the following months, Turok, Golden Eye, and Yoshi Story were released. The impact caused by the N64 forced Sony to allow 2D games in the west. It also forced Sony to remember that children exist.
In March 1997, N64 surpassed the Sega Saturn, the installed base was 2.3 million units, Sega Saturn never reached 2 million units in the US.
This only reinforces the insanity that was the launch of the Saturn in the US, Sega had money for the marketing campaign, it was $50 million (the same number as the PS1 and N64) but obviously this money was spent in the wrong way, the games were made wrong (the main use of the Sega Saturn should be in games like Guardian Heroes) and the console has a design that made its price expensive, what Sega taught the market was that idiots can't design hardware, produce commercially viable games, and market effectively. If such people exist within a team, they need to be removed.