I think the most damning thing about the N64 is revealed through the list wars. All of the N64 lists, proving the greatness of the console, start strong, but they get highly dubious really quickly.
Beyond that, even some of the most cherished games were still essentially just glorified proofs of concept. Golden Eye was always a little janky. It hasn't aged well. It was a great time, but there were always problems.
I think the fact that it had four controller ports, and the fact that Nintendo's first party games really nailed the multiplayer component, is the one saving grace. However, it is hard to see the console as a huge success when you consider how few games there were, how many genres were essentially just absent, and the quality of it's competitors lineup. As a teenager, the system basically forced me to buy a Playstation. At the time that was something I REALLY didn't want to do, as Nintendo was home to me.
I feel if you compared Square and Capcom's output on the PS1 to the entire N64 lineup, the PS1 would probably have more killer titles.