Back then, I knew one person who had tried it and felt just like you. But damn for the rest of us it was a whole new world. Tomb Raider and Crash were just starting to figure things out, and sm64 leapfrogged them completely.
The key was the level design. Nobody really had that figured out. The inspiration was there in most games that had open maps like metroid and fps games that used items to gate things and get you around, but even that was changed appreciably for SM64. Those worlds were super dense and the way the stars got more obtuse and not simply "harder" as you went along in a given world was brilliant.
Sometime in the last 30 years you must have noticed your opinion on this is an outlier!!