You're right. Super Mario 64 is a platforming game with adventure game elements and Banjo-Kazooie is an adventure game with platforming elements - they're very similar to one another, but both have different intentions to their design.
But at the core of their design, large hub worlds with multiple objectives is still at the basis of their experience - you go into a large open level, you complete a task to get a token, there's usually more than one task to complete.
The inbetween is where the difference is - Super Mario 64 asks you to use your core set of abilities to solve challenges the majority of the time, which is where people are celebrating it's allowance for experimentation and high level play. Banjo-Kazooie on the other hand usually has objectives tied either to using special abilities or just for paying attention to the environment.
Best example are their final levels - Click Clock Woods and Tick Tock Clock. Both have a similar vertical design, and both have a central gimmick that relates to time - in Mario, it's having obstacles speed up or slow down depending how you enter the level, while in Banjo it's a level that shifts through different seasons, with objectives and obstacles rearranged depending.
Where as climbing up Tick Tock Clock is part of the fun of that game, Click Clock Woods is more in the vain of an adventure game where you go between different seasons to effect one another.
The reason I mention this is that - having replayed both these games recently - returning to Banjo-Kazooie was nowhere near as satisfying as I had already completed these objectives before, whereas with Super Mario 64 there was ground to improve and be more risky which was way more enjoyable.
Different strokes for different folks I guess