Shit, you're right - I forgot about that cheeky mention in Uncharted 2!
People were hypothesizing that the Museum
is where Sam got captured and Nate never saw him again. It's why he was so against going back there.
But of course, the loot persuaded him otherwise