Two suggestions so... I chose.stamina
I only got 3korok seeds, and the doublet. What other secrets should I stay and find? Memories?
You can go, the Warm Doublet is the main thing to get on the Plateau as it'll be useful for the entire game.
Memories can't be gotten until you start the quest for them.