Im going hearts all the way, there are foods you can cook to restore stamina while in the middle of climbing or swimming.
I put two points in stamina and am happy to keep it there. Saves me some food usage on climbs and helps with both sprinting longer, swimming longer, and taming wilder horses.
That said, I don't see the need to go beyond that, max. Right now I feel a bit freer to impulsively explore without considering my food reserves, but have a lot of room for heart upgrades.