I'd say don't worry about side quests until you get fast travel. It'll make them more bareable.
The world is so small, though, and traversal is so smooth with fast movement and even the mounts. I almost don't use fast travel.
In my opinion you should skip ALL sidequests. Seriously.
On my first runthrough, I did ALL sidequests asap, which resulted in me being totally overleveled for wanting to beat Papa Servo, I was level 55, where I should actually be around 15. I ruined the entire main story for me.
Which is why I gave up my save and did it all over again.
I started to change the difficulty on boss fights. Worked very well. I still did a ton of damage, but I actually had to be careful because so did the enemy. The last few boss fights were awesome. I can't wait to unlock chapter select to redo the ones I did on Normal.
I'd rather be way overleveled and nerf the numbers by wielding sub-bar weapons like the cypress stick than miss out on all the good stuff loaded into the narrative of the quests, be it the serious stuff or the jokes. Plus later Servo quests and a few others are a clear outlier as the enemy levels themselves make it pretty clear you are supposed to do them later. You don't get to lv. 55 during route A without significant grinding.
Yeah, I did almost everything I could and finished it on level 39 or 40.