I'm 9 hours in and on Chapter 6 already.
I don't feel like I'm rushing, but I don't really ever do sidequests in games because I don't find them fun or compelling. I'm almost always chasing the story because it compels me.
I'm usually like you. But this game's combat system is so fucking good, that I'm even enjoying doing hunts and other stuff.