I was able to fight all three ogres and clear that quest. Done many quests. But bandits wreck me. Trying to do the last few quests I have before going into stage 2, this one which is the tome one. I can't do it.
So yeah I will miss the gold intake but I am sure that is fine! Just as long as I can keep my quest progress and kill the freaking bandits.
Bandits continue to wreck throughout the game on a fresh hard mode run. I'm above level 60, on the penultimate quest, and they're still one shotting me. Much scarier than chimeras and cockatrice and whatever the hell else.