u and me both. in fact, the higher you level, the harder he'll become. The bonus armor is just too much. Stalling him and doing dot damage works, but you can't stay alive against 4 guys. I had success with a freeze + fire sword, so you have 2 turns to heal or only get damage from 2 guys per turn, but I still lost closely (level 10). I'll try him on level 11 again, then I get my new armor with lots of +armor (which also boosts my damage). Otherwise I'll wait till level 14 and do him with the katana, which ignores 500 armor or so by default
you don't need to grind. the enemies scale with your level.