Probably something to do with the placement and where you stand behind it, seems kind of luck-based a lot of the time. If he charges, pull out your shield. You should be able to bonk him down right next to the golem. Then you can get all the golem hits, and freeze him at the same time.
Also blood magic to get yourself down to Sacrifice-level HP, which does a respectable amount of damage.
The same trick can be used on Jack-O-Lantern (Singleplayer version). Summon him, place a golem on the left side of the entrance nearby, and stand on it's left side. He'll sit there and take every hit. You can then hit him with blood magic through your golem. Less than a minute for the win.