Don't be a wimp and kill him.
Died like 10 times or so. Felt very Dark Souls-y!Shoot regular arrows at his head to stun him, run towards him and mount with A, and hack away, get thrown off and repeat.
I will fucking do that with my new character, mark my words! This fucker is going down!