Now, the problem with resistance is that it is a baseline deduction off of damage. We know some things about it. First some terms
AR = Attack Rating
mod = a modifier of how strong the attack is. It seems this is based on the animation speed difference between an R1 and R2 attack.
base damage = AR * mod (so R1 means that base damage = AR)
dmg = base damage - ([<appropriate defence> + 10(?)] / 2)
Here I suggest +10 for simplicity, I am currently calculating the actual value for this, it seems to be a bit elusive. There is an 'offset' in defense, though, that's for sure.
We know when AR << def (read as 'very much smaller'), it starts 'bottoming out' in a way that can simply be considered an 1 / x function. It approaches 0, but veeery slowly. I saw the non-linearity kick in when def got close to 1.5x AR. I have yet to test the bottom of that scale. That means when AR >> def, it might boost the negation. But I doubt that.
So resistance seems linear when
def/AR = X
0 < X < AR*1.5
How much does health vs resistance help in PvP. Here's the case of me PvPing myself:
I have 1066 HP. My weapon has an AR of 250 in both phy and lightning. My slash resistance with this armor is 396, and my lightning res is 321. 250 AR vs 396 in DEF - we're in the non-linear territory, but my testing tells me that this gives a phy damage of ~65. 250 AR vs 321 lightning DEF gives dmg of 85
Each attack does 150 dmg to myself. As a side-note to myself, my Queelag's would've done 216 damage to myself (Flame DEF of 292). If I'm correct. Hmm.
What's interesting seeing here is that if I'm up against a heavily armed opponent, I should use a weapon that has high AR versus one stat (like the Queelag has 336 AR vs flame), because more damage will get through in the end, even though both swords have near the exact same total AR.
So, furthermore, I would've survived 7.1 R1 hits from myself. Now I remove my Smough's helm an put on Mask of a Mother. I have now 1225 HP and 356 phy DEF and 308 lightning DEF
Total attack: 165
I would've survived: 7.4 hits. This means that getting more HP from this mask is better than getting more resistance.
Before I could've taken 4.9 hits from Queelag's, and now 5.3.
Favor's Ring seems to be better than Havel's, because the loss of resistance through the heavy equipment is negated by more HP. Negated and then some.
What about the case of insanely strong attacks?:
We've all been hit by attacks that take away half our damage, via magic or huge weapons, or bosses. Let's look at what their AR normally is. Let's assume they do 1 damage type, and that we have a resistance of 300. Half my damage is 500 HP. The base damage of the attack has to be 650. Let's assume we have half the resistance, but the base damage is the same. That gives us dmg = AR - (def + 10) / 2 = 650 - 80 = 570
We see that halving the resistance doesn't mean the attack will do twice the damage. Actually, staying light against hard hitting opponents seem to be key, because we can evade better.
What about normal hitters? Say they have an AR of 400, and we have a res of 300, that means a damage of 250. Half your res - 150, and the attack does 325. You can survive probably as many super-hard hitting magics with low res, but aside from non-super heavy weapons and insane magic, weapons that have near the same base damage that you have RES, you will have a pretty big benefit from resistance.
tl;dr: Against insanely hard hitting opponents, resistance is pretty moot. An attack that does half your HP with some resistance means that it'll probably do about 10% more damage with half that resistance. Against normally hard hitting enemies doing a tenth of your HP, half the resistance can easily mean 50% harder attack, meaning you won't be able to survive 10 attacks, but only 7.5. Resistance is good versus either katana/spear/etc opponents in PvP, or normal enemies in the game, but against mages, heavy weapon users and bosses, you're much better off with high HP. In my ramblings above, even using my katana against myself would mean that I'd survive longer with the Mask of a Mother than any helmet. Remember that VIT is effectively a non-linear increase to ALL resistances, and it gives more resistance the harder the opponent hits.
Should you increase RES or VIT? Resistance increases by half a point per level anyway, increasing RES gives an additional half a point. Increasing my VIT from 29 to 30 gives me 33 VIT, and from 30 to 31 gives me 22.5 - but half a resistance will take away .25 damage per hit. That means I have to be hit 132 times per health bar for it to be better. If I add 8 levels to resistance, that will give me 4 more resistance. Against a 400 base damage, with a res of 400 / 404 res, that will decrease my damage by 2. RES is not worth it (though it gives good poison resistance)