The stats make a HUGE difference.
White church set is essential for nightmare frontier, Crowfeather/Ashen for frenzy, Charred for fire, Henryk for bolt, Knight's set for blood...etc.
The difference is actually massive, some people just don't bother because the only stat people are looking at is physical defense, Bloodborne handles armor in a more RPGish way than Demon's or Dark in that each armor set gives you certain advantages & disadvantages instead of a linear upgrade.
The above doesn't apply to me though, because fashion > function.