It's much more of a third person shooter/stealth game than a horror game. I've beaten the game three times on normal difficulty. Don't even fancy trying hard. Hard/survivor/grounded difficulties, first and foremost, deny the player ammo and supplies. Call me a noob but I really enjoy blasting enemies with the shotgun or getting a nice headshot with the rifle or stealth-ing a section with the bow or setting a group on fire with a molotov.. That's what's fun to me. I do enjoy the challenge of higher difficulties for certain games, but TLoU isn't one of them. And it's not like you are forced to use listen mode.
TL;DR: Playing TLoU on harder difficulties mostly seems to deny the player the very supplies which make the game more fun for me.