I found a falchion that is better than my fire sword and I even upgraded it. For some reason though it does even less dmg to that dude than my firesword. The game really isn't great at explaining stuff. I have a mace in my inventory anyway, so I could upgrade that thing a bit. But still, I feel like I'm barely inflicting any dmg.
If Smelter is too tough, you can come back to him later. There is a shortcut past Smelter for a reason. No shame in skipping for now and cleaning up later when you're better equipped.