People do that, the early game will MURDER you, but gimmick builds like that are a thing. I recommend it more for a repeat playthrough than a first. Focus on Dex or Ma. Once Dex/Ma are 200 you can start doing silly shit.
It's not as bad as Dark Souls with all your levels in resistance, but it's bad.
Sounds fun. I'll keep to something sane then. I like the sound of guns. i'll make another run with magic and see which one goes better.