I have a Breton mage at 33 with maxed destruction, conjuration at about 60 and restoration at about 55 (alteration/illusion are in their 20s, and not worth talking about). Leveling has been 30 level ups of magicka and 2 on health, none on stamina.
Between Breton base magic resistance of 25% and the atronach stone 50% spell absorption, I don't really worry too much about magic. That 25% resist could easily be replicated on gear if you are not a Breton. I never use wards, and usually just rely on overwhelming amounts of dirt cheap low cost destruction spells to kill things.
I also wear no armour and run around in robes, gloves and boots that have been enchanted, with a circlet, necklace and ring. In other words, I have 0 armor whatsoever. If I get hit by anything heavy, I die. Draugr Deathlords, a single arrow from a leveled enemy (or 2 or 3 from base enemies), a bear swipe/2 or 3 wolf bites, one bite from a dragon that landed in my face, etc. Yes, I could cast some mage armor spells, but I instead rely on killing things before they get the chance to kill me. Playing on normal difficulty, but whatever. I find it challenging enough considering I could easily play in a fashion that would make me tougher to kill (ie use some armor, cast some alteration spells, etc), but that's the trade off for going hardcore magicka loaded wizard with full on wizard robes.
There is nothing wrong with going pure destro. It is entirely playable and your character is anything but busted beyond reparation. If you have trouble with magic, enchant some magic resist and grab the atronach stone. Utilize wards if necessary to absorb some magic attacks. But saying the game is broken because now the enemy can cast powerful magic spells and you do nothing to combat that with so little health...well, yeah, you're probably gonna die. Lots.