A non-viable skill or build is one which is not applicable from start to finish of the game.
Note that "finish of the game" does not necessarily mean end of the story campaign. In the example of the madman, the game ended when the PC finally bitten off more than he could chew.
Guns in Arcanum is a major skill around which you can build your character, yet they're pretty damn ineffective. The rate of fire is very slow, the damage output is very low and ammo is extremely scarce; all of those factors make the game ridiculously frustrating. It's not about difficulty but about effectiveness, whether they work or not. That is, they don't solve the problems they're is supposed to. For at least first half of the game, guns are ineffective. It may have gotten better in the second half, I don't know, but that doesn't matter because they weren't useful in the first half. They're non-viable.
An awful lot of games offer classes that reward the player early but aren't so good later on, or vice versa. I suppose you could argue that all of those games are terrible, but I still don't see why this makes guns nonviable. Early guns really
were frustrating and inaccurate, and the only reason they got as much use as they did was because they were relatively easy to train with in comparison to a longbow. Maybe the real problem is that Arcanum got it backwards--a gun build should have been easier than something like an archer build for the first half of the game, to reflect the reasons guns were actually used in real life. From my perspective, in an intelligently designed world, any style you choose has to be viable, simply because if it were totally useless it wouldn't exist in that world.
All that being said: the real measure of whether the gun build was useful was whether it meaningfully altered the game experience for you. It seems to me like it did--you probably had to hunt for ammo, pick your battles very carefully, and die a lot. That may not be the kind of gameplay that you actually enjoy, but it still sounds like a perfectly reasonable candidate for a major skill (assuming, again, that there was an actual good reason for your character to specialize in guns, like ease of use or firepower in the late game, in fitting with the game world).
Okay, what about a character in, again, Fallout 2 who doesn't have any investment in combat skills? It's more difficult than a build with a combat skill focus, but it's effective, it works, it solves problems, from start to finish of the game. And you get a drastically different experience.
Sure, that's a great example. I think most of us agree that both combat and noncombat skills should be viable builds, provided it's done intelligently. That doesn't mean that I should be able to get out of every single fight by smooth talking or bash my way around every problem, though (not that I think that's what you're saying). Just because I can specialize in guns doesn't mean guns need to solve every problem in the game, at least not from my perspective.
I think you're right that we don't really disagree about this... I think you just don't want skills that sound cool but are actually useless in most situations to be selectable as major skills from the getgo. I do
kind of agree with that, but only because if something really requires training (hence is worth making a major skill) there has to be a reason for the player character to pursue that skill in the first place. Obviously there are a lot of people who are talented at fairly useless things, but those things don't need to show up in the character selection sheet, as they can be roleplayed without any substantial impact on the game. It's just that in a fully realized world, I don't see that as being an issue.