Nintendo's profit margins right now are a long way away from it being a good idea to drop 60% of its revenue stream to abandon hardware. It would only begin to make sense when the the company starts losing maybe 20 or 30 percent of its worth year over year or something like that. So far Nintendo's been seeing YoY losses of maybe five percent of that worth (off the top of my head).
To go third party Nintendo would probably have to chop off at least half of its staff. A lot of its game design structure also probably depends on having hardware revenue as a bedrock, not to mention static hardware to work on. I'm not even sure I can imagine Nintendo developing games for a user interface Miyamoto & co didn't design. Basically, Nintendo would have to become a different company than it is today. As for how long it can hold out in its current state, I think Nintendo, with it's current money reserves, can survive at least one more console failure.
As for potential third party sales, I for one think people actually would follow Nintendo's games to PlayStation or Xbox. I think the one thing a lot of console gamers have desperately wanted for at least a decade now is essentially a PlayStation or Xbox that plays Nintendo games, whether that meant Nintendo making a successful console that's like a PlayStation or Xbox, or Nintendo going third party. I think a lot of PlayStation and Xbox owners, especially the hardcore ones, are still fans of Nintendo games and would be more likely to buy them if they didn't have to buy a whole other console to do so.
On top of this, Sony and Microsoft I think both admit they don't do as well as Nintendo when it comes to games for the younger market, and they'd both like to have the younger market in addition to their existing markets. Sony sees Nintendo's younger audience as gamers who'll potentially graduate to its own games. If there was even a whisper a third party Nintendo was into the idea of exclusivity deals, Sony and Microsoft would both be at Kyoto sucking copious amounts of dick to get Nintendo on board. I wouldn't be surprised if Microsoft offered to front the entire budget for a Mario platformer, Mario Kart, or most likely Pokemon. If the possibility for any exclusivity deal occurred and Sony still had a handheld, Kaz would pray to the heavens for Nintendo to put Pokemon on it.
But the ultimate question is how much more would those games sell? It might be worth it if Mario, Mario Kart, and Pokemon could sell the Call of Duty numbers they used to, but that would only be after Nintendo incurs far, far more losses than it already has.
Hardware is over half their revenue but nearly invisible to their margins. Console hardware only makes big money on third party licensing and network paywalls - both of which are alien to Nintendo as it is today. At the end of the day they can certainly do it if they wanted to but I'm uncertain it would end well for them.
Nintendo's hardware woes aren't the problem, they're a symptom of the real issue. Their biggest problem is creative stagnation. Nintendo churns out beautiful, polished and nearly perfect platformers in a world where they're increasingly a niche genre on home consoles. By cranking out Mario, Yoshi, Kirby and Donkey Kong games year after year, Nintendo appears to be stuck doing the same thing over and over again. Where's the diversity to expand and grow the audience? More importantly where are the signs they're growing up with their audience? That home console audience is older and increasingly demands more complex games. Sony in particular has been incredibly adept at getting the same studios that were churning out Nintendoesque platformers like Sly, Crash and Jax to make complex, adult oriented and still amazing games like Infamous, Uncharted and The Last Of Us. Nintendo's problem isn't that the Wii U isn't powerful enough for what they do, its that nothing that they do needs more powerful hardware.
Nintendo isn't going to fare any better as a publisher then they did as a platform owner if they don't start increasing the diversity of their games.
The thing is, Nintendo is aware of this shift in how everybody else, especially western developers, makes games, and they seem to be actively disinterested in it. Miyamoto has expressed awareness of the western consumer desire for increasingly complex games. But Nintendo has always been the company of accessibility and relative simplicity in game design. Nintendo doesn't like using hardware for the sake of using hardware, but rather taking advantage of hardware when its games might actually need it.
In some ways that kind of explains Japanese game development in general. Outside the big AAA fare, most Japanese developers don't seem to care about pushing hardware to the limit. Just look at most Japanese PS3 games right now -- the majority don't look anywhere close to the likes of Uncharted or GTA V. With the exception of Zelda, Nintendo has never really behaved like a AAA developer.
As for that affecting software sales, I don't think it does. Not in a world where Pokemon can still sell 12 million copies, Animal Crossing can sell eight million, and Mario Kart can sell upwards of five million. And that's on Nintendo's own hardware with few other games to justify it.
If you're talking about Nintendo getting more complex and "hardcore" games to draw people to its hardware, I think that'll only happen when Nintendo get's into bed with more western developers the way it has with Platinum. Maybe, MAYBE if Nintendo came up with another Zelda or Metroid-like franchise, but it's really impossible to tell when or if that'll happen again. X?