Assuming everybody that bought a Xbox would have bought a GameCube has no basis in reality. We have no exact way of knowing what would have changed, but a 100% conversion rate is simply not on the cards. The main draw of the Xbox was shooters, multiplayer, DVD playback, newness and interested PC players. None of which the GameCube offered. The GameCubes main draw was Nintendo's offerings, so those who cared for it picked it up. The GameCube was also dirt cheap so I doubt cost was a factor. But 20 vs 30 million would have had little impact. The Wii would have still been the Wii and enjoyed the success it did.
In the end the decision that led to Nintendo focusing on handheld was to do with the failures of the Wii U and not the GameCube, arguing otherwise ignores the massive success of the Wii, which was a home console. But Nintendo going its own way has been on the cards since the 64, and abundantly clear with the Wii launch.
As for the modern console space, Nintendo has zero incentive to actually move from the hybrid (mainly handheld) approach they have now. Having another console is just going to split development and sales with very little benefit. People buy Nintendo because it's Nintendo, not because they want to play CoD on it. With DLSS future Switch hardware will have little issues in displaying 4k content anyway in docked mode. The current approach for Nintendo clearly works for them, it combines the utter domination they have enjoyed in handhelds for over 30 years without needing to have a home console to split the playerbase.
With Nintendo not interested who else would bother and have the resources competing? Apple? Amazon? Google? All seem unlikely, especially with the failures in gaming that Amazon and Google have had.