I disagree with you, and I'll counter your argument with an analogy.
Let's take the example of cellphones and cameras.
In the past cellphone cameras were crap (no discussion there), so if people wanted a portable camera that took "fine" photos, you bought a slim camera, and if people wanted to take great photos, they bought big ass cameras.
Now phone cameras are very good, so cellphones took the "casual" photo market or the "fine" photo market, but the market for big cameras is intact, because cellphones will never be able to replicate those cameras, it's impossible.
With gaming it's the same, Nintendo benefited from the underserved casual gaming market, so it sold lots and lots of systems to people that are not dedicated gamers, they wanted good quality titles, "fine" games to pass the time. Now cellphones are taking that market, because there is no way Nintendo or Sony can effectively retain those markets and they are not interested in competing against $1 games.
But there is no possible way that the cellphone market can bite into the dedicated gamer market, because if they tried, the phones themselves loose the appeal of being practical phones (no battery, extremely expensive, bad controls, etc), so even though the phone market effectively took a big share of the handheld market, there is no way for it to take more of it.
The portable market is probably reduced, although I do believe that the "core" market expanded due to game companies efforts during this generation, so it now may be a 100 million person market instead of 200 millions, it is a drastic difference, but it still is much bigger than the 70 million GBA market.