The seventh gen problem for Japan was a resource allocation problem. Mobile was booming and the DS, PSP and later Wii became the dominant platforms in the latter half of the 2000s. There was a huge shift in how the Japanese consumed games and so naturally the money followed them. These systems were all very underpowered compared to PS360. In the DS' case, they were targeting PS1-tier visuals, while Wii was a GameCube. That made it almost impossible to port HD games directly across and necessitated having teams making totally different versions of games. In many cases Japanese developers just shifted their focus to those platforms entirely for the generation. This also took away valuable experience making HD games, so that by the time the Japanese developers returned to the fold that Western studios had been occupying, they were far behind technologically. Thankfully they have not only caught up but have even exceeded Western studios.
A minor correction and an addition:
1) Unlike in the west, japanese third-parties never fully embraced the Wii cause the kind of "low" budgets were absorbed by the NDS and PSP projects so while Wii ended up outselling PS3 in Japan, mostly on the back of first-party games, I wouldn't say it attained a dominant position.
This is easily demonstrable by looking at the stark difference in third-party games sales on Wii between USA and Japan:
2) There are historical reasons that made the transition to more complex games more painful for japanese game publisher behind what were the dominant platforms in Japan at the time.
- the japanese gaming industry was heavily reliant on custom made software development practice (and each time they would start from scratch even if it was a sequel on the same console) due to the cultural heritage tied toh the arcade boards (and their custom hardware) and a widespread "winner takes all mentality" that meant game developers tended to aggregate around one or two winners (typically Nintendo and PlayStation) and develop for a very restricted number of dominant hardware per generation.
As an example of the above here a citation from a key manager in one of the biggest japanese third-party:
Yoshihiro Maruyama (Executive vice president, Square U.S.)
"Square was focusing on PlayStation only whereas EA was a multiplatform game company. They kept publishing on Nintendo as well as Sega as well as PC. So there was a bit of a different approach because they could spread out marketing across different platforms, but you know, we only published for the PlayStation. So there was that kind of discussion between us — why were we just doing it? [Laughs] We told them that the Japanese game market was always a “winner takes all” type of market. It used to be Nintendo. Now it’s PlayStation, so we had no intent to make any games for other consoles like Sega or Nintendo. But the U.S. market was twice as big. So they understood it, but they still wanted us to make games for other consoles. Which Square, you know, firmly refused."
- Due to the "Galápagos syndrome" Japan has many industry in which they excel but software isn't one of them, with the exception of game software which requires a lot of creative input besides software engineering proficency and for which they enjoyed a head start thanks to Space Invaders (arcade boom) and Nintendo (console boom in Japan and reviving of the console industry in North America).
- Japanese game publishers were reliant on the home market which could have very different tastes compared to other big markets like USA or the european main markets (UK, Germany, France, Italy, Spain).
Beyond tastes, if one look thoroughly at sales data for a long stretch of time for the japanese market it should notice that the budget for the some of the most popular games in Japan tended/tends to be lower compared the software that is more popular in USA.
[As a generalization] The japanese game publishers were more focused on "cheapness" than western publishers and their mentality of "go big or go home".
To face the challenge of arising costs japanese publishers allied between them so in the '00s we got the formation of Square Enix, Sega Sammy, Bandai Namco, Koei Tecmo, Spike Chunsoft, Takara Tomy and Konami absorbed Hudson Soft.
Even thus japanese game publishers remained risk adverse.
Now it's clear that with these ingredients japanese game publishers were set to struggled mightly in the initial phase of transition to HD games (meaning more complex games) that required higher budgets and higher software proficency, and to lessen the risks it enforced to adhere to a multi-platform strategy with a view that encompasse a worldwide audience instead of looking at the home market and then hoping the game would be received well in the west.
It should be noted that despite always been the biggest console publisher in the World, Nintendo should be treated differently from the rest of japanese game publishers because they function differently (Nintendo is a game publisher with its own platform business) and their leadership always displayed higher than average foresight.