I think the big part the industry in Japan was missing is that there is no single hardcore audience, they have to (re)build them every gen there.
I think the Valkyria interview that just happened is one of my favorite examples of that.
In 2008, Valkyria Chronicles 1 released. Between the base game and the budget re-release, it sold 141,589 + 95,667 = 237,256 copies.
In 2011, Valkyria Chronicles 3 released, and sold 152,659 copies, which was at least still up from the initial release of Valkyria Chronicles 1.
Now, in 2016: http://www.neogaf.com/forum/showthread.php?t=1270939
GAMER: Speaking of which, the Valkyria series is ridiculously popular overseas, isn't it?
Ozawa: A surprising thing about that... when we revealed the first YouTube trailer for Azure Revolution in November last year, all of the comments on the video were in English! (Laughs) We had each line translated for us, one by one.
It's not surprising to me that there were English comments on the Japanese trailer, because it's a popular series overseas. However, what's really striking here is that there were zero comments in Japanese.
Valkyria Chronicles released in 2008 and then had a re-release on PC in 2014 that was clearly driven by everyone remembering positive word of mouth about the PS3 game. Since the PSP was a non-entity by the time of Valkyria Chronicles 2, this is a series that largely started and stopped in 2008 overseas. However, its Western popularity was enduring since the people playing games in 2008 are still playing games like this today. In Japan, it feels like they're basically having to launch a new IP only five years later, which makes the amount of changes they're making make a lot more sense in context.