Why? If anything, Akai Katana should've worked better because it's a hori game and it makes full use of a 16:9 screen. Mushi 1 is an old game that already has a port (though it's a bad one) and most shooter fans probably own it.
Deathsmiles 2X is also a hori and makes full use of a 16:9 screen, and most important of all, it's the direct sequel to Cave's most successful shooter of this generation and itself is their 2nd best-selling 360 port of this generation. There is also no way for owners of EU 360's to play it. This would have been a far better choice than any other Cave game.
Mushihime-sama is the prequel to Cave's 3rd best-selling console port of this generation (possibly tied with Espgaluda 2), and both the 360 and PS2 releases were region-locked (and as you mentioned, the PS2 port wasn't very good). It was also received quite favorably in arcades. It's true the 360 port didn't do that well in Japan, but I would place most of the blame on the availability of the PS2 port lowering the total demand in Japan. Though in retrospect, it probably would have been a good idea to skip this one too in favor of SDOJ.
Akai Katana bombed both in the arcades and on consoles. Its performance in the arcades at least should have been a red flag for any potential publisher. RSG had probably licensed it before seeing the port's initial sales numbers. After playing Akai Katana for myself, I think I got a good idea as to why it wasn't so popular; the scoring system is extremely arcane and loaded with micromanagement, and I strongly believe that's something that turned away a lot of not-so-hardcore shooter fans.
I don't see how RSG has "poor understanding of the shooter market". Shmup fans should've bought all of those games, unless they hate the bullet hell genre, because they're excellent. The problem is, there aren't many fans of the genre, and the few extra gamers who bought Death Smiles out of curiosity already had their fix and don't need more.
Not all shmups are created equal. The more hardcore fans, those who have a tendency of buying just about any shooter that gets released no matter what, are for the most part out of touch with the more casual fans who make up the majority of the overall fanbase. The shmup fanbase as a whole favors some shooters over others, and usually for good reason. We aren't running a charity here, nor should we. It is up to companies like Cave to release shooters that the market desires. If they choose to ignore sales trends and proper analysis of what sold how much and for what reasons, they should not expect to turn much of a profit. This goes for overseas publishers as well.
Look at all of Cave's best selling shooters lately and you should notice that nearly all of them have many things in common: colorful fantasy settings, "moe" characters, mostly intuitive/straight-forward scoring systems that are relatively easy to understand, multiple game modes to choose from, and in the case of the top 2 games; stage select and per-stage rank select. Perhaps Akai Katana wouldn't have done quite as badly if it had all of those things from the very beginning. Deathsmiles 2, Mushihime-sama Futari, Espgaluda 2, and the DDP series demonstrate that there's still a sustainable market for "good" high-budget shooters post-Deathsmiles 1, but it's up to developers like Cave to make them. I guess we'll know for sure once the SDOJ port gets released, but it seems to be doing quite well in arcades from what I saw, and that's usually a very good sign.
And if you want to more directly see RSG's "poor understanding" in action, go to the shmups forum and read the posts by a certain RSG employee, one who joined the forum and then asked the forum members to help him understand how to play DFK and Akai Katana, and this was after RSG had already released both games, and long after Cave sent RSG gameplay guides and superplays in English...
http://shmups.system11.org/viewtopic.php?f=1&t=43792
Deathsmiles 2? I mean, the Deathsmiles name is established somewhat due to the original coming stateside, but bringing over a game that appears like it's more of the same, except with bad, sub-Dreamcast graphics? That sounds like a worse idea than what they actually ended up bringing Stateside. Daifukkatsu at least had the Dodonpachi name going for it, which I'd think is probably more recognizable to shmup fans.
I don't think RSG misunderstood the shmup market in the West, the problem is that there is no shmup market in the US, especially for games costing more than $5.
No real argument regarding DFK. It's just not as good a choice as DS2X in my honest opinion. I also think you're downplaying the strength of the Deathsmiles brand.
For what it's worth, Cave stated that they were happy with the sales they got from the NA digital release of Deathsmiles 2X.
And in terms of gameplay, Deathsmiles 2X is actually more accessible than Deathsmiles 1, and has a more intuitive scoring system. I doubt there are very many shmup fans, casual or otherwise, who buy shooters mainly based on how good they look from a technical standpoint. I will agree that the artsyle isn't as good though.
From what I've heard, Deathsmiles 1 managed to sell over 25k copies in North America, and a similar number in Europe (more certain of the former than the latter). Part of the problem here is that Microsoft has a "50k units minimum for 1st print run" rule for retail 360 games sold in the US. This is extremely discouraging to publishers of niche games, especially to ones as niche as shmups. Their rules about that aren't nearly as strict in Japan or Europe. There are at least 25k people in North America who are willing to spend much more than $5 on a "good" retail shooter, but with the rules MS has in place in NA, ~25k just isn't enough to satisfy publishers, who then have to pay certain retailers (like Gamestop) a penalty in order to keep unsold copies in stock, on top of having to pay MS for more copies than the publisher can reliably sell.