It did better in Japan (~200k LTD?), but it's still a steep decline over previous iterations. It doesn't make any business sense to keep releasing BA games if that's the type of sales that it's gonna get.
From reading the Iwata Asks on it, I get the impression that Iwata and Dr. Kawashima have sort of a camaraderie between them that plays into the game being given a greenlight. For example, the only reason they started a Brain Age 3 was because Dr. Kawashima had an idea for a new type of training that he wanted to use on the Japanese Winter Olympic athletes in the Skeleton event, so he went to Nintendo and asked them to make it for him. Once they had that, they started experimenting with more ideas (even using his brain blood flow helmet device to confirm that the exercises were hitting the right parts of the brain) and worked it into the format they did now. They put a lot of care to get the character of Dr. Kawashima in the game "just right" to be encouraging, so the player will keep trying instead of getting frustrated and stopping, since the game is pretty hard. A game-wide achievement system is in place to encourage you to keep going. Many of the fun exercises from Brain Age 1/2 reappear under the supplemental and Brain Training sections. In Japan, you could even choose to share your results for the first 30 days over SpotPass with Dr. Kawashima's University directly to help their studies of how continued training affected brain functions.
This game really seemed more like a personal project (like Electroplankton) than a "we need this to sell millions" project. From that sense, they probably are fine with the Japanese sales (can't see how < 5K first month could be anything but disappointing for retail in the U.S., though), and whether there's another really depends on Dr. Kawashima's area(s) of research than on internal Nintendo directives.
If they really wanted to make this more of a casual hit, then they probably should've gone with the iOS nickel-and-dime method of charging $0.99 for the first training (or just outright F2P), and then making everything else add-on content. If they wanted to go that route, that'd work, but it's just a personal thing that I'd rather buy the whole game up front than have these nebulous extra fees hanging around after purchasing to get the full game. Like you said, if I wanted to keep the games going (if there are even to be more), I should be okay with whatever the market is dictating to get there, though.
