Here's an easy one. Just the process of giving a birth at a hospital can cost you up to 1,000,000 yen. Then once they hit junior high school age, if they're in public school, even though that's free if you want your kid to get into a decent high school you're going to have to pay for cram school. Then it's high school time and you have to do the exact same thing if you want them to get into a university. Oh by the way, you have to do the same thing if you want them to get a good job or onto a graduate course , too. So education in Japan being free is a total myth actually.
That's just the education part of having kids. Japan is expensive as shit and that is something immigration cannot fix because the vast vast majority of immigrants in Japan are from poorer Asian countries (think Vietnam, The Philippines, etc) who take on low income jobs, and can barely afford to survive as it is. There will never be an influx of wealthy Westerners looking to build families because, most Westerners treat their stay in Japan as one long holiday. A few might get married, but most don't want to integrate at a local level, or learn the language.
I think the trend is irreversible unless there some huge upswing in the economy (unlikely) and people can feel their own future is more secure, but it will level out at some point.