FFXIII sold 1.5m in its first week, so, yes, 500k less than that would obviously be drastically down.
I think 1 million would actually be pretty great, as it is above everyone's predictions, especially when you count in the fact the declining consoles sales, and the fact that no PS4 game has yet to even reach that number in Japan.
XIII may have sold well, but it came off of previous good games to build the hype. The XIII series really soured people on Final Fantasy. If XV reaches a million sold it would be a testament that the series can still thrive in Japan's changing market (imo).