You think they waste this 5 years of engine, game research and assets just because this game did not sell enough. I'm sure they invested this amount of time because they want this as a franchise(even sony too thats why they still market this game by adding Galahad in Super time ultra force, LBP costumes, PS4 cover plates, huge marketing it had for launch, bundles etc.,) which is why we did not have proper ending in this game, perhaps they spent so much time on game engine and world instead of focusing on content in their time to get it released even with a delay. I can see they can do this even better in sequel addressing the problems it had with 10+ hours content and multiplayer with less cinematic. This game problem was just did not have enough content, too much cinematic gameplay and no multiplayer for a $60 game in these days of multiplayer domination and users asking replayablity.
The bundles and guest appearances don't say anything about Sony wanting to go forward with the IP. They still have a bomba game to salvage and they are doing exactly that.
We don't know how flexible the engine is but judging by The Order, to me at least, it looks like it was made for one thing and one thing only, and that's to push visuals and anything that benefits the realism the game is going for unlike something like FOX engine which looks very flexible. Wouldn't be surprised if it were solely made for this IP only and with no grand scheme in mind. By that I mean the engine will need to be heavily retooled to work with other IP's/franchises.
Sony have let go of developers that have done much better like Liverpool. They are doing great but to fund a sequel to the order is not using your revenue wisely and unlike a decade ago they desperately need this revenue. Realistically RAD better start looking at other publishers.