This looks sick. It’s still janky and not at all possible in real-time for now, but it gives an interesting glimpse into the future.
Letting the filter handle the image while allowing the game-logic to manage simulation consistency (something LLMs still aren’t great at) seems promising.