Please read what I said more carefully. I've given you an exact elaboration of the scenario you suggested and a very obvious problem it could introduce (offloaded compute clogging the rendering pipeline in an unknown moment)
the interviews I have read in this describe exactly how the key to this has been keeping the Box working in harmony with the server.
the algorithms to do that have no doubt been worked on by MS Research for a while now figuring out how to implement it in the APIs and in the way the box handles the data so it works seamlessly as possible.
