It can't be and shouldn't be a matter of raw performance. I mean it's just a user interface, there's no reason you should need to quadruple your performance to prevents app from hanging and button presses to be registered after several seconds.
I don't know exactly how MS could mess it up when they're supposed to have the most expertise on that field, but there's definitely something wrong in how they're handling the Xbox OS. I'm not even talking about design, which is (or was) mostly fine for me, but just responsiveness.
And don't start me on Cortana, which runs better on my old Lumia phone than on a gaming console.