See, so was I - and it was fine BUT the TV remembers settings for each input, so my Xbox would automatically have game mode and a few other changes I did enabled, each time I switch to it. When I was using my receiver, It was in effect only one input in the TV, so I'd have to manually do this every time. Unless I am missing something?
Now I just use ARC to my AV receiver for the surround sound...it feels a bit...wasted..
This is where I am now too. I run the PS4 and Tivo separately from the receiver which has bluray player, PC, chromecast...
PS4 so I can keep game mode settings on its own input. Tivo because it does not have HDMI-CEC so it does not auto-switch inputs.
ARC back to receiver.