LOL.
Surely 99.99% of people would not notice or think that it is that . What's more, you will probably even detect the extra sharpness of the image sooner than artifacts that only occur in specific details of specific scenes where VRS has application. As I say, in daytime and bright scenes it does not apply.
And this is the case in the vast majority of cases. It is usually applied to areas/details further away from the scene and darkened areas. While you play it will hardly be something that catches your attention and it certainly does not deteriorate the IQ in general, much less because it is only limited to objects, details and specific shadow areas, not to the general image as you try to affirm.