It's not crazy at all. When you're playing on a TV or monitor your brain is very aware that you are sat in a room. When you use a VR headset it creates the illusion in your brain that you are moving about, but your body is actually still, so this miss match is what can make you feel ill. It's the opposite to when you're in a vehicle and you're not looking out the window (looking down at your phone etc.), your body is moving but your brain can't see why/where.
I've been thinking about what actually happens to how you think when you are in VR, in particular for me some moments in games have been literally utterly terrifyingdifferent to be scared like when you watch a scary moviemore like: I'm actually going to die from this horrific situation.
So VR is not like being somewhere else: when I play Skyrim I don't think I'm in Skyrim, my consciousness knows I'm in VR playing a game; but I think something else is happening as well: your unconsciousness does think you are in a different real place. your unconscious brain is behaving as though large elements of your sensory input are actually real, i.e. the bits of your brain that calculate trajectories are treating the world as a real 3D place.
This is why in Farpoint, when I get careless and let a Brute get close to me and it starts attacking me, I behave like I'm really being ripped apart by a 10 foot space spider and uncontrollably scream and shriek (and it feels like its beyond my control). Also because being attacked by aforementioned space spider is a serious risk you brain prioritises that feeling over your conscious knowledge of just playing a game.
In Skyrim I'm finding fighting the dragons when they are in close proximity on the ground, especially when they start advancing on you, somewhat uncomfortable. And this time I can't blame my arachnophobia: its not as intense as with spiders, but there is still this deep unsettling feeling that I'm in mortal danger. Its awesome!