Support NeoGAF

[Wollan]

I'm interested in knowing how much GPU performance one can possibly hope to get back from foveated rendering at this point as devs like Valve, Nvidia and Sony have already started using various pixel masking techniques in the periphery vision (saving upwards of 25-50% of GPU performance). I'm suspecting that eye-tracking has to be pretty precise to allow another huge rendering gain on top of this again (I read somewhere that the human eye has full focus on only 3% on the overall image that we see so that's the end-potential).

Obviously eye-tracking has huge benefits outside of mere performance with the million game design possibilities it opens up.

 

[Crispy75]

I'm interested in knowing how much one can possibly hope to get back from foveated rendering at this point as devs like Valve, Nvidia and Sony have already started using various pixel masking techniques in the periphery vision (saving upwards of 25-50% of GPU performance). I'm suspecting that eye-tracking has to be pretty precise to allow another huge rendering gain on top of this again.

Obviously eye-tracking has huge benefits outside of mere performance with the million game design possibilities it opens up.

Improvements with proper tracking and foveated rendering are estimated to be 200-400% rather than 25%. It's totally worth doing.

 

[Onemic]

That's way unrealistic. Sometime between a year or two in my personal opinion. I suspect HTC and Oculus will have to react to early foveated birds at some point.

Is that eye tracking?

 

[Raticus79]

It also introduces other challenges though. in particular, even if you solve the tracking issue perfectly in hardware, you'd need to then render 200 FPS to make it work as far as I know. Probably almost every non-trivial VR game would be CPU bound on many systems when trying that.

One optimization we could make there is skipping rendering some frames when the eye isn't moving. If the pipeline for a frame-on-demand system was quick enough, maybe we could get away with still only displaying 90 FPS overall, with additional frames on demand for saccades. Cuts down on the power draw quite a bit.

Is that eye tracking?

Yup, that was referring to "early birds" beating them to the punch on foveated rendering. The fovea is the small point (a few degrees) at the centre of our vision where we can see the most detail. Foveated rendering is only rendering full res at that focus region where the player is looking.

 

[Crispy75]

One optimization we could make on there is skipping rendering some frames when the eye isn't moving.

Carmack, that you?

 

[Onemic]

Was Vorpx always a product you had to pay for? I could have sworn it was free.

Yup, that was referring to "early birds" beating them to the punch on foveated rendering. The fovea is the small point (a few degrees) at the centre of our vision where we can see the most detail. Foveated rendering is only rendering full res at that focus region where the player is looking.

Ah, thanks.

 

[Croatoan]

So, got to try the Vive last night at the Houston VR Meetup and I was blown away. Some of my Takeaways.

The Good
-Controllers and headset were comfortable and were really light
-VR clicked with me instantaneously (meaning it felt natural).
- Space Pirate trainer and Budget Cuts were awesome though budget cuts could be more interactive (more on that later)
- Presence is definitely a thing with these head sets. People are not messing with you.

The Bad
- Whats with the textured (scalloped) lenses? When I first put the headset on I was looking at the lens and it was extremely distracting. I was expecting like a smooth screen .
- Focus seems really bad. While Playing Job Simulator the whole world lost focus a lot which made everything look like a blurry mess. I don't know if it is because the headset was a little loose, or my eyes just kept trying to focus on the weird lenses.
- Are you never supposed to "Look with your eyes"? Often times only the middle of the screen was in focus or had good resolution. If I tried to look around with my eyes shit got blurry towards the periphery. I assumed that was just how it worked and explains why Foveated Rendering is sought after right now.
- I have really good peripheral vision and the screens seem too small (as in I could see the black of the headset in my peripheral). I hope we can someday get some type of curved wrap around screens (lenses or whatever you call them).
- The screen door effect never really goes away. I noticed the pixels, or whatever, instantly and in every game I tried. You tend to ignore it when you get pulled in ,and there is action, but when the focus goes blurry (again is the common?) and you get pulled out of the game the SDE becomes super noticeable.
- The Resolution is pretty disappointing coming from traditional monitors and TVs.
- VR feels really natural so when things don't work like you think they should it pulls you out of the experience. For example, when playing budget cuts I tried to stab a robot with the knife and it did nothing. Also, when I ran out of knives I tried throwing a dead robots body and it also did nothing. Furthermore punching a robot does nothing. Its these "do nothings" that pull you out of it. The game is still fun within its own rules though.

I really really liked VR. Most of my beefs have to do with the limited technology we have now and early game ideas.

One odd thing though, after removing the headsets I got a weird sensation that lasted for a few hours. Like my mind was having trouble adjusting to the depth of reality or something. It was odd and went away during the long drive home. Does this happen to anyone else?

Again, a big Thank You to Krejlooc and everyone else that puts on the Houston VR Meetup. Was a great evening talking to devs and enthusiasts. Hope to have something to bring to the show in a few months!

 

[Onemic]

From other accounts the blurriness is most likely from you not putting the headset on correctly.

 

[Croatoan]

From other accounts the blurriness is most likely from you not putting the headset on correctly.

is there a dial or something to change the focus? I found that if I mashed the headset tighter to my face everything was better.

 

[FlyinJ]

From other accounts the blurriness is most likely from you not putting the headset on correctly.

Yes, this was your problem.

What I do every time I put someone new in the HMD is bring up the steam overlay. I then tell them to slide the HMD up and down their face until the text on the steam overlay is most clear.

It also helps to tighten the HMD so it doesn't slip away from the "sweet spot" from head tilting.

 

[Croatoan]

Yes, this was your problem.

What I do every time I put someone new in the HMD is bring up the steam overlay. I then tell them to slide the HMD up and down their face until the text on the steam overlay is most clear.

It also helps to tighten the HMD so it doesn't slip away from the "sweet spot" from head tilting.

Gotcha, what about the eye look thing? Is that just how the current screens work?

 

[Raticus79]

Gotcha, what about the eye look thing? Is that just how the current screens work?

Yeah, it's a limitation of the lenses. I heard it's improved from earlier versions at least. By the way, that pattern is due to it being a Fresnel lens to allow the headset to be lighter and smaller.

One odd thing though, after removing the headsets I got a weird sensation that lasted for a few hours. Like my mind was having trouble adjusting to the depth of reality or something. It was odd and went away during the long drive home. Does this happen to anyone else?

Hmm, maybe there's some adjustment involved in going back to normal focus distances after the eyes have been relaxing and focusing at infinity for a while in VR?

 

[taoofjord]

Gotcha, what about the eye look thing? Is that just how the current screens work?

I'm pretty sure I calibrated my Vive correctly and still see blur on anything beyond what's directly in front of me. It seems to just be a weakness in the tech at the moment. I'm right there with you on a lot of "the bad" you listed. My main issue is pixel density/clarity and I'm already awaiting the next iteration with higher DPI screens.

That said, VR continue to blow me away and I've put 20+ hours into it at home now. Early adopters definitely have to deal with some technical drawbacks but it certainly doesn't make VR any less impressive to me. I feel the same as I did when I bought the first iPhone and iPad releases. I loved being a part of the early scene and getting super excited when the major technical leaps were made (App Store, Retina screens, early iOS feature updates). It's nice to have that feeling again as smartphones and tablets don't really wow anymore.

 

[Croatoan]

I'm pretty sure I calibrated my Vive correctly and still see blur on anything beyond what's directly in front of me. It seems to just be a weakness in the tech at the moment. I'm right there with you on a lot of "the bad" you listed. My main issue is pixel density/clarity and I'm already awaiting the next iteration with higher DPI screens.

That said, VR continue to blow me away and I've put 20+ hours into it at home now. Early adopters definitely have to deal with some technical drawbacks but it certainly doesn't make VR any less impressive to me. I feel the same as I did when I bought the first iPhone and iPad releases. I loved being a part of the early scene and getting super excited when the major technical leaps were made (App Store, Retina screens, early iOS feature updates). It's nice to have that feeling again as smartphones and tablets don't really wow anymore.

Am I correct in assuming that Foveated Rendering and associated eye tracking will fix this issue?

 

[Croatoan]

Yeah, it's a limitation of the lenses. I heard it's improved from earlier versions at least. By the way, that pattern is due to it being a Fresnel lens to allow the headset to be lighter and smaller.

Hmm, maybe there's some adjustment involved in going back to normal focus distances after the eyes have been relaxing and focusing at infinity for a while in VR?

I have perfect vision(like I ace vision tests) but for whatever reason I squint a lot. I don't know why, neither has any real doctor I have seen. It just occurred to me that I might not have been doing my normal squint thing in VR and the muscles in my eyes are confused .

My issue gives me headaches all the time btw. Maybe VR will fix me.


fuck sorry for the double post. Meant to add this to the one above .

 

[Crispy75]

Am I correct in assuming that Foveated Rendering and associated eye tracking will fix this issue?

Not really - blurring towards the edges is a limitation of the optics. Higher resolution screens will help, but without a fundamental redesign of how the image is focused, the edges will always be worse off.

 

[Croatoan]

Not really - blurring towards the edges is a limitation of the optics. Higher resolution screens will help, but without a fundamental redesign of how the image is focused, the edges will always be worse off.

Hmm, I thought the point of it was to make where the eye is focusing clear and blur the periphery for performance gains. If the eye is looking towards the edge of the lens and its still blurry I don't really see the point of FR.

 

[Guardian Monkey]

FR will track your eye so when you look to the corner the image will be clear.

I am still wanting wireless before any resolution bump imo. Regardless of what everyone says after two weeks with the Vive the cords do get in the way.

 

[GuitarAtomik]

FR will track your eye so when you look to the corner the image will be clear.

Yeah, but if the image is blurry because of the optics, won't it still be blurry?

 

[pj]

Yeah, but if the image is blurry because of the optics, won't it still be blurry?

Would curved screens help with edge blurriness?

 

[Crispy75]

Would curved screens help with edge blurriness?

The optics get really complicated. A flat screen emits a bunch of parallel rays. Easy to focus. Doing the same for converging rays - not so easy. There is a huge amount of R&D to be done before we replace the flat screens & magnifying glasses design.

 

[Fafalada]

you'd need to then render 200 FPS to make it work as far as I know. Probably almost every non-trivial VR game would be CPU bound on many systems when trying that.

Not necessarily - even if eye movement necessitates refresh at 200fps, that tick would need only reshading - ie. basically zero cost on the CPU to do between frame updates to "reveal" detail.
On the other hand - we already have working multi-resolution examples with completely static foveated-blur, and they tend to be perceptually very similar(depending on how agressive you go) to full-screen updating. Of course idea behind Eye-tracking is to go a lot more agressive, but it does raise questions about how well we would really perceive the detail shifting if it wasn't updated at 200hz.

 

[Raticus79]

Not necessarily - even if eye movement necessitates refresh at 200fps, that tick would need only reshading - ie. basically zero cost on the CPU to do between frame updates to "reveal" detail.
On the other hand - we already have working multi-resolution examples with completely static foveated-blur, and they tend to be perceptually very similar(depending on how agressive you go) to full-screen updating. Of course idea behind Eye-tracking is to go a lot more agressive, but it does raise questions about how well we would really perceive the detail shifting if it wasn't updated at 200hz.

Cool, that sounds like it could work.

Wonder if anyone made a demo where the focus region is your mouse pointer. Even without eye tracking, it could be fun to get supersampling on that focus area in an otherwise normal resolution picture.

 

[wonderpug]

The optics get really complicated. A flat screen emits a bunch of parallel rays. Easy to focus. Doing the same for converging rays - not so easy. There is a huge amount of R&D to be done before we replace the flat screens & magnifying glasses design.

If a flat screen emitted parallel rays, you'd have a display doing something like this:

If you were looking at this sentence:
The quick brown fox jumps over the lazy dog.

Your left eye could maybe see "The quick", but nothing else, and your right eye could see "over the", but nothing else. You wouldn't be able to move your eyes to see the rest of the sentence if the rays were parallel, you'd have to move your entire head left and right.

Rather than being parallel rays, the light from a display goes in every direction. It wouldn't project letters onto your face like in that picture above; each pixel is shining everywhere on your face at once.

 

[Crispy75]

Yes you're right. Ignore my handwaving

 

[Candescence]

It's eminently practical, because lighthouse is inside-out it scales basically infinitely to more tracked points.

Well, I was more thinking about how the limits of scaling down sensor devices so they're light and small enough to be worn on, say, fingers.

But I can see body-based trackers coming in sets of four - one torso tracker, a pair of tracker sets that are worn on the elbows with 'gloves' attached for hand and finger sensors, and two sets for the knee and feet. IK at that point immediately extrapolates the way your body moves from there, since the human body can only move in certain ways due to the way the joints work. The number of tracking devices is minimized while still achieving good results.

 

[viveks86]

So you'd actually see the lowfi imagery strobing as you flicked your eye over the scene. So maybe what they need to do (if the hardware can do it) is turn off low-persistence during a saccade.

Yup. The fix will most likely come from changes to screen/optics and other reprojection trickery. Brute forcing rendering at the same rate as tracking is an unattainable goal for several years at least IMO

Should be interesting to see how Fove does it this year. I believe they are going with 120hz tracking, 60 fps rendering with reprojection to 90.

 

[Croatoan]

So, if Oculus and Valve/HTC iterate quickly (1 to 2 years) what are the realistic features or changes that would make it into cv2 for both companies?

 

[Chairmanchuck]

So, if Oculus and Valve/HTC iterate quickly (1 to 2 years) what are the realistic features or changes that would make it into cv2 for both companies?

Maybe eye-tracking?

 

[Monger]

From other accounts the blurriness is most likely from you not putting the headset on correctly.

It took me a few days to really find the right fit. The focus was pretty narrow before I finally got it dialed in.

It still blurs at the far edges but you naturally adapt to how you look around so it becomes far less intrusive.

 

[Crispy75]

Maybe eye-tracking?

and a resolution bump. 1440?

 

[GuitarAtomik]

So, if Oculus and Valve/HTC iterate quickly (1 to 2 years) what are the realistic features or changes that would make it into cv2 for both companies?

1 year: Better optics, resolution, FoV, and headset ergonomics

2 years: Above + Foveated rendering

Though I think foveated rendering is probably doable in a year, I don't see them focusing on it without improving some of that other stuff first and I think throwing it in so soon is going to mean a bigger, heavier headset. Wireless I imagine is going to be further down the line.

 

[bj00rn_]

1 year: Better optics, resolution, FoV, and headset ergonomics

2 years: Above + Foveated rendering

Though I think foveated rendering is probably doable in a year, I don't see them focusing on it without improving some of that other stuff first and I think throwing it in so soon is going to mean a bigger, heavier headset. Wireless I imagine is going to be further down the line.

I just wonder if we're going to get to 4k HMDs without foveated rendering and/or some gpu giant leap for mankind.. Better optics is sorely needed for sure. I'm frankly surprised that the optics aren't already better than they are. It kinda even worries me.. and especially in the context of foveated rendering. Perhaps we need completely different technology to get high end to move on from where it is today.

 

[viveks86]

I'm frankly surprised that the optics aren't already better than they are. It kinda even worries me..

Yeah this is what worries me the most. Do lenses evolve at the rate that electronics do? I guess not?

 

[Onemic]

I just wonder if we're going to get to 4k HMDs without foveated rendering and/or some gpu giant leap for mankind.. Better optics is sorely needed for sure. I'm frankly surprised that the optics aren't already better than they are. It kinda even worries me.. and especially in the context of foveated rendering. Perhaps we need completely different technology to get high end to move on from where it is today.

Is that 4K per eye, or combined? Either way it would take a while, like not for the next 2-3 years at the very least.

 

[Zalusithix]

Yeah this is what worries me the most. Do lenses evolve at the rate that electronics do? I guess not?

Light behavior and lens material properties are quite well known at this point. We've been working with optics for thousands of years now. Electronics, in comparison, have been around for decades. They're nowhere in the same ballpark when it comes to the rate of evolution. We're literally clawing at the the last possible straws with lenses for improvements.

 

[sammex]

So do we reckon Oculus/HTC will publicly release sales figures? What would you estimate they'd be happy with to greenlight working on a second hardware iteration, or do you think they've maybe already started the process based off preorder demand?

 

[Tain]

So do we reckon Oculus/HTC will publicly release sales figures? What would you estimate they'd be happy with to greenlight working on a second hardware iteration, or do you think they've maybe already started the process based off preorder demand?

I would be shocked if both weren't working on second iterations.

 

[GuitarAtomik]

I just wonder if we're going to get to 4k HMDs without foveated rendering and/or some gpu giant leap for mankind.. Better optics is sorely needed for sure. I'm frankly surprised that the optics aren't already better than they are. It kinda even worries me.. and especially in the context of foveated rendering. Perhaps we need completely different technology to get high end to move on from where it is today.

I don't think we'll get to 4k for a few years. I think they'll focus on a combo of incrementally better resolution along with pixel density (or some other way to lessen the effect of SDE).

 

[Zalusithix]

I would be shocked if both weren't working on second iterations.

If they haven't begun at least preliminary work on a successor, then they've failed. There's far too many ways to improve the headsets at this point to ship a product and sit on your ass twiddling your thumbs. There should be near constant R&D work being done.

 

[viveks86]

Light behavior and lens material properties are quite well known at this point. We've been working with optics for thousands of years now. Electronics, in comparison, have been around for decades. They're nowhere in the same ballpark when it comes to the rate of evolution. We're literally clawing at the the last possible straws with lenses for improvements.

That's what I feared

So basically we are fucked? :'(

 

[Zalusithix]

That's what I feared

So basically we are fucked? :'(

Depends what you mean by fucked. Everything can be worked around. It's just a matter of weight, size, and complexity. Ideally we have to work our electronics around the limitations of optics instead of the other way around.

 

[Crispy75]

Incremental improvements to this technology will only go so far. You want something like https://en.wikipedia.org/wiki/Virtual_retinal_display to really make a leap

 

[sammex]

Incremental improvements to this technology will only go so far. You want something like https://en.wikipedia.org/wiki/Virtual_retinal_display to really make a leap

Looks awfully like this

https://www.youtube.com/watch?v=wW0WRqnLYMw

You sure it won't suck out my neural energy?

 

[Croatoan]

I don't think we'll get to 4k for a few years. I think they'll focus on a combo of incrementally better resolution along with pixel density (or some other way to lessen the effect of SDE).

Are pixel density and resolution tied to each other?

What makes near lightfield displays one of the possible future technologies for vr?

Like are we always going to be using lenses or can you like use lasers or something to draw directly on the retina or whatever(isnt that how VR worked in RPO).

 

[GuitarAtomik]

Are pixel density and resolution tied to each other?

Not necessarily (unless I'm using the term "pixel density" wrong). A 100" screen that maxes at 4k is going to have more space between each pixel than a 50" screen maxing at 4k. It's all in how the actual pixels are placed. The PSVR for example uses lower resolution screens than Vive and Rift but apparently the screen door effect is about the same (and from some reports actually less noticeable).

What makes near lightfield displays one of the possible future technologies for vr?

Like are we always going to be using lenses or can you like use lasers or something to draw directly on the retina or whatever(isnt that how VR worked in RPO).

Given the apparent limitations of the lenses, that might be the solution eventually. I don't know enough about the field to know if we're there yet though.

Now having said all that, I'm not an expert. Plenty of people on here are more knowledgable about this stuff and will probably correct me on it.

 

[BeEatNU]

For people who have both.....

If you had to do it all over again knowing what you know now. What would you choose?
*I currently have a rift, end up canceling my vive april order*

 

[Owlowiscious]

silversurfer's ratcheting: the game: http://store.steampowered.com/app/462480/?snr=1_7_7_230_150_1

interesting way to implrement ratcheting without breaking immersion. I wonder if anyone else will get ratcheting going

 

[Sam Bishop]

That's what I feared

So basically we are fucked? :'(

Are you that defeatist? A decade ago, we barely had a smartphone in the sense we have them now. 2G was the norm for mobile data transmission. YouTube was a place where people uploaded cat videos and relatives commented on them.

We have come SO far in the last decade, and the advancements since then have only accelerated. There is literally no stopping the progression of you putting something the size of a contact lens in and seeing the entire world change.

4K in 10 years will be a blip, a mound we step over on the way to completely photo-real, wireless, fully immersive VR. This is the best time to be alive. The BEST.

I got to live through the Internet becoming a thing. You guys will get to live through reality itself becoming passe.

 

[Lunar FC]

Has anybody tried getting NewRetroArcade running on the Vive?