Much like a lot of people on GAF, you're either a 10/10 or a 0/10.
That's why Steam uses thumbs-up / thumbs-down, and why market researchers use net promoter rather than taking a 0-10 rating at face value. It's also why Uber counts 1-3 star reviews as 1 star (so if you're one of those weirdos that gives out 3 stars, know that it doesn't matter).
People don't think in terms of numbers when it comes to experiences.
How much fun did you have on a scale of 1-10?
That's a ridiculous question when you really sit down and think about it. A better question is "Did you have fun?"
OpenCritic's been looking at a % recommended system, but we're starting to see some flaws there too. One issue is that some publications simply recommend games disproportionately often. Kotaku recommends 85%+ of the games it reviews. Around 50% of all games IGN reviews get an 8.0 or better.
Some publications also just don't have scales that let them do much. Telegraph and others use 1-5 stars, with no half stars, which means the jumps are from 100, to 80, to 60. If you say 4 or 5 stars is recommended, then you've got the same problem: they're recommending over 80% of their games. Giant Bomb has given out 4 stars or better to 63% of all games on OpenCritic. How do we draw the line on where it's "recommended" or "not recommended"?
The next feature for OpenCritic is going to be showing where a game falls on the distribution of all games that have been reviewed (ex: Tales from the Borderlands is in the top 3% of games scored on OpenCritic. Fallout 4 is in the top 4.5% of all games scored on OpenCritic). After that, we'll be looking at highlighting how publications actually score, showing each publication's rating distributions. From there, we can hopefully get some feedback and find some ways forward on a better scoring system (such as % critics that recommend this game).
Edit: One more thing on selection bias... It's easy to forget that there's a huge selection bias when it comes to games and review scores. Shitty games get reviewed significantly less often. And there are a
lot of shitty games out there. It's easy to forget that there's a massive selection bias when it comes to these scores.