Support NeoGAF

[hawk2025]

I find it funny that people complain about score aggregators, especially with respect to how 5 or 10-scale reviews are aggregated.

...But if the original reviewer went with a five-star rating, they already decided that their opinion would be aggregated into chunks, and putting it into a scale that goes from 0 to 100 is quite literally only a linear transformation of that.

No information is lost from the transformation, and we can always invert it. Some of the points being made here make no sense.

I get that, but it's not like a "5" point scale can't accommodate it.

If reviewers, all of whom are keenly aware of how Metacritic operates, don't like how 4/5 is interpreted, they should move to a 10-point scale. It's not on Metacritic in the slightest.

Exactly. People are questioning a mix of basic math and a reviewer's own decision to how to summarize and represent their opinions. It's not a good look. The only situation where Metacritic would be losing information when transforming scores is if they collected scores from a source that grades beyond a 100-point scale.

 

[ByWatterson]

That'd be (at least) a 10 point scale.

I get that, but it's not like a "5" point scale can't accommodate it.

If reviewers, all of whom are keenly aware of how Metacritic operates, don't like how 4/5 is interpreted, they should move to a 10-point scale. It's not on Metacritic in the slightest.

 

[Izuna]

Metacritic should have a score converter, where it recalculates the scores according to another scale. SO many games would be 4/5

 

[ByWatterson]

Metacritic of about 85 is a great score for a First person shooter. When was the last time a FPS got a Meta of 90+ ? Looking in my crystal ball i doubt their will be a FPS in the next 5 years other than Half Life 3 that gets a Metacritic of 90 or over, if it ever came out

The Taken King is at 90, I believe, if that counts.

 

[LegalEagleMike]

The Taken King is at 90, I believe, if that counts.

Only if you look at the system that has 8 reviews. Taken King is at 85 for PS4 where all the reviews are.

 

[carl32]

The Taken King is at 90, I believe, if that counts.

I just checked its 85, i ain't got a clue what the last Fps was tho, it maybe a Halo game

 

[Synth]

I get that, but it's not like a "5" point scale can't accommodate it.

If reviewers, all of whom are keenly aware of how Metacritic operates, don't like how 4/5 is interpreted, they should move to a 10-point scale. It's not on Metacritic in the slightest.

It FACTUALLY can't... there's no discussion for this. It doesn't matter if you use the numbers 1 through 5, 1 through 10, or 0 through FF... the moment you represent a 9 accurately, you're not on a 5 point scale. It cannot accomodate it. Fact.

It should not be on the review outlets to change their preferred review scale to sync with a completely external website that operates solely for their own gain. Many of these outlets massively predate Metacritic, and may potentially outlive it too it something else comes along and does to Metacritic what Metacritic did to Gamerankings. They're writing the review for their website. When they rate it a 9, that's the score they want it to have, not the aggregate that Metacritic will produce after.

 

[shoreu]

It's hilarious to read how people are coping with the low review scores. The game does suck and HaloGaf's ego wont allow them to admit it.

Lol pitiful attempt.

 

[ByWatterson]

Only if you look at the system that has 8 reviews. Taken King is at 85 for PS4 where all the reviews are.

Ah, true.

 

[ByWatterson]

It FACTUALLY can't... there's no discussion for this. It doesn't matter if you use the numbers 1 through 5, 1 through 10, or 0 through FF... the moment you represent a 9 accurately, you're not on a 5 point scale. It cannot accomodate it. Fact.

It should not be on the review outlets to change their preferred review scale to sync with a completely external website that operates solely for their own gain. Many of these outlets massively predate Metacritic, and may potentially outlive it too it something else comes along and does to Metacritic what Metacritic did to Gamerankings. They're writing the review for their website. When they rate it a 9, that's the score they want it to have, not the aggregate that Metacritic will produce after.

Sorry....what exactly are you suggesting Metacritic should do....smell test/ballpark how it translates? Apply subjective reasoning to hard math?

 

[digoutyoursoul]

This thread is a great reference point for future use

 

[Amentallica]

Fuck it. Just use z scores and standard deviations to rate games.

 

[BokehKing]

I just checked its 85, i ain't got a clue what the last Fps was tho, it maybe a Halo game

Far cry 3 and borderlands 2 got in the 90s
Far cry 4 was only like 88 or 89

 

[balohna]

Everything reviews worse these days, 85 is fantastic in 2015. It was always good, but the days of every big release getting 90+ are over. You'll always have a site that just doesn't get it or is offended by it or is bored of the series, etc. Or a reviewer that just thinks it's mediocre.

 

[Izuna]

Common core is destroying America guys

 

[EGM1966]

That's because the Xbox One version only has only 8 reviews recorded, whilst the PS4 version has 71….

To be fair though that's a great example of the issue with the whole MC number as a comparison and evaluation point. Different games or even (as in this case) the same game on different platforms can have different numbers of reviews that MC will also weight in some mysterious way further making it invalid to use the resulting number as a comparison point.

Comparing MC scores is intrinsically false as the underlying factors the make up the percentage may not be apples to apples.

A decent game with less reviews is more likely to have a better MC. A game with more reviews is more likely to have a lower MC.

As a list of handy links MC is fine, as the "final number" for comparison its hugely suspect yet people have fallen into the trap of using it all the time (and not just for games). It's dumb but I'll stop ranting...I think the spirit of my old statistics teacher just possessed me for a moment there.

 

[Synth]

Sorry....what exactly are you suggesting Metacritic should do....smell test/ballpark how it translates? Apply subjective reasoning to hard math?

I'm not suggesting Metacritic do anything. It's not their problem. It's not IGN's problem. It's not Edge's problem. There shouldn't be a problem. The only issue is trying to assume some exact science for turning incompatible scoring systems into something completely objective in order to place a cutoff point where a game crosses a numeric line into greatness. That's not on Metacritic. It's on the people here (and elsewhere) that try to pass it off as that. Edge's 5 doesn't even equal IGN's 5... yet 100 scores of different scales are supposed to equal 72 scores of other varying scales... It's silly to not understand how that's not really going to be without issue.

Different people are reviewing, different numbers of people are reviewing, they're using different scales, they're being given different strengths... the metascores between different games (or even the same game on different platforms, or on different years, etc) are not directly comparable without a decent margin of error due to the imprecise nature of the metascore.

It really shouldn't be that difficult to understand tbh.

 

[hawk2025]

It FACTUALLY can't... there's no discussion for this. It doesn't matter if you use the numbers 1 through 5, 1 through 10, or 0 through FF... the moment you represent a 9 accurately, you're not on a 5 point scale. It cannot accomodate it. Fact.

It should not be on the review outlets to change their preferred review scale to sync with a completely external website that operates solely for their own gain. Many of these outlets massively predate Metacritic, and may potentially outlive it too it something else comes along and does to Metacritic what Metacritic did to Gamerankings. They're writing the review for their website. When they rate it a 9, that's the score they want it to have, not the aggregate that Metacritic will produce after.

You are making *no* sense. The aggregate that Metacritic produces is an aggregate. By definition, it is not the reviewer's original score, and it's not meant to be.

I'm seriously amazed that people don't understand exceedingly simple concepts like weighted averages and linear transformations.

 

[carl32]

Far cry 3 and borderlands 2 got in the 90s
Far cry 4 was only like 88 or 89

Just checked, average across platforms is lower than 90 for all.

 

[Synth]

You are making *no* sense. The aggregate that Metacritic produces is an aggregate. By definition, it is not the reviewer's original score, and it's not meant to be.

I'm seriously amazed that people don't understand exceedingly simple concepts like weighted averages and linear transformations.

How am I making no sense? I'm responding to a suggestion that the REVIEWERS should change their scoring to fit Metacritic's methodology. Are you even reading this stuff fully?

 

[FireFly]

I find it funny that people complain about score aggregators, especially with respect to how 5 or 10-scale reviews are aggregated.

...But if the original reviewer went with a five-star rating, they already decided that their opinion would be aggregated into chunks, and putting it into a scale that goes from 0 to 100 is quite literally only a linear transformation of that.

No information is lost from the transformation, and we can always invert it. Some of the points being made here make no sense.

Well, the information is lost by the 5 point scale itself. Which is fine when you are dealing with a single review in isolation, but not so fine when you are making comparisons to reviews where the reviewer was allowed to use a finer grained scale.

This is clearest with a binary yes/no rating. How do we compare a yes, to a score of 90%?

 

[Hoo-doo]

This thread has basically become 'Why didn't metacritic rate my game higher: the thread'.

An interesting discussion, but it basically only appears after a big hyped game doesn't quite garner the critical reception people anticipated, i've noticed.

Two points are truly not going to make a difference. Metacritic is not an objective measure of a game's quality so why are we treating it as such.

 

[Synth]

This is clearest with a binary yes/no rating. How do we compare a yes, to a score of 90%?

100 or 0 obviously. Do you not know how statistics work? It translates perfectly.

 

[hawk2025]

How am I making no sense? I'm responding to a suggestion that the REVIEWERS should change their scoring to fit Metacritic's methodology. Are you even reading this stuff fully?

If the problem is a 4/5 not "truly representing" their opinion as a 80/100, and if it's something that bothers the writers, they absolutely should. If they are happy with the current transformation or simply think it's irrelevant, then they shouldn't.

Any coarser number-based scale can be transformed into a thinner scale with no loss of information. If a writer wants to express more information with their score, then they need to make the scale thinner. If not, a 80/100 is a perfect (in fact, a mathematically identical representation) of a 4/5.

Be it 4 out 5 stars, unicorns, Destructoid-robots, or whatever measure a reviewer wishes to use. All Metacritic does is transform any given objective scale into a thinner objective measure.

 

[ByWatterson]

Well, the information is lost by the 5 point scale itself. Which is fine when you are dealing with a single review in isolation, but not so fine when you are making comparisons to reviews where the reviewer was allowed to use a finer grained scale.

This is clearest with a binary yes/no rating. How do we compare a yes, to a score of 90%?

Probably be better to pick a 60% cutoff like RT and go yes/no, making the average rating a sub-category.

 

[Synth]

This thread has basically become 'Why didn't metacritic rate my game higher: the thread'.

An interesting discussion, but it basically only appears after a big hyped game doesn't quite garner the critical reception people anticipated, i've noticed.

Two points are truly not going to make a difference. Metacritic is not an objective measure of a game's quality so why are we treating it as such.

Quote all the posts on this subject over the last few pages, that say the score should be higher as a result. I (along with others) are arguing that they're simply not equivalent.. A 4 on a five point scale could either be a 7 or a 9 on a 10 point scale.

Although again, like AgentP, I've been here long enough to know your angle.

If the problem is a 4/5 not "truly representing" their opinion as a 80/100, and if it's something that bothers the writers, they absolutely should. If they are happy with the current transformation or simply think it's irrelevant, then they shouldn't.

Any coarser number-based scale can be transformed into a thinner scale with no loss of information. If a writer wants to express more information with their score, then they need to make the scale thinner. If not, a 80/100 is a perfect (in fact, a mathematically identical representation) of a 4/5.

Be it 4 out 5 stars, unicorns, Destructoid-robots, or whatever measure a reviewer wishes to use. All Metacritic does is transform any given objective scale into a thinner objective measure.

The measure isn't objective. That's the point. The only ways for it to be objective would negate the need for a Metacritic.

 

[Synth]

dp

 

[Any Questions]

This thread has basically become 'Why didn't metacritic rate my game higher: the thread'.

An interesting discussion, but it basically only appears after a big hyped game doesn't quite garner the critical reception people anticipated, i've noticed.

Two points are truly not going to make a difference. Metacritic is not an objective measure of a game's quality so why are we treating it as such.

Dipped into the thread. Skimmed over a few pages, read about the campaign which sounds great and then got bored on how to review a game.

Sounds like a fun title, if I had the console I would get. But I don't.

 

[BokehKing]

Just checked, average across platforms is lower than 90 for all.

Why are we going all consoles when halo is only on one?
If we are going by Xbox One, the taken king scored a 90 according to MC

 

[Disxo]

Nice reviews I hope I can get an xone to play this game at low price.
Btw:
Lol guys MC doesnt have to be your main concern.

 

[FireFly]

Any coarser number-based scale can be transformed into a thinner scale with no loss of information. If a writer wants to express more information with their score, then they need to make the scale thinner. If not, a 80/100 is a perfect (in fact, a mathematically identical representation) of a 4/5.

I think the point is not that the transformation loses anything, but that it it doesn't gain back the information lost by using a less granular scale. It's easy to think, "well 4/5 = 80%, so that's what the reviewer would have given the game". But that's untrue. He might have given it 88%.

 

[ByWatterson]

The measure isn't objective. That's the point. The only ways for it to be objective would negate the need for a Metacritic.

But reviews themselves are subjective. So what does it matter if a subjective score, reduced to a number, is translated to another number, proportionally?

4/5 means 4/5, and is literally equivalent to 8/10. If that's not what the reviewer means to communicate, he should make the scale bigger to communicate more information via a narrower review number.

I guess my question is - what exactly is the problem?

 

[carl32]

Why are we going all consoles when halo is only on one?
If we are going by Xbox One, the taken king scored a 90 according to MC

Its only on xbox one as far as i know, thats why and im on about FPS games on any platform, i don't see why it really matters. Taken King is identical on Ps4 and X1 so it makes sense to average the score

 

[Synth]

I think the point is not that the transformation loses anything, but that it it doesn't gain back the information lost by using a less granular scale. It's easy to think, "well 4/5 = 80%, so that's what the reviewer would have given the game". But that's untrue. He might have given it 88%.

The other point is that people act as though the reviewers should accomodate Metacritic. Why? If I tell you out of 5 what I rate Halo 5, why should I ensure it aligns with how you're going to score it? It's my opinion not all of ours.

But reviews themselves are subjective. So what does it matter if a subjective score, reduced to a number, is translated to another number, proportionally?

4/5 means 4/5, and is literally equivalent to 8/10. If that's not what the reviewer means to communicate, he should make the scale bigger to communicate more information via a narrower review number.

I guess my question is - what exactly is the problem?

It doesn't matter. Not until someone tries to portray all this subjectivity as objective. Which is what's been going on in this thread, and how this discussion started.

 

[Hoo-doo]

Quote all the posts on this subject over the last few pages, that say the score should be higher as a result. I (along with others) are arguing that they're simply not equivalent.. A 4 on a five point scale could either be a 7 or a 9 on a 10 point scale.

Although again, like AgentP, I've been here long enough to know your angle.

My angle? What does that have to do with what I just posted, exactly?

I'm just saying that even with your 'stars-to-numbers non equivalency' argument taken into account, metacritic still wouldn't be an objective measure of a game's quality in any way. There simply is no such thing. You're arguing in favor of a faulty system that's going to be just as faulty as the current one.

 

[TheOddOne]

Why are we going all consoles when halo is only on one?
If we are going by Xbox One, the taken king scored a 90 according to MC

8 critics vs 51 critics.

 

[Synth]

My angle? What does that have to do with what I just posted, exactly?

I'm just saying that even with your 'stars-to-numbers non equivalency' argument taken into account, metacritic still wouldn't be an objective measure of a game's quality in any way. There simply is no such thing. You're arguing in favor of a faulty system that's going to be just as faulty as the current one.

Your angle was...

"This thread has basically become 'Why didn't metacritic rate my game higher: the thread'.

... which is why I asked you to quote some posts to back that up.

I haven't even suggested an alternate system... hell, one of my previous posts was specifically that there simply is no system that wouldn't be flawed. I'm arguing against how the system is being represented (as factually accurate, because statistics), not what the system is. If you were reading the posts, and not some imaginary "it should score 99 average" dialog, then you'd probably have noticed that.

 

[Japanmanx3]

This thread has basically become 'Why didn't metacritic rate my game higher: the thread'.

An interesting discussion, but it basically only appears after a big hyped game doesn't quite garner the critical reception people anticipated, i've noticed.

Two points are truly not going to make a difference. Metacritic is not an objective measure of a game's quality so why are we treating it as such.

 

[Hoo-doo]

Your angle was...

... which is why I asked you to quote some posts to back that up.

I haven't even suggested an alternate system... hell, one of my previous posts was specifically that there simply is no system that wouldn't be flawed. I'm arguing against how the system is being represented (as factually accurate, because statistics), not what the system is. If you were reading the posts, and not some imaginary "it should score 99 average" dialog, then you'd probably have noticed that.

Let's get it out of the way, the numbers presented for Halo 5 did not meet everyone's lofty expectations. Agreed?
So is it really unthinkable that this giant off-topic discussion criticizing metacritic's method of score aggregation has at least a slight connection with this?

I mean, that's all I said, and it is an interesting discussion to have.
I just think it stems from general disappointment in the scores at this point. Many wish it scored higher and think the game deserves higher.

The system is flawed and we all know it. All we can do right now is make the process of score aggregation more transparent.
OpenCritic currently already improves on the metacritic formula by binning the weighting system, at least.

 

[PhineasRed]

I don't know if this has already been covered, but when dealing with skewed data sets it usually makes more sense to use the median rather than the average. I would be curious how that changes the scores on some of the more popular titles, especially ones that had a wide distribution of scores. A cursory view of the Halo 5 review data would put it at an 88 (half above 88, half below 88). Also, the original Destiny would have been 80 instead of 76. I also might have no idea how do this correctly.

More generally I think it's safe to say that Metacritc is best used a blunt tool to help make purchase decisions, not empirical evidence of game superiority/inferiority.

 

[BokehKing]

👮👮👮👮👮

Freeze

Let's all agree on one thing
85 is a good score
Halo 5 is a good game
A meta critic rating will not effect sales, how many people do you know that even knows of the site called metacritic?

I know zero, and I'm a pretty popular guy

 

[Synth]

Let's get it out of the way, the numbers presented for Halo 5 did not meet everyone's lofty expectations. Agreed?
So is it really unthinkable that this giant off-topic discussion criticizing metacritic's method of score aggregation has at least a slight connection with this?

I mean, that's all I said, and it is an interesting discussion to have.
I just think it stems from general disappointment in the scores at this point. Many wish it scored higher and think the game deserves higher.

The system is flawed and we all know it. All we can do right now is make the process of score aggregation more transparent.
OpenCritic currently already improves on the metacritic formula by binning the weighting system, at least.

Yes, I'll agree that the metascore is lower than many would probably have expected. Especially if you assume that metacritic scores are consistent, which would lead to thoughts like "Halo 4 scored 87... so surely Halo 5 has to score higher". That's not a reason to claim that comments surrounding the scoring are a result of people not being able to accept the score. Not once in this thread have I suggested that it should score higher than 85 (and why would I? I haven't even played my copy yet)... I've been responding to posts asking why there's a focus on 85 vs something like 90.. or even a claim that you can accurately represent a 9 on a 5 point scale...

This discussion (which I have found interesting despite certain posts, yours included, that attempt to steer it elsewhere) has led to me posting quite a bit over the last few pages. And seeing as other posts are making very similar points to mine, then I can only infer that your first line is being applied to me, and those making points like mine... which is honestly bullshit. If I thought Halo 5 inherently deserved a better metascore, then that's what I'd say. Having people like yourself come along and try to say it for me, to invalidate the things I actually am saying is really, really annoying.

Yes these sorts of discussions often come up in review threads when the scores spark debate... but that's because conversations are very often situational. I don't tend to sit there and suddenly think "aha! I should make a thread to discuss metacritic averages!". The thought simply never comes to me, until I see a post whilst browsing something I'm actually interested in. It's also why you're not likely to see me make the same points in a thread for a game I don't have any interest... I'm probably just not in that thread in the first place. That doesn't mean I wouldn't make the same points if someone bat-signalled me into it to make me aware of the discussion.

 

[Azzanadra]

85 is still a good score, so I don't know why people are disappointed. The biggest games of the year so far were what, Bloodborne, TW3, MGSV and Arkham Knight, right? Even AK did not get 90+. If Halo's 5 campaign is as bad as people say, that leaves the multiplayer... I don't think that alone would warrant near perfect scores.

Though in all honesty, if I was to interject my own opinion, I don't think any Halo game sans the first one really deserve that 90+ metascore. Especially with this new trilogy, which seems to be pandering towards CoD fans.

 

[PhineasRed]

👮👮👮👮👮

Freeze

Let's all agree on one thing
85 is a good score
Halo 5 is a good game
A meta critic rating will not effect sales, how many people do you know that even knows of the site called metacritic?

I know zero, and I'm a pretty popular guy

Yeah, can't argue with any of that.

 

[- J - D -]

Halo 5 would've probably scored higher if it weren't for the batmobile.

 

[ethomaz]

Yes 85 is pretty good score.
It is just low for the Halo standards I guess so people expected something close to 90.

 

[PooBone]

I'm nearing the end of the game and I'm flabbergasted by the lack of mission starring Master Chief.

 

[Anon67]

👮👮👮👮👮

Freeze

Let's all agree on one thing
85 is a good score
Halo 5 is a good game
A meta critic rating will not effect sales, how many people do you know that even knows of the site called metacritic?

I know zero, and I'm a pretty popular guy

Basically

 

[PhineasRed]

Let's get it out of the way, the numbers presented for Halo 5 did not meet everyone's lofty expectations. Agreed?
So is it really unthinkable that this giant off-topic discussion criticizing metacritic's method of score aggregation has at least a slight connection with this?

I mean, that's all I said, and it is an interesting discussion to have.
I just think it stems from general disappointment in the scores at this point. Many wish it scored higher and think the game deserves higher.

The system is flawed and we all know it. All we can do right now is make the process of score aggregation more transparent.
OpenCritic currently already improves on the metacritic formula by binning the weighting system, at least.

Maybe I came in late to the discussion, but I didn't feel that the discussion on the usefulness of MC was because people were disappointed in the scores. In this review climate, I think the reviews are great as a whole, and I'm not even a huge halo guy (I suck at the multiplayer but love the campaigns). Discussing better ways to interpret the data sets (as flawed as they may be) is interesting.

 

[PanicFreak]

I don't know if this has already been covered, but when dealing with skewed data sets it usually makes more sense to use the median rather than the average. I would be curious how that changes the scores on some of the more popular titles, especially ones that had a wide distribution of scores. A cursory view of the Halo 5 review data would put it at an 88 (half above 88, half below 88). Also, the original Destiny would have been 80 instead of 76. I also might have no idea how do this correctly.

More generally I think it's safe to say that Metacritc is best used a blunt tool to help make purchase decisions, not empirical evidence of game superiority/inferiority.

Medians do tell use more about data because extreme outliers can significantly affect the average. That's why the median income is reported for economies and not the average.