Karunamon
Member
I floated this concept over on Reddit a few months ago and got some good feedback, though I'd like the general feelings of the people here too.
tl;dr: This is a framework for a game review site that does not exist yet, and goes pretty heavy into how this site would work, and what it can and can't do.
I want your feedback about this. What sucks, what doesn't, how a bad actor might manipulate this idea to be an unethical piece of crap.
This is gonna be long.
----
This hypothetical site is called ShouldYouPlay (SYP).
Overview & The Scoring System
SYP is a site that attempts to score video games for consumers. It's that simple. Its audience is the slightly-hardcore gamer. The person who knows what League of Legends is. The person who has gamed competitively. The kind of person who has 100% achievements in a game or two. The person who has an account here (
).
Scores generally operate on a 4-point scale, with a 4 being the best score, and a 1 being the worst score. Some scores have a -1 rating for extreme issues. These scores are summed to generate a final score for the game. The absolute highest score a game can receive is 25, the absolute lowest is 1. Scores are whole numbers, no decimals are allowed.
The total score for a game is split into two parts, an objective and a subjective section.
This reflects that it is possible for a game to be visually and audibly stunning, without a single bug, but yet still be un-fun to play, badly monetized, and made by a terrible developer.
The characteristics are:
These are represented as a fractional number with a total. This isn't a fraction, it's just a way to represent the numbers above and below the line.
As an example, Death Stranding, as reviewed by me, would rate, in order, 3, 4, 4 and 3, 3, 4, 1, for an above the line of 11 and a below the line of 11, with a score of 22. The site would reveal this score as 11/11=22
----
Score Definitions: Above the line
Visuals
Score definitions: Below The Line
Enjoyment
This score is the most subjective, reflecting the opinion of the reviewer on how enjoyable the game was to play. Keep in mind that the other scores mentioned can and do have an impact on this. Think of this as the reviewer's willingness to buy the game again if they knew what they did after they reviewed it.
Developer (Publisher/Responsible Party)
This score measures the behavior of a developer as it applies to their product, marketing, customer service, etc (named related conduct). It does NOT take into account political or social stances or any other behavior unrelated directly to their games. Read "developer" as "publisher" or "responsible party" where applicable; the company responsible for making the day-to-day decisions about the game.
If for some reason this is ambiguous, SYP defaults to and rates the lower score among the parties.
Related conduct specifically includes, but is not limited to:
Monetization
This score measures how the responsible party tries to get extra cash above and beyond the price of the game. Complaints about the price would factor into enjoyment, not here.
Note that expansion packs (DLC that adds significant content *and* playtime to a game) are excluded from this. Horse armor is cosmetic DLC (4), paid cheats are (3), your average Asian MMO cash shop bait is a (2) or a (1). Many mobile games are a (-1) .
Top Dog / Real Dog
* +1/-1: An extra point can be granted or removed from any game that the reviewer thought was particularly particularly notable for any reason, positively or negatively, regardless of all other scores. These should be granted sparingly, and SYP reviewers are only allowed to use a certain number of these in a given time frame.
----
Site Practices
The ShouldYouPlay pledge:
Edge cases and FAQs
On developer scores
A problem we recognize with the developer score is the temptation to bring issues unrelated to the game under review into the spotlight. An example of an unrelated issue in this case would be the revelation that the developers of a game hold distasteful political views - visualize your your (least) favorite form of extremism, it would count here.
The problem with taking these into account is, primarily, that they have nothing to do with the content of the game under review (unless those views slap the player in the face via their design, in which case they would necessarily be surfaced in the other scores).
Both Scientologists and fundamentalists, Nazis and communists, SJWs and MRAs, and so forth, have the ability, if not the likelihood, to create something that is a good game. Recognizing this, we specifically exclude political beliefs as being a part of any review score, above or below the line, if they aren't part of a game's content.
However, one exception must be made, as it crosses this line in the sand: Adding or removing content from a game in response to politically-motivated outrage. We absolutely would expect games that censor their content, becoming lesser in the process, to show up in the developer and enjoyment scores. We absolutely would expect games that drop anvils on their player's heads about some political concern to do the same.
This does not mean that games are not allowed to have political content. The difference is whether the game is compromised in some way to serve as a vehicle for the developer's beliefs. Metal Gear without politics isn't Metal Gear anymore. The devs of Hotline Miami are known to be socialists - neither Kojima or Devolver Digital would be rated on their developer scores for this.
On graphics & sound
Graphics and sound design are to be reviewed within their own genres and intended design. Context matters. An intentionally retro JRPG that tries to look like a NES title would not get marked down in the visuals or score categories just because Final Fantasy 15 exists. But the same JRPG, if it compares unfavorably to, say, Cheetahmen, would be deserving of its low scores.
On qualified reviewers
Reviewers for a game are expected to be familiar with and generally appreciative of the genre of a game they are reviewing. Your grandma probably isn't the best person to review Doom (2019). The FPS god probably isn't super into Madden. We've seen other sites pull this kind of crap, it sucks, and SYP pledges not to do it.
On sentimental value and retro reviews
SYP will avoid reviewing any game that is more than 10 years old. Not only because retro reviews are a silly concept, but also because it risks nostalgia goggles getting in the way of objectivity, especially below the line.
On objectivity in general
It may seem contradictory to assign a score to a singular component of a game, and claim that this is objective, when the reviewer is answering the "did you think characteristic X of game Y falls into bucket Z?" question.
Each reviewer will have their own biases, of course, but we believe most gamers have the ability to know when something like graphics or sound are uncharacteristically good, uncharacteristically bad, or so on, and it comes down to whether those things take you further into the game's experience, or take you out of them and stand out.
The bucketing concept (score X in category Y means Z) partially serves to enforce this. We see very little disagreement among gamers whether a game's graphics, on their own, are generally good or generally bad. Given the state of the industry, "ok graphics" would fall into the "generally good" bucket.
A particular score being below the line does not mean that a reviewer is free to abandon all sense of objectivity and rate what they want. The buckets still exist. The concept of the line is intended to separate things that can and cannot be objectively compared. Reviewers are to remain, as said up above, "relentlessly objective, honest, forthcoming, and on-topic."
On scores in general
One piece of feedback received frequently from a previous version of this proposal is that numeric scores cannot be trusted and are too hard to understand. So the question is, why scores? What makes this different from the others?
tl;dr: This is a framework for a game review site that does not exist yet, and goes pretty heavy into how this site would work, and what it can and can't do.
I want your feedback about this. What sucks, what doesn't, how a bad actor might manipulate this idea to be an unethical piece of crap.
This is gonna be long.
----
This hypothetical site is called ShouldYouPlay (SYP).
Overview & The Scoring System
SYP is a site that attempts to score video games for consumers. It's that simple. Its audience is the slightly-hardcore gamer. The person who knows what League of Legends is. The person who has gamed competitively. The kind of person who has 100% achievements in a game or two. The person who has an account here (
).Scores generally operate on a 4-point scale, with a 4 being the best score, and a 1 being the worst score. Some scores have a -1 rating for extreme issues. These scores are summed to generate a final score for the game. The absolute highest score a game can receive is 25, the absolute lowest is 1. Scores are whole numbers, no decimals are allowed.
The total score for a game is split into two parts, an objective and a subjective section.
- Objective elements of the game, such as its graphics, sound, and technical characteristics, are referred to as "above the line".
- More subjective elements, such as enjoyment, monetization practices, and developer behavior, as well as the "top dog/real dog" designation (reviewer thought this game was notably good or notably bad), are considered "below the line".
This reflects that it is possible for a game to be visually and audibly stunning, without a single bug, but yet still be un-fun to play, badly monetized, and made by a terrible developer.
The characteristics are:
- Visuals: How it looks
- Sound: How it sounds
- Technical: How it runs
- Enjoyment: Would I buy this again if I knew then what I know now?
- Monetization: Is the dev trying to reach into my pocket?
- Developer: How does this dev behave?
- Top Dog / Real Dog:
These are represented as a fractional number with a total. This isn't a fraction, it's just a way to represent the numbers above and below the line.
As an example, Death Stranding, as reviewed by me, would rate, in order, 3, 4, 4 and 3, 3, 4, 1, for an above the line of 11 and a below the line of 11, with a score of 22. The site would reveal this score as 11/11=22
----
Score Definitions: Above the line
Visuals
- 4: This game is a triumph of either advanced graphics or artistic design. It is notable for its graphics even if everything else was terrible.
- 3: This game has overall good graphics, with some issues that prevent it from being perfect.
- 2: This game has overall poor graphics, with significant room for improvement to either clarity or aesthetics. At this point, the graphics are detracting from the gameplay.
- 1: This game's graphics suck and significantly make it a worse game. The score is automatically a 1 if any flipped or stolen assets are used.
- 4: This game is enjoyable to listen to even while not playing. Effects are satisfying, music is enjoyable.
- 3: This game has overall good sound design.
- 2: This game has overall bad sound design.
- 1: This game has painful sound design - its poor music or SFX actually make it less enjoyable to play.
- 4: This game is rock solid, and no technical issues of any kind were encountered.
- 3: This game is solid, but minor glitches should be expected. Most games fit here.
- 2: This game has bugs that make it annoying to play. An average player must avoid or work around bugs in normal play.
- 1: This game is buggier than a bait store, or has severe bugs encounterable by an average player that can ruin competitive games, lose progress, damage hardware, or cause the game to be unwinnable.
Score definitions: Below The Line
Enjoyment
This score is the most subjective, reflecting the opinion of the reviewer on how enjoyable the game was to play. Keep in mind that the other scores mentioned can and do have an impact on this. Think of this as the reviewer's willingness to buy the game again if they knew what they did after they reviewed it.
- 4: Game of the Year contender. I find it very difficult to come up with room for significant improvement. I would buy this game at full price, twice. Very rare.
- 3: Good. Overall enjoyable, with minor issues that get in the way of enjoyment. I would buy this game at full price, and absolutely would on sale.
- 2: Mediocre. The game can be somewhat fun, but has a number of issues that make it worse than it could be. I would think about buying this game on sale if it were appealing to me for some reason, but wouldn't bother at full price.
- 1: Awful. This game has significant issues that render it mostly un-fun to play. I would absolutely not buy this game. I might play it if someone gave me a copy. Guilty pleasures can sometimes be found here.
- -1: Hot garbage. This score is reserved for games that simply could not be played or otherwise severely fail at being "a game". These games aren't just bad. Abysmal barely even qualifies. I would not play this game even if it were given to me for free. I might not play it even if I was paid to do so. It's so bad I might play it for charity. Think Daikatana or Big Rigs.
Developer (Publisher/Responsible Party)
This score measures the behavior of a developer as it applies to their product, marketing, customer service, etc (named related conduct). It does NOT take into account political or social stances or any other behavior unrelated directly to their games. Read "developer" as "publisher" or "responsible party" where applicable; the company responsible for making the day-to-day decisions about the game.
If for some reason this is ambiguous, SYP defaults to and rates the lower score among the parties.
- 4: Great developer. The developer is honest and has little or no history of malfeasance, corruption, or scandals. It is responsive to player input, deals with disputes in a fair manner, and is overall a good citizen. Gaming as a whole is better for this developer existing. This is very rare.
- 3: Average developer. There may be a closet skeleton or two, or perhaps they are not as responsive to players as they should be. They are still overall, honest and decent, and their missteps can be plausibly explained as error rather than malice.
- 2: Smelly developer. The developer engages in tricks such as false or misleading advertising, or has a significant history of fraud, corruption, rip-offs, or bias. The developer cannot be considered generally honest.
- 1: Rotten developer. The developer is literally EA Games or takes after them. Their reputation is horrible, and everyone knows why. They are significantly dishonest, and their presence drags down the entire gaming community.
Related conduct specifically includes, but is not limited to:
- Development practices
- Marketing practices (to include social media posts as they relate to the game under review or its customer base, to include statements by employees of responsible parties, whether made "officially" or otherwise)
- Customer service
- Honesty
- Politically/ideological controversies inasmuch as they don't relate directly to the characteristics of the game under review
- Working conditions or political beliefs of the responsible parties
- Legal issues of the responsible parties, its employees, or partners
- Identities of the responsible party's employees or partners
Monetization
This score measures how the responsible party tries to get extra cash above and beyond the price of the game. Complaints about the price would factor into enjoyment, not here.
- 4: Entirely free, or is a one-off cost to purchase. There are either no microtransactions at all, or they exist but are purely cosmetic and do not impact gameplay. No ads are present.
- 3: Average monetization. Ads may be present, but are un-intrusive or optional (watch ad to receive item), and meaningful progress can be made without watching ads. Game-impacting DLC may exist.
- 2: Badly monetized. Ads are frequent or annoying, or there is a Pay2Win aspect with boosts or consumables. It is still possible to make meaningful progress without paying, but it is demonstrably inferior to the experience a paying player receives.
- 1: Aggressively monetized. A player can expect constant ad popups or demands to buy ingame currency. The game is structured in a way that making meaningful progression in a reasonable time frame is not possible without spending money, or the player is expected to pay to make annoyances go away.
- -1: Disqualified. This game has both a purchase cost, AND game-impacting DLC or intrusive advertisement. This is not acceptable under any terms, and so the game loses a point for this score. Advertisement is "intrusive" if it is not an organic part of the game world - for example, a sports game is expected to have billboard ads in its stadiums. A -1 score is also automatically earned for evidence of on-disc DLC (asking paying for that which you already purchased) or cut and resold content.
Note that expansion packs (DLC that adds significant content *and* playtime to a game) are excluded from this. Horse armor is cosmetic DLC (4), paid cheats are (3), your average Asian MMO cash shop bait is a (2) or a (1). Many mobile games are a (-1) .
Top Dog / Real Dog
* +1/-1: An extra point can be granted or removed from any game that the reviewer thought was particularly particularly notable for any reason, positively or negatively, regardless of all other scores. These should be granted sparingly, and SYP reviewers are only allowed to use a certain number of these in a given time frame.
----
Site Practices
The ShouldYouPlay pledge:
- The goal: You, as a consumer of video games, you have a right to be honestly informed about the content out there. SYP intends to fill that niche by being relentlessly objective, honest, forthcoming, and on-topic in its reviews.
- The "no BS" rule: Each game stands on its own merits. With the exception of the developer score, more information on this below). Each game can and will be rated individually on its own merits, that is, independent of any prequels, sequels, state of the industry, and so forth.
The only exception here are multi-part games as part of a continuing storyline; an example would be the Mass Effect trilogy. It is required in these cases to take prior games into account to make sense of the game's storyline. Consequently, any views on storyline concerns may factor into the enjoyment score. SYP is a game review site. Not a game industry review site.
The only pseudo-exception is the developer score, which does take past related conduct into account. It is valid to want or not want to support developers based on their behavior, and so that is reflected in one of the scores. More information on this is below.E
- The "sponsor lock" rule: SYP will never accept sponsorships of any form (ads, honorable mentions, swag, review copies whether early or after release, trips, ingame currency, irl currency, or any other thing of value, broadly defined) from publishers, developers, or any other parties connected to a game a review is posted for.
If a company does sponsor SYP, SYP reviewers may make personal opinion content of it at its own discretion, but this will not be construed to be a formal review, no score will be given, and any sponsorship deals will be disclosed *above the fold* in that content. Such notices will be retroactively added to any related content should a sponsorship deal be entered into after a review is posted.
Further, any prior personal opinion content centered around that company's content will have appropriate disclaimers added. Personal opinion content will be easily recognizable as such to set it apart from actual scored reviews. It is also noted that review aggregator sites MUST NOT take personal opinion posts as reviews for the purposes of scoring a title.
- The *gate rule: Any personal ties, even as far as casual acquaintances, with the developers/publishers/PR people/other related people (broadly defined) to the content being reviewed, will be disclosed *above the fold* in all related content.
Notwithstanding this, people employed (broadly defined) by a connected person (broadly defined) may not review any related content.
- The "on topic" rule: SYP will review all games according to these criteria and absolutely no others, regardless of any past, present, or future events or controversies.
- The "final answer" rule: SYP scores are final and committed as of publication time.
If circumstances such as significant post-review changes that would materially impact the score are made, a full re-review will be done, and the past content will be amended (deprecated) to note that a new version of the content exists.
Under no other circumstances will SYP change scores after they have been posted unless a material error in the review's content was made (factually/typographically), in which case the correction will be conspicuous in a way that a reasonable person can see both the old and the new content and understand the specific change made. Such corrections will be noted above the fold.
- The "live service" rule: (Exception to the "final answer" rule) SYP recognizes that live service games and MMOs can be, and often are, made significantly better or worse via patches.
For live service games, reviews should be taken as representative of the version of the game reviewed, and not necessarily any future releases. Such reviews will prominently include the version and/or release date of the patch under review above the fold. SYP may deprecate old reviews for live service games with another review when patches make material changes to the experience. In this case, the old and new reviews will all conspicuously link to one another.
Edge cases and FAQs
On developer scores
A problem we recognize with the developer score is the temptation to bring issues unrelated to the game under review into the spotlight. An example of an unrelated issue in this case would be the revelation that the developers of a game hold distasteful political views - visualize your your (least) favorite form of extremism, it would count here.
The problem with taking these into account is, primarily, that they have nothing to do with the content of the game under review (unless those views slap the player in the face via their design, in which case they would necessarily be surfaced in the other scores).
Both Scientologists and fundamentalists, Nazis and communists, SJWs and MRAs, and so forth, have the ability, if not the likelihood, to create something that is a good game. Recognizing this, we specifically exclude political beliefs as being a part of any review score, above or below the line, if they aren't part of a game's content.
However, one exception must be made, as it crosses this line in the sand: Adding or removing content from a game in response to politically-motivated outrage. We absolutely would expect games that censor their content, becoming lesser in the process, to show up in the developer and enjoyment scores. We absolutely would expect games that drop anvils on their player's heads about some political concern to do the same.
This does not mean that games are not allowed to have political content. The difference is whether the game is compromised in some way to serve as a vehicle for the developer's beliefs. Metal Gear without politics isn't Metal Gear anymore. The devs of Hotline Miami are known to be socialists - neither Kojima or Devolver Digital would be rated on their developer scores for this.
On graphics & sound
Graphics and sound design are to be reviewed within their own genres and intended design. Context matters. An intentionally retro JRPG that tries to look like a NES title would not get marked down in the visuals or score categories just because Final Fantasy 15 exists. But the same JRPG, if it compares unfavorably to, say, Cheetahmen, would be deserving of its low scores.
On qualified reviewers
Reviewers for a game are expected to be familiar with and generally appreciative of the genre of a game they are reviewing. Your grandma probably isn't the best person to review Doom (2019). The FPS god probably isn't super into Madden. We've seen other sites pull this kind of crap, it sucks, and SYP pledges not to do it.
On sentimental value and retro reviews
SYP will avoid reviewing any game that is more than 10 years old. Not only because retro reviews are a silly concept, but also because it risks nostalgia goggles getting in the way of objectivity, especially below the line.
On objectivity in general
It may seem contradictory to assign a score to a singular component of a game, and claim that this is objective, when the reviewer is answering the "did you think characteristic X of game Y falls into bucket Z?" question.
Each reviewer will have their own biases, of course, but we believe most gamers have the ability to know when something like graphics or sound are uncharacteristically good, uncharacteristically bad, or so on, and it comes down to whether those things take you further into the game's experience, or take you out of them and stand out.
The bucketing concept (score X in category Y means Z) partially serves to enforce this. We see very little disagreement among gamers whether a game's graphics, on their own, are generally good or generally bad. Given the state of the industry, "ok graphics" would fall into the "generally good" bucket.
A particular score being below the line does not mean that a reviewer is free to abandon all sense of objectivity and rate what they want. The buckets still exist. The concept of the line is intended to separate things that can and cannot be objectively compared. Reviewers are to remain, as said up above, "relentlessly objective, honest, forthcoming, and on-topic."
On scores in general
One piece of feedback received frequently from a previous version of this proposal is that numeric scores cannot be trusted and are too hard to understand. So the question is, why scores? What makes this different from the others?
- There is no such thing as "average". Reviewers are required to choose from great, good, bad, terrible for scores above the line. "Average" is a meaningless concept in reviews where the end goal is to inform you whether it's worth buying or not.
- There's no need to tl;dr a concept as simple as a take on the 7 facets of a review here. That's what the review text is for.
- SYP wants to carve out its own reviewing niche, not be an also-ran. If you want wisdom of the crowds and a 0-100 score, Meta/Open critic exists. If you want a simple thumbs up/thumbs down, Steam has you covered.
- SYP caters more to the hardcore-ish gamer, not the average consumer, not your gran. The score system is a bit unique, but understandable.
- I think that by "bucketing" scores (score X means characteristic Y was met in category Z) removes the problem of scores not being trustworthy, especially given the concept of "the line".
Last edited:
