Don't step in the Pharaoh

Speak your mind, or lack thereof. There may occasionally be on-topic discussions.
User avatar
Crowbar
Posts: 680
Joined: Fri Jul 25, 2008 7:40 pm

Re: Don't step in the Pharaoh

Post by Crowbar »

I think a 1-10 scale is the maximum anyone would ever need, but I also like the idea of doing away with a numerical system entirely and just ending reviews with a very short summary of the strengths and weaknesses of the title (maybe two, one a short sentence, another just a few words). I don't know if that would be practical in the long term, though.

User avatar
Green Gibbon!
BUTT CHEESE
Posts: 4648
Joined: Fri May 21, 2004 11:39 am
Now Playing: Bit Trip Complete
Location: A far eastern land across the sea
Contact:

Re: Don't step in the Pharaoh

Post by Green Gibbon! »

Numerical ratings try to put into mathematical terms something that is innately not mathematical. Basically, they are bullshit.

User avatar
Neo
Posts: 974
Joined: Mon Jun 19, 2006 6:18 pm
Location: Portugal

Re: Don't step in the Pharaoh

Post by Neo »

Numerical ratings attempt to rank games as being objectively "better" or "worse" than others, but the very definition of "better" and "worse" is entirely subjective. Thus, to extract any sort of insight you need to round up a lot of ratings from a lot of different people, which means not just the fanboys and the haters but every shade of gray in between. The problem being that anyone in this shade of gray doesn't care enough to bother rating the game at all, making the numbers skewed.

EDIT: Also, the problem of "8" equaling "average".

User avatar
Green Gibbon!
BUTT CHEESE
Posts: 4648
Joined: Fri May 21, 2004 11:39 am
Now Playing: Bit Trip Complete
Location: A far eastern land across the sea
Contact:

Re: Don't step in the Pharaoh

Post by Green Gibbon! »

Neo wrote:Numerical ratings attempt to rank games as being objectively "better" or "worse" than others, but the very definition of "better" and "worse" is entirely subjective.
Ergo, numerical ratings are completely useless to anyone but the person who made them, and it's unlikely even he would come up with the same number every single time.

User avatar
Crisis
Posts: 531
Joined: Wed Mar 25, 2009 5:15 pm

Re: Don't step in the Pharaoh

Post by Crisis »

Rating scales are obviously going to be subjective. But that's not the same as being completely arbitrary. And even if they only apply to a single individual, so does any other aspect of his or her opinion.

It's supposed to be an efficient way of conveying a lot of information. Not everybody has time to read through potentially hundreds of reviews by a critic to find the ones they recommend. A lot of people use scores as a "filter" for games that aren't worth their time, e.g. "What games on the PSP did IGN give an 8.0 or higher?". Then you can read the reviews of the games that remain to see if they're something you'd enjoy.

I'm guessing that's also the point of metacritic. By taking an overview of hundreds of subjective opinions, you can get a vague idea of the critical consensus on the subject. That's useful as a starting point when investigating a game (or movie, or book). (It's a problem when you end your investigation there, but that's another discussion.)

It's also how academic grading works. You create a list of subjective, but non-arbitrary criteria, and you give it to a bunch of teachers or professors who have to judge whether a piece of work meets the criteria. Those judgements are going to be subjective, but we generally accept them anyway because no objective standard exists.

Anyway, here's what I came up with, along with some notes. Obviously, my criteria reflect what I think is important in a game. But I don't think they're arbitrary, at any rate. To clarify, a game really has to satisfy every condition in the description before it can be moved up a tier:


-----

0/10: A game that completely fails to entertain or deliver anything of credible artistic merit, and is generally offensive to its target audience.

1/10: A game that displays extremely limited imagination, a bare-bones appreciation for design, and severe technical issues.

2/10: Generally the same as 1/10, but with a handful of redeeming features. Only of interest to enthusiasts.

3/10: A reasonable amount of thought and polish in places, but little consistency in design and some technical issues. (I'd say this is maybe 50-60 points on metacritic.)

4/10: A generally solid experience marred by a small number of severe design or technical oversights. Worth considering spending time and money on if you’re already somewhat interested in the genre, but badly in need of further development.

5/10: Similar to 4/10, but with less obvious design or technical oversights. To get the extra point, the game must be a polished product that contributes something meaningful to its genre.

6/10: A game with some relevant and original ideas that push at the boundaries of expectations. Typically, these games do not have notable technical issues. Worth investigating, even if you aren’t particularly interested in the genre.

7/10: A consistently high-quality game, abundant with thoughtful design decisions. These games are very positive representations of their genre.

8/10: A landmark game that represents our highest expectations. Technical issues may be present due to the experimental nature of these games, but not debilitating ones. Recommended even to people who are sceptical of video games as a medium of art or of entertainment.

9/10: These games exceed our highest preconceived expectations and are the best representation of the medium we can hope for. They ought to stand tall with celebrated pieces of literature, film, etc.

10/10:
A hypothetical standard to which there is no conceivable criticism.

-----

Each point is supposed to build on the last one, either by adding extra criteria or raising the bar on existing criteria.

Some criteria are clearer than others. I'm sure lots of people would argue that Skyrim is "a consistently high-quality game, abundant with thoughtful design decisions", because they enjoyed it and because they didn't think it wasn't badly designed. That's a valid opinion - but personally I found the quality to be very inconsistent, with some sections of the story going a mile a minute in between long, boring dungeon crawls. You could also argue that it was "a game with some relevant and original ideas that push at the boundaries of expectations", and I'd agree - kind of. But it definitely has notable technical issues, and it's not really of any interest to people who aren't already interested in the genre.

Having regular bugs or crashes also invalidates the consistency clause, although afterwards I allude to the possibility of leniency for technical issues for particularly ambitious games. For instance, I think World of Warcraft (the original) qualifies as a 7/10 because it was arguably the most ambitious game of its time, and because the dungeon design is as yet unmatched, certainly in MMOs; the expansion packs I would give a 3/10, 2/10, and 3/10, in chronological order, because they were obvious hack jobs which further broke the game while adding next to nothing of value in return.

Basically, I think that having a subjective opinion is fine, as long as you can argue a justification for it. This scale is an attempt to make a framework for such arguments.

Another example - Sonic 2006. I suspect most people who've played it would put it at 1 or 2 on the scale above. (Or maybe a 0 if they felt particularly wronged.) But that doesn't mean nobody would be interested. It has historical and educational value, at least for the current generation of game enthusiasts, which is what I had in mind when I added a criteria to 2/10 that required the game to be interesting to a niche audience at the very least. Another example: 50 Cent: Blood on the Sand. It sounds like a terrible game, but if you're a massive 50 Cent fan or want a "so bad it's good" experience, you might enjoy it.

One day, I'll learn to make a point in under two pages.

User avatar
Radrappy
Posts: 1329
Joined: Wed May 26, 2004 10:53 pm
Now Playing: MvC3, Vanquish, Skies of Arcadia Legends
Contact:

Re: Don't step in the Pharaoh

Post by Radrappy »

Green Gibbon! wrote:Numerical ratings try to put into mathematical terms something that is innately not mathematical. Basically, they are bullshit.
How else is an art form supposed to be credibly categorized? It sounds good on paper to abolish scores, but would probably end up confusing people more than it helped. People just need to be more responsible with the scores they give.

User avatar
Blount
Posts: 291
Joined: Thu Jan 15, 2009 2:44 pm

Re: Don't step in the Pharaoh

Post by Blount »

What's wrong with just sticking to final thoughts? The only inconvenient I can see with that is the disappearance of aggregate scores, but those are even more bullshit by default anyway. When I'm dubious about a game, I usually just visit forums or read the general consensus about it on Wikipedia, never bothering with numeric scores or grades. I do understand some of Crisis's points, though.

Another thing people seem to forget all the time is that just because a game had a good rating upon release doesn't necessarily mean it has aged well. And even if it has, better games have probably come out since then, but thanks to scores having to keep in mind what else is on the market at the time, they are likely to be ranked lower than said older, potentially worse game.

User avatar
gr4yJ4Y
Posts: 1366
Joined: Wed Jun 09, 2004 10:14 am
Now Playing: Breath of the Wild (Switch), Resident Evil VII (PS4)
Location: Crescent Knoll

Re: Don't step in the Pharaoh

Post by gr4yJ4Y »

Part of the problem with 5 or 7 or 8 being considered average is there's a load of shovelware that never gets reviewed and never gets a numerical score. It's hard to say Deus Ex: Human Revolution is "above average" when a site doesn't touch half the games that come out and the average for the ones they do review is an 8.5. EGM used to have somewhat of a bell curve with their reviews. Back in the day they would review just about every game that came out - everything from Metal Gear Solid to Sesame Street: Words and Math. They gave Super Man 64 a 0 (individual score) and Rouge Squadron an 8 (mean between several scores). No one was mad that they gave Rouge Squadron (a major title at the time) an 8. An 8 was considered a great thing. I'm going entirely from memory, but I think it might've gotten game of the month. There were fall issues, with all the big games coming out, where there'd be only one or two titles to score 9 average. 10's were rare and if a game got a 10, it was a sign that it was a work or art, a true masterpiece that pushes the medium. The games that got 10's from that time certainly did that - Metal Gear Solid, Ocarina of Time, Tony Hawk's Pro Skater 2, Metroid Prime. (There were other problems with their reviews however, such as not being very comprehensive and being very short.) It's a little frustrating that there isn't such strict standards for most reviewers and multiple games can be reviewed as "masterpieces" every month now.

The topic of game reviews comes up a lot on Reddit, where someone came up with the idea of putting what each score means right next to the number in the review. For example, if Crisis was a game reviewer and gave Sonic Colors a 7/10, it would say at the bottom of his review "7/10: A consistently high-quality game, abundant with thoughtful design decisions. This game is a very positive representation of its genre." That way the reader wouldn't have to go hunt down a scale saying what each score meant and the reviewer would have to reflect if the game really matched its score. What the score meant would be as much a part of the review as the numerical score itself.

I like Crisis's strict scoring system. If I were to be a reviewer I would have a different description for each number because I feel that I can still have a fantastic experience even with technical issues (and all those Skyrim fans would probably agree as well). But it's fine that Crisis feels different than me. Maybe Crisis could gain a following of readers who feel similar to him and I could gain a following for my style of reviews. Understanding a review means understanding how each reviewer rates games.

User avatar
Segaholic2
Forum God
Posts: 3516
Joined: Fri May 21, 2004 11:28 am
Now Playing: Your mom

Re: Don't step in the Pharaoh

Post by Segaholic2 »

Radrappy wrote:How else is an art form supposed to be credibly categorized? It sounds good on paper to abolish scores, but would probably end up confusing people more than it helped. People just need to be more responsible with the scores they give.
I give the Mona Lisa a 5 out of 10. Cute smile but not enough cleavage.

User avatar
Radrappy
Posts: 1329
Joined: Wed May 26, 2004 10:53 pm
Now Playing: MvC3, Vanquish, Skies of Arcadia Legends
Contact:

Re: Don't step in the Pharaoh

Post by Radrappy »

see, that was irresponsible scoring.

User avatar
Tsuyoshi-kun
Posts: 946
Joined: Tue Sep 07, 2004 11:33 am
Now Playing: Super Smash Bros. for Wii U
Location: Philadelphia, PA, U.S.A.

Re: Don't step in the Pharaoh

Post by Tsuyoshi-kun »

Being a writer of anime reviews for a web site (which I won't post a link to, as to adhere to the rules), I wish at times that I myself was not limited to the oppressive 1-5 scale on there, or the 1-10 scale on MyAnimeList. (Though you can choose on the latter to leave blank numbers.) With that said, I feel that in the case of my site, the 1-5 star number is just a fancier way of me saying whether to buy / rent before buying / rent only / avoid a particular anime. Oftentimes my own opinion of a series will not match up with the star rating, which I often add at the last moment! Does that make it any better at all to think of it that way, or is it still stupid?

To keep on-topic; I haven't tried the trailer for Rayman Origins yet, but I also haven't played a new Rayman game in 12 years. The warm response to Origins makes me to want to set up X-Box Live (I don't have a modem, so I have to lug my basement TV to my main computer) to do so.

User avatar
Green Gibbon!
BUTT CHEESE
Posts: 4648
Joined: Fri May 21, 2004 11:39 am
Now Playing: Bit Trip Complete
Location: A far eastern land across the sea
Contact:

Re: Don't step in the Pharaoh

Post by Green Gibbon! »

Well if you want a consensus, there's only two answers: I liked it or I didn't like it. Thumbs up or thumbs down. One or two. There's no objective way to quantify anything inbetween. What's a 7? I sort of kind of liked it? Versus an 8, I kind of almost really liked it? Is there a formula for this I missed in high school? Someone please teach it to me!

Nothing wrong with dicking around with numbers if that's what floats your boat, but no matter how complex a "system" you devise, it is just that - dicking around. You cannot, with any seriousness, apply a number to an abstract idea.
Cute smile but not enough cleavage.
This is a good review. There's actual information in it. (Actually, I don't even think her smile is that cute myself, but at least now we're talking about something tangible!)

It's also how academic grading works.
Now this is a whole other monkey and one which I have become abundantly, painfully aware of in recent years. And it's still bullshit.

User avatar
Tsuyoshi-kun
Posts: 946
Joined: Tue Sep 07, 2004 11:33 am
Now Playing: Super Smash Bros. for Wii U
Location: Philadelphia, PA, U.S.A.

Re: Don't step in the Pharaoh

Post by Tsuyoshi-kun »

Green Gibbon! wrote:Well if you want a consensus, there's only two answers: I liked it or I didn't like it. Thumbs up or thumbs down. One or two. There's no objective way to quantify anything inbetween. What's a 7? I sort of kind of liked it? Versus an 8, I kind of almost really liked it? Is there a formula for this I missed in high school? Someone please teach it to me!
I completely agree with this, but oftentimes my opinion on something is not as black and white as simply liking it or disliking it, since everything has its good points and flaws. (Even games I really like, like Super Mario Bros. 3, I'll admit there's flaws in.) And even then, you could divide liking/disliking something into so many levels (kind of liking, greatly disliking) that it itself can become a number system.
Green Gibbon! wrote:Nothing wrong with dicking around with numbers if that's what floats your boat, but no matter how complex a "system" you devise, it is just that - dicking around. You cannot, with any seriousness, apply a number to an abstract idea.
I also agree. And if a number is all I wrote, or if I wrote short reviews, the review would be meaningless. But I often write 2/3-page reviews of the series I've watched. (I think my longest is the length of a typical college paper.) I have no problems with numerical ratings for the most part if they're backed up by a solid review explaining why they came up with what they had and (preferably) not listed until the end of the review.

Video games are the most pointless things to give numerical scores to, and it took me a long time to finally come to this conclusion. They require so much more time and energy than a typical movie or even television show (especially RPGs), I'm surprised people deal with reviews of them at all. Two people who play the same game for different lengths (say, 5 and 10 hours each) are going to have completely different takes on it, plus their opinion might change over time. At least with something like a movie or anime, it has a fixed beginning and end. (Unless you fast-forward through it, which is a colossal dick move for anyone who's getting paid to review it to do.)

User avatar
Green Gibbon!
BUTT CHEESE
Posts: 4648
Joined: Fri May 21, 2004 11:39 am
Now Playing: Bit Trip Complete
Location: A far eastern land across the sea
Contact:

Re: Don't step in the Pharaoh

Post by Green Gibbon! »

I have no problems with numerical ratings for the most part if they're backed up by a solid review explaining why they came up with what they had and (preferably) not listed until the end of the review.
Explain to me exactly what information that number possesses that the preceding text would have failed to give.

User avatar
Tsuyoshi-kun
Posts: 946
Joined: Tue Sep 07, 2004 11:33 am
Now Playing: Super Smash Bros. for Wii U
Location: Philadelphia, PA, U.S.A.

Re: Don't step in the Pharaoh

Post by Tsuyoshi-kun »

Absolutely nothing, but so is saying you liked something. A paragraph about how much you like a movie that doesn't use a star rating is far less useful than a two/three-page review of it that would otherwise use a star rating that explains in greater why you do. I fail to see how this is up to debate. The longer a review for something is, and the more detailed it is, the more useful it is, regardless of how it's reviewed.

User avatar
Green Gibbon!
BUTT CHEESE
Posts: 4648
Joined: Fri May 21, 2004 11:39 am
Now Playing: Bit Trip Complete
Location: A far eastern land across the sea
Contact:

Re: Don't step in the Pharaoh

Post by Green Gibbon! »

Okay, then why do you need a star rating at the end of that 3-page review?!

User avatar
Tsuyoshi-kun
Posts: 946
Joined: Tue Sep 07, 2004 11:33 am
Now Playing: Super Smash Bros. for Wii U
Location: Philadelphia, PA, U.S.A.

Re: Don't step in the Pharaoh

Post by Tsuyoshi-kun »

In my case with anime reviews, because of a policy enforced 15 years ago that even the head administrators of the site cannot break out of. In any other case, because people are lazy and only look at the shiny numbers. It's unfortunate, but it's long been accepted the norm. (Leonard Maltin is the only move critic I trust simply because he doesn't use number ratings; he gets to the point fast and quick with his reviews.) Either way you're missing the point. A long, detailed review is a long, detailed review, regardless of how you do so or what bells and whistles you add to it. I suppose the way you present it can be annoying and unfortunately detrimental, though.

User avatar
Crisis
Posts: 531
Joined: Wed Mar 25, 2009 5:15 pm

Re: Don't step in the Pharaoh

Post by Crisis »

Green Gibbon! wrote:Explain to me exactly what information that number possesses that the preceding text would have failed to give.
It doesn't. It's not supposed to. It's supposed to be a summary of the sentiment of the preceding text. Which isn't particularly useful unless you have other scores as a reference and ideally an established set of criteria (which is what I was trying to come up with).
Green Gibbon! wrote:Well if you want a consensus, there's only two answers: I liked it or I didn't like it. Thumbs up or thumbs down. One or two. There's no objective way to quantify anything in-between.
Nobody is claiming true objectivity. But I think it's clearly more complicated than that. I liked Mario 64 and I also like The Binding of Isaac, but they're not the same thing to me; one of them blew my mind and the other is jut a particularly fun distraction.

Game ranks are typically an ordinal scale. (An ordinal scale is one in which the rank order is significant, but the distance between ratings isn't. So 1st is better than 2nd, and 2nd is better than 3rd, but there is no way of telling how much better 1st is from 2nd or 3rd.) This suggests to me that most people have a set of internally consistent criteria they apply to games, but that they're bad at articulating them. A number on its own is meaningless - a single score on an ordinal scale provides no information whatsoever - but if you attach a set of criteria to each point, you can make the scale a little more meaningful.

----

On a related topic topic. Does anyone read any good game critic blogs? The ones I've found haven't been great.

User avatar
Delphine
Horrid, Pmpous Wench
Posts: 4720
Joined: Fri May 21, 2004 1:05 pm
Now Playing: DOVAHKIIN
Location: Los Angeles, CA
Contact:

Re: Don't step in the Pharaoh

Post by Delphine »

Crisis wrote:Does anyone read any good game critic blogs? The ones I've found haven't been great.
That's because there aren't any. Even the ones with halfway decent content are sexist, homophobic cesspools.

And yeah, rating systems are shit. Even citing bugs and technical errors doesn't accurately reflect a good or bad rating; when I buy a Bethesda game I expect those issues and ignore them. Gravity-defying horses and giants that send you careening into space add to my fun.

User avatar
Crisis
Posts: 531
Joined: Wed Mar 25, 2009 5:15 pm

Re: Don't step in the Pharaoh

Post by Crisis »

Delphine wrote:That's because there aren't any. Even the ones with halfway decent content are sexist, homophobic cesspools.
Yeah, I suspected as much.

Still, there must be something out there, right? You know, the other part of Sturgeon's law.

User avatar
Radrappy
Posts: 1329
Joined: Wed May 26, 2004 10:53 pm
Now Playing: MvC3, Vanquish, Skies of Arcadia Legends
Contact:

Re: Don't step in the Pharaoh

Post by Radrappy »

Delphine wrote: Even citing bugs and technical errors doesn't accurately reflect a good or bad rating; when I buy a Bethesda game I expect those issues and ignore them. Gravity-defying horses and giants that send you careening into space add to my fun.
gonna call bullshit on this seeing as technical shortcomings are one of the only truly objective justifications to knock off points. So you've gone into the experience expecting bugs? Great, but the bugs are still there to hamper the enjoyment for most people. Also, I like to believe that a credible journalist will separate his/her enjoyment and the over all quality of the title.

User avatar
Esrever
Drano Master
Posts: 2981
Joined: Mon May 24, 2004 2:26 am
Contact:

Re: Don't step in the Pharaoh

Post by Esrever »

I get what you all are saying, but I mean, geeze. When I find a mainstream game reviewers that is too complex and nuanced to ever be properly summarized by a numerical score, I will let you guys know!

User avatar
Crisis
Posts: 531
Joined: Wed Mar 25, 2009 5:15 pm

Re: Don't step in the Pharaoh

Post by Crisis »

Radrappy wrote:gonna call bullshit on this seeing as technical shortcomings are one of the only truly objective justifications to knock off points. So you've gone into the experience expecting bugs? Great, but the bugs are still there to hamper the enjoyment for most people. Also, I like to believe that a credible journalist will separate his/her enjoyment and the over all quality of the title.
There's the issue that a product should work as advertised. It would be different if I bought "Skyrim: Beta Edition". But it's not unreasonable to expect a reasonable amount of stability from a £40 piece of software.

Then there's the credibility issue. Why is Skyrim full of bugs? Is Bethesda incapable or unwilling to hire competent software engineers? Did they rush the product to market? Do they not value immersion? Did they not manage their funds properly?

Sometimes there is a justifiable explanation. Super Mario 64 was the first game released on an engine which was made from scratch at a time when there were no alternatives available, so a certain number of bugs were inevitable if the product was ever going to ship. World of Warcraft wasn't coded to handle 10 million players and servers regularly went down in the first couple of years, something that Blizzard acknowledged by offering financial compensation towards those who were affected.

But when a game is released at full price on an existing engine as part of an established franchise, without a demo, and there are still bugs, then that suggests there is an underlying sloppiness.

User avatar
gr4yJ4Y
Posts: 1366
Joined: Wed Jun 09, 2004 10:14 am
Now Playing: Breath of the Wild (Switch), Resident Evil VII (PS4)
Location: Crescent Knoll

Re: Don't step in the Pharaoh

Post by gr4yJ4Y »

Crisis wrote:but that they're bad at articulating them.
Game reviewers are bad at articulating what a game is. Maybe the language just isn't developed enough. Maybe the medium needs to mature more before that can happen. It's very hard to describe a game experience to someone who has never played the game in mention before. And even so, I suspect that most reviewers skip over the part where they should be describing what a game is trying to accomplish. So many omit a mention of what the game is trying to do and they instead come up with a list of pros and cons - which can be deceptive if its not accurately weighed. Often the game that is envisioned from reading a review doesn't match how the actual game plays until after it's been played. The review will make sense to someone who has played the game and they can agree or disagree with each statement in the review, but for who the review is being written for - those who haven't played the game and are trying to learn more about it - it is telling a misleading story.

User avatar
Green Gibbon!
BUTT CHEESE
Posts: 4648
Joined: Fri May 21, 2004 11:39 am
Now Playing: Bit Trip Complete
Location: A far eastern land across the sea
Contact:

Re: Don't step in the Pharaoh

Post by Green Gibbon! »

I get what you all are saying, but I mean, geeze. When I find a mainstream game reviewers that is too complex and nuanced to ever be properly summarized by a numerical score, I will let you guys know!
That's the thing, though, it's not about summarizing. I can say for example there's not enough graphical variation between stages so they get repetitive, but the way the stars glitter when you land a hit along with the beat is really cool - shit, I mean how the hell do I "summarize" that with a number? Is that a 6? Or a 5? 7.3? It just doesn't make any sense to me. It's fluff, it adds nothing, it conveys nothing, it summarizes nothing, at least not effectively. Even if I create some complex list of criteria, it's still useless to anyone who isn't completely familiar with my criteria, and for that they may as well take the time to read my actual thoughts. It's just a lot of extra work for nothing.

I dunno, maybe it wouldn't bother me so much if people didn't take it so dead seriously.

Post Reply