Review Assessment Scale

[BobBretall]

We just had a conversation at the end of episode #053 about our rating system that we use for our reviews. A bit more informal on the show, but I do write them down on the Blog.....

Right now we have:
* Thumbs Up!
* neutral
* Thumbs Down

The problem here is that this is not very granular & some "Thumbs Up" books we like a lot more than others.

We could go to a standard 10 point scale, but then, is a 7/10 a Thumbs Up? Where is neutral? Is a 4/10 a Thumbs down, or does a book have to be a 2 before it's a Thumbs down? Lots of variables.....

We could combine them and say (for instance) "This book was a definite Thumbs up, an 8/10!" but John worries that we'd be inconsistent from week-to-week on what a 7 or 8 is......

Maybe we could get cute and adopt the 10-point Overstreet scale: "That book was GREAT! NM(9." -or- "An OK middle of the road book, I'll give it a VG(4.5)"

Anyway, help us out! Put some thoughts on what you'd like to see in a Review Assessment Scale!

[Trev]

When my wife and I are reviewing beers we use a 4 star scale where:

1 * = I would never order this/Makes me ill to think about drinking it -- do *you* want it?
2 * = I would never order it, but might drink it if it's the only thing around
3 * = A good stand by. Would order this again
4 * = A favorite. I would gravitate towards it and it would be among my first choices

We use half stars to denote 'somewhere in between'.

I think you could probably apply a similar system to comics.

1 * = sell it now. Why did I buy it? Damn, it's taking up space too.
2 * = well, that's an issue. Done with this series for a while (for mini's you would drop it there)
3 * = I'll keep reading this. Not bad. Let's see where it goes.
4 * = A classic. Where are my mylars?

With half stars denoting a similar 'in between' feeling.

[zack kruse]

I always think these types of opinions are hard to quantify. Not only are they personal opinion, but, in my experience, can be swayed by outside forces or what that particular book is going up against that week.

From listening to you guys for quite a while, I have no doubt that you guys can be objective in your assessments, though.

Maybe, instead of a thumbs up/thumbs down or 10-scale model, you could just give your thoughts on the book and then say whether or not you thought it was worth the cover price. I suppose that isn't too much different than what you do now, but, for the listener, it gives them an objective standard of value and one that hits close to home.

A thumbs up is certainly a good review, but is the listeners' $2.99 or $3.99 (or discounted price) better spent on that book than another? Then support that assertion with your opinion on the book.

I also think doing it that way, while certainly establishing your opinion, makes a negative review less of an indictment of the book steers it more towards offering listeners other titles that they may be more interested in. So, let's say you don't like a book--like really don't like it--then instead of giving it a 1/10 or a thumbs down you can direct listeners to another, similar title that may grab them more.

That way you've given your thoughts and, at the same time, can help steer listeners towards comics that may be more in their interest.

I don't know if that is helpful for you or not. I hope I've been clear in my thoughts.

[JohnMayo]

I don't know if that is helpful for you or not. I hope I've been clear in my thoughts.

That is very helpful, Zack. I like the idea of "is it worth the cover price" and if it isn't, what like it would be. I'll have to give that some more thought.

[BobBretall]

I don't know if that is helpful for you or not. I hope I've been clear in my thoughts.

That is very helpful, Zack. I like the idea of "is it worth the cover price" and if it isn't, what like it would be. I'll have to give that some more thought.

Even a "is it worth the price" rating is pretty subjective & open to variations week-to-week, though.

Maybe more of a "would you buy it again knowing what you now know?" kind of thing, but that's really what my interpretation of "Thumbs Down" was.

Another thing to consider with this kind of granularity is that John & I have a vastly different threshold (from most listeners) on what will cause us to stop buying something, or even go back & not buy again. Given this thought, I suppose a "not worth the money" is more appropriate, since we can feel we got ripped off, and go into that knowing full well that we'd get ripped off again to keep up a run or something, since we are just that sadly addicted...

[Fnord Serious]

Just finished listening to your discussion at the end of episode #53.

The 3 point scale (Thumbs Up, Neutral, Thumbs Down) works fine in the context of any given episode because it is accompanied by your discussion of the book in question and the listener can tell how much you did or didn't like it.

If you want more granularity, I would suggest a five point scale. Something along the lines of:

5- Great book, strongly recommended
4- Good book, no problems with it, just didn't knock my socks off
3- A few problems with execution but still worth the money
2- Okay, but only if you can find it in the quarter bins
1- Don't bother

I think a ten point scale would be too complex and inconsistent.

[JohnMayo]

I've added a web page with a summary of the titles we discussed in the Weekly Comics Spotlight episodes and our thumbs up/neutral/thumbs down ratings for them:

http://comicbookpage.com/Podcast/ShowNo ... ewList.php

[Frank Castle]

I like the idea of 1-5 stars. I also like the idea of "is this worth cover price?" Or even something like:

Buy this from DCBS or off the Rack
Wait for Trade
Pick up from a Bargain Bin
Don't buy it, even for out of a quarter box

[JohnMayo]

I'm leaning towards a five point scale measuring if we recommend the title or not.

I mean, I'm going to keep getting some of these things even if they aren't all that good so telling you that I'm going to drop a title isn't going to happen all that often. Conversely, if I don't think a title is all that good, I'm not going to recommend it to other people even if I still plan on getting it.

So, my current thoughts on a rating system:

+2: This is a great comic. I recommend it without reservations.
+1: This is a good comic. It it a step above average and I recommend it.
0: This is an average comic. It isn't bad but it doesn't stand out from the crowd. This is the baseline of what I'd consider to be a good and acceptable comic.
-1: This is a comic that just didn't "click" with me for some reason. It isn't horrible but I really can't recommend it.
-2: This is a comic that not only didn't entertain me but seems unlikely to entertain most people.

This sort of rating system would allow us to differentiate the "good" from the "great" which we can't do with the current thumbs up/neutral/thumbs down system.

Also, I want to stress that the middle rating denotes what I consider to be an average acceptable comic book. It is a perfectly good comic book but one that really isn't any better (or worse) than most of what I've read over the years. This is what I see as the "default" rating.

If we go with this sort of system, I'm open to suggestions for icons for the different ratings that I can use on the new Weekly Comics Spotlight Review List webpage.

Thoughts?

[BobBretall]

Having positive & negative numbers grouped around a zero center-point would seem to appeal to hardcore math types (that's you, John), but seems a bit generally unclear for someone wandering in off the street & seeing a "+2" as a rating (given that we don't continually re-publish a legend/code key).

Since a 1 to 5 "star" system or a 1-10 point numeric scale seems to be generally used in society for rating lots of other things, I'd just as soon keep with one of those, since they are most generally understandable.

[JohnMayo]

A -2, -1, 0, +1, +2 scale and a 1, 2, 3, 4, 5 scale are the same thing. I'm fine with going with the 1 to 5 scale.

[BobBretall]

Most of the suggestions seem to be able to be consolidated to a 5 point scale (really a 6 point scale, since something REALLY bad can get a zero):

We can twizzle the specific verbiage as necessary:

5- Outstanding!! One of the best I have read.
4- Great! Glad to be reading it.
3- Good solid comic. No major problems, just didn't knock my socks off.
2- OK. A few problems with execution but still worth reading.
1- Poorly executed comic. I really struggled to read through this book. Try it only if you really love the character/creator or if you can find it in the quarter bins.
0- Terrible Comic. I Regret having wasted my money on this book.

For me, these gradations map as follows to our previous scale:

Thumbs up! -> 4 or 5 (though I may have given some 3's a Thumbs up if I was in a good mood)

neutral -> 2 or 3

Thumbs Down -> 0 or 1 (though I may have given a 2 a Thumbs down when I was in a bad mood).

Hey! All ratings scales are pretty subjective!

[Pictor]

I think that by far, the actual review and opinions therein are key to my purchase decision, not the recommendation at the end, and so I'm not sure a point scale would be all that useful. Useful things in the podcasts are descriptions of the art style, synopsis of the story, opinions on wordiness and quality of writing, whether it fits in continuity and so forth. This way, the listener can assign their own “point value”, based on the factors that they feel are most important to them. Some with a love of a certain types of art may choose to weigh that more heavily, while others will focus on writing. Even going deeper into the art side of things, some may favor style, while others might look more at its storytelling effectiveness. I think a brief summary of these attributes may be more effective than an arbitrary point scale, a quick list of the most crucial points at the end of the review might work, something like:

-[Pretty/Ugly/Detailed/Messy/…etc] Artwork
-[Unique/Confusing/Generic/Wordy] Writing/Story
-Reminds me a bit of the story/art in [example] book.

[BobBretall]

I think that by far, the actual review and opinions therein are key to my purchase decision, not the recommendation at the end, and so I'm not sure a point scale would be all that useful.

The rating is mostly there for the Blog (which has a much less detailed review than we do verbally on the show), as well as for a summary of all books we've reviewed in the past.

I'd agree that it is of limited value while listening to the show since you just heard us rattle on for 5 minutes about specific reasons we did or did not care for a title.

[JohnMayo]

I've done a quick conversion of the Weekly Comics Spotlight Review List page to the new rating system.

We'll go through and adjust the ratings for past items when we get a chance.