I mentioned a few weeks ago that I was hoping to share a little bit about the process I use to evaluate products for the ENnies. However, before I get into too much detail, I think it’s important to note that I see judging and reviewing as very different tasks.

This is not to say that reviews aren’t valuable–they are! However, reviewing a game product is, essentially, like comparing it to every other game that has ever existed. Reviewers are trying to determine how good or bad the product is when compared to an enormously large field of contemporaneously available products (and even some that are just fondly remembered). Because that field is so large, if reviewers attempt to assign any values to the products they evaluate, they’re essentially limited to assigning them ostensibly objective values (three stars!), rating products on a numeric scale (seven out of ten!), or some other, similar substitute.

That, on the other hand, is not what an ENnies judge like myself needs to do. My job is to evaluate products released or entered over a single twelve-month period, comparing them only to the other products that are submitted by publishers or highly motivated fans. Comparing Warhammer 40K: Dark Heresy to Space:1889 might be an interesting exercise, but doing so isn’t really going to help me determine whether Dark Heresy, which was entered this year, deserves to be nominated for an award.

Leaving aside the fact that my opinions about what should be nominated will have to be compared to four others judges’, I need to make a solid snapshot of the upper crust of the entries: somehow, I need to rank them. Trying to list the entries from best to worst would be an unrealistically daunting task, and even if it weren’t, it would be terribly imprecise, and potentially biased.

So what will I do instead? I’ll use a pair-ranking system. Most simply, pair ranking ensures that I’ve compared each product entered to every other product in a category, one pair at a time. Let’s assume, for a second, that I’m evaluating only three products - Product A, Product B, and Product C. To pair rank them, I need to carefully compare each pair, and determine a winner, like so:

Product A vs. Product B? I like Product B better. Product B wins. I tally the scores like so:

• Product A (0)
• Product B (1)
• Product C (0)

Product A vs. Product C? I like Product A better. Product A wins. I tally the scores like so:

• Product A (1)
• Product B (1)
• Product C (0)

Product B vs. Product C? I like Product B better. Product B wins. I tally the scores like so:

• Product A (1)
• Product B (2)
• Product C (0)

Obviously, Product B wins overall.

1. Product B
2. Product A
3. Product C

This process works more or less exactly as stated for categories that are largely matters of personal preference - Best Cover Art, for example. I can look at the covers, and pair-rank them as I do. Other categories, like Best Rules or Best Game, are slightly more complex, and are treated slightly differently, even though I’ll use the same basic system as I judge them. Want to learn more? Stay tuned.

Before this blog goes all the way live, I’m curious if you loyal readers like the look of the site? What do I need to do differently? Is this thing a visual disaster?

As some of you may know, I was elected to be a judge for the ENworld Gen Con RPG Awards this year. Indeed, if you know about it, it’s probably because I e-mailed you and asked you to vote for me. Thank you to those of you who did. Due at least in part to your efforts, I will be inundated with Roleplaying Game (RPG) products over the next 9 months, ultimately nominating 5 products in a variety of categories that will be then be further narrowed to a first- and second-place winner by thousands of RPG fans, online.

For those of you not already intimately familiar with the awards, I should note that I won’t be alone in this process. There are 4 other judges that I’ll be working with to judge this stuff, and considering the vastly disparate set of “likes” and “dislikes” we all share, coming to a consensus on what deserves to be recognized is going to be a very difficult process. Nevertheless, it should be a great deal of fun, and certainly satisfying when it’s all said and done.

Over the next little while, I hope to detail a little bit about how I plan to evaulate the products that are entered this year, and perhaps highlight some of the more interesting items that cross my desk. In any case, I hope you’ll come along for the ride with me. It should be reasonably safe, and I can say with certainty that, if you read this blog, you won’t be destroyed by space lasers (at least not because of me).

If you’d like to get a little headstart on where I’m going with all this, I recommend that you read The Thinker’s Toolkit, by Morgan D. Jones. I’ll be using the weighted pair-ranking system described in that book to do the bulk of my assessments, and if you’re really interested in the process, it’s the best explanation of it that I’ve ever come across. It won’t really help illuminate what kind of games I like (which will also deeply affect this process), but it should give you an idea of how I’ll compare one thing to another.