I mentioned a few weeks ago that I was hoping to share a little bit about the process I use to evaluate products for the ENnies. However, before I get into too much detail, I think it’s important to note that I see judging and reviewing as very different tasks.

This is not to say that reviews aren’t valuable–they are! However, reviewing a game product is, essentially, like comparing it to every other game that has ever existed. Reviewers are trying to determine how good or bad the product is when compared to an enormously large field of contemporaneously available products (and even some that are just fondly remembered). Because that field is so large, if reviewers attempt to assign any values to the products they evaluate, they’re essentially limited to assigning them ostensibly objective values (three stars!), rating products on a numeric scale (seven out of ten!), or some other, similar substitute.

That, on the other hand, is not what an ENnies judge like myself needs to do. My job is to evaluate products released or entered over a single twelve-month period, comparing them only to the other products that are submitted by publishers or highly motivated fans. Comparing Warhammer 40K: Dark Heresy to Space:1889 might be an interesting exercise, but doing so isn’t really going to help me determine whether Dark Heresy, which was entered this year, deserves to be nominated for an award.

Leaving aside the fact that my opinions about what should be nominated will have to be compared to four others judges’, I need to make a solid snapshot of the upper crust of the entries: somehow, I need to rank them. Trying to list the entries from best to worst would be an unrealistically daunting task, and even if it weren’t, it would be terribly imprecise, and potentially biased.

So what will I do instead? I’ll use a pair-ranking system. Most simply, pair ranking ensures that I’ve compared each product entered to every other product in a category, one pair at a time. Let’s assume, for a second, that I’m evaluating only three products - Product A, Product B, and Product C. To pair rank them, I need to carefully compare each pair, and determine a winner, like so:

Product A vs. Product B? I like Product B better. Product B wins. I tally the scores like so:

• Product A (0)
• Product B (1)
• Product C (0)

Product A vs. Product C? I like Product A better. Product A wins. I tally the scores like so:

• Product A (1)
• Product B (1)
• Product C (0)

Product B vs. Product C? I like Product B better. Product B wins. I tally the scores like so:

• Product A (1)
• Product B (2)
• Product C (0)

Obviously, Product B wins overall.

1. Product B
2. Product A
3. Product C

This process works more or less exactly as stated for categories that are largely matters of personal preference - Best Cover Art, for example. I can look at the covers, and pair-rank them as I do. Other categories, like Best Rules or Best Game, are slightly more complex, and are treated slightly differently, even though I’ll use the same basic system as I judge them. Want to learn more? Stay tuned.