Writing Judging Criteria Before You Score Anything

The most common failure in a small selection process is not bias in the judging. It is that the criteria did not exist until after the judges had played. Once that has happened, every criterion is really a description of a decision already made, and the resulting rubric explains the outcome rather than producing it.

Writing the rubric first is unglamorous and it is the only part of the process that can be checked from outside.

Why order of operations matters

In usability work there is a long-running precedent for fixing the checklist before evaluating a product. Heuristic evaluation, as described in its literature, has an evaluator examine an interface against a set of recognised usability principles — and the principles come from an established body, not from the evaluator’s reaction to the product in front of them. The Nielsen Norman Group’s ten heuristics are the best-known public example of such a list, and they are published independently of any particular product.

That structure is the useful import. The list is public, the evaluation is separate, and anyone can ask whether the same list was applied the same way twice.

Five things to fix in writing

A rubric that can survive scrutiny needs five decisions recorded before the first judgment:

1. The criteria themselves, as short named statements. “Rule clarity” is a criterion; “feels fair” is a reaction.
2. The weights, as explicit numbers that sum to something. A weight is a statement about priority, and it should be written down while nothing is at stake.
3. The evidence standard — what counts as a valid source for each criterion. This is where most rubrics quietly cheat: a criterion said to be about the game being evaluated against the store page’s marketing copy.
4. The failure mode, stated per criterion. What would make a game fail this specific test?
5. The order of assessment, if a later criterion depends on an earlier one. It usually does.

Make each criterion falsifiable

The clearest way to test whether a criterion is worth anything is to ask what evidence would show a game failing it. If no evidence could, the criterion is decoration.

Take accessibility, which is one of the easiest to write badly. The Game Accessibility Guidelines publish a full list ordered by audience and by difficulty of implementation, splitting items into basic, intermediate and advanced and grouping them under motor, cognitive and other impairment categories. Basic items include adjustable game speed, remappable controls, adjustable control sensitivity, and making interface elements large and well spaced. Cognitive basics include avoiding flickering images and repetitive patterns, letting players progress through text prompts at their own pace, and using simple clear language.

That list turns a vague criterion into a set of checkable questions. The record can then say which items were present in the options menu and which were absent. It does not need to grade a studio’s good intentions.

For a game that presents itself as a web page, the W3C’s Web Content Accessibility Guidelines give a second, equally checkable standard. Success Criterion 1.4.3 requires a contrast ratio of at least 4.5:1 for text, with large-scale text at 3:1. Success Criterion 1.4.10 requires content to be presented without loss of information or functionality and without two-dimensional scrolling at a width equivalent to 320 CSS pixels. Both are measurable with free tools and a browser.

Write down what you will not measure

The most useful sentence in a rubric is often a refusal. If the desk will not publish sales figures, team sizes or development budgets because it cannot verify them, that sentence belongs in the rubric, not in a defensive footnote later.

Two categories are worth pre-committing to ignore:

• Production scale. A criterion that rewards budget is not a criterion about the game, and it will monotonically favour large studios in a rubric that claims to be about small ones.
• Genre expectations. Judging a puzzle game against what “puzzle games usually do” imports a moving target. Judge it against its own stated rules and against the criteria as written.

Weight for the argument, not for the drama

Weights are where a rubric makes its values public. If rule clarity carries a quarter of the weight, the desk is saying that a beautiful game with an unlearnable rule set cannot be saved by its presentation. That is a position someone can disagree with, which is exactly why it should be published before the first game is opened.

Write the weights so that they can be defended one at a time. If a weight cannot be explained in a sentence, it is probably a proxy for something else.

Publish and freeze

Finally, date the rubric. If a weight changes, the change takes effect on a stated date onwards, and records written before it keep the weights they were written under. Deleting the earlier version is the one move that makes the whole exercise pointless, because it destroys the only evidence that the rubric was not reverse-engineered.

None of this makes a selection process objective. It makes it auditable, which is a different and more honest claim.

Sources

• Game Accessibility Guidelines, “Full list” — https://gameaccessibilityguidelines.com/full-list/
• W3C WAI, “Understanding SC 1.4.3 Contrast (Minimum)” — https://www.w3.org/WAI/WCAG22/Understanding/contrast-minimum.html
• Wikipedia, “Heuristic evaluation” — https://en.wikipedia.org/wiki/Heuristic_evaluation
• Nielsen Norman Group, “10 Usability Heuristics for User Interface Design” — https://www.nngroup.com/articles/ten-usability-heuristics/