Judging Order in a Contest
Scores drift upward through a session. Re-randomising per judge is the standard fix.
Contest judging is the one ordering problem where a single random order is insufficient: because every judge sees the same sequence, the order effect is shared rather than cancelled, and only re-randomising per judge removes it.
Judges score later entries differently from earlier ones. The effect is well documented, it goes in different directions in different contexts, and it is large enough to change outcomes.
A single random order does not fix it, and that is the part most contests get wrong.
Why one random order is not enough
If all the judges see the entries in the same sequence, then whatever order effect exists is applied identically to every judge. It does not average out — it accumulates, because every judge's scores are shifted the same way by the same positions.
Randomising that single order removes the systematic advantage of a particular entry being first, and leaves the order effect fully intact for whichever entry happens to draw that position. The contest is fairer in the sense that nobody was chosen to go first, and just as distorted in the result.
Re-randomising per judge is what actually cancels it. If judge A sees entry seven first and judge B sees it last, the order effect on entry seven differs between them and averages out across the panel.
Twelve entries, four judges
- One shared random order: every judge scores entry 12 last. The late-slot effect is applied four times.
- Four independent random orders: entry 12 is seen in four different positions.
- The order effect on any entry now averages across positions instead of accumulating.
Which direction the effect runs
It is not always a penalty for going late, and this is worth knowing before designing around it. In some settings judges become more generous as a session progresses; in others, more critical. In several documented contexts, judges anchor on the first few entries and score subsequent ones relative to them.
What is consistent is that position matters. Which direction it runs in your contest depends on the format, the fatigue level and what the judges are comparing against, and you are unlikely to know in advance.
That uncertainty is itself the argument for re-randomising per judge: it neutralises an effect whose direction you do not need to know, which is a considerably more robust position than trying to correct for a direction you guessed.
Practical implementation
Generate one order per judge before the contest and hand each judge their own sheet. The order randomizer produces an independent shuffle each time you press it, so four judges is four presses.
Where entries are performed live rather than reviewed independently, per-judge randomisation is impossible — everyone watches the same performance in the same order. In that case the mitigations are breaks, and scoring against a rubric rather than against the previous entry.
For anything reviewed asynchronously — written entries, recordings, portfolios — per-judge randomisation costs nothing and should be the default.
- Asynchronous review: one independent random order per judge. Always do this.
- Live performance: per-judge ordering is impossible; use breaks and an absolute rubric.
- Either way: score against the criteria, not against the previous entry.
- Record which order each judge used, so the scores can be checked for order effects afterwards.
Checking whether it mattered
If you recorded each judge's order, you can check for order effects afterwards by comparing scores against position. A consistent trend across judges — later entries scoring systematically higher or lower — tells you the effect was present and how big it was.
That is worth doing once for a recurring contest, because it converts an assumption into a measurement and tells you whether the mitigations are needed at your scale.
It also gives you something to say if a result is challenged. "We randomised per judge and checked for position effects" is a considerably stronger answer than an assertion that the judging was fair.
Frequently asked questions
Does randomising the judging order fix order effects?
Not if every judge sees the same order. The effect is applied identically to all of them, so it accumulates rather than averaging out.
What does fix it?
An independent random order per judge. Then each entry is seen in different positions by different judges, and the effect averages across the panel.
Do judges score later entries higher or lower?
It varies by context and format. What is consistent is that position matters, which is why neutralising the effect beats trying to correct for a direction.
What if the entries are performed live?
Per-judge ordering is impossible. Use breaks and score against an absolute rubric rather than against the previous entry.
How do I generate per-judge orders?
Press the order randomizer once per judge. Each press is an independent shuffle, so four judges is four sheets.
Should I record which order each judge used?
Yes. It lets you check afterwards whether position correlated with score, which converts an assumption into a measurement.
Is one order fairer than alphabetical?
Marginally — it removes the systematic advantage of a name. It does nothing about the order effect itself.
What do I say if a result is challenged?
That you randomised per judge and checked for position effects. That is much stronger than asserting the judging was fair.
Tools for this job
Order Randomizer
Shuffle any list into a fair random order — presentation order, turn order, anything.
Random Seed Generator
Generate a seed, then reproduce the exact same draw from it — the auditable kind of random.
Turn Order Generator
Straight, reversing, or snake order across rounds — first-player advantage, handled.
Background reading
Random Sampling for Audits and Surveys
Sampling frames, sample size, and the mistake that invalidates the whole exercise.
How to Prove a Draw Was Fair Afterwards
What evidence actually convinces someone who was not in the room when you drew.
The Law of Large Numbers, Practically
What 'it evens out' actually promises, on what timescale, and what it never promises.