Adaptive Comparative Judgement is a system that I've been working on for several years in association with Goldsmiths college and Digital Assess

This site is a relatively new version of the system that has been specifically designed to give users all the tools they need to run their own sessions.
The frontend has been built using React, talking to a REST API written in Symfony.
The ACJ calcuations are preformed on a clustered Java platform.
The secondary statistics and reports and built through a Python backend.

The technique of comparative judgement combined with Rasch statistical analysis is a tried and tested approach. However, certainly in an educational context, it's always been a labour intensive approach and has generally been conducted manually.

The concept is very simple. Given a set of stimuli - be they questions or pictures or exam papers, a "judge" is shown 2 items from the set. They have been instructed to make a comparison based on some pre-determined holistic criteria, e.g. which item is better written, which item is most creative, which item shows better problem solving skills.
The judge only has to chose which of the two items best fits the criteria - which is "better".
This process is repeated until a fully representative sample of judgements have been made.
Using Rasch analysis, these items can then be put into a rank order from best to worst.
With a manual approach, you would endeavour to generate a full range of judgements, ensuring that every item has been compared to every other item.
This approach is only practicable for relativly small data sets.

What ACJ offers, through the algorithms it employs, and by delivering this system online, is a much more efficient method to reach a reliable rank order. It can arrive at the answer more quickly. We have shown over many trials that trained judges are usually quicker and considerably more reliable when compared to more traditional "marking" techniques.