Adaptive Comparative Judgement in Educational Assessment

Adaptive Comparative Judgement (ACJ) is an innovative assessment methodology in which assessors make holistic judgments by comparing pairs of student work and selecting the superior item. Building on traditional Comparative Judgement (CJ), which produces reliable rank orders through pairwise comparisons and statistical models such as the Bradley–Terry model, ACJ employs dynamic pair-selection algorithms to maximise information gain and reduce the number of comparisons required. This approach addresses common challenges in large-scale and high-stakes assessment, including marker bias, inconsistency and cognitive load. By iteratively adapting which items are compared—whether through entropy-driven selection, reference-based strategies or Bayesian active learning—ACJ enhances efficiency without sacrificing validity. Its flexibility has been demonstrated across diverse domains, from essay writing and creative performance to design-thinking artefacts and theoretical mathematics tasks. Digital platforms have further enabled collaborative standard-setting across regions and nations, underpinning a shift towards more transparent, reliable and scalable practices in educational measurement.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Adaptive Comparative Judgement in Educational Assessment publication trend

The graph below shows the total number of articles in adaptive comparative judgement in educational assessment across all publications each year (not limited to Nature Index journals).

Technical terms

Comparative Judgement (CJ): A holistic assessment method in which judges compare two items and choose the better one, producing a quality scale via statistical models.

Adaptive Comparative Judgement (ACJ): An extension of CJ that dynamically selects the most informative pairs for comparison to enhance efficiency and reliability.

Bradley–Terry model: A probabilistic ranking model used to convert pairwise comparison data into a continuous measure of item quality.

Bayesian active learning: An approach that integrates Bayesian inference with selection criteria (such as entropy) to choose comparisons likely to maximise information gain.

References

1. A Bayesian active learning approach to comparative judgement within education assessment. Computers and Education Artificial Intelligence (2024).
2. A Review of the Valid Methodological Use of Adaptive Comparative Judgment in Technology Education Research. Frontiers in Education (2022).
3. Comparative judgement in education research. International Journal of Research & Method in Education (2023).
4. The Accuracy and Efficiency of a Reference-Based Adaptive Selection Algorithm for Comparative Judgment. Frontiers in Education (2022).
5. On the Bias and Stability of the Results of Comparative Judgment. Frontiers in Education (2022).
6. Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability. International Journal of Educational Technology in Higher Education (2016).
7. Examining the Validity of Adaptive Comparative Judgment for Peer Evaluation in a Design Thinking Course. Frontiers in Education (2021).
8. Assessing covariation as a form of conceptual understanding through comparative judgement. Educational Studies in Mathematics (2022).
9. Using Teachers’ Judgments of Quality to Establish Performance Standards in Technology Education Across Schools, Communities, and Nations. Frontiers in Education (2022).