Which AI model should mark a short answer, review an item or draft feedback? The usual answer is the most capable one available. For assessment, that answer is often wrong. The choice of a model should rest on measured criteria, on the tasks it will actually perform.
Benchmarks set by education, not only by model developers
Today, model developers decide what AI is and what it can do. Those working in education adopt the tools as they stand, with their strengths and weaknesses. There are no shared criteria, no common benchmarks, and no collective voice to state what education needs. Innovations follow one another at a relentless pace, and adopting a tool that may soon be obsolete has a cost: decisions are delayed.
The e-assessment community is well placed to change this. Assessment is its core expertise; measuring what models can and cannot do on assessment tasks extends that expertise to the questions AI raises. The community already brings together the perspectives that such measurement needs: institutions; data scientists, statisticians and developers; researchers in psychometrics, subject pedagogy, educational sciences, computer science and ethics; and users, from teachers and assessors to representatives of students and families.
Why measure rather than presume
Three reasons make measurement necessary.
- Sovereignty. Open-source models, hosted locally, keep student data and processing under institutional control. They differ widely in quality, language coverage and bias.
- Cost. The right model is the smallest one that does the job, not the most capable one.
- Trust. A wrong score looks like a score. A model that hallucinates, contradicts itself between runs or follows the wording of a prompt rather than the rubric produces errors that do not look like errors.
Principles behind the benchmark
The benchmark follows the principles set out in the framework and in the talk Trusting AI in Digital Assessment:
- Real assessment tasks. Models are measured on the tasks they will perform, such as marking, item review and feedback, before they are adopted.
- Stability across runs. The same response is marked several times. The score and the main points should come back near-identical. Variance across runs is a quality metric, not a detail.
- Comparison with teachers. Several models are run over the same sample and compared with teacher marking. Agreement between independent models is a strong signal; disagreement shows where the task is under-specified.
- Ambiguity ruled out first. Instability often comes from the question, not from the model. An ambiguous item cannot be marked consistently by anyone, human or machine.
An indicator, not a ranking
The aim is an indicator of how well each open-source model suits e-assessment, task by task. It is meant to inform a choice that depends on context: the language, the subject, the stakes of the test and the infrastructure available.