Reasonable Doubt: Why ChatGPT is not (quite) ready to grade and give feedback on law assessments

Description

The emergence of Open AI’s ChatGPT in late 2022 triggered considerable debate among academics across Higher Education. Initially, debate focused on students’ use of ChatGPT, and later iterations of Generative AI (GenAI), in creating their academic work. Fears of cheating were rife and, to some extent, have proven well-grounded.

However, the discussion then moved to academic use of GenAI, including its potential use to grade student work, and provide feedback. Some want to take the Red Pill, exploring the reality where GenAI plays a role in marking student work – potentially more quickly, and with less bias, releasing academic teams to pursue other, hopefully more rewarding, activities. For others, it is the Blue Pill, and a return to the false, but comfortable reality where GenAI marking and the associated risks of academic redundancies never manifest.

This case study is based on a custom-built, ChatGPT 4.5-based, GPT (“MarkGPT”), which graded and provided feedback on Level 5 Law student work completed in 2023/24, on one compulsory module, and one designated module. Methodologically, the project combined quantitative analysis of the divergence between human and AI grading, with a Braun & Clarke thematic analysis of the main theses which emerged in both the human and AI feedback.

The project concluded that, as of Spring 2025, ChatGPT lacks the capacity for consistency (with itself, and with human markers), and accuracy in grading, but that there are significant similarities with the human markers in terms of the feedback themes.

ChatGPT then, is not quite there. Yet.

Period	19 Jun 2025
Event title	Reimagining Education: The Future of Learning & Teaching: Inaugural I-LEAD Mini conference
Event type	Conference
Location	London, United KingdomShow on map
Degree of Recognition	International