Generative artificial intelligence vs. law students: an empirical study on criminal law exam performance

Claims that GPT-4 can outperform more than 90% of human test-takers in the US Uniform Bar Examination have sparked heated debates about the impact of Generative AI (GenAI) on legal education, academic integrity, and the future of legal practice. Yet GenAI’s capabilities in broader legal examination contexts–including in jurisdictions outside the US–are unclear. This study addresses this gap by evaluating GenAI’s performance against students who took the ‘Criminal Law’ final exam at an Australian law school in Spring 2023. Various AI models and prompt engineering techniques were used to generate 10 distinct answers to the exam question. Five criminal law tutors, unaware of AI involvement, graded a mix of AI-generated and student responses. Then, the tutors were briefed on the AI-generated papers they marked and engaged in reflective semi-structured interviews. The study found that GenAI performed below the student average in questions that required detailed legal and critical analysis. However, all GenAI papers performed better than students in open-ended questions and essay writing tasks. These results provide a benchmark for the capabilities and limitations of GenAI in higher education and provide insights into the potential implications of its application to legal assessments and education, curriculum development, and the future workforce.