2026 ASEE Annual Conference & Exposition

Experiences in Reverse Engineering Student Mistakes Using Generative AI

Presented at Computers in Education (CoED): AI in Education (3 of 9) -- M508B

The use of generative artificial intelligence has become commonplace among students, but less so among their faculty. In this paper, we present our experience using large language models (LLMs) to determine what misconceptions 89 undergraduate students have based on the incorrect responses to a four-part exam problem on CPU scheduling policies. If we are successful, our goal is to incorporate such feedback into a CPU scheduling simulation used by undergraduate Operating Systems students. To answer this question, we first manually analyze each of the 112 incorrect solutions to identify underlying misconceptions and random mistakes. We then ask three production-quality LLMs (ChatGPT, Claude, and Gemini) to do the same task and compare their responses to ours. The LLM responses varied widely in quality and accuracy. The LLMs tended to perform better when students made systematic errors and on simpler policies. Interestingly, the LLMs tended to perform better when the instructors initially disagreed on the misconception, and the error was systematic. Faculty may find this usage of artificial intelligence highly beneficial, but we recommend reviewing the responses before sharing them with students. We thus conclude that the feedback provided by these LLMs is not yet ready for use directly in the CPU simulator.

Authors

Dr. Maria R Ebling United States Military Academy [biography]

Maria R. Ebling is an Assistant Professor at the United States Military Academy at West Point in the Department of Electrical Engineering and Computer Science. She earned the Ph.D. and M.S. degrees in Computer Science from Carnegie Mellon University and the B.S. in Mathematics with a Computer Science option from Harvey Mudd College. Her research interests lie at the intersection of distributed systems and human-computer interaction, especially around pervasive computing, and the Internet of Things. Prior to joining the faculty of West Point, Dr. Ebling spent three years as the Chief Technology Officer at Medaptive Health, a small health-tech startup. Before that, she spent nearly 20 years at the IBM T. J. Watson Research Center, in various roles ranging from research staff member to director and was a member of the IBM Academy of Technology. Dr. Ebling is an ACM Distinguished Scientist. She served as the Editor-in-Chief of IEEE Pervasive Computing from 2014-2017 and continues to serve the publication as a member of the Advisory Board. Over the years, she has served on numerous program committees in mobile computing.
Mr. Geoffrey Moores http://orcid.org/0000-0002-9496-960X United States Military Academy [biography]

MAJ. Geoff Moores is an instructor in the Department of Electrical Engineering & Computer Science at the United States Military Academy (USMA) in West Point, New York. He is an active duty Cyber Officer with a masters degree in Computer Science from UMD. His research interests include network science, optimization algorithms, and computer science education.

Download paper (379 KB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session