2026 ASEE Annual Conference & Exposition

Amid rapid industry change, new AI tools and the growing popularity of “vibe-coding” applications are fueling interest in learning computer science across a wide range of instructional settings. This growing demand may intensify an existing challenge in computing education: heavily skewed instructor-to-learner ratios that make it difficult to provide timely formative feedback on assessments and projects. Autograding tools help address this problem, but even strong autograders often require substantial manual configuration and typically provide limited insight into the underlying reasons for test-case failures or the misconceptions reflected in student submissions. Because learners often turn to LLM-based tools to understand why something failed, this creates an opportunity to integrate them into assessment workflows in ways that better support learning.

This paper analyzes an LLM-based grading workflow using thousands of historical student submissions from 70 Python code-writing exercises in an asynchronous MOOC-style course on Codio. The assessments span a wide range of autograder complexity, including simple input-output tests, unit-test-based checks, and custom autograders that parse student submissions.

Our analysis shows that, for this dataset, the LLM-based workflow achieves high agreement with the existing auto-graders across a wide variety of test configurations, while simultaneously producing fine-grained, test-case-level explanations for students. We illustrate how this workflow can surface rubric-like feedback from existing auto-graders with minimal additional configuration, and we discuss practical opportunities and limitations for integrating LLM-generated grading and feedback into CS courses.