2025 ASEE Annual Conference & Exposition

Recent research has demonstrated significant advancements in the applications of Large Language Models (LLMs) in educational environments, particularly in delivering immediate, personalized student feedback. This study examines the impact of Codio Coach, a specialized AI learning assistant integrated into the Codio platform, on student engagement and performance in asynchronous MOOC-style computer science courses.. It utilizes Large Language Models (LLMs) to provide support without supplying direct answers. It consists of three modules: Summarizer, which simplifies assignment instructions; Error Explanation, which clarifies programming error messages; and Hints, which provides Socratic-style hints by posing questions or suggestions to guide students toward solutions.

Analysis revealed an immediate and sustained uptake in assistant usage, with "Explain this error" being the most frequent interaction (56.3%), confirming engagement and highlighting student need for error comprehension support. Assignments where Coach was enabled showed improved student performance, with a 12% increase in Mean Grade and a 15% increase in Median Grade. Furthermore, an impressively low error event rate (0.12%) observed in these AI-assisted courses suggests early signs that such tools may contribute to more effective programming environments.

These findings provide valuable evidence for the efficacy of tailored AI learning assistants in enhancing student engagement and performance in CS education. We recommend educators guide students in leveraging custom, context-specific assistants to improve learning and develop critical AI application skills.