2025 ASEE Annual Conference & Exposition

Extensive research highlights the value of gathering student feedback throughout the semester. While more time-consuming to analyze than traditional Likert scales, free-response feedback often yields more nuanced insights. However, instructors may struggle to implement meaningful course changes without a structured approach to parsing such feedback. The development of large language models (LLMs) capable of addressing various text-processing tasks allows instructors to streamline the analysis of students' course reflections. We propose an iterative, short-text reflection analysis approach for real-time dynamic course intervention development. Using the most common proficiency challenges extracted from students' reflections, we develop support materials designed to improve students' knowledge and skills, including lessons, videos, and other resources. Then, we analyze the impact of these newly developed lessons on students. We implement this approach through our reflection-based intervention development cycle (CIDC). This method helps instructors gather reliable evidence about course effectiveness, enabling them to design new lessons or adapt existing ones to support students' needs more effectively. In this work, we present the findings of a use case in an asynchronous online computer science course and test both manual and LLM-based methods of implementing the various stages of the intervention development cycle. Our study suggests that students may be receptive to lessons (offered as extra credit activities), with 80.25% completing at least one of the lessons and nearly half (48.3%) completing at least four out of six. We also present the results of experiments with automated methods evaluated by human labelers. These findings explore use cases for automation, demonstrate the complexity of the task, and justify the need for further development.