2026 ASEE Annual Conference & Exposition

LLM Use to Evaluate Student Weekly Reflections in First Year Design Class

Presented at Computers in Education (CoED): Learning, Engagement & Inclusion (2 of 9) -- M408B

This empirical paper investigates whether Large Language Models (LLMs) are effective in assisting professors in summarizing and scoring students' weekly reflections. Reflections support improved retention and enable instructors to identify those who may need personalized assistance. Driven by the need for scalable, timely, and actionable feedback analysis in large classes, this study examines the reliability of LLM summarizations. The LLM-generated qualitative data is then validated by correlating it with student self-reported quantitative improvements in a first-year engineering course.

Weekly student feedback collected from this course was pre-processed and analyzed using the LLM ChatGPT-4o-mini. The analysis included LLM-based summarization and rubric-based Likert scoring on a 1 to 5 scale. The summarization rating categories included roadblocks, tone/mood, perceived learning, team sentiment, persistence, and engagement quality. These categories were based on Likert responses and open-ended questions from the feedback form, in which students were asked to discuss breakthroughs and roadblocks on their design project, teamwork, and general feedback about the class. Manual analysis of past student reflections served as a benchmark to evaluate the accuracy and consistency of LLM-generated summaries and sentiment scores. The comparison emphasized how effectively both methods identified students who may require additional support or intervention. After the LLM demonstrated performance comparable to manual grading, Spearman's correlation tests were used to examine how LLM-derived rubric scores align with students’ end-of-quarter Likert responses and to illustrate the potential for using these validated scores to establish concurrent validity.

Results from this study indicate that LLM-generated scoring based on rubrics is comparable to human scoring. These results suggest that LLMs can serve as effective, scalable graders of short reflection forms in large engineering classes when paired with clear rubrics and prompts. This work highlights how integrating LLM-generated reflection scores establishes a foundation for supporting timely interventions for at-risk students in the future.

Authors

Anthony Gwun Hynn Chin University of California, San Diego [biography]

Anthony Chin is a master's student in Mechanical and Aerospace Engineering at the University of California San Diego. Their Research focuses on utilizing education technology and data analysis to improve student learning outcomes in spatial visualization and engineering classes. They have worked on projects involving large scale educational feedback and analysis and are interested in applying AI tools for instructional support and student success.
Dr. Lelli Van Den Einde eGrove Education [biography]

Van Den Einde is a Teaching Professor in Structural Engineering at UC San Diego and the President of eGrove Education, Inc. She has decades of experience teaching hands-on, project-based curricula, spanning high school camps, K-12 outreach, and undergraduate design courses. Dedicated to fostering diversity, she creates supportive environments for students of all backgrounds. Her teaching approach emphasizes scaffolding multidisciplinary skills to boost student self-efficacy and foster meaningful learning outcomes. Dr. Van Den Einde's research focuses on student engagement in large classrooms, developing adaptable design-build-test projects, and creating software to enhance spatial visualization skills, a key factor in improving STEM retention and success.
Dr. Nathan Delson University of California, San Diego [biography]

Nathan Delson, Ph.D. is a Senior Teaching Professor at the University of California at San Diego. He received a PhD in Mechanical Engineering from MIT and his interests include robotics, biomedical devices, product design, engineering education, and maker spaces. In 1999 he co-founded Coactive Drive Corporation (currently General Vibration), a company that provides force feedback solutions. In 2016 Nate co-founded eGrove Education an educational software company focused on teaching sketching and spatial visualization skills.
Jishan Kharbanda University of California, Los Angeles
Hollis Voinov University of California, San Diego [biography]

Hollis Voinov is an undergraduate student in Structural Engineering at the University of California San Diego. He joined the research team with an interest in analyzing how software can train spatial visualization and how best to support students in large classes.
Andrea Tueanh Huynh University of California, San Diego [biography]

Undergraduate Structural Engineering student at University of California, San Diego.

Note

The full paper will be available to logged in and registered conference attendees once the conference starts on June 21, 2026, and to all visitors after the conference ends on June 24, 2026

« View session