2024 ASEE Annual Conference & Exposition

Board 62: Work in progress: A Comparative Analysis of Large Language Models and NLP Algorithms to Enhance Student Reflection Summaries

Presented at Computers in Education Division (COED) Poster Session

The advent of state-of-the-art large language models has led to remarkable progress in condensing enormous amounts of information into concise and coherent summaries, benefiting fields like education, health, and public policy, etc. This study contributes to the current effort by investigating two NLP approaches’ effectiveness in summarizing students’ reflection text. This approach includes Natural Language Processing (NLP) algorithms customized for summarizing students’ reflections and ChatGPT, a state-of-the-art large language model. To conduct the study, we used the CourseMIRROR application to collect students’ reflections from two sections of the engineering course at a large Midwestern university. Over the semester, students were asked to reflect after each lecture on two aspects of their learning experience, i.e., what they found 1) interesting and 2) confusing in the lecture? In total, we collected reflections from 42 lectures, and the average class size was 80 students in each section. To inform the study, we generated a reflection summary for all reflection submissions in each lecture using both NLP approaches and human annotators. Furthermore, we evaluated the quality of reflection summaries by assessing the ROUGE-N measure for each lecture’s reflection summary generated by all three approaches. These summaries were then aggregated for each approach by averaging the ROUGE-N scores. Subsequently, we used ANOVA to determine significant differences between the average ROUGE scores of the two NLP approaches and human-generated reflection summaries. Preliminary findings suggest that NLP algorithms outperformed ChatGPT for creating reflection summaries. This finding implies that, despite being trained on a large corpus of textual data, the prominent large language model ChatGPT still requires improvements to surpass or match the performance of NLP algorithms tailored for solving custom problems.

Authors

Dr. Ahmed Ashraf Butt http://orcid.org/0000-0003-2047-8493 Carnegie Mellon University [biography]

Dr. Ahmed Ashraf Butt is an Assistant Professor at the University of Oklahoma. He recently completed his Ph.D. in the School of Engineering Education at Purdue University and pursued post-doctoral training at the School of Computer Science, Carnegie Mellon University (CMU). He has cultivated a multidisciplinary research portfolio bridging learning sciences, Human-Computer Interaction (HCI), and engineering education. His primary research focuses on designing and developing educational technologies that facilitate various aspects of student learning, such as engagement. Additionally, he is interested in designing instructional interventions and exploring their relationship with first-year engineering (FYE) students’ learning aspects, including motivation and learning strategies. Prior to his time at Purdue, Dr. Butt worked as a lecturer at the University of Lahore, Pakistan, and has been associated with the software industry in various capacities.
Eesha tur razia babar University of California, Irvine [biography]

Eesha Tur Razia Babar holds a master's degree in Electrical and Computer Engineering from the University of California, Irvine. She completed her undergraduate studies in Electrical Engineering at the University of Engineering and Technology in Lahore, Pakistan. Her primary research interests include educational technology, educational data mining, and educational data science.
Dr. Muhsin Menekse Purdue University, West Lafayette [biography]

Muhsin Menekse is an Associate Professor at Purdue University with a joint appointment in the School of Engineering Education and the Department of Curriculum & Instruction. Dr. Menekse's primary research focuses on exploring K-16 students' engagement and learning of engineering and science concepts by creating innovative instructional resources and conducting interdisciplinary quasi-experimental research studies in and out of classroom environments. Dr. Menekse is the recipient of the 2014 William Elgin Wickenden Award by the American Society for Engineering Education. He is also selected as an NSF SIARM fellow for the advanced research methods for STEM education research. Dr. Menekse received four Seed-for-Success Awards (in 2017, 2018, 2019, and 2021) from Purdue University's Excellence in Research Awards programs in recognition of obtaining four external grants of $1 million or more during each year. His research has been generously funded by grants from the Institute of Education Sciences (IES), the U.S. Department of Defense (DoD), Purdue Research Foundation (PRF), and the National Science Foundation (NSF).
Ali Alhaddad Purdue University, West Lafayette

Download paper (1.8 MB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session