2024 ASEE Annual Conference & Exposition

A Comparative Analysis of Natural Language Processing Techniques for Analyzing Student Feedback about TA Support

Presented at DSA Technical Session 8

This paper advances the exploration of Natural Language Processing (NLP) for automated coding and analysis of short-answer, text-based data collected from student feedback. Specifically, it assesses student preferences for Teaching Assistant (TA) support in engineering courses at a large public research university. This work complements existing research with an in-depth comparative analysis of NLP approaches to examining qualitative data within the realm of engineering education, utilizing survey data (training set = 1359, test set = 341) collected from 2017 to 2022. The challenges and intricacies of multiple types of classification errors emerging from five NLP methods are highlighted: Latent Dirichlet Allocation (LDA), Non-Negative Matrix Factorization (NMF), BERTopic, Latent Semantic Analysis (LSA), and Principal Component Analysis (PCA). These results are compared with results from traditional thematic analysis conducted by a domain expert to assess their efficacy. Two principal findings emerged for TA teaching practice and for the use of NLP in education research. Firstly, the conclusions derived from each coding technique are consistent, demonstrating that students want, in order of priority, extensive TA-student interactions, problem-solving support, and experiential/laboratory learning support from TAs. Secondly, the research offers insights into the effectiveness of NMF and PCA in processing educational survey data. A comparison of NMF and PCA topic models, based on accuracy, precision, recall, and F1 scores, reveals that while PCA outperforms NMF in terms of precision (identifying truly relevant topics), NMF excels in recall (capturing a broader range of student responses).

Authors

Neha Kardam University of Washington [biography]

Neha Kardam is a PhD candidate in Electrical and Computer Engineering at the University of Washington, Seattle. She is an interdisciplinary researcher with experience in statistics, predictive analytics, mixed methods research, and machine learning techniques in data-driven research.
Dr. Denise Wilson http://orcid.org/0000-0002-2367-8602 University of Washington [biography]

Denise Wilson is a professor and associate chair of diversity, equity, and inclusion in electrical and computer engineering at the University of Washington, Seattle. Her research interests in engineering education focus on the role of self-efficacy, belonging, and instructional support on engagement and motivation in the classroom while her engineering workplace research focuses on the role of relatedness, autonomy, and competence needs on persistence and fulfillment.
Sep Makhsous http://orcid.org/0000-0002-3618-9736 University of Washington [biography]

Sep Makhsous is an Assistant Teaching Professor in the Department of Electrical and Computer Engineering at the University of Washington and the Director of the ARC Lab (Autonomy, Robotics, and Collaboration). His work focuses on engineering education, with an emphasis on hands-on, interactive learning methods that bridge the gap between theory and practice. Sep’s research explores inclusive robotics education, developing culturally-aware human-robot interfaces and tools to support students with disabilities, ensuring accessibility and equity in STEM fields.

In addition to his educational research, Sep collaborates with NASA, JCATI, and the Air Force on motor controller development for hybrid-electric aviation and integrates these industry-driven challenges into his teaching. He brings experience from the NSF I-Corps program and successful startup ventures, helping students connect classroom learning to real-world applications.

Through the ARC Lab, Sep focuses on advancing robotics education, fostering an inclusive approach to technology development, and preparing students to engage with emerging challenges in robotics and autonomy.

Download paper (1.88 MB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session