2024 ASEE Annual Conference & Exposition

In this study, we evaluate the use of generative AI (GAI) models for qualitative coding of open-ended student responses, compared to traditional natural language processing (NLP) methods. The main objective was to explore an in-house GAI method to develop themes from students’ feedback responses. A systematic four-step process of text extraction, embedding, clustering, and code generation was employed on responses from a large engineering course regarding the transition to online learning during COVID-19. A locally deployed GAI model (Dolphin-Mistral 2.6) was used for privacy-preserving text extraction, with the UAE-Angle embedding model enabling the clustering of similar responses. GAI was then leveraged to generate qualitative codes and themes from the clusters. Human evaluation (i.e., human in the loop process) found the GAI-generated codes displayed high similarity to human-generated codes, with minor terminology distinctions. Key themes emphasized the importance of instructor feedback, communication strategies, engagement approaches, and resource accessibility for effective online learning experiences. Treemap visualizations aided the interpretation of the hierarchical code structure. While human input was still required for consolidating overlapping sub-codes, the study demonstrates GAI's potential to semi-automate qualitative coding tasks traditionally performed manually, while ensuring data privacy through local deployments. Future work could explore more advanced GAI models to further streamline the clustering and code generation workflow.