2024 ASEE Annual Conference & Exposition

Evaluating ChatGPT's Efficacy in Qualitative Analysis of Engineering Education Research

Presented at Educational Research and Methods Division (ERM) Technical Session 22

This study explores the potential of ChatGPT, a leading-edge language model-based chatbot, in crafting analytic research memos (ARMs) from student interview transcripts for use in qualitative data analysis. With a rising interest in harnessing artificial intelligence (AI) for qualitative research, our study aims to explore ChatGPT's capability to streamline and enhance this process.

The research is part of a mixed-methods project examining the relationships between engineering students' team experiences, team disagreements, and engineering identities. Our team had previously developed an interview protocol for collecting qualitative data and initiated analysis using coding methods and ARMs for individual transcripts. We designed an ARM Development Guidelines document to ensure consistency among four team members in the ARM creation process. The guidelines include a set of key questions that each ARM should aim to address.

Our objective is to assess ChatGPT's proficiency in creating ARMs based on our development guidelines and compare its outputs with human-written ARMs for accuracy and depth of insight. For this purpose, we selected two student interview transcripts. A structured analysis protocol for ChatGPT was devised in adherence to the ARM Development Guidelines.

Two team members, experienced in qualitative analysis and ARM composition, drafted ARMs for the chosen transcripts using the same guidelines, enabling a direct outcome comparison. Subsequently, a rigorous validation process was conducted, using rubrics to assess narratives from both methods. The manual ARM authors performed a self-assessment, while the other researchers conducted a blind evaluation of the human-generated and AI-generated ARMs. We used two rubrics for this comparison. A general rubric gauged accuracy, clarity, analysis time, and usefulness. A specialized rubric was used to determine if the ARMs address the topics laid out in the ARM guidelines, such as self-identification, perceptions of engineering, teamwork descriptions, connections between identity and team experiences, comparisons with other interviews, and reflections.

In this paper, we describe our research methodology, present our findings, evaluate the advantages and limitations of ChatGPT in qualitative analysis within engineering education research, and provide guidance for future research directions. We aim to shed light on the capabilities of ChatGPT in qualitative analysis and contribute to the ongoing dialogue on harnessing AI for research in engineering education. Our findings will inform researchers and practitioners about the benefits, challenges, and best practices associated with integrating AI-powered tools such as ChatGPT into qualitative research methods.

Authors

Dr. Xiaorong Zhang San Francisco State University [biography]

Dr. Xiaorong Zhang is an Associate Professor in Computer Engineering in the School of Engineering at San Francisco State University (SFSU). She is the Director of the Intelligent Computing and Embedded Systems Laboratory (ICE Lab) at SFSU. She has broad research experience in human-machine interfaces, neural-controlled artificial limbs, embedded systems, and intelligent computing technologies. She is a recipient of the NSF CAREER Award to develop the next-generation neural-machine interfaces (NMI) for electromyography (EMG)-controlled neurorehabilitation. She is a senior member of the Institute of Electrical and Electronics Engineers (IEEE) and a member of the Society of Women Engineers (SWE). She has served in professional societies in various capacities including the Chair of the IEEE Engineering in Medicine and Biology Society (EMBS) San Francisco Chapter (2018-present), an Associate Editor of the IEEE Inside Signal Processing E-Newsletter (2016-2018), an Outreach Co-Chair of the Society of Women Engineers (SWE) Golden Gate Section (2017-2018), a Co-Chair of the Doctoral Consortium at 2014 IEEE Symposium Series on Computational Intelligence, a Program Committee Member of various international conferences, and a regular reviewer of a variety of journals and conferences in related fields.
Dr. Stephanie Claussen San Francisco State University [biography]

Stephanie Claussen is an Assistant Professor in the School of Engineering at San Francisco State University. She previously spent eight years as a Teaching Professor in the Engineering, Design, and Society Division and the Electrical Engineering Departments at the Colorado School of Mines. She holds a B.S. in electrical engineering from MIT and a M.S. and Ph.D in electrical engineering with a Ph.D. minor in education from Stanford University.
Fatemeh Khalkhal San Francisco State University [biography]

Dr. Khalkhal is an assistant professor in mechanical engineering at San Francisco State University. She has a PhD in Chemical Engineering from Ecole Polytechnique de Montreal. Her research experience and interest are in developing structure-property relationships in complex fluids and broadening the participation of women and underrepresented minorities in engineering.
Yiyi Wang San Francisco State University [biography]

Yiyi Wang is an assistant professor of civil engineering at San Francisco State University. In addition to engineering education, her research also focuses on the nexus between mapping, information technology, and transportation and has published in Accident Analysis & Prevention, Journal of Transportation Geography, and Annuals of Regional Science. She served on the Transportation Research Board (TRB) ABJ80 Statistical Analysis committee and the National Cooperative Highway Research Program (NCHRP) panel. She advises the student chapter of the Society of Women Engineers (SWE) at SFSU.

Download paper (1.88 MB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session