2024 ASEE Annual Conference & Exposition

This work-in-progress paper explores how students and faculty are employing large language models (LLMs) like ChatGPT in engineering and computing education contexts through a systematic literature review (SLR). As seen in the myriad opinion pieces and articles in the popular press and much to the concern of instructors, students are leveraging AI models such as ChatGPT to complete their assignments – bringing discussions of academic dishonesty to the forefront. However, the use of LLMs like ChatGPT is not entirely fraught with threats to education, work has also emerged about faculty experimenting with incorporating these models into their teaching and evaluation methods. Moreover, categorizing all student use of LLMs as a violation of academic integrity is unproductive; similar work has emerged to explore student perceptions and use cases. Despite the proliferation of manuscripts offering methods for incorporating LLMs in their teaching, much of the advice either does not elaborate on practical use cases across disciplines or does not offer any data to support the efficacy of the use case. Thus, we will delve into the implementation of different approaches to using LLMs like ChatGPT in engineering and computing education, examining how these tools are being leveraged for pedagogical and assessment purposes.

The research question guiding this work is: “how are students and faculty using LLMs (i.e., ChatGPT) in engineering and computing education contexts for instruction and assessment?” To provide a comprehensive understanding of the current landscape, an SLR was conducted, specifically culling papers from Arxiv – assuming that much of the work related to using LLMs in education was still under peer review. We first selected a set of “sentinel articles,” which are articles selected beforehand that we were interested in extracting for analysis to help develop a set of keywords and form the search string. The search string was a combination of general terms such as “large language model” and specific models “GPT-3.5.” These were combined with keywords like “education” to capture a breadth of papers. The initial search returned 717 papers. Our main inclusion criterion was that the paper must be situated in an engineering or computing education context. After evaluating the papers at the abstract level, 51 papers were identified as meeting the inclusion criterion, whereas 49 needed to be checked further for alignment – typically consisting of papers that were situated in education broadly. The remaining 617 papers were determined to be out of scope for the purposes of this study. We are currently reviewing the 110 papers further to refine the final sample of papers by applying checks for quality to pull studies based on actual implementations of the technology.

Our initial results suggest that the papers are converging around a set of common use cases for ChatGPT and similar models, such as leveraging ChatGPT for authoring learning outcomes. Papers also detail the development and utilization of distinct fine-tuned engines designed for personalized interaction with students, serving as customized tutors by generating practice problems in areas like computing, physics, and mathematics. Diverse applications in assessment are also present, such as generating multiple-choice questions and feedback using LLMs. We plan to include detailed, concrete examples of ChatGPT's practical applications.

We anticipate this research to be primarily useful for faculty in engineering and computing education and educational researchers exploring the potential uses and impacts of LLMs. By extracting actionable first steps for instructors from our sample of manuscripts, we aim to contribute more comprehensive strategies to the literature by synthesizing the necessary components to translate this emerging technology into the classroom.