2024 ASEE Annual Conference & Exposition

Background: Large Language Models (LLMs) have begun to influence software engineering practice since the public release of GitHub's CoPilot and OpenAI's ChatGPT in 2022. As an interactive “assistant” that can answer questions and prototype software, LLMs could potentially revolutionize the way software engineering is practiced – and thus may inform how software engineering is taught. LLMs offer varying experiences among users.
While some schools have banned ChatGPT (Dibble, 2023), researchers have proposed strategies to address potential issues inherent in the use of LLMs (Lo, 2023; Gimpel et al., n.d.; de Fine Licht, 2023). The Association for Computing Machinery (ACM) identifies curriculum guidelines with essential competences for Computer Science undergraduate degree programs. The 2023 guidelines which incorporates the use of LLMs are still in beta version and soliciting feedback (CS2023 – ACM/IEEE-CS/AAAI Computer Science Curricula, n.d.). It is, therefore, important to evaluate students’ perception of LLMs and possible ways of adapting the computing curriculum to these shifting paradigms.

Purpose: The purpose of this study is to explore computing students’ experiences and approaches to using LLMs during a semester-long software engineering project.

Design/Method: In this paper we will present data collected from a senior-level software engineering course at a large public university in the Midwest. This course uses a project-based learning (PBL) design with a semester-long team project. In Fall 2023, the students were required to use LLMs such as ChatGPT and CoPilot as they completed their projects. A sample of these student teams were interviewed in the middle and at the end of the semester to understand (1) how they used LLMs in their projects, and (2) whether and how their perspectives on LLMs changed over the course of the semester. We are analyzing the data qualitatively to identify themes related to students’ usage patterns and learning outcomes.

Results/Discussion: We will report on students’ thinking over the course of the semester and how they developed strategies to use LLMs. We will discuss observed trends and how the use of LLMs shaped the teams’ dynamics and deliverables. The results will help us characterize the impact that the incorporation of LLMs had on the students’ learning. Based on our findings, we will make recommendations for future software programming courses seeking to incorporate LLMs. Our results can also inform professional development programs and policies associated with the integration of AI-driven technology in education.

References
CS2023 – ACM/IEEE-CS/AAAI Computer Science Curricula. (n.d.). Retrieved October 30, 2023, from https://csed.acm.org/
de Fine Licht, K. (2023). Integrating Large Language Models into Higher Education: Guidelines for Effective Implementation. Computer Sciences & Mathematics Forum, 8(1), Article 1. https://doi.org/10.3390/cmsf2023008065
Dibble, M. (Director). (2023, February 6). Schools Ban ChatGPT amid Fears of Artificial Intelligence-Assisted Cheating. https://www.voanews.com/a/schools-ban-chatgpt-amid-fears-of-artificial-intelligence-assisted-cheating/6949800.html
Gimpel, H., Hall, K., Decker, S., Eymann, T., Lämmermann, L., Mädche, A., er, R., Maximilian, R., Caroline, S., Manfred, S., Mareike, U., Nils, V., & rik, S. (n.d.). Unlocking the power of generative AI models and systems such as GPT-4 and ChatGPT for higher education: A guide for students and lecturers.
Lo, C. K. (2023). What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410