2025 ASEE Annual Conference & Exposition

In the past few years, rapid research of Natural Language Processing (NLP) has enabled Artificial Intelligence (AI) tools to answer a wide array of questions with reasonable accuracy by an AI tool. Furthermore, the rapid inclusion of this technology into Computer Science practice requires an investigation on how it can affect our pedagogical strategies. That is, with the rise of Large Language Model (LLM) based Generative AI tools (Gen-AI), they have shown tremendous efficiency in solving Computer Science related questions found in common course curricula. Thus, this paper presents a novel investigation into the performance of these Gen-AI tools within the context of two courses in the Computer Science and Software Engineering (CSE) undergraduate curricula at Miami University.

We introduce a systematic benchmarking methodology to measure LLM performance against course material found in CSE 174 (Fundamentals of Problem Solving and Programming) and CSE 274 (Data Abstraction and Data Structures). This includes exams, lab assignments, and projects. Our benchmarking results reveal both the strengths and limitations of these Gen-AI tools in Computer Science educational tasks, providing crucial insights for curriculum adaptation. Utilizing these results, we provide insight on how they might assist in the evolution of Computer Science education. Specifically, we highlight the areas where Gen-AI tools could enhance learning and where human skills remain explicitly required. This work continues the discussion on how AI capabilities can be integrated into the academic setting. Such benchmarking efforts are essential to ensure our students receive the knowledge they require to move from the Computer Science academic setting into the professional setting which largely utilizes such Gen-AI tools. Thus, preparing the next generation of Computer Scientists to be effective AI-augmented professionals.