2024 ASEE Annual Conference & Exposition

Generative AI (GenAI) chatbots have taken the world by storm with their ability to summarize information and produce complex natural language responses. At the same time, the demand for computer science education has grown enormously.

We developed a custom GenAI chatbot and released it to over 700 students to assist with a course programming project. The tool was trained on the project specification, lecture content, lab materials, and past course forum posts relevant to the project. The course is a high-enrollment upper-level computer science elective at a large public research university. The project is a simple distributed system using processes, threads, and sockets. The goal of the tool was to provide an additional course resource that supplies valuable information to students comparable to advice they would receive from course instructors.

We measured the effectiveness of our tool with a student survey, evaluation by course instructors, and comparison with a state-of-the-art general purpose chatbot. The overall survey data indicated high rates of correctness and helpfulness in the Bot responses. We found that hallucination was not common, and most incorrect responses were identifiable by students. The Bot also performed better than general purpose bots for project-specific help.

Our experience can provide insights for faculty using GenAI to assist students in their courses. A customized chatbot can be helpful to students and augment traditional course resources.