2025 ASEE Annual Conference & Exposition

The rise of ChatGPT, and other generative AI tools, has led to a number of debates in higher education. Multiple news articles have noted the many ways students are already using it in classes and how instructors have had to adapt. Given that ChatGPT has been able to improve quickly and dramatically at solving a broad range of exam and homework problems, and that spending on these technologies continues to grow across industries, how AI is being used across fields makes it difficult to ignore in engineering education.

These changes have forced instructors to consider how to use (or ban) AI in their classrooms. For instance, some see these tools as a means access–helping raise all students, especially those from disadvantaged backgrounds, to a minimum level of knowledge–which may allow students to develop more complex skills, such as critical thinking, application, and synthesis. Critical thinking in engineering is a complex set of skills that engineers need to tackle ever-evolving challenges (e.g., articulating and challenging assumptions in problem solving, designing experiments, and collaborating with other engineers), skills that map to multiple ABET Student Outcomes. As more AI-powered technologies are used in engineering practice, teaching critical thinking skills across the engineering curriculum will become more important than ever–as AIs easily generate correct-sounding false arguments.

In our Material and Energy Balances (MEB) course–the first technical chemical engineering course our sophomores and transferring juniors take–one of the learning objectives is for students to be able to critique solutions and determine the qualities of stronger proposals. We have previously targeted this objective through case-based activities, group discussions, and peer review; and saw an opportunity for AI to further support these activities.

In our presentation, we will share our use of a customized retrieval-augmented generator (RAG) chatbot built on ChatGPT that we designed to serve as a stand-in for a novice engineer with which students can practice critical thinking skills, with a primary focus of critiquing proposed solutions. This is opposite to how many other instructors have employed chatbots in their teaching (e.g., as automated tutors or experts). As far as the authors are aware, this application of an intentionally errant AI-chatbot has not yet been shared in the engineering education literature. We will share our approach to training and developing our MEB Novice chatbot, and its use in our MEB class. We will discuss how we selected misconceptions to be targeted for errors, how we tuned the bot to be errant enough for our teaching goals (e.g., with the goal of dropping the accuracy rate of the MEB Novice bot from ~90% to ~70%), and how we evaluated the accuracy of the MEB Novice bot. We also will share how we compared different iterations of our trained MEB Novice bot against each other and commercially available AI chatbots to which students may have access. By sharing our experience, we hope to encourage our colleagues to try experimenting with other AI-powered techniques that are likely to become more common in engineering education and higher education at large.