2026 ASEE Annual Conference & Exposition

With the rapid growth of generative AI chatbots, use of these tools by students is likewise increasing. Students often encourage chatbots to provide direct solutions, undermining key learning goals, rendering assignments ineffective, and bypassing reliable source materials. This paper seeks to embrace shifting preferences towards AI assistance, offering a chatbot design that encourages better practices for students learning new software libraries.

We developed a chatbot intended to assist students with navigating and understanding technical documentation. The bot uses course project specifications and developer documentation as context for the GPT-4o model. The system is tasked with assisting student learning in a similar manner to course instructors. Additional design features include automatic highlighting of related text in the corresponding documentation, instructor-provided starter prompts to guide appropriate use, and support for sharing conversations to encourage collaboration. These features are intended to emulate strategies teaching assistants often employ to aid students while promoting self-directed exploration. The goal is to encourage direct student interaction with developer documents and offer an alternative course resource that can provide assistance similar to instructional staff at any time.

To evaluate our system, we deployed it as a resource during the final project of an upper-level undergraduate computer science course. One of the goals of the course is that students learn to use developer documentation. We collected server usage logs, surveyed the user population, and compared our chatbot with a generic online GenAI tool to inform our results. Our study included 448 students across 186 project groups. We analyzed over 680 chatbot interactions logged during a three-week deployment period and compared responses using expert evaluation on a sample of 93 student prompts. This comparison considered response helpfulness, appropriateness, and the presence of misleading information or code snippets. This analysis provided insight into how students used the tool in practice and how the quality of its responses compared to those from a general-purpose chatbot.

Overall, students reported increased comfort with documentation, with 48% responding positively and 41% responding neutrally. Our bot returned misleading information at a rate of 14%, compared to ChatGPT's 47%, and gave high quality responses twice as often. These findings suggest that context-aware generative AI systems are often more reliable than their generic counterparts and may promote more productive learning behaviors. Embedding such tools in computing courses may be a promising approach to AI adoption and warrants further study.