2025 ASEE Annual Conference & Exposition

In this Empirical Research Full Paper, we explore the effects of chatbot usage on student performance in self-regulated learning tasks conducted in a classroom setting.
The increasing use of generative artificial intelligence (AI) and large language models (LLMs) in STEM education have resulted in thought-provoking conversations regarding its potential benefits and dangers.
While sophisticated LLM-based chatbots developed for pedagogical purposes (i.e., context-aware information retrieval, conversational feedback, problem-solving, etc.) may offer unprecedented accessibility and efficiency in multidisciplinary subjects, they also threaten academic integrity and rigor through abuse or hallucination.
In this exploratory study, we attempt to determine the effects of chatbot usage on student learning in the context of an upper-division embedded systems lab.
We designed five self-regulated learning tasks---completed by students (N=49 of 60) at the beginning of each lab module---each including a short assessment.
We then employed a pseudo-random counterbalanced longitudinal design on four tasks, where students used LLM-based chatbots to prepare for half of their assessments.
In the fifth task we re-randomized participation groups for a standalone experiment with different motivational conditions.
These experiments attempted to measure the effects of chatbot use on short-term performance of students' comprehension and problem-solving.
We report experimental results for the longitudinal design, as well as the standalone design and discuss our observations.
In addition, we present students' self-reported utilization strategies and sentiments regarding their use of chatbots in preparation for the assessments alongside our own analyses of their chatlogs to compare and contrast students' perceptions and their actual interaction patterns.
We note from the longitudinal study that, contrary to students' generally positive attitude toward it, the use of LLM-based chatbots did not appear to have any predictive power on performance outcomes.
Finally, we call for continued empirical research on the efficacy of LLM-based technologies in STEM education and propose future research directions in exploring their impact on teaching and learning.

Keywords: Learning technology, Performance, Student perception, Experimental research, Self-regulated learning