2026 ASEE Annual Conference & Exposition

The rapid proliferation of generative AI tools in education presents both opportunities and challenges for engineering educators, particularly in machine learning (ML) courses where students develop complex result interpretation skills. This study examines how AI assistance affects undergraduate students' ability to interpret machine learning results through a semester-long empirical investigation in an authentic classroom context. Using a repeated-measures pre-post design, we collected data from 19 students across five sequential ML assignments, measuring confidence, multidimensional understanding (technical accuracy, metrics interpretation, real-world applications, model limitations, causal reasoning), and self-reported learning versus dependency through paired questionnaires administered before and after AI usage. Results demonstrate that AI assistance produced moderate but meaningful improvements in student confidence (9.4% average increase, Cohen's d=0.47), though individual experiences varied substantially: 41% improved, 48% showed no change, and 11% declined. AI effectiveness varied dramatically by concept complexity, with Support Vector Machines and Neural Networks showing statistically significant improvements (p=0.012, Cohen's d=0.816) while basic concepts showed minimal effects. Analysis across understanding dimensions revealed AI particularly supports higher-order interpretive skills, especially causal reasoning (+15.6%) and metrics interpretation (+15.3%), while showing limited impact on domain-specific contextualization. Students' baseline confidence increased naturally across the semester, providing evidence against harmful dependency. These findings suggest evidence-based practices for AI integration: structured phasing requiring independent work before AI assistance, explicit instruction in critical AI evaluation, strategic deployment for mid-to-high complexity concepts, and monitoring individual responses. As AI becomes ubiquitous in engineering education, our research demonstrates the importance of nuanced, empirically-grounded integration strategies.