2026 ASEE Annual Conference & Exposition

Graduate education in Modeling and Simulation (M&S) faces the challenge of integrating Generative AI (GenAI) to preserve learning objectives while supporting skill development. However, systematic frameworks for pedagogical integration remain underdeveloped. This work-in-progress reports on structured LLM scaffolding in a graduate M&S course using instructor-designed prompts and assignment-embedded constraints. We implemented a within-subjects design where 15 graduate students in an M&S program completed two simulation modeling tasks: (i) one with unrestricted AI use and (ii) one with four structured scaffolding prompts guiding paradigm selection, scoping, validation, and critique. Data included pre- and post-surveys measuring confidence and understanding, correctness of paradigm selection, comparative preferences, qualitative reflections, and AI interaction transcripts. Findings reveal a paradoxical scaffolding effect where students reported higher confidence in work quality with scaffolding (+0.47 on a 10-point scale), but lower confidence in understanding their modeling decisions (-0.40) and a reduced sense that AI helped them learn (-0.20). Different students failed under different conditions with no overlap: three corrected errors with scaffolding, while three made new errors only under scaffolding. Transcript analysis identified four metacognitive patterns: (i) resistance to needed scaffolding despite correction; (ii) disruption of expert strategies, leading to new errors; (iii) false confidence in incorrect scaffolded answers; and (iv) accurate self-assessment of scaffolding benefits and risks. Results suggest that effective AI integration requires explicit teaching of metacognitive awareness, monitoring, regulation, and evaluation, alongside AI literacy skills. Faculty should use adaptive scaffolding that is responsive to demonstrated competence, with explicit metacognitive instruction on recognizing knowledge gaps and critically evaluating AI output.