2026 ASEE Annual Conference & Exposition

A Multimodal Framework for Embodied Cognition in Oral Explanations

Presented at Computers in Education (CoED): Learning, Engagement & Inclusion (7 of 9) -- T508B

This study advances a multimodal analytic framework for investigating embodied cognition during oral explanations in statistics education. Grounded in theories of embodied, situated, and distributed cognition, gesture is conceived not just as an accessory to speech but also as reasoning that is enacted through coordinated movement and language. Using computer vision, gesture trajectories, rhythm, and spatial anchoring are captured, while speech-to-text large language models (LLMs) transcribe and semantically analyze verbal explanations. A central Inference Agent integrates these modalities to reveal how gesture and discourse converge or diverge as indicators of conceptual understanding. Rather than claiming autonomous interpretation, the system functions as an epistemic instrument that visualizes the coupling between gesture and conceptual understanding. By aligning theories of embodied cognition with computational observation, this work reframes oral assessment as a dialogic event in which knowing unfolds through motion, voice, and interpretation—positioning understanding itself as an embodied relation between cognition, expression, and inference.

Authors

Dr. Jason W Morphew http://orcid.org/0000-0001-5971-214X Purdue University – West Lafayette [biography]

Dr. Jason Morphew is an assistant professor at Purdue University in Engineering Education and serves as the director of undergraduate curriculum and advanced learning technologies for SCALE, and is affiliated with the INSPIRE research institute for Pre-College Engineering and the Center for Advancing the Teaching and Learning of STEM. Dr. Morphew's research focuses on the application of principles of learning derived from cognitive science and the learning sciences to the design and evaluation of learning environments and technologies that enhance learning, interest, and engagement in STEM. Morphew earned his Ph.D. from the University of Illinois Urbana-Champaign in Educational Psychology from the Cognitive Science of Teaching and Learning Division where he examined self-regulated learning of engineering students enrolled in introductory physics courses.
Mr. Amirreza Mehrabi Purdue Engineering Education [biography]

Amirreza Mehrabi is an accomplished scholar at Purdue University specializing in machine learning and AI-aided modeling and Engineering Education. As an interdisciplinary researcher, he designs and deploys intelligent platforms that tackle complex decision-making challenges, integrating advanced techniques such as transformer models, reinforcement learning, and statistical inference. His work spans across adaptive testing, cognitive fatigue detection by ML, agentic AI agents tutors, and multimodal analytics, leading to innovative solutions with measurable real-world impact. Amirreza’s expertise is reflected in his custom architectures, model optimization research, and psychometrics—combining educational measurement science with deep learning. Amirreza’s collaborative spirit, technical mastery, and commitment to transparent, bias-aware modeling foster a vibrant culture of innovation.
Junior Anthony Bennett http://orcid.org/https://0009-0004-6441-3805 Purdue University – West Lafayette (College of Engineering) [biography]

Junior Anthony Bennett is a Graduate Research Assistant and Lynn Fellow at Purdue University, West Lafayette, Indiana, USA. He is pursuing an Interdisciplinary Ph.D. program in Engineering Education majoring in Ecological Sciences and Engineering (ESE). His research focus is the 'Impact of Extended Reality (XR) Technologies on Learning'. He worked for over a decade in higher education and held multiple positions of responsibilities including Lecturer of Mathematics, Engineering Physics and Industrial Engineering Core Courses; He had served as Program Leader, Academic Advisor, Applied Sciences and Engineering Cooperative Education Coordinator, Program Coordinator & Chair of Infrastructure Committee in higher education. He earned a Master of Science in Manufacturing Engineering Systems from Western Illinois and was recognized for being an outstanding graduate student. He earned a Bachelor of Education in TVET Industrial Technology – Electrical from the University of Technology, Jamaica. He is a Certified Manufacturing Engineer (CMfgE) with the Society for Manufacturing Engineers (SME). He is the immediate past president of the ESE Interdisciplinary Graduate Program Graduate Student Organization and recent co-chair of the 18th Annual 2024 ESE Symposium at Purdue University. He is the 2025 recipient of the School of Engineering Education (ENE) Outstanding Service/Leadership Award at Purdue University, West Lafayette.
Aashvi Majmundar Purdue University – West Lafayette (College of Engineering)

Download paper (5.4 MB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session