2025 ASEE Annual Conference & Exposition

AI-Driven Multimodal System for Enhancing Non-Verbal Communication in Public Speaking

Presented at DSAI Technical Session 5: Educational Technology and Innovative Tools

This work-in-progress paper introduces an AI coach to train engineering students on nonverbal communication. Engineering students often find it difficult to combine verbal and nonverbal cues in their interactions to communicate their ideas effectively. Conventional pedagogical approaches in engineering place emphasis on spoken communication while ignoring the crucial function of nonverbal cues, including head movements, body language, and facial emotions. This study proposes a personalized AI-driven coaching system designed to teach nonverbal communication skills to bridge the communication gap in the engineering community. Leveraging state-of-the-art multimodal AI models, our methodology integrates a comprehensive pipeline to evaluate nonverbal cues—namely, head pose, eye contact, and facial expressions, as well as verbal cues—namely, speech transcript. For head pose analysis, we utilize the MediaPipe library to classify head orientations. Facial expression evaluation relies on models trained on the FER-2013 dataset, with dominant emotions extracted and normalized from video frames. Verbal communication is assessed using OpenAI’s Whisper for transcription and Meta’s Llama3 for context-aware evaluation. Unlike traditional classroom settings, this approach integrates advanced AI to provide personalized, real-time feedback, bridging the gap between technical and interpersonal competencies. Our AI system’s performance was evaluated against human evaluators for assessing students’ communication skills and providing personalized feedback. On the Likert scale, our AI system outperformed human evaluators by 24.25% in clarity, comprehensiveness, personalization, and specificity of the feedback.

Authors

Dr. Brainerd Prince Plaksha University [biography]

Brainerd Prince is the Associate Professor of Practice and the Director of the Center for Thinking, Language and Communication at Plaksha University. He teaches courses such as Reimagining Technology and Society, Ethics of Technological Innovation, and Art of Thinking for undergraduate engineering students and Research Design for PhD scholars. He completed his PhD on Sri Aurobindo’s Integral Philosophy from OCMS, Oxford – Middlesex University, London. He was formerly a Research Tutor at OCMS, Oxford, and formerly a Research Fellow at the Oxford Centre for Hindu Studies, a Recognized Independent Centre of Oxford University. He is also the Founding Director of Samvada International Research Institute which offers consultancy services to institutions of research and higher education around the world on designing research tracks, research teaching and research projects. His first book The Integral Philosophy of Aurobindo: Hermeneutics and the Study of Religion was published by Routledge, Oxon in 2017. For more information, please visit: https://plaksha.edu.in/faculty-details/dr-brainerd-prince
Siddharth Siddharth Plaksha University [biography]

Dr. Siddharth is the Director of Human-Technology Interaction Lab at Plaksha University. He recently came to Plaksha from the Human Factors Center of Excellence at Microsoft USA where he worked as Senior Human Factors Researcher. He is passionate about working on research themes with practical applications in the technology industry.

At Plaksha, Dr. Siddharth is investigating and modeling human physiological responses to improve cognitive well-being, and enhance human safety, productivity, and creativity. By integrating domain-specific knowledge from the fields of signal processing, computational neuroscience, computer vision, and machine learning, his research group is utilizing human physiological responses to track various cognitive states such as emotions, stress, awareness, mental load, and more. His research and recommendations have practical applications in the technology industry for the development of safer and more user-centered designs that promote cognitive well-being.

In the past, Dr. Siddharth led the Microsoft Human Factors Center of Excellence’s efforts to design personalized user experiences through neurophysiological insights into users' preferences enabling them to be more productive and establish meaningful connections with each other. He has also worked at Facebook Reality Labs, Samsung Research America, French National Center for Scientific Research (CNRS), and National University of Singapore. His research has been awarded fellowships and research grants from various institutions including the National Science Foundation, Army Research Lab, and the Kavli Institute of Brain and Mind. It has also been published in many international conferences and reputed journals including IEEE Transactions and Nature Scientific Reports and has spawned multiple patents.

Dr. Siddharth received his Ph.D. from the University of California San Diego where he also served as the Graduate Fellow Editor of the Prospect Journal of International Affairs and Graduate Student Member of the South Asia Initiative. His first book, Founding Generations: Democracy's Origins and Parallels in America and India, was released recently. For more up-to-date information about him, please visit https://ssiddharth.in/about-me
Subham Jalan
Hibah Ihsan Muhammad Plaksha University, Punjab
Chaitanya Modi

Download paper (2.37 MB)

» Download paper

« View session