2025 ASEE Annual Conference & Exposition

An Assessment of ChatGPT 4o's Performance on Mechanical Engineering Concept Inventories

Presented at ME Division Technical Session 2 - Harnessing AI and Machine Learning to Transform ME Education

Large Language Models (LLMs) like OpenAI’s ChatGPT-4o show promise for enhancing engineering education through real-time support and personalized feedback. However, their reliability in interpreting the conceptual diagrams central to mechanical engineering remains uncertain. This study evaluates ChatGPT-4o’s performance on four concept inventories—Force Concept Inventory, Materials Concept Inventory, Mechanics Baseline Test, and Mechanics of Materials Concept Inventory—using assessments by two Mechanical Engineering professors based on correctness, depth of explanation, and application of theoretical knowledge. While ChatGPT-4o demonstrates the ability to provide robust explanations, it often lacks the contextual depth required for higher-order concept mastery, especially when reasoning from diagrams. These findings align with existing literature highlighting AI’s limitations in discipline-specific support. Future research should refine AI responses to better align with engineering problem-solving approaches and explore hybrid models integrating AI assistance with human instruction, potentially leading to more effective AI-augmented learning platforms in mechanical engineering education.

Authors

Dr. Rujun Gao http://orcid.org/0000-0002-3095-8215 Texas A&M University [biography]

Dr. Rujun Gao has completed her Ph.D. in Mechanical Engineering at Texas A&M University and holds an M.S. in Mechanical Engineering from Zhejiang University, China. Her research focuses on Generative AI, Natural Language Processing (NLP), Large Language Models (LLMs), LLM Agents, and the development of educational technology products.
Hillary E. Merzdorf Cornell University [biography]

STEM Instructional Design Associate, eCornell, Cornell University
Xiaosu Guo http://orcid.org/0009-0007-0571-3746 University of Texas at Dallas
Sami Melhem Texas A&M University [biography]

Sami Melhem is an undergraduate student pursuing a Bachelor of Science in Computer Science at Texas A&M University, where he is also planned to enroll in a concurrent Master of Science program in Computer Science. Sami serves as an undergraduate research assistant in the department of Mechanical Engineering under the guidance of Dr. Srinivasa. His research interests include the simulation of manufacturing processes including robotic sheet forming and magnetic polishing, and the development of AI-driven educational tools.

Beyond academics, Sami is deeply involved in the Aggie Data Science Club, where he serves as Projects Officer, overseeing and mentoring multiple student-led activities each semester. He also plays clarinet as the concertmaster for the TAMU Wind Symphony and principal clarinet for the TAMU Chamber Orchestra, demonstrating his passion for both technology and the arts. Sami aspires to leverage his technical expertise to develop scalable AI solutions that address real-world challenges in education and sustainability.
Dr. Kristi J. Shryock Texas A&M University [biography]

Dr. Kristi J. Shryock is the Frank and Jean Raymond Foundation Inc. Endowed Associate Professor in Multidisciplinary Engineering and Affiliated Faculty in Aerospace Engineering in the College of Engineering at Texas A&M University. She also serves as Director of the Craig and Galen Brown Engineering Honors Program. She received her BS, MS, and PhD from the College of Engineering at Texas A&M. Kristi works to improve the undergraduate engineering experience through evaluating preparation in areas, such as mathematics and physics, evaluating engineering identity and its impact on retention, incorporating non-traditional teaching methods into the classroom, and engaging her students with interactive methods.
Prof. Arun R Srinivasa http://orcid.org/0000-0002-9825-7482 Texas A&M University [biography]

Dr Arun Srinivasa is the J. N. Reddy Chair in Applied Mechanics and Associate Dean for Student Success. He has been with TAMU since 1997. Prior to that he was a faculty at University of Pittsburgh. He received his undergraduate

Note

The full paper will be available to logged in and registered conference attendees once the conference starts on June 22, 2025, and to all visitors after the conference ends on June 25, 2025

« View session