2026 ASEE Annual Conference & Exposition

WIP: Assessing the Reliability of the AI Adoption and Motivation Survey Instrument in Engineering Education

Presented at NSF Grantees Poster Session I

The growing integration of artificial intelligence (AI) using large language models (LLMs) into engineering education has proven to be both an asset to student learning and a potential limitation to student growth. With the recent wave of AI use in engineering education, the ways the technology is utilized and the characteristics of students who rely on it have been the subject of much research. How students utilize AI in their education can have profound implications on their learning outcomes; however, no consensus has emerged on how to measure the relationship between AI use and learning.

This paper discusses the development of an AI adoption and motivation survey , a scale designed to measure the relationship between AI use and learning, and reports on its psychometric properties, specifically when used within an engineering student population. The survey was created as an adaptation of three existing surveys —the Motivated Strategies for Learning Questionnaire (MSLQ), the Reliance on Technology Questionnaire, and the Student AI Survey—paired with the intention to examine relationships among specific constructs of AI use and learning measured by each survey. Previous work has established the reliability of each parent instrument, but population-specific reliability of each instrument has not been reported . The purpose of this paper is to establish the reliability of the instrument such that it can be used for future research among engineering student populations. This work is framed within the context of Novick’s Classical Test Theory (CTT), which states that the quality of an instrument can be measured through psychometric properties, including internal consistency. This framework is particularly suited to this work, as it establishes fundamental indicators of the instrument’s quality before advancing to more complex measurement models, such as factor analysis and item response theory.

Data for this study were collected through survey responses from N=131 students enrolled in the College of Engineering at a large R1 western university. Internal consistency of the instrument was assessed using Cronbach’s Alpha and McDonald’s Omega. Results of the analysis indicate that the survey has good reliability when used with an engineering student population. The results of this study have implications for engineering education researchers and educators, as it introduces a psychometrically grounded instrument to assess the relationship between AI use and learning. This work serves as a steppingstone towards further work to establish the validity of the instrument through use of factor analysis and item response theory. This work represents a foundational step in validating an AI use instrument for engineering students. Future research will extend this work by examining the instrument’s construct validity through factor analysis and item response theory, and by assessing its reliability across broader student populations.

Authors

Daniel Kane http://orcid.org/0000-0002-0220-9962 Utah State University [biography]

Daniel Kane is a third-year Ph.D. student in the department of engineering education at Utah State University. His research interests include spatial ability, accessibility for students with disabilities, artificial intelligence in education, and enhancing electric vehicle charging system infrastructure. Daniel has contributed significantly to the development of the Tactile Mental Cutting Test (TMCT) which is a significant advancement in assessing spatial ability for blind and low-vision populations. His research has helped inform teaching methods and develop strategies for improving STEM education accessibility. Currently, he is studying how AI tools are utilized by students across USU’s colleges to optimize their educational value. Daniel has also served as president of the ASEE student chapter at USU where he initiated outreach activities at local K-12 schools and promoted student engagement in research.
Michaela Harper http://orcid.org/0009-0007-5985-8676 Utah State University [biography]

Michaela Harper is a doctoral student at Utah State University, pursuing a Ph.D. in Engineering Education. She holds a Bachelor's degree in Environmental Studies, focusing on STEM and non-traditional education approaches, and a Master's degree in Engineering Education, where she explored faculty perspectives on Generative Artificial Intelligence (GAI). Michaela's current research delves deeply into the effects of disruptive technologies on engineering education, driven by her passion for uncovering the foundational nature of phenomena and applying an exploratory and explanatory approach to her studies. Her work aims to illuminate how technological advancements reshape educational landscapes through student and faculty engagement.
Dr. Wade H Goodridge http://orcid.org/0000-0002-5811-7629 Utah State University [biography]

Wade Goodridge is a tenured Associate Professor in the Department of Engineering Education at Utah State University. His research often has a focus on what drives us to learn, how we learn and how well we do it, and what can improve that learning. This work is often directed with a spatial thinking emphasis. Dr. Goodridge has investigated different learning modalities (fully online, hybrid, blended courses), cognitive and metacognitive learning processes (self-regulation, engineering problems solving, neural efficiency in engineering problem solving) learning motivators (engineering self-efficacy), and effective and personalized engineering education interventions (tactile engineering curriculum, physical teaching models, XR environments and AI informed systems of delivering and monitoring engineering learning). Dr. Goodridge has created a valid and reliable instrument to assess spatial ability in blind and low vision (BLV) learners thus opening new areas of research centered on tactile spatial ability. He has created tactile and visual learning tools and has offered professional development experiences on how to improve the education of tomorrows engineers.
Dr. Cassandra McCall http://orcid.org/https://0000-0002-0240-432X Utah State University [biography]

Dr. Cassandra McCall is an Assistant Professor in the Engineering Education Department at Utah State University (USU). Her research focuses on the intersections of disability, identity formation, and culture and uses anti-ableist approaches to enhance universal access for students with disabilities in STEM, particularly in engineering. At USU, she serves as the Co-Director of the Institute for Interdisciplinary Transition Services. In 2024, Dr. McCall received a National Science Foundation CAREER grant to identify systemic opportunities for increasing the participation of people with disabilities in engineering. Her award-winning publications have been recognized by leading engineering education research journals at both national and international levels. Dr. McCall has led several workshops promoting the inclusion of people with disabilities and other minoritized groups in STEM. She holds B.S. and M.S. degrees in civil engineering with a structural engineering emphasis.
Linda Davis Ahlstrom Utah State University [biography]

Linda Ahlstrom PhD student currently studying Engineering Education at Utah State University. Interested in the Univerity to Industry interface and the use of AI tools in engineering. MS Electrical Engineering Cal State Long Beach. Worked in industry: Biomedical, Software Development and Aerospace.
Dr. Oenardi Lawanto Utah State University [biography]

Dr. Oenardi Lawanto is a professor in the Department of Engineering Education at Utah State University, USA. He received his B.S.E.E. from Iowa State University, his M.S.E.E. from the University of Dayton, and his Ph.D. from the University of Illinois at Urbana-Champaign. Dr. Lawanto has a combination of expertise in engineering and education and has more than 30 and 14 years of experience teaching engineering and cognitive-related topics courses for his doctoral students, respectively. He also has extensive experience in working collaboratively with several universities in Asia, the World Bank Institute, and USAID to design and conduct workshops promoting active-learning and life-long learning that is sustainable and scalable. Dr. Lawanto’s research interests include cognition, learning, and instruction, and online learning.

Note

The full paper will be available to logged in and registered conference attendees once the conference starts on June 21, 2026, and to all visitors after the conference ends on June 24, 2026

« View session