2025 ASEE Annual Conference & Exposition

This work-in-progress research paper describes an instrument development effort to understand how engineering students use external resources to supplement their learning and the associated metacognitive strategies they employ with those resources. With the increasing prevalence of generative AI tools like ChatGPT, traditional problem-solving platforms (e.g., Chegg) and educational video resources (e.g., YouTube, Khan Academy), engineering students now have unprecedented access to external learning resources. Considering the pace of technological developments lowering barriers to entry for students to get on-demand assistance with their studies, there is a growing need to understand how students engage with these different resources.

Our instrument is being developed to explore two key research questions: (1) How do engineering students use external resources, including generative AI, to assist in problem-solving within their coursework? and (2) What factors drive their preference for certain tools over others? Later in our project, we will explore how students use metacognitive strategies with various external resources – which is a separate research question not directly relevant to the instrument itself. Therefore, we plan to include scales related to metacognitive strategies to enable purposive sampling for student interviews based on the results of the instrument distribution.

Our theoretical framework to ground the instrument development process is the technology acceptance model. The model's premise is that individuals choose to adopt new technologies based on the perceived usefulness and ease of use of that technology, which has been used to explore the adoption of e-learning tools, learning management systems, and video conferencing platforms among faculty and students.

To source items for our instrument, we are reviewing the literature on student adoption of external resources (i.e., “homework help” websites like Chegg, video platforms like YouTube, and generative AI systems like ChatGPT). We are using databases including Education Research Complete, ERIC, and Arxiv to search for papers; this search is currently in progress. After extracting questions, we will align them with the technology acceptance model to construct a draft instrument. Later in Fall 2024, we will pilot the instrument with a sample of 10-30 undergraduate engineering students – members of the intended population – through cognitive interviewing to determine the relevance and comprehensibility of the questions. By the time of the draft, we will have the findings to share from the cognitive interviews.

After our cognitive interviewing phase, the instrument will be administered to at least 200 undergraduate engineering students at a large Midwestern university in Spring 2025. We plan to analyze the instrument using exploratory factor analysis if items are primarily customized and confirmatory factor analysis if more minor changes are made to specific scales. By the time of the conference, we expect to present initial survey findings from the Spring data collection.

We anticipate this research will enable us to better understand how students leverage external resources in engineering education to provide better support for maximizing their utility. We invite suggestions for improving the instrument by the draft submission and seek feedback on refining the instrument for broader use at other institutions in later semesters.