2026 ASEE Annual Conference & Exposition

Learning application (app) testing typically must involve examining how learners engage with the app through pedagogical or content and the interface. Both components are key to ensure a smooth user experience to ensure the app functions as intended to facilitate learning, instead of possibly creating misconceptions among students’ existing knowledge. Think-aloud methods have demonstrated effectiveness in capturing user experiences with app testing. Think-aloud methods are useful to have users verbalizing their thought process while engaging in activities, exploring their thinkings behind their actions during the activities. Although there has been abundant research in engineering education that engages students thinking while learning, such as conceptual learning, there is scarcity in the literature that about using think-aloud protocols, along with video data, to test engineering educational apps.

As part of a research project to develop a tablet-based application to supplement and enhance student learning in spatial visualization skills, our team has conducted the app testing with the think-aloud methods to explore the students’ thought process while using the app. In this think-aloud method, we also collected video data to observe students’ hand movements while using the app to enhance the verbalizations. This paper documents the process of (1) conducting the think-aloud protocol and (2) analyzing the verbal and video data for potential future work. We strive to answer the research questions: (1) How are the procedures for a think-aloud app testing procedures implemented and (2) how are the video data analyzed among four researchers, with three researchers being new to qualitative research? Our paper provides details of implementations of the think-aloud protocol and engage the readers with our process of analyzing the data through the iterations of coding and discussions to ensure consistent interpretations of codes (interrater reliability).

We have found that implementations of the think-aloud protocols to be useful in capturing the students’ thought process and hand movements while using the app. In addition, the analysis of the data has yielded findings that have substantially improved the app. These findings include providing more flexibility for students to explore the different blocks while assembling blocks to complete a part, removing a default feature where the blocks “jumping” back to the default orientation if the student incorrectly place the block, and having the color-coded surface (a distinct feature of this app) to be the default interface when the app is first opened.