2025 ASEE Annual Conference & Exposition

This Evidence-based Practice Work in Progress Paper, presents a systematic approach to design a high-quality context-informed research measurement tool – a human-centered design (HCD) depth of thinking rubric that gauges undergraduate engineering students' use of qualitative and quantitative data in a HCD task. The curriculum for undergraduate engineering students is heavily focused on developing quantitative skills. However, engineering professionals may want or need to expand their skill set to also include qualitative methods. To that end, this research project will introduce and study qualitative methods training included in an existing industrial engineering course. Students in this mixed methods group, along with a comparison group of students who received standard quantitative-only methods training, are then asked to work through an HCD problem that includes both quantitative and qualitative data. Because of the relative sparseness of qualitative methods training for HCD problem-solving in engineering, studying the impact of this additional training requires the development of a valid, context-informed, highly discriminant measurement tool sensitive enough to capture any potential differences in student thinking that may emerge.

For the given design problem, students will be provided with 10 qualitative interview summaries in addition to standard quantitative anthropometric data tables to support their work on a design problem focused on workstation design. We used generative AI (i.e., ChatGPT) to produce 10 fictitious interview transcripts as a starting point, adjusting the prompts as needed to construct realistic looking interviews. After editing the transcripts to introduce more variability and distinction across the 10 interview transcripts, intentional "design seeds" were planted within the interview texts for students to potentially discover during their qualitative analysis. Our goal was to have recurrent design seeds (e.g. comments about absence of adequate lumbar support by the desk chair), appearing across multiple interview transcripts in a variety of conversational ways, that students could discover during their analysis of the interviews and include in their workstation designs.

We then developed a HCD depth of thinking rubric to include a tiered framing of depth of student thinking, that mapped to the design seeds we planted in the transcripts. Design seeds mapped to the following tiers: (1) explicit, (2) implicit, and (3) external. Explicit design seeds are ones that were directly tied to the workstation problem statement and implicit design seeds were indirectly mentioned. A third category of external design seeds was not mentioned directly or indirectly in the problem statement. This tiered approach allows us to assess the depth of student thinking in their approach to the design problem, and how their analysis of the qualitative interview transcripts supported their design thinking. The HCD depth of thinking rubric also includes a class of "primary" quality indicators based on quantitative data that students work with (anthropometric data to design chair height, desk dimensions, reach envelope, etc.). "Secondary" quality indicators are those based on the qualitative data. This structure built into the measurement tool permits subscore analyses as well as the tiered analyses and will be tested, and refined, on the comparison group of students first.