2025 ASEE Annual Conference & Exposition

This paper explores the conceptual challenges and opportunities in measuring personal epistemology and epistemic cognition (PE&EC) in higher education with a special focus on the unique challenges of engineering education. Our approach is structured in two segments: a retrospective evaluation of current PE&EC measurement tools and a prospective strategy for enhancing their application and evidence of validity.

We begin by examining existing instruments that assess PE&EC, highlighting the complexities of validity and the challenges in using these metrics for broad, population-level insights. We especially focus on the relationship of theories of PE&EC, the instruments designed to measure PE&EC, and the special challenges that result from measuring such foundational beliefs. We particularly explore the types of inferences related to PE&EC that are useful to researchers as well as the instrument design decisions that can make supporting those inferences easier or more difficult.

Our discussion focuses on three critical observations in PE&EC research: (1) the inherent difficulties in measurement at the intersection of attitudes, beliefs, and reasoning; (2) ongoing debates about the nature of validity in PE&EC research, particularly the relationship between psychometrics, measurement, and validity; and (3) the role of the theoretical underpinnings of epistemological beliefs in the practical interpretability of instruments. In particular, we focus on the over-reliance on factor analysis and assumptions of latent structures in standard PE&EC assessments and advocate for a greater focus on aligning measurement with utility and individuals rather than theories.

Moving forward, we propose a novel framework for PE&EC measurement that enables flexibility for diverse research contexts, populations, and designs. Overall, to paraphrase XKCD, our goal is to not create another measurement instrument to ‘fix’ the inability of every other instrument to create a consensus measure. Rather than another instrument, we contribute a framework for measurement. Our framework is grounded in factorial survey design, regression techniques, and modern approaches to validity. We hypothesize that this approach will ease many of the existing barriers to quantitative PE&EC research while enabling the reuse and potential unification of existing instruments. Doing so enhances the usability of existing instruments rather than replacing them. Particular to engineering education, this advance can enable the efficient longitudinal study of key questions about PE&EC such as why engineering students progress less and later on existing theories.