2023 ASEE Annual Conference & Exposition

In this Work in Progress (WIP) paper, we describe preliminary results from a systematic literature review (SLR) examining how engineering education researchers humanize data-driven quantitative methods by using “person-centered” approaches. Engineering education researchers commonly use quantitative methods to seek causal or correlational relationships between observed phenomena of interest. However, under the assumptions of many traditional statistical methods, there is a tendency to minimize the voice of underrepresented engineering student groups because of low sample sizes. This minimization is accomplished by grouping underrepresented students into heterogenous categories or removing them altogether as outliers. As a result, these methods can incorrectly generalize the findings based on the dominant group to the entire engineering student body. Moreover, data-driven research is often associated with objectivity, encapsulated by the phrase “letting the data speak for itself.” Drawing conclusions from incomplete demographic samples and generalizing biased results can further damage the benefits of minoritized groups in the engineering student body and exacerbate the existing inequality.

Despite these challenges, thoughtfully applied data-driven methods have the potential to value the experience of minoritized individuals. In particular, Godwin et al. (2021) introduce the concept of person-centered approaches to the community. A person-centered approach recognizes heterogeneity and attempts to find relationships among variables to form latent groupings among individuals in the sample. More traditional statistical analyses, called variable-centered approaches, are concerned with prediction and work best with homogeneity in the sample. To explore how engineering education researchers employed person-centered approaches, we followed Borrego et al.’s (2014) process for conducting a systematic literature review (I.e., PRISMA) to analyze publications from four journals from the last decade (2011-2021): Journal of Engineering Education, European Journal of Engineering Education, International Journal of Engineering Education, and IEEE Transactions on Education. We formed our search string based on common data-driven techniques, such as cluster analysis and random forest, including more general keywords like machine learning. The initial data set yielded by the search was 236 articles. After screening abstracts to remove articles that did not apply the techniques to human subjects and were not empirical articles employing data-driven methods, we retained 98 articles. A full-text review using the same exclusion criteria removed 19 articles, resulting in our final dataset (n = 79).

Our analyses are still in progress. The planned analysis is to apply the principles of QuantCrit (Garcia et al., 2018; Gillborn et al., 2018) combined with a categorization scheme of person and variable-centered analyses to elicit themes of how engineering education researchers make analytical decisions when applying data-driven methods. We anticipate that this SLR could help the field to understand how researchers use data-driven quantitative methods and offset the issues with these methods using person-centered approaches. This SLR also will reveal what kinds of engineering education problems are more likely to be studied with data-driven methods.

References:

Borrego, M., Foster, M. J., & Froyd, J. E. (2014). Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields: Systematic Literature Reviews in Engineering Education. Journal of Engineering Education, 103(1), 45–76. https://doi.org/10.1002/jee.20038

Garcia, N. M., López, N., & Vélez, V. N. (2018). QuantCrit: Rectifying quantitative methods through critical race theory. Race Ethnicity and Education, 21(2), 149–157. https://doi.org/10.1080/13613324.2017.1377675

Gillborn, D., Warmington, P., & Demack, S. (2018). QuantCrit: Education, policy, ‘Big Data’ and principles for a critical race theory of statistics. Race Ethnicity and Education, 21(2), 158–179. https://doi.org/10.1080/13613324.2017.1377417

Godwin, A., Benedict, B., Rohde, J., Thielmeyer, A., Perkins, H., Major, J., Clements, H., & Chen, Z. (2021). New Epistemological Perspectives on Quantitative Methods: An Example Using Topological Data Analysis. Studies in Engineering Education, 2(1), 16-34. https://doi.org/10.21061/see.18