2023 ASEE Annual Conference & Exposition

This work-in-progress paper expands on a collaboration between engineering education researchers and machine learning researchers to automate the analysis of written responses to conceptually challenging questions in statics and dynamics courses (Authors, 2022). Using the Concept Warehouse (Koretsky et al., 2014), written justifications of ConcepTests (CTs) were gathered from statics and dynamics courses in a diverse set of two- and four-year institutions. Written justifications for CTs have been used to support active learning pedagogies which makes them important to investigate how students put together their problem-solving narratives of understanding. However, despite the large benefit that analysis of student written responses may provide to instructors and researchers, manual review of responses is cumbersome, limits analysis, and can be prone to human bias.

In efforts to improve the analysis of student written responses, machine learning has been used in various educational contexts to analyze short and long texts (Burstein et al., 2020; Burstein et al., 2021). Natural Language Processing (NLP) uses transformer-based machine learning models (Brown et al., 2020; Raffel et al., 2019) which can be used through fine-tuning or in-context learning methods. NLP can be used to train algorithms that can automate the coding of written responses. Only a few studies for educational applications have leveraged transformer-based machine learning models further prompting an investigation into its use in STEM education. However, work in NLP has been criticized for heightening the possibility to perpetuate and even amplify harmful stereotypes and implicit biases (Chang et al., 2019; Mayfield et al., 2019).

In this study, we detail the aim to use NLP for linguistic justice. Using methods like text summary, topic modeling, and text classification, we identify key aspects of student narratives of understanding in written responses to mechanics and statics CTs. Through this process, we seek to use machine learning to identify different ways students talk about a problem and their understanding at any point in their narrative formation process. Thus, we hope to help reduce human bias in the classroom and through technology by giving instructors and researchers a diverse set of narratives that include insight into their students’ histories, identities, and understanding. These can then be used towards connecting technological knowledge to students’ everyday lives.

References

Authors. (2022, June). [Title omitted for blind review].

Burstein, J., Horbach, A., Kochmar, K., Laarmann-Quante, R., Leacock, C., Madnani, Nitin., Pilan, I., Yannakoudakis, H. & Zesch,T.. (2021) Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics

Burstein,J., Kochmar, K., Leacock, C., Madnani, Nitin., Pilan, I., Yannakoudakis, H. & Zesch,T.. (2020). Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics.

Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. and Agarwal, S., (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

Chang, K.-W., Prabhakaran, V., and Ordonez, V. (2019). Bias and Fairness in Natural Language Processing [Paper presentation]. In Proceedings of the 2019 Conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP): Tutorial abstracts. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (China), Hong Kong, China. Association for Computational Linguistics.

Koretsky, M. D., Falconer, J. L., Brooks, B. J., Gilbuena, D. M., Silverstein, D. L., Smith, C., & Miletic, M. (2014). The AIChE Concept Warehouse: A web-based tool to promote concept-based instruction. Advances in Engineering Education, 4(1), 7:1-27.

Mayfield, E., Madaio, M., Prabhumoye, S., Gerritsen, D., McLaughlin, B., Dixon-Román, E., and Black, A. W. (2019, August). Equity beyond bias in language technologies for education [Paper presentation]. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications (Italy), Florence, Italy (pp. 444–460). Association for Computational Linguistics.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.