2025 ASEE Annual Conference & Exposition

This work-in-progress explores the application of pre-trained, open-source transformer models designed to run efficiently on local hardware for natural language processing (NLP) in classifying student short-answer responses within the context of the narrative-based engineering ethics game/assessment, Mars! An Ethical Expedition (Mars!). Building on the contemporary learning theory of situated cognition and concepts of seamless (stealth) assessment, the game immerses students in decision-making scenarios tied to ethical dilemmas on a Mars colony, encouraging context-dependent ethical reasoning. The primary focus is on analyzing the justifications students provide for their in-game decisions using NLP-based text analytics. Traditional ethical reasoning assessment tools, such as the Engineering Ethical Reasoning Instrument (EERI), have been critiqued for their limitations in capturing in-situ ethical decision-making. In response to these limitations, Mars! was developed to provide a rich, narrative-driven environment that allows for a more context-sensitive assessment of students' ethical reasoning as they engage with complex, first-person dilemmas.
We propose using transformer-based machine learning techniques to analyze student responses, with a primary focus on assessing response completeness. The completeness classifier categorizes responses as irrelevant or incomplete, partial, or complete, providing instructors with a scalable method to evaluate student engagement with ethical dilemmas. Beyond completeness, models will categorize student justifications based on perspective (e.g., first-person vs. third-person reasoning), motive (e.g., self-interest vs. social good), and reasoning approach (e.g., strict rule application vs. situated reasoning). These additional classifications are designed to support future research into how students frame and justify ethical decisions. By evaluating these responses at scale, the study aims to develop more efficient and accurate instructor-friendly tools for assessing ethical reasoning in authentic, first-person contexts.
Initial results suggest that locally deployed transformer models for text classification may supplement quantitative ethical reasoning assessments like the EERI by providing additional nuanced analysis of student ethical judgments. This project contributes to a growing body of research on the use of text analytics for formative assessment in engineering ethics education, with implications for enhancing student learning and promoting ethical decision-making in professional engineering contexts.