2023 ASEE Annual Conference & Exposition

As technology advances and databases grow larger, people require high-level skills to process information effectively. To address complex problems while maintaining a comprehensive view of the situation, one valuable competency is Systems Thinking (ST). ST is a systematic approach that allows individuals to navigate different levels of a system without losing sight of the big picture. For instance, software development involves numerous components, including user needs, environments, change management, performance metrics, budget, workflows, and more. A systems thinker must understand the causal relationships between these components to provide a comprehensive and optimal solution. They use mental models to identify interdependencies between inputs, processes, transactions, automation needs, and desired outputs. Successful systems thinkers offer solutions that address the root causes of problems rather than simply treating symptoms. ST is a vital skill in engineering, but it also applies to environmental and ecological issues, socio-economic problems, medical cases, nursing, and geography education.
Assessing ST typically involves using self-reported measures of systems-thinker characteristics, behavior-based assessments, and affective domain learning. While these traditional methods are quick and easy, they do not provide ongoing, formative assessments that can guide teaching and learning. To address this gap, new approaches like stealth assessment are emerging. Stealth assessment involves diagnosing ST performance based on evidence from students' interactions with multimedia and using Evidence-Centered Design (ECD) frameworks to create optimal ST achievement conditions. This work-in-progress study proposes the use of a video game designed under ECD and stealth assessment principles to teach ST through simulations and problem-solving strategies.
A further validation study aims to evaluate the game's effectiveness in measuring ST achievement in real-life situations beyond Software Engineering. The study will focus on middle school students and will consist of two phases. In the first phase, participants will play the game individually and describe their thought processes to identify any necessary changes to the assessment. The second phase will involve enrolling 1,000 seventh-grade students in selected schools to play the game during several class periods. The study will collect and store game logs, demographic information, and performance-based measures to analyze the effectiveness of the game.