Measuring Systems Thinking Using Stealth Assessment

Andrea Ramirez-Salgado; Eric Wright

Download Paper | Permalink

Conference: 2023 ASEE Annual Conference & Exposition
Location: Baltimore , Maryland
Publication Date: June 25, 2023
Start Date: June 25, 2023
End Date: June 28, 2023
Conference Session: Systems Engineering Division (SYS) Technical Session 1
Tagged Division: Systems Engineering Division (SYS)
Tagged Topic: Diversity
Page Count: 28
DOI: 10.18260/1-2--43575
Permanent URL: https://peer.asee.org/43575
Download Count: 326

Paper Authors

biography

Andrea Ramirez-Salgado University of Florida orcid.org/0000-0002-0001-2672

visit author page

Andrea is a doctoral student at the University of Florida specializing in Educational Technology within the Curriculum and Instruction program. She has a master’s degree in Education and ICT and a bachelor’s degree in Software Systems Engineering. Andrea has been teaching undergraduate and graduate courses for the past thirteen years covering topics such as algorithms, process engineering, instructional design, and applications of technology in education. Her research interests include understanding the implications of hands-on software and hardware learning approaches for developing engineering identity and fostering engineering persistence in students. Additionally, she is part of a research team funded by the National Science Foundation that aims to design and develop gamified activities to teach hardware principles at high school and undergraduate levels with a focus on equity principles.

visit author page

biography

Eric Wright University of Florida

visit author page

PhD student in quantitative research methodology in education at the University of Florida, former eLearning instructional designer and developer. Primary interest is in critical theory approaches to quantitative research.

visit author page

Download Paper | Permalink

Abstract

As technology advances and databases grow larger, people require high-level skills to process information effectively. To address complex problems while maintaining a comprehensive view of the situation, one valuable competency is Systems Thinking (ST). ST is a systematic approach that allows individuals to navigate different levels of a system without losing sight of the big picture. For instance, software development involves numerous components, including user needs, environments, change management, performance metrics, budget, workflows, and more. A systems thinker must understand the causal relationships between these components to provide a comprehensive and optimal solution. They use mental models to identify interdependencies between inputs, processes, transactions, automation needs, and desired outputs. Successful systems thinkers offer solutions that address the root causes of problems rather than simply treating symptoms. ST is a vital skill in engineering, but it also applies to environmental and ecological issues, socio-economic problems, medical cases, nursing, and geography education. Assessing ST typically involves using self-reported measures of systems-thinker characteristics, behavior-based assessments, and affective domain learning. While these traditional methods are quick and easy, they do not provide ongoing, formative assessments that can guide teaching and learning. To address this gap, new approaches like stealth assessment are emerging. Stealth assessment involves diagnosing ST performance based on evidence from students' interactions with multimedia and using Evidence-Centered Design (ECD) frameworks to create optimal ST achievement conditions. This work-in-progress study proposes the use of a video game designed under ECD and stealth assessment principles to teach ST through simulations and problem-solving strategies. A further validation study aims to evaluate the game's effectiveness in measuring ST achievement in real-life situations beyond Software Engineering. The study will focus on middle school students and will consist of two phases. In the first phase, participants will play the game individually and describe their thought processes to identify any necessary changes to the assessment. The second phase will involve enrolling 1,000 seventh-grade students in selected schools to play the game during several class periods. The study will collect and store game logs, demographic information, and performance-based measures to analyze the effectiveness of the game.

Citation
Format

Ramirez-Salgado, A., & Wright, E. (2023, June), Measuring Systems Thinking Using Stealth Assessment Paper presented at 2023 ASEE Annual Conference & Exposition, Baltimore , Maryland. 10.18260/1-2--43575

TY - CPAPER
AB - As technology advances and databases grow larger, people require high-level skills to process information effectively. To address complex problems while maintaining a comprehensive view of the situation, one valuable competency is Systems Thinking (ST). ST is a systematic approach that allows individuals to navigate different levels of a system without losing sight of the big picture. For instance, software development involves numerous components, including user needs, environments, change management, performance metrics, budget, workflows, and more. A systems thinker must understand the causal relationships between these components to provide a comprehensive and optimal solution. They use mental models to identify interdependencies between inputs, processes, transactions, automation needs, and desired outputs. Successful systems thinkers offer solutions that address the root causes of problems rather than simply treating symptoms. ST is a vital skill in engineering, but it also applies to environmental and ecological issues, socio-economic problems, medical cases, nursing, and geography education.
Assessing ST typically involves using self-reported measures of systems-thinker characteristics, behavior-based assessments, and affective domain learning. While these traditional methods are quick and easy, they do not provide ongoing, formative assessments that can guide teaching and learning. To address this gap, new approaches like stealth assessment are emerging. Stealth assessment involves diagnosing ST performance based on evidence from students&#39; interactions with multimedia and using Evidence-Centered Design (ECD) frameworks to create optimal ST achievement conditions. This work-in-progress study proposes the use of a video game designed under ECD and stealth assessment principles to teach ST through simulations and problem-solving strategies.
A further validation study aims to evaluate the game&#39;s effectiveness in measuring ST achievement in real-life situations beyond Software Engineering. The study will focus on middle school students and will consist of two phases. In the first phase, participants will play the game individually and describe their thought processes to identify any necessary changes to the assessment. The second phase will involve enrolling 1,000 seventh-grade students in selected schools to play the game during several class periods. The study will collect and store game logs, demographic information, and performance-based measures to analyze the effectiveness of the game.
AU - Andrea Ramirez-Salgado
AU - Eric Wright
CY - Baltimore , Maryland
DA - 2023/06/25
PB - ASEE Conferences
TI - Measuring Systems Thinking Using Stealth Assessment
UR - https://peer.asee.org/43575
DO - 10.18260/1-2--43575
ER -