Initial Validity Evidence for a Survey of Skill and Attitude Development on Engineering Teams

Justin Charles Major; Richard Tyler Cimino

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: July 12, 2024
Conference Session: Educational Research and Methods Division (ERM) Technical Session 29
Tagged Division: Educational Research and Methods Division (ERM)
Tagged Topic: Diversity
Permanent URL: https://peer.asee.org/47625

Request a correction

Paper Authors

biography

Justin Charles Major Rowan University orcid.org/0000-0002-3111-8509

visit author page

Dr. Justin C. Major (he/him) is an Assistant Professor of Experiential Engineering Education at Rowan University where he leads ASPIRE Lab (Advancing Student Pathways through Inequality Research in Engineering). Justin’s research focuses on engineering belonging and marginalization mechanisms, adverse childhood experiences, and feminist approaches to EER, and connects these topics to broader understandings of student success in engineering. Justin completed his Ph.D. in Engineering Education (’22) and M.S. in Aeronautics and Astronautics (’21) at Purdue University, and two B.S. in Mechanical Engineering and Secondary Mathematics Education at the University of Nevada, Reno (’17). Atop his education, Justin is a previous NSF Graduate Research Fellow and has won over a dozen awards for research, service, and activism related to marginalized communities, including the 2020 ASEE ERM Division Best Diversity Paper for his work on test anxiety. As a previous homeless and food-insecure student, Justin is eager to challenge and change engineering engineering education to be a pathway for socioeconomic mobility and broader systemic improvement rather than an additional barrier.

visit author page

biography

Richard Tyler Cimino New Jersey Institute of Technology orcid.org/0000-0003-4171-4133

visit author page

Dr. Richard T. Cimino is a Senior Lecturer in the Otto H. York Department of Chemical and Materials Engineering at New Jersey Institute of Technology. His research interests include the intersection of engineering ethics and process safety, and broadening inclusion in engineering, with a focus on the LGBTQ+ community.

visit author page

Download Paper | Permalink

Abstract

This research paper discusses an emerging project that 1) seeks to gather validity evidence for a survey of engineering student teaming attitudes and skill development and 2) identifies what skills and attitudes engineering students develop over time and to what degree.

Given teamwork’s importance in engineering education and practice, the skill is essential for engineering students to learn. ABET (2023) states that teams “consist of more than one person working toward a common goal and should include individuals of diverse backgrounds, skills, or perspectives.” According to ABET Criterion 3, an effective team includes “members [who] together provide leadership, create a collaborative and inclusive environment, establish goals, plan tasks, and meet objectives.” As we have explored the teamwork literature, we have identified that teamwork, as the discipline has defined it, lacks specificity , thus leaving open the criterion for assessment. This project emerged as an opportunity to both “measure” and precisely define teamwork skills and attitudes.

In the summer/fall of 2023, we developed a pilot survey of engineering teaming attitudes and skills and administered it at two mid-Atlantic institutions. The survey was given in both in- and out-of-class contexts. Items relating to many attitudes (e.g., identity, motivation, belonging, etc.) with strong validity with engineering students were taken as-is from a project Studying Underlying Characteristics of Computing and Engineering Student Success (SUCCESS) developed by Berger et al. (2018), some of which were specifically adapted to the teaming environment (e.g. gratitude was changed to gratitude towards teaming). Other items included other factors with strong validity evidence such as patience (non-engineering; adapted from Schnitker, 2012), engineering team conflict and roles (unpublished, work by Kirn, Godwin, and colleagues), leadership (non-engineering; adapted from Wielkiewicz et al., 2000), and perceptions of inclusion and discrimination in engineering (Bahnson et al., 2022). We also collected demographics using Fernandez and colleagues' (2016) recommendations for inclusive survey design, expanding options to be more representative of additional socioeconomic backgrounds and neurodivergence. Finally, we asked students to answer qualitative questions to gather initial content validity for our items and factors and to add context to our findings.

Survey administration ended in early October 2023, and validity analyses are underway. Our pilot administration led to a sample size of n=606 with representation across years, discipline, race/ethnicity, gender, socioeconomic background, (dis)ability and neurodivergence identities, military background, and sexual orientation. Our validity work, given our small sample size, focused on factors that we have newly adapted to our context or engineering altogether. This paper focuses on exploratory factor analysis. We used parallel analysis to determine the number of factors present across our items, followed by exploratory factor analysis to identify those factors and their makeup. In this paper, we will share our present findings and our future plans.

Citation
Format

Major, J. C., & Cimino, R. T. (2024, June), Initial Validity Evidence for a Survey of Skill and Attitude Development on Engineering Teams Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. https://peer.asee.org/47625

TY - CPAPER
AB - This research paper discusses an emerging project that 1) seeks to gather validity evidence for a survey of engineering student teaming attitudes and skill development and 2) identifies what skills and attitudes engineering students develop over time and to what degree.

Given teamwork’s importance in engineering education and practice, the skill is essential for engineering students to learn. ABET (2023) states that teams “consist of more than one person working toward a common goal and should include individuals of diverse backgrounds, skills, or perspectives.” According to ABET Criterion 3, an effective team includes “members [who] together provide leadership, create a collaborative and inclusive environment, establish goals, plan tasks, and meet objectives.” As we have explored the teamwork literature, we have identified that teamwork, as the discipline has defined it, lacks specificity , thus leaving open the criterion for assessment. This project emerged as an opportunity to both “measure” and precisely define teamwork skills and attitudes.

In the summer/fall of 2023, we developed a pilot survey of engineering teaming attitudes and skills and administered it at two mid-Atlantic institutions. The survey was given in both in- and out-of-class contexts. Items relating to many attitudes (e.g., identity, motivation, belonging, etc.) with strong validity with engineering students were taken as-is from a project Studying Underlying Characteristics of Computing and Engineering Student Success (SUCCESS) developed by Berger et al. (2018), some of which were specifically adapted to the teaming environment (e.g. gratitude was changed to gratitude towards teaming). Other items included other factors with strong validity evidence such as patience (non-engineering; adapted from Schnitker, 2012), engineering team conflict and roles (unpublished, work by Kirn, Godwin, and colleagues), leadership (non-engineering; adapted from Wielkiewicz et al., 2000), and perceptions of inclusion and discrimination in engineering (Bahnson et al., 2022). We also collected demographics using Fernandez and colleagues&#39; (2016) recommendations for inclusive survey design, expanding options to be more representative of additional socioeconomic backgrounds and neurodivergence. Finally, we asked students to answer qualitative questions to gather initial content validity for our items and factors and to add context to our findings.

Survey administration ended in early October 2023, and validity analyses are underway. Our pilot administration led to a sample size of n=606 with representation across years, discipline, race/ethnicity, gender, socioeconomic background, (dis)ability and neurodivergence identities, military background, and sexual orientation. Our validity work, given our small sample size, focused on factors that we have newly adapted to our context or engineering altogether. This paper focuses on exploratory factor analysis. We used parallel analysis to determine the number of factors present across our items, followed by exploratory factor analysis to identify those factors and their makeup. In this paper, we will share our present findings and our future plans.

AU - Justin Charles Major
AU - Richard Tyler Cimino
CY - Portland, Oregon
DA - 2024/06/23
PB - ASEE Conferences
TI - Initial Validity Evidence for a Survey of Skill and Attitude Development on Engineering Teams
UR - https://peer.asee.org/47625
ER -