Validity Evidence for the Sophomore Engineering Experiences Survey

Fanyi Zhang; Beth M. Holloway; Eric Holloway

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: June 26, 2024
Conference Session: Educational Research and Methods Division (ERM) Technical Session 27
Tagged Division: Educational Research and Methods Division (ERM)
Page Count: 48
DOI: 10.18260/1-2--48252
Permanent URL: https://peer.asee.org/48252
Download Count: 203

Paper Authors

biography

Fanyi Zhang Purdue University orcid.org/0000-0002-1523-9087

visit author page

Fanyi is a third-year Ph.D. student at Purdue University. She majors in Industrial-Organizational Psychology and works as a Graduate Research Assistant for the Women in Engineering program. Her area of interest focuses on conceptualizing and promoting flourishing and understanding the dynamics of positive relationships. Her current research agenda includes developing a mentor support framework and promoting the effective design of mentor training.

visit author page

biography

Beth M. Holloway Purdue University orcid.org/0000-0003-2528-6079

visit author page

Beth Holloway is the Senior Assistant Dean for Student Access and Success, the Leah H. Jamieson Director of Women in Engineering (WiE), and a Professor of Engineering Practice in the School of Mechanical Engineering in the College of Engineering at Purdue University.

visit author page

biography

Eric Holloway Purdue University orcid.org/0000-0002-0343-1709

visit author page

Prof. Eric Holloway currently serves as a Professor of Engineering Practice in the School of Mechanical Engineering at Purdue University. He also holds a courtesy faculty appointment in the School of Engineering Education. His research focuses on assessment development and the professional formation of students.

visit author page

Download Paper | Permalink

Abstract

This research paper presents validity evidence for a sophomore engineering experience survey that provides an initial understanding of how sophomores experienced their second year of engineering studies. While the sophomore year is a pivotal transition for engineering students, existing research and practices have largely overlooked this crucial period. Unfortunately, not much is known about how sophomore engineering students are thriving (or not) in their college experiences. As such, there is a need to assess these students and understand more about their college experiences so interventions can be planned and implemented. The primary aim of this research is to establish validity evidence for the scales used in the Sophomore Engineering Experiences Survey (SEES). The survey was adapted from Schreiner’s Sophomore Experiences Survey, and guided by Tinto’s framework of student departure to provide a multifaceted understanding of sophomore engineering students’ experiences. Specifically, we ask the following research questions: 1) What validity evidence is needed for each of the five scales in the SEES? 2) What is the validity evidence for each of the five scales in SEES, and how is it interpreted? 3) To what extent do SEES scores vary among the demographic groups of gender and race/ethnicity for sophomore engineering students? Surveys were administered each Spring semester from 2013 to 2022 to sophomore engineering students (i.e., students at the end of their fourth semester) at a large PWI institution in the Midwest, yielding a dataset of 1,766 clean responses. Based on prior theory and research, we determined whether there was sufficient prior validity evidence for adapting Schreiner’s survey and what additional validity evidence was needed for the sophomore engineering use case. Adopting Kane’s argument-based approach, we gathered evidence to find support for the validity of the interpretations of the five scales of the SEES, specifically for reliability and factor structure. We then employed either a) Confirmatory Factor Analyses (CFA) only, or b) both Exploratory Factor Analysis (EFA) and CFA based on this understanding. For the scales that underwent both EFA and CFA, we performed a stratified random split based on the year the survey was conducted, allowing us to use separate samples for the two analyses. Internal consistency was calculated for all scales in the SEES. Our findings provided supporting evidence for the reliability and factorial validity of the interpretations of each scale in the SEES. Specifically, all scales in the SEES exhibited acceptable Cronbach’s alpha, supporting internal consistency of the proposed score uses. We also found support for appropriate factor structure for all five scales in the SEES. We performed group analyses for gender and race/ethnicity groups, and the differences aligned with previous theories and established research. We conclude that the Sophomore Engineering Experiences Survey has sufficient validity evidence for assessing the experiences of sophomore engineering students and, therefore, can be used to 1) offer empirical insights into the current state of sophomore engineering experiences, 2) identify factors that contribute to positive or negative experiences, 3) further elucidate group differences, and 4) provide actionable guidance for students, advisors, and administrators.

Citation
Format

Zhang, F., & Holloway, B. M., & Holloway, E. (2024, June), Validity Evidence for the Sophomore Engineering Experiences Survey Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--48252

TY - CPAPER
AB - This research paper presents validity evidence for a sophomore engineering experience survey that provides an initial understanding of how sophomores experienced their second year of engineering studies. While the sophomore year is a pivotal transition for engineering students, existing research and practices have largely overlooked this crucial period. Unfortunately, not much is known about how sophomore engineering students are thriving (or not) in their college experiences. As such, there is a need to assess these students and understand more about their college experiences so interventions can be planned and implemented. The primary aim of this research is to establish validity evidence for the scales used in the Sophomore Engineering Experiences Survey (SEES). The survey was adapted from Schreiner’s Sophomore Experiences Survey, and guided by Tinto’s framework of student departure to provide a multifaceted understanding of sophomore engineering students’ experiences. Specifically, we ask the following research questions: 1) What validity evidence is needed for each of the five scales in the SEES? 2) What is the validity evidence for each of the five scales in SEES, and how is it interpreted? 3) To what extent do SEES scores vary among the demographic groups of gender and race/ethnicity for sophomore engineering students? Surveys were administered each Spring semester from 2013 to 2022 to sophomore engineering students (i.e., students at the end of their fourth semester) at a large PWI institution in the Midwest, yielding a dataset of 1,766 clean responses. Based on prior theory and research, we determined whether there was sufficient prior validity evidence for adapting Schreiner’s survey and what additional validity evidence was needed for the sophomore engineering use case. Adopting Kane’s argument-based approach, we gathered evidence to find support for the validity of the interpretations of the five scales of the SEES, specifically for reliability and factor structure. We then employed either a) Confirmatory Factor Analyses (CFA) only, or b) both Exploratory Factor Analysis (EFA) and CFA based on this understanding. For the scales that underwent both EFA and CFA, we performed a stratified random split based on the year the survey was conducted, allowing us to use separate samples for the two analyses. Internal consistency was calculated for all scales in the SEES. Our findings provided supporting evidence for the reliability and factorial validity of the interpretations of each scale in the SEES. Specifically, all scales in the SEES exhibited acceptable Cronbach’s alpha, supporting internal consistency of the proposed score uses. We also found support for appropriate factor structure for all five scales in the SEES. We performed group analyses for gender and race/ethnicity groups, and the differences aligned with previous theories and established research. We conclude that the Sophomore Engineering Experiences Survey has sufficient validity evidence for assessing the experiences of sophomore engineering students and, therefore, can be used to 1) offer empirical insights into the current state of sophomore engineering experiences, 2) identify factors that contribute to positive or negative experiences, 3) further elucidate group differences, and 4) provide actionable guidance for students, advisors, and administrators.
AU - Fanyi Zhang
AU - Beth M. Holloway
AU - Eric Holloway
CY - Portland, Oregon
DA - 2024/06/23
PB - ASEE Conferences
TI - Validity Evidence for the Sophomore Engineering Experiences Survey
UR - https://peer.asee.org/48252
DO - 10.18260/1-2--48252
ER -