Validating Assessment Instruments for Use in Engineering Education: A Primer for Conducting and Interpreting Factor Analysis

Susan L. Amato-Henderson; Jon Sticklen

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: June 26, 2024
Conference Session: Educational Research and Methods Division (ERM) Technical Session 27
Tagged Division: Educational Research and Methods Division (ERM)
Page Count: 21
DOI: 10.18260/1-2--48250
Permanent URL: https://peer.asee.org/48250
Download Count: 125

Paper Authors

biography

Susan L. Amato-Henderson Michigan Technological University

visit author page

Susan Amato-Henderson is an Associate Professor Emeritus of Psychology in the Department of Cognitive and Learning Sciences at Michigan Technological University. She received her Ph.D. in Experimental Psychology from the University of North Dakota. Her research interests broadly include STEM education, and focus on individual differences in terms of motivation, self-regulated learning, self-efficacy, grit, resilience, and similar attributes as they can be leveraged to increase academic success indicators. Since retirement she serves as a research consultant.

visit author page

biography

Jon Sticklen Michigan Technological University orcid.org/0000-0003-1778-4027

visit author page

Jon Sticklen is an Associate Professor with the Engineering Fundamentals Department (EF) and Affiliated Faculty with the Department of Cognitive and Learning Sciences (CLS). He served as Chair of EF from 2014-2020, leading a successful effort to design an active-learning focused upgrade of the MTU first-year engineering program. His main research interests currently are in early engineering education, particularly remote education, in systems engineering education at the undergraduate level, and in AI. His background has spanned 40 years, with active research in computer science/AI, and in large scale educational reform in CS and engineering. His work has been supported by NSF, NASA, DARPA, and on the Commerical side by McDonnell Douglas (now Boeing), GE Aircraft Engines, and The MathWorks.

visit author page

Download Paper | Permalink

Abstract

Much of the researchers’ recent work focuses on validating the MUSIC inventory of Jones and associates. This inventory assesses students’ self-perceptions of motivation-related factors in an academic environment. Previous validation work took place in general education or free elective courses. It has not seen widespread use in engineering education. Thus, our near-term goal is to establish the MUSIC inventory, or an adaptation of it, for use in engineering education research.

Establishing the validity of an instrument does not occur in isolation from its use in a given context. For the MUSIC inventory to be a valid tool in engineering, we sought to validate the inventory through a series of studies. Confirmatory factor analysis (CFA) of the five-factor scale measuring the constructs of eMpowerment, Usefulness, Success, Interest, and Caring has indicated that the Interest factor may be problematic (i.e., cross-loading with items on the usefulness factor, among others).

In this report, we will describe our work in validating the MUSIC inventory to exemplify the process of establishing the validity of a pre-existing instrument when used in relatively unexplored domains while conducting discipline-based education research (DBER). More specifically, we strategically examine potential validation issues related to the Interest factor of the MUSIC inventory when used in a first-year engineering education (FYE) course.

Our overriding goal is to offer a primer to DBER researchers who may not be familiar with confirmatory and exploratory factor analysis techniques (CFA and EFA) used in multi-dimensional instrument validation. We take as a case the techniques used in our current efforts to validate the MUSIC inventory within engineering education. We discuss confirmatory factor analysis (CFA) and the underlying assumptions necessary to perform CFA with ordinal data, such as normality and homogeneity of variance. We will describe the various model options when conducting CFA and summarize how to assess a model’s fit. If model fit is lacking, we demonstrate returning to exploratory factor analysis (EFA) to assist in interpreting the model. Throughout, we will use our work with the MUSIC inventory as a case study to illustrate the methodology.

Our case study underscores the central importance of replication studies. Although not common in engineering education, replication is the core method used to validate original study conclusions, utilize assessment tools within new contexts, apply conclusions, or utilize cognitive models within varying contexts (e.g., external validation). If new validation efforts result in contradictions from the original work, one must either abandon the method(s) or update the tool, model, or findings given the new conditions and/or results. For us, we believe that the MUSIC inventory (or a modified version of the tool) can be an effective accelerant in identifying the importance of intrinsic motivation in students for learning.

Citation
Format

Amato-Henderson, S. L., & Sticklen, J. (2024, June), Validating Assessment Instruments for Use in Engineering Education: A Primer for Conducting and Interpreting Factor Analysis Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--48250

TY - CPAPER
AB - Much of the researchers’ recent work focuses on validating the MUSIC inventory of Jones and associates. This inventory assesses students’ self-perceptions of motivation-related factors in an academic environment. Previous validation work took place in general education or free elective courses. It has not seen widespread use in engineering education. Thus, our near-term goal is to establish the MUSIC inventory, or an adaptation of it, for use in engineering education research.

Establishing the validity of an instrument does not occur in isolation from its use in a given context. For the MUSIC inventory to be a valid tool in engineering, we sought to validate the inventory through a series of studies. Confirmatory factor analysis (CFA) of the five-factor scale measuring the constructs of eMpowerment, Usefulness, Success, Interest, and Caring has indicated that the Interest factor may be problematic (i.e., cross-loading with items on the usefulness factor, among others).

In this report, we will describe our work in validating the MUSIC inventory to exemplify the process of establishing the validity of a pre-existing instrument when used in relatively unexplored domains while conducting discipline-based education research (DBER). More specifically, we strategically examine potential validation issues related to the Interest factor of the MUSIC inventory when used in a first-year engineering education (FYE) course.

Our overriding goal is to offer a primer to DBER researchers who may not be familiar with confirmatory and exploratory factor analysis techniques (CFA and EFA) used in multi-dimensional instrument validation. We take as a case the techniques used in our current efforts to validate the MUSIC inventory within engineering education. We discuss confirmatory factor analysis (CFA) and the underlying assumptions necessary to perform CFA with ordinal data, such as normality and homogeneity of variance. We will describe the various model options when conducting CFA and summarize how to assess a model’s fit. If model fit is lacking, we demonstrate returning to exploratory factor analysis (EFA) to assist in interpreting the model. Throughout, we will use our work with the MUSIC inventory as a case study to illustrate the methodology.

Our case study underscores the central importance of replication studies. Although not common in engineering education, replication is the core method used to validate original study conclusions, utilize assessment tools within new contexts, apply conclusions, or utilize cognitive models within varying contexts (e.g., external validation). If new validation efforts result in contradictions from the original work, one must either abandon the method(s) or update the tool, model, or findings given the new conditions and/or results. For us, we believe that the MUSIC inventory (or a modified version of the tool) can be an effective accelerant in identifying the importance of intrinsic motivation in students for learning.

AU - Susan L. Amato-Henderson
AU - Jon Sticklen
CY - Portland, Oregon
DA - 2024/06/23
PB - ASEE Conferences
TI - Validating Assessment Instruments for Use in Engineering Education: A Primer for Conducting and Interpreting Factor Analysis
UR - https://peer.asee.org/48250
DO - 10.18260/1-2--48250
ER -