Board 77: Exploring the Relationship between Item Stability and Item Characteristics: Exploratory Graph Analysis
- Conference
- Location
-
Portland, Oregon
- Publication Date
-
June 23, 2024
- Start Date
-
June 23, 2024
- End Date
-
June 26, 2024
- Conference Session
-
Educational Research and Methods Division (ERM) Poster Session
- Tagged Division
-
Educational Research and Methods Division (ERM)
- Page Count
-
15
- DOI
-
10.18260/1-2--48376
- Permanent URL
-
https://peer.asee.org/48376
- Download Count
-
58
Paper Authors
Chia-Lin Tsai University of Northern Colorado
Chia-Lin Tsai is an associate professor in the Department of Applied Statistics and Research Methods at the University of Northern Colorado. Her research interests include psychometrics studies and first-generation college students’ academic experience.
Lisa Y Flores University of Missouri, Columbia
Lisa Y. Flores, Ph.D. is a Professor of Psychological Sciences at the University of Missouri. She has expertise in the career development of Latino/as and Latino/a immigrant issues and has over 100 peer reviewed journal publications, 19 book chapters, and 3 co-e
Rachel L Navarro University of North Dakota
Rachel L. Navarro, Ph.D. is Professor of Counseling Psychology and Associate Dean for Research and Faculty Development for the College of Education and Human Development at the University of North Dakota (UND). She is the former department chair for UNDâ€
Pat Garriott
Dr. Garriott received his PhD from the University of Missouri. He is a member of the American Psychological Association (APA), Division 17 (Counseling Psychology) of the APA, and the Society for Vocational Psychology. His work has been recognized by Divi
Han Na Suh Georgia State University - Perimeter College
Sarah Lynn Orton P.E.
University of Missouri, Columbia
orcid.org/0000-0002-7896-039X
Abstract
(This is a Methods Paper.) In this study, we aim to provide some insights into what item characteristics are related to item stability through the newly developed exploratory graph analysis (EGA; Golino & Epskamp, 2017) and bootstrap exploratory graph analysis (bootEGA; Christensen & Golino, 2021). Item stability describes how often an item is placed in the same dimension, as identified by the empirical data, across multiple samples (Christensen & Golino, 2021). Specifically, using a newly designed engineering interest measure as an example, we explored the relationship between item stability and the following item characteristics: 1) network loading, 2) item redundancy, 3) item mean, 4) item-total correlation, 5) item discrimination, and 6) item location. These item characteristics were selected as they are often used as item quality indices within different measurement frameworks, including network psychometrics, Classical Test Theory (CTT), and Item Response Theory (IRT). In this current study, we focused on addressing the following research questions: • RQ1: What is the relationship between item stability and the item quality indices? • RQ2: Do stable and unstable items differ in the average value of the item quality indices? • RQ3: How well do the item quality indices perform when classifying stable and unstable items?
Our sample consisted of 476 Latinx engineers who completed all 89 items of the Engineering Interests Scale (EIM). The Engineering Interest Measure was developed to assess individuals’ interests in engineering-related tasks and skills for engineers in the workplace. A sequence of steps was taken to develop the EIM instrument, including reviewing the literature, specifying construct domains, generating items, and refining measurement items. We conducted EGA on the initial item pool (89 items) to assess the dimensionality of the instrument and conducted bootEGA with 500 random samples to verify the item and structural stability of the dimensionality findings from the EGA. The analyses were conducted using the EGAnet package (Hudson & Alexander, 2023) in R. We analyzed the relationship between item stability and item quality indices produced from CTT item analysis and IRT analysis. Specifically, we used 1) Pearson correlation to analyze the association between the item quality indices and raw item stability value (RQ1), 2) independent-sample t-tests to compare mean differences in the item quality indices between stable and unstable items (RQ2), and 3) the receiver operating characteristic (ROC) curve plot and the area under the curve (AUC) to determine the classification accuracy (RQ3).
Our results showed that item stability had a positive and moderate relationship with network loadings, item-total correlation, and IRT item discrimination. Our findings showed that stable items, on average, were characterized by higher network loadings, higher item-total correlation, higher item discrimination, and higher item difficulty (i.e., lower item means and higher IRT item location) compared to unstable items. Finally, in the ROC curve plot, we observed that network loadings, item mean, item-total correlation, and IRT item difficulty perform well in classifying stable and unstable items. Completed results and implications for the application of exploratory graph analysis in education research will be discussed in the full paper.
Tsai, C., & Flores, L. Y., & Navarro, R. L., & Garriott, P., & Suh, H. N., & Orton, S. L. (2024, June), Board 77: Exploring the Relationship between Item Stability and Item Characteristics: Exploratory Graph Analysis Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--48376
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2024 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015