Validity evidence for measures of statistical reasoning and statistical self-efficacy with engineering students

Todd M. Fernandez; David S. Ancalle

Download Paper | Permalink

Conference: 2023 ASEE Annual Conference & Exposition
Location: Baltimore , Maryland
Publication Date: June 25, 2023
Start Date: June 25, 2023
End Date: June 28, 2023
Conference Session: Research Methodologies – Session 2
Tagged Division: Educational Research and Methods Division (ERM)
Page Count: 20
DOI: 10.18260/1-2--44597
Permanent URL: https://peer.asee.org/44597
Download Count: 299

Paper Authors

biography

Todd M. Fernandez Georgia Institute of Technology orcid.org/0000-0002-1387-2575

visit author page

Todd is a lecturer in the Wallace H. Coulter Department of Biomedical Engineering at Georgia Institute of Technology. His research interests are engineering students beliefs about knowledge and education and how those beliefs interact with learning engineering.

visit author page

biography

David S. Ancalle Department of Civil and Environmental Engineering, Kennesaw State University

visit author page

David S. Ancalle is a Lecturer in the Department of Civil and Environmental Engineering at Kennesaw State University, and a PhD student in the Woodruff School of Mechanical Engineering at Georgia Institute of Technology. Ancalle earned a B.S. from the University of Puerto Rico at Mayaguez and a M.S. from the University of Illinois at Urbana-Champaign, both in civil engineering. He has a passion for teaching undergraduate engineering courses, which has driven his teaching career for the past six years. He recently began working in the area of Engineering Education and plans to continue this path after completing his graduate studies.

visit author page

Download Paper | Permalink

Abstract

This research paper will contribute evidence that two instruments developed to measure statistical learning and statistical confidence retain construct validity with engineering students. This validation study is part of a larger project to evaluate the effectiveness of a homework intervention to improve learning in an undergraduate engineering statistics course. The two instruments we used to measure effectiveness are not new, but they have not been well studied in engineering specific courses or populations. In such cases, it is important to continually evaluate such instruments prior to using them as evidence to make claims about changes in pedagogy to ensure such claims have validity.

The two instruments we implement are the Statistical Self-Efficacy (SSE) instrument and the Statistical Reasoning Assessment (SRA). The SSE is a self-reported measure of confidence on introductory statistics topics. It contains 14 items that students respond to using a 6 point likert-like confidence scale. We evaluate the performance of the SSE using confirmatory factor analysis (CFA) to test for the expected structure of a self-report self-confidence and to test invariance across both demographics and length of the course. The SRA is a scored test of introductory statistics topics. It contains 20 multiple-choice items each with one correct answer. In addition to responding to the items, we also collected self-reported confidence to enable confidence weighted analyses. We evaluate the performance of the SRA using a confidence weighted Rasch modeling analysis. Similar to the goals of the CFA analysis, Rasch models look at instrument level fit of responses to a preconceived model of good measurement. Again, we test to ensure that the SRA shows the expected internal structure and evaluate differential item function - an analogue to invariance for scored tests.

Results from analyses of both instruments will be reported in the full paper. Analyses will use data collected in Spring 2022. Data comes from two sections of an undergraduate course on Biomedical Statistics at a large research-focused engineering university in the United States (n,enrollment=188). Both sections were part of a homework intervention study that randomly assigned each student to one of two homework grading schemes. The two instruments evaluated in this study were given to students in the first and last week of the semester (pre and post design). We received a total of 229 responses to each instrument, 118 from the ‘pre’ distribution and 111 from the ‘post’ distribution. In addition to the instruments, we collected basic demographic data at the end of the pre test.

Citation
Format

Fernandez, T. M., & Ancalle, D. S. (2023, June), Validity evidence for measures of statistical reasoning and statistical self-efficacy with engineering students Paper presented at 2023 ASEE Annual Conference & Exposition, Baltimore , Maryland. 10.18260/1-2--44597

TY - CPAPER
AB - This research paper will contribute evidence that two instruments developed to measure statistical learning and statistical confidence retain construct validity with engineering students. This validation study is part of a larger project to evaluate the effectiveness of a homework intervention to improve learning in an undergraduate engineering statistics course. The two instruments we used to measure effectiveness are not new, but they have not been well studied in engineering specific courses or populations. In such cases, it is important to continually evaluate such instruments prior to using them as evidence to make claims about changes in pedagogy to ensure such claims have validity.

The two instruments we implement are the Statistical Self-Efficacy (SSE) instrument and the Statistical Reasoning Assessment (SRA). The SSE is a self-reported measure of confidence on introductory statistics topics. It contains 14 items that students respond to using a 6 point likert-like confidence scale. We evaluate the performance of the SSE using confirmatory factor analysis (CFA) to test for the expected structure of a self-report self-confidence and to test invariance across both demographics and length of the course. The SRA is a scored test of introductory statistics topics. It contains 20 multiple-choice items each with one correct answer. In addition to responding to the items, we also collected self-reported confidence to enable confidence weighted analyses. We evaluate the performance of the SRA using a confidence weighted Rasch modeling analysis. Similar to the goals of the CFA analysis, Rasch models look at instrument level fit of responses to a preconceived model of good measurement. Again, we test to ensure that the SRA shows the expected internal structure and evaluate differential item function - an analogue to invariance for scored tests.

Results from analyses of both instruments will be reported in the full paper. Analyses will use data collected in Spring 2022. Data comes from two sections of an undergraduate course on Biomedical Statistics at a large research-focused engineering university in the United States (n,enrollment=188). Both sections were part of a homework intervention study that randomly assigned each student to one of two homework grading schemes. The two instruments evaluated in this study were given to students in the first and last week of the semester (pre and post design). We received a total of 229 responses to each instrument, 118 from the ‘pre’ distribution and 111 from the ‘post’ distribution. In addition to the instruments, we collected basic demographic data at the end of the pre test.
AU - Todd M. Fernandez
AU - David S. Ancalle
CY - Baltimore , Maryland
DA - 2023/06/25
PB - ASEE Conferences
TI - Validity evidence for measures of statistical reasoning and statistical self-efficacy with engineering students
UR - https://peer.asee.org/44597
DO - 10.18260/1-2--44597
ER -