June 26, 2011
June 26, 2011
June 29, 2011
Educational Research and Methods
22.1352.1 - 22.1352.13
Studying the Reliability and Validity of Test Scores for Mathematical and Spatial Reasoning Tasks for Engineering StudentsBackground and MotivationThe purpose of this paper is to continue a program of research in assessment and test design forthe measurement of three constructs that are keys to academic success in engineeringengineering. The constructs include abilities to: a) select mathematical applications relevant tosolving varied problems in engineering; b) translate two-dimensional images to three-dimensional and vice versa when solving engineering problems; and, c) understand how theengineering quantities (e.g. force, work, power, and flow rate ) are described by themathematical representations (e.g. integration, differentiation, or interpolation) presented instatics, dynamics, thermodynamics, and fluid mechanics.The researchers have designed several problem-solving measures in engineering where scoresfor the three constructs are hypothesized to predict academic success. This research studydemonstrates not only how psychometric models can be applied to study the contribution of theconstructs in problem solving but also how to improve the quality of items, option sets, andscoring keys to increase the reliability and validity of test scores. . Several psychometric andstatistical methods are employed to determine the value of a): scoring responses as partiallycorrect to gauge students’ misconceptions; b) evaluating the role of time relative to accuracy inthe completion of tasks; and, c) studying the relations among the three constructs given students’level of success in engineering courses. The item sets include multiple-choice and constructedresponse formats. Reliability and validity estimates for these two item formats are compared andcontrasted.Test Design StrategyCurrent design for all measures will include computer-based administration where responsetimes for test completion can be recorded along with interactivity for certain tasks (e.g.,manipulation of screen images). Items on the mathematical relevance test present a set ofengineering problems where a particular mathematical application is required to solve thoseproblems. The spatial-visualization task enables students to move between two- to-threedimensional displays to select an engineering principle that must be understood (e.g. linear orangular momentum, and mass flow rate) . The final test includes items that present a series ofmathematical functions in the stem that are required to specify the properties of one or morerelations among variables in physics. Students must select the response that establishes thecondition for why the mathematical application is needed given the properties of the physicsvariables.Data Source and Psychometric ModelingThe data source represents approximately 300 undergraduate students who have declared theirmajor in engineering. Item Response Theory (IRT) models are applied to examine test responsecharacteristics.. The researchers will illustrate how psychometric reports can provide evidencethat items,options, or constructed-response rating categories are contributing to the reliability ofscores... Test scores are then correlated with course grades to establish validity.Conclusions and SignificanceReliable and valid test scores are needed for domain-specific measures in engineering to not onlyprofile patterns of strength and weakness for students enrolled in various programs, but also totest instructional interventions that may facilitate academic progress. This presentationintroduces test design strategies and psychometric evaluation of scores of three variablesconsidered key to academic success in engineering.
Pauley, L. L., & Kulikowich, J. M., & Sedransk, N., & Engel, R. S. (2011, June), Studying the Reliability and Validity of Test Scores for Mathematical and Spatial Reasoning Tasks for Engineering Students Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. https://peer.asee.org/18504
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2011 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015