June 14, 2015
June 14, 2015
June 17, 2015
K-12 & Pre-College Engineering
26.177.1 - 26.177.11
An Assessment Tool to Evaluate Student Learning of Engineering (Fundamental)While STEM subjects have traditionally been taught separately in K-12 schools the newinitiatives share a focus on integrated approaches to teaching STEM. For example, the recentlyreleased Next Generation Science Standards (NGSS) addressed the need for explicit integrationof science with engineering. Science teachers are expected to teach intersecting concepts andcore disciplinary science using scientific and engineering practices. One challenge that manyscience teachers face is the lack of measurement instruments for student learning required in theNGSS. Measuring the student understanding of engineering, connections between science andengineering, and engineering practices requires the development of new assessments. Thepurpose of this study was to develop, scale, and validate an assessment in engineering. The useof item response theory to assess item functioning was a focus of the study. The work is part ofa larger project focused on increasing student learning in STEM-related areas in Grades 4 – 8through an engineering design-based, integrated approach to STEM instruction and assessment.The test construction process began with an assessment development team of academicresearchers with collective expertise in engineering, science, and mathematics. Researcherswrote 21 multiple-choice items and content and face validity of the items were established byexpert review. Next the assessment was piloted in two waves with the goal of identifying finalversions of each assessment via item analyses and scaling students’ responses to produce anestimate of their proficiency. In the first wave the assessment was administered to a small groupof about 10-20 middle school students. The purpose was to obtain and analyze preliminary itemresponse data as well as obtain feedback on characteristics of the assessments and the testingenvironment that affected performance (e.g., readability, clarity of items). Based on informationprovided by wave 1 piloting the assessments were modified by the assessment developmentteam. In the second wave of piloting the revised assessments were administered toapproximately 150 students. Data from wave two were used to conduct extensive item analysesof the assessments. We initially performed a factor analysis of the data for each assessment formiddle school students. The goal of these analyses was to assess the likelihood that itemsrepresented a single underlying construct. The results suggested that a single (major) factorunderlies the assessment, a finding that was robust to different methods of factoring (principalaxis, maximum likelihood) and factor rotation (varimax, oblique). These results suggests that thegoal of constructing items for the assessment that reflected a single factor was generally met, andalso helped to justify the use of item response theory (IRT) to generate proficiency scores forstudents. Next the Rasch IRT model was fitted to the data for each item on each assessment toestimate that item’s ability to contribute to estimates of student proficiency. The Rasch modeladequately fits the data of all 21 items. In this presentation, we will present the development and reliability of the engineeringassessment and share the assessment tool with the participants.
Moore, T. J., & Guzey, S. S., & Holly , J. (2015, June), An Assessment Tool to Evaluate Student Learning of Engineering (Fundamental) Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.23516
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015