Performance Assessment in Elementary Engineering: Evaluating Student (RTP)

Cathy P. Lachapelle; Christine M. Cunningham

Download Paper | Permalink

Conference: 2016 ASEE Annual Conference & Exposition
Location: New Orleans, Louisiana
Publication Date: June 26, 2016
Start Date: June 26, 2016
End Date: June 29, 2016
ISBN: 978-0-692-68565-5
ISSN: 2153-5965
Conference Session: K-12 & Pre-College Engineering Division: Fundamental & Research-to-Practice: K-12 Engineering Resources: Best Practices in Curriculum Design (Part 2)
Tagged Division: Pre-College Engineering Education Division
Tagged Topic: Diversity
Page Count: 16
DOI: 10.18260/p.25884
Permanent URL: https://peer.asee.org/25884
Download Count: 774

Paper Authors

biography

Cathy P. Lachapelle Museum of Science

visit author page

Cathy Lachapelle leads the EiE team responsible for assessment and evaluation of our curricula. This includes the design and field-testing of assessment instruments and research on how children use EiE materials. Cathy is particularly interested in how collaborative interaction and scaffolded experiences with disciplinary practices help children learn science, math, and engineering. Her work on other STEM education research projects includes the national Women's Experiences in College Engineering (WECE) study. Cathy received her S.B. in cognitive science from the Massachusetts Institute of Technology and her Ph.D. in educational psychology from Stanford University.

visit author page

biography

Christine M. Cunningham Museum of Science orcid.org/0000-0003-1922-7101

visit author page

Dr. Christine Cunningham is an educational researcher who works to make engineering and science more relevant, accessible, and understandable, especially for underserved and underrepresented populations. A vice president at the Museum of Science, Boston since 2003, she founded and directs Engineering is Elementary™, a groundbreaking project that integrates engineering concepts into elementary curriculum and teacher professional development. As of September 2014, EiE has served 6.2 million children nationwide and 71,000 educators. Cunningham has previously served as director of engineering education research at the Tufts University Center for Engineering Educational Outreach, where her work focused on integrating engineering with science, technology, and math in professional development for K-12 teachers. She also directed the Women’s Experiences in College Engineering (WECE) project, the first national, longitudinal, large-scale study of the factors that support young women pursuing engineering degrees. Cunningham is a Fellow of the American Society for Engineering Education and was awarded the 2014 International Society for Design and Development in Education Prize. She holds B.A. and M.A. degrees in biology from Yale and a Ph.D. in Science Education from Cornell University.

visit author page

Download Paper | Permalink

Abstract

With the new emphasis on engineering practices and engineering design (NGSS Lead States, 2013), teachers of and researchers studying K-12 engineering need to find ways to measure students’ developing engineering skills. To efficiently measure student learning of engineering practices, there is need for a tool to capture student performances in a way that readily affords evaluation and scalability. The problem we pursue in this paper is how to accomplish this measurement. Can we evaluate individual elementary student skills with a quick pencil and paper design task? What aspects of performance does the measure capture? How do these aspects of performance match up with the intended measurement goals? Can the instrument be efficiently and reliably coded using a rubric, so that researchers and teachers can make use of it? To address this need, we developed a performance assessment. Performance assessments are a form of contextual assessment where students engage in tasks within a context that affords the use of practices of interest to the assessor (Klassen, 2006). Students are presented with three quick design challenges to choose from. Each is presented as a written thought experiment with follow-up questions. During development of the instrument, as part of our process of gathering evidence for validity, we conducted think-aloud protocols with a dozen students in the target age range (8-11) who were learning engineering, to inform design and ensure that students were interpreting the instrument as intended. To characterize quality of performance on the written assessment, we developed a rubric that focuses solely on aspects of the NGSS Engineering DCI and Practices that we expected to see, in particular (a) generating multiple possible solutions, (b) evaluating a potential solution against criteria and constraints, (c) planning an investigation, and (d) communicating information (a design plan). Using the rubric, we individually scored 1531 written assessments from 276 grades 3-5 classrooms. Participating classrooms hailed from schools in 3 states, from a variety of urban / suburban / rural contexts, with a variety of racial / ethnic demographics. In this paper, we present the instrument and coding rubric. We calculate inter-rater reliability for coders and present descriptive statistics for student scores to demonstrate the utility of the instrument for distinguishing a range of performances. To build a case for validity for use of the assessment to measure student learning of practices, we compare video of 30 students working on design challenges in their student groups, collected from 10 of the participating classrooms, to the same students’ performance on the assessment. This also informs the use and limits of utility of the written performance assessment for measuring elementary students’ engineering skills and understanding-in-use. Finally, we describe the time needed to score the assessments, and discuss its utility for larger-scale research studies.

Klassen, S. (2006). Contextual assessment in science education: Background, issues, and policy. Science Education, 90(5), 820–851. NGSS Lead States. (2013). Next Generation Science Standards: For States, By States. Washington, DC: The National Academies Press.

Citation
Format

Lachapelle, C. P., & Cunningham, C. M. (2016, June), Performance Assessment in Elementary Engineering: Evaluating Student (RTP) Paper presented at 2016 ASEE Annual Conference & Exposition, New Orleans, Louisiana. 10.18260/p.25884

TY - CPAPER
AB - With the new emphasis on engineering practices and engineering design (NGSS Lead States, 2013), teachers of and researchers studying K-12 engineering need to find ways to measure students’ developing engineering skills. To efficiently measure student learning of engineering practices, there is need for a tool to capture student performances in a way that readily affords evaluation and scalability. The problem we pursue in this paper is how to accomplish this measurement. Can we evaluate individual elementary student skills with a quick pencil and paper design task? What aspects of performance does the measure capture? How do these aspects of performance match up with the intended measurement goals? Can the instrument be efficiently and reliably coded using a rubric, so that researchers and teachers can make use of it?
To address this need, we developed a performance assessment. Performance assessments are a form of contextual assessment where students engage in tasks within a context that affords the use of practices of interest to the assessor (Klassen, 2006). Students are presented with three quick design challenges to choose from. Each is presented as a written thought experiment with follow-up questions. During development of the instrument, as part of our process of gathering evidence for validity, we conducted think-aloud protocols with a dozen students in the target age range (8-11) who were learning engineering, to inform design and ensure that students were interpreting the instrument as intended.
To characterize quality of performance on the written assessment, we developed a rubric that focuses solely on aspects of the NGSS Engineering DCI and Practices that we expected to see, in particular (a) generating multiple possible solutions, (b) evaluating a potential solution against criteria and constraints, (c) planning an investigation, and (d) communicating information (a design plan). Using the rubric, we individually scored 1531 written assessments from 276 grades 3-5 classrooms. Participating classrooms hailed from schools in 3 states, from a variety of urban / suburban / rural contexts, with a variety of racial / ethnic demographics.
In this paper, we present the instrument and coding rubric. We calculate inter-rater reliability for coders and present descriptive statistics for student scores to demonstrate the utility of the instrument for distinguishing a range of performances. To build a case for validity for use of the assessment to measure student learning of practices, we compare video of 30 students working on design challenges in their student groups, collected from 10 of the participating classrooms, to the same students’ performance on the assessment. This also informs the use and limits of utility of the written performance assessment for measuring elementary students’ engineering skills and understanding-in-use. Finally, we describe the time needed to score the assessments, and discuss its utility for larger-scale research studies.

Klassen, S. (2006). Contextual assessment in science education: Background, issues, and policy. Science Education, 90(5), 820–851.
NGSS Lead States. (2013). Next Generation Science Standards: For States, By States. Washington, DC: The National Academies Press.
AU - Cathy P. Lachapelle
AU - Christine M. Cunningham
CY - New Orleans, Louisiana
DA - 2016/06/26
PB - ASEE Conferences
TI - Performance Assessment in Elementary Engineering: Evaluating Student (RTP)
UR - https://peer.asee.org/25884
DO - 10.18260/p.25884
ER -