Asee peer logo

HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering

Download Paper |


2015 ASEE Annual Conference & Exposition


Seattle, Washington

Publication Date

June 14, 2015

Start Date

June 14, 2015

End Date

June 17, 2015





Conference Session

Fundamental: Metrics & Assessment for K-12 Engineering Education

Tagged Division

K-12 & Pre-College Engineering

Tagged Topic


Page Count


Page Numbers

26.848.1 - 26.848.19



Permanent URL

Download Count


Request a correction

Paper Authors


Cathy P. Lachapelle Museum of Science

visit author page

Cathy Lachapelle leads the EiE team responsible for assessment and evaluation of our curricula. This includes the design and field-testing of assessment instruments and research on how children use EiE materials. Cathy is particularly interested in how collaborative interaction and scaffolded experiences with disciplinary practices help children learn science, math, and engineering. Her work on other STEM education research projects includes the national Women's Experiences in College Engineering (WECE) study. Cathy received her S.B. in cognitive science from the Massachusetts Institute of Technology and her Ph.D. in educational psychology from Stanford University.

visit author page


Yoonkyung Oh Pennsylvania State University

visit author page

Yoonkyung Oh is a research associate in the College of Education at Pennsylvania State University. She received her Ph.D. in educational policy from University of Wisconsin-Madison. Her research focuses on investigating family, school, and community as contexts for children’s education and development. She is interested in applying experimental, quasi-experimental, and longitudinal research methods to understand the effects of educational practices, policies, and interventions.

visit author page


Muhammad Faiz Shams Museum of Science

visit author page

Muhammad Shams is a Research Associate who has been with the Museum of Science, Engineering is Elementary team for almost 2 years. He graduated from the University of Massachusetts Dartmouth with a Bachelors of Science in Mathematics. Prior to his time with EiE, Muhammad worked as an undergraduate researcher in the field of applied numerical analysis. Muhammad assists the team with quantitative analysis, paper writing, and database management.

visit author page


Jonathan D Hertel EiE/Museum of Science

visit author page

Jonathan manages the Examining the Efficacy of Engineering is Elementary (E4) project (an NSF-funded study of the efficacy of the EiE curriculum), overseeing and organizing a research effort that involves 240 teachers in the different states. He also provides evaluation support for the Engineering Adventures and Engineering Everywhere projects. He holds an Ed.M. in learning and teaching from the Harvard Graduate School of Education. In 2013-2014, he was named a CADRE Fellow as part of the NSF Community for Advancing Discovery Research in Education (CADRE) project.

visit author page


Christine M Cunningham Museum of Science

visit author page

Dr. Christine Cunningham is an educational researcher who works to make engineering and science more relevant, accessible, and understandable, especially for underserved and underrepresented populations. A vice president at the Museum of Science, Boston since 2003, she founded and directs Engineering is Elementary™, a groundbreaking project that integrates engineering concepts into elementary curriculum and teacher professional development. As of September 2014, EiE has served 6.2 million children nationwide and 71,000 educators. Cunningham has previously served as director of engineering education research at the Tufts University Center for Engineering Educational Outreach, where her work focused on integrating engineering with science, technology, and math in professional development for K-12 teachers. She also directed the Women’s Experiences in College Engineering (WECE) project, the first national, longitudinal, large-scale study of the factors that support young women pursuing engineering degrees. Cunningham is a Fellow of the American Society for Engineering Education and was awarded the 2014 International Society for Design and Development in Education Prize. She holds B.A. and M.A. degrees in biology from Yale and a Ph.D. in Science Education from Cornell University.

visit author page

Download Paper |


HLM modeling of pre/post-assessment results from a large-scale efficacy study of elementaryengineering (Curriculum Evaluation)As engineering enters K-12 classrooms, it is important that curricula and activities engage students andfoster student learning of engineering concepts. This paper reports on an efficacy study that isexamining the effects of a set of critical curriculum design components on student learning ofengineering and science concepts at the elementary level. Critical components include that: (a)engineering content is introduced in a context, (b) students learn about and use the engineering designprocess, (c) engineering challenges specify a challenge and constraints and permit many possiblesolutions, (d) children use math and science to design solutions, (e) children use failure constructivelyand design iteratively, and (e) students work collaboratively. These components are present in thetreatment curriculum. The comparison curriculum has the same learning objectives but does notembody these principles.Our large-scale research study involves ~250 teachers from 150 schools in 3 regions (Massachusetts,Maryland, and North Carolina). Teachers were randomly assigned (at the school level) to treatment andcomparison conditions. After engaging in a curriculum-specific summer professional developmentworkshop, teachers implemented a curriculum unit which was approximately two weeks in length withtheir students, who were in grades 3-5. A large number of assessments, surveys, observations, and videodata were collected; those that will inform this paper include: • Student demographic data • Student pre-/post-assessments that probe their knowledge of engineering concepts and science concepts • Teacher instruction logs that report the content of lessonsWe are using an ANCOVA model implemented using Hierarchical Linear Modeling (HLM, Bryk &Raudenbush, 1992) to measure effect sizes of the treatment and moderating variables on student post-assessment scores. At Level 1, we model student data; at Level 2, we model classroom variables; and atLevel 3, we model the school-level variables.In this paper, we will present the models we have developed using HLM, including the range ofmoderators included in the model, and why we have included them. We will explain the advantages ofusing HLM to examine data from a large number of students, and we will interpret HLM output.Mostimportantly, we will describe the results of our statistical model. A preliminary round of analysis foundthat the treatment curriculum, student socio-economic status (SES), and student prior knowledge affectstudent outcomes. Therefore we expect the analysis we are currently undertaking to indicate that: • use of the treatment curriculum, as compared to the control curriculum, improves student understanding of science and engineering, • student SES negatively affects student outcomes, and • student prior knowledge as assessed by the pre-assessment predicts student outcomes.Finally, we expect to report that school factors such as the percent of low-income students in the schoolare important moderators of student outcomes.Our study is one of the first large-scale efficacy study of an engineering curriculum. The role that thecritical components play in student learning is important and should guide the work of other curriculumdesigners.Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.

Lachapelle, C. P., & Oh, Y., & Shams, M. F., & Hertel, J. D., & Cunningham, C. M. (2015, June), HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24185

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015