Lachapelle, C. P., & Oh, Y., & Shams, M. F., & Hertel, J. D., & Cunningham, C. M. (2015, June), HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering  Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24185

\bibitem{asee_peer_24185}
  Lachapelle, C. P., \& Oh, Y., \& Shams, M. F., \& Hertel, J. D., \& Cunningham, C. M. (2015, June), \emph{HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering}
  Paper presented at 2015 ASEE Annual Conference \& Exposition, Seattle, Washington.
  10.18260/p.24185

Cathy P. Lachapelle, Yoonkyung Oh, Muhammad Faiz Shams, Jonathan D Hertel, and Christine M Cunningham.  "HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering".  2015 ASEE Annual Conference & Exposition, Seattle, Washington, 2015, June.  ASEE Conferences, 2015.  https://peer.asee.org/24185 Internet.  02 May, 2024

\bibitem{asee_peer_24185}
  Cathy P. Lachapelle, Yoonkyung Oh, Muhammad Faiz Shams, Jonathan D Hertel, and Christine M Cunningham.
  "HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering".
  \emph{2015 ASEE Annual Conference \& Exposition, Seattle, Washington, 2015, June}.
  ASEE Conferences, 2015.
  https://peer.asee.org/24185 Internet.  02 May, 2024

@INPROCEEDINGS{asee_peer_24185
author = "Cathy P. Lachapelle, Yoonkyung Oh, Muhammad Faiz Shams, Jonathan D Hertel, and Christine M Cunningham"
title = "HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering"
booktitle = "2015 ASEE Annual Conference \& Exposition"
year = "2015"
month = "June"
address = "Seattle, Washington"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/24185}
number = {10.18260/p.24185}
}

TY  - CPAPER
AB  - HLM modeling of pre/post-assessment results from a large-scale efficacy study of elementaryengineering (Curriculum Evaluation)As engineering enters K-12 classrooms, it is important that curricula and activities engage students andfoster student learning of engineering concepts. This paper reports on an efficacy study that isexamining the effects of a set of critical curriculum design components on student learning ofengineering and science concepts at the elementary level. Critical components include that: (a)engineering content is introduced in a context, (b) students learn about and use the engineering designprocess, (c) engineering challenges specify a challenge and constraints and permit many possiblesolutions, (d) children use math and science to design solutions, (e) children use failure constructivelyand design iteratively, and (e) students work collaboratively. These components are present in thetreatment curriculum. The comparison curriculum has the same learning objectives but does notembody these principles.Our large-scale research study involves ~250 teachers from 150 schools in 3 regions (Massachusetts,Maryland, and North Carolina). Teachers were randomly assigned (at the school level) to treatment andcomparison conditions. After engaging in a curriculum-specific summer professional developmentworkshop, teachers implemented a curriculum unit which was approximately two weeks in length withtheir students, who were in grades 3-5. A large number of assessments, surveys, observations, and videodata were collected; those that will inform this paper include:    •   Student demographic data    •   Student pre-/post-assessments that probe their knowledge of engineering concepts and science        concepts    •   Teacher instruction logs that report the content of lessonsWe are using an ANCOVA model implemented using Hierarchical Linear Modeling (HLM, Bryk &Raudenbush, 1992) to measure effect sizes of the treatment and moderating variables on student post-assessment scores. At Level 1, we model student data; at Level 2, we model classroom variables; and atLevel 3, we model the school-level variables.In this paper, we will present the models we have developed using HLM, including the range ofmoderators included in the model, and why we have included them. We will explain the advantages ofusing HLM to examine data from a large number of students, and we will interpret HLM output.Mostimportantly, we will describe the results of our statistical model. A preliminary round of analysis foundthat the treatment curriculum, student socio-economic status (SES), and student prior knowledge affectstudent outcomes. Therefore we expect the analysis we are currently undertaking to indicate that:    •   use of the treatment curriculum, as compared to the control curriculum, improves student        understanding of science and engineering,    •   student SES negatively affects student outcomes, and    •   student prior knowledge as assessed by the pre-assessment predicts student outcomes.Finally, we expect to report that school factors such as the percent of low-income students in the schoolare important moderators of student outcomes.Our study is one of the first large-scale efficacy study of an engineering curriculum. The role that thecritical components play in student learning is important and should guide the work of other curriculumdesigners.Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis         methods. Newbury Park, CA: Sage.
AU  - Cathy P. Lachapelle
AU  - Yoonkyung Oh
AU  - Muhammad Faiz Shams
AU  - Jonathan D Hertel
AU  - Christine M Cunningham
CY  - Seattle, Washington
DA  - 2015/06/14
PB  - ASEE Conferences
TI  - HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering
UR  - https://peer.asee.org/24185
DO  - 10.18260/p.24185
ER  -