June 14, 2015
June 14, 2015
June 17, 2015
K-12 & Pre-College Engineering
26.848.1 - 26.848.19
HLM modeling of pre/post-assessment results from a large-scale efficacy study of elementaryengineering (Curriculum Evaluation)As engineering enters K-12 classrooms, it is important that curricula and activities engage students andfoster student learning of engineering concepts. This paper reports on an efficacy study that isexamining the effects of a set of critical curriculum design components on student learning ofengineering and science concepts at the elementary level. Critical components include that: (a)engineering content is introduced in a context, (b) students learn about and use the engineering designprocess, (c) engineering challenges specify a challenge and constraints and permit many possiblesolutions, (d) children use math and science to design solutions, (e) children use failure constructivelyand design iteratively, and (e) students work collaboratively. These components are present in thetreatment curriculum. The comparison curriculum has the same learning objectives but does notembody these principles.Our large-scale research study involves ~250 teachers from 150 schools in 3 regions (Massachusetts,Maryland, and North Carolina). Teachers were randomly assigned (at the school level) to treatment andcomparison conditions. After engaging in a curriculum-specific summer professional developmentworkshop, teachers implemented a curriculum unit which was approximately two weeks in length withtheir students, who were in grades 3-5. A large number of assessments, surveys, observations, and videodata were collected; those that will inform this paper include: • Student demographic data • Student pre-/post-assessments that probe their knowledge of engineering concepts and science concepts • Teacher instruction logs that report the content of lessonsWe are using an ANCOVA model implemented using Hierarchical Linear Modeling (HLM, Bryk &Raudenbush, 1992) to measure effect sizes of the treatment and moderating variables on student post-assessment scores. At Level 1, we model student data; at Level 2, we model classroom variables; and atLevel 3, we model the school-level variables.In this paper, we will present the models we have developed using HLM, including the range ofmoderators included in the model, and why we have included them. We will explain the advantages ofusing HLM to examine data from a large number of students, and we will interpret HLM output.Mostimportantly, we will describe the results of our statistical model. A preliminary round of analysis foundthat the treatment curriculum, student socio-economic status (SES), and student prior knowledge affectstudent outcomes. Therefore we expect the analysis we are currently undertaking to indicate that: • use of the treatment curriculum, as compared to the control curriculum, improves student understanding of science and engineering, • student SES negatively affects student outcomes, and • student prior knowledge as assessed by the pre-assessment predicts student outcomes.Finally, we expect to report that school factors such as the percent of low-income students in the schoolare important moderators of student outcomes.Our study is one of the first large-scale efficacy study of an engineering curriculum. The role that thecritical components play in student learning is important and should guide the work of other curriculumdesigners.Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
Lachapelle, C. P., & Oh, Y., & Shams, M. F., & Hertel, J. D., & Cunningham, C. M. (2015, June), HLM Modeling of Pre/Post-assessment Results from a Large-scale Efficacy Study of Elementary Engineering Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24185
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015