Asee peer logo

Developing a Measure to Capture Middle School Students’ Interpretive Understanding of Engineering Design

Download Paper |

Conference

2021 ASEE Virtual Annual Conference Content Access

Location

Virtual Conference

Publication Date

July 26, 2021

Start Date

July 26, 2021

End Date

July 19, 2022

Conference Session

Assessing Hard-to-Measure Constructs in Engineering Education: Assessment Design and Validation Studies

Tagged Division

Educational Research and Methods

Page Count

25

DOI

10.18260/1-2--36934

Permanent URL

https://peer.asee.org/36934

Download Count

50

Request a correction

Paper Authors

biography

Jeremiah Pina Smith College

visit author page

Jeremiah Pina is a postdoctoral researcher at Smith College. He received a B.S. in Psychology and a Ph.D. in Educational Psychology from the University of Georgia. His current research focuses on developing alternative assessment methods for use in the pre-college STEM education settings.

visit author page

biography

Glenn W. Ellis Smith College

visit author page

Glenn Ellis is a Professor of Engineering at Smith College who teaches courses in engineering science and methods for teaching science and engineering. He received a B.S. in Civil Engineering from Lehigh University and an M.A. and Ph.D. in Civil Engineering and Operations Research from Princeton University. The winner of numerous teaching and research awards, Dr. Ellis received the 2007 U.S. Professor of the Year Award for Baccalaureate Colleges from the Carnegie Foundation for the Advancement of Teaching and the Council for Advancement and Support of Education. His research focuses on creating K-16 learning environments that support the growth of learners’ imaginations and their capacity for engaging in collaborative knowledge work.

visit author page

biography

Al Rudnitsky Smith College

visit author page

Al Rudnitsky teaches Introduction to the Learning Sciences; Thinking, Knowing and the Design of Learning Environments, How Do We Know What Students are Learning?, and instructional methods in elementary and middle school mathematics and science. He has authored books on curriculum design and teaching children about scientific inquiry. Current research interests focus on creating environments for “good talk” in elementary and middle school classrooms, and also on advancing the use of knowledge building pedagogy in higher education. His most recent article (2013) is entitled “Tasks and Talk: The Relationship Between Teachers’ Goals and Student Discourse,” in Social Studies Research and Practice.

visit author page

author page

Rebecca Mazur Collaborative for Educational Services Orcid 16x16 orcid.org/0000-0001-9521-9192

biography

Beth McGinnis-Cavanaugh Springfield Technical Community College

visit author page

Beth McGinnis-Cavanaugh is a professor at Springfield Technical Community College, where she teaches courses in physics, engineering mechanics, and structures. A graduate of the engineering transfer program at STCC, McGinnis-Cavanaugh holds a B.S. and M.S. in Civil Engineering from the University of Massachusetts Amherst. She focuses on developing meaningful educational strategies to recruit and retain a diverse student body in engineering and designs innovative learning environments at all levels of the engineering pipeline. Her work in these areas is particularly focused on full inclusion and equity for community college women in engineering and related STEM fields.

visit author page

biography

Isabel Huff Springfield Technical Community College

visit author page

Isabel is a curriculum designer and training specialist for Transforming Engineering Education for Middle Schools (TEEMS). She joined the “TEEMS team” in 2010 as a Smith College undergraduate. After spending eight years developing her love of storytelling in education and passion for encouraging women in STEM, Isabel took a one-year leave to get a master’s degree in education and technology at Stanford. Since returning, her work has focused more heavily on design and teacher professional development.

visit author page

Download Paper |

Abstract

By emphasizing the engineering design process as a central concept of the subject area, the Next Generation Science Standards (NGSS) have created an impetus for advancing a model of K-12 engineering education that supports opportunities for students to meaningfully engage with the idea and practice of iterative design. Assessing meaningful learning in the K-12 context, however, is a challenge, as traditional engineering assessment practices are rarely able to measure or contribute to high levels of cognitive engagement.

One aspect of meaningful learning is that it incorporates assessments that ask students to go beyond recalling information by rote, and instead invites them to engage in constructing deeper understanding (Mayer, 2002). Doing so requires instruments that can get at deeper levels of students’ capacity to think with, or interpret, engineering concepts rather than remembering information or applying fixed procedures to situations similar to those they encountered during instruction.

This paper describes our effort to design and develop one such instrument using the theories and frameworks of preparation for future learning (Schwartz, Bransford, & Sears, 2005) and adaptive design expertise (Neeley, 2007).

Starting with an analysis of the assessment items available to teachers in our partner school system, we begin by unpacking how traditional engagement with engineering concepts through assessment creates conditions that perpetuate common pitfalls in pre-college engineering education. We then describe our own process of developing test items, at first as an attempt to accommodate the principles of traditional assessment, creating at total of 17 items across three separate item pools, and increasingly gravitating toward the performance assessment paradigm, finally honing in on three performance assessment tasks.

We then provide the results of a study carried out using assessment results generated by our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non-treatment (n = 241) groups, wherein the former participated in a two-week, NGSS-aligned unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach (Egan, 2005). The latter group were taught using the district’s existing engineering design curriculum.

A series of non-parametric comparative analyses (independent two-group Mann-Whitney tests) found statistically significant differences in the pattern of student responses related to (1) fluency, (2) reflective ideation, and (3) figural elaboration of the performance assessment tasks. These results may shed light on ways that innovating engineering assessment can both contribute to and provide insight about students’ capacity to generate ideas, reflect, and demonstrate their understanding of engineering design. The utility of these findings are further discussed in terms of their relevance to teachers, researchers, curriculum designers, and other stakeholders.

Pina, J., & Ellis, G. W., & Rudnitsky, A., & Mazur, R., & McGinnis-Cavanaugh, B., & Huff, I. (2021, July), Developing a Measure to Capture Middle School Students’ Interpretive Understanding of Engineering Design Paper presented at 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference. 10.18260/1-2--36934

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2021 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015