Evaluating Student Responses in Open-Ended Problems Involving Iterative Solution Development in Model-Eliciting Activities

Mark T. Carnes; Heidi A. Diefes-Dux; Monica E. Cardella

Download Paper | Permalink

Conference: 2011 ASEE Annual Conference & Exposition
Location: Vancouver, BC
Publication Date: June 26, 2011
Start Date: June 26, 2011
End Date: June 29, 2011
ISSN: 2153-5965
Conference Session: Open-Ended Problems and Student Learning
Tagged Division: Educational Research and Methods
Page Count: 26
Page Numbers: 22.647.1 - 22.647.26
DOI: 10.18260/1-2--17928
Permanent URL: https://peer.asee.org/17928
Download Count: 563

Paper Authors

biography

Mark T. Carnes Purdue University, West Lafayette

visit author page

Mark Carnes is a licensed Professional Engineer (PE) and is currently a doctoral student and a future faculty fellow in the School of Engineering Education at Purdue University. Before coming to Purdue, he spent over 30 years as an electronics designer of control and power conversion circuits. He received an M.S. from the University of Michigan (1982) and a B.S. from the University of Notre Dame (1975), both in Electrical Engineering.

visit author page

biography

Heidi A. Diefes-Dux Purdue University, West Lafayette orcid.org/0000-0003-3635-1825

visit author page

Heidi Diefes-Dux is an Associate Professor in the School of Engineering Education at Purdue University. She received her B.S. and M.S. in Food Science from Cornell University and her Ph.D. in Food Process Engineering from the Department of Agricultural and Biological Engineering at Purdue University. Since 1999, she has been a faculty member in Purdue’s First-Year Engineering Program, the gateway for all first-year students entering the College of Engineering. She is currently the Director of Teacher Professional Development for the Institute for P-12 Engineering Research and Learning (INSPIRE). Her research interests center on implementation and assessment of mathematical modeling problems.

visit author page

biography

Monica E. Cardella Purdue University, West Lafayette orcid.org/0000-0002-4229-6183

visit author page

Monica E. Cardella is an Assistant Professor of Engineering Education and is the Co-Director of Assessment Research for the Institute for P-12 Engineering Research and Learning (INSPIRE) at Purdue University. Dr. Cardella earned a B.Sc. in Mathematics from the University of Puget Sound and an M.S. and Ph.D. in Industrial Engineering at the University of Washington. At the University of Washington she worked with the Center for Engineering Learning and Teaching (CELT) and the LIFE Center (Learning in Informal and Formal Environments). She was a CASEE Postdoctoral Engineering Education Researcher at the Center for Design Research at Stanford before beginning her appointment at Purdue. Her research interests include: learning in informal and out-of-school time settings, pre-college engineering education, design thinking, mathematical thinking, and assessment research.

visit author page

Download Paper | Permalink

Abstract

Evaluating Student Responses in Open-Ended Problems Involving Iterative Solution DevelopmentOpen-ended problems are an important part of the engineering curriculum because, when welldesigned, they closely resemble problem-solving situations students will encounter asprofessional engineers. However, valid and reliable evaluation of student performance on open-ended problems is a challenge given that numerous reasonable responses are likely to exist for agiven problem and multiple instructors and peers may be evaluating student work. In previouswork, evaluation tools (including a rubric, task-specific supports, and scorer training) for open-ended problems, specifically Model-Eliciting Activities (MEAs), were developed using threeeducational research perspectives: the models and modeling perspective, design researchmethodology, and multi-tiered teaching experiment methodology. This ensured that theevaluation tools evolve with fidelity to the characteristics of high performance that professionalscare about and with increased reliability.One of the features of the MEA implementation that models real-world engineering problemsolving is the iterative nature of solution development. This study uses the previously developedevaluation tools to examine the changes the student teams made between solution iterations thatwould indicate that they were making progress towards higher quality solutions over the courseof these iterations. The Just-in-Time Manufacturing MEA was implemented in spring 2009 in alarge first-year engineering course. Each student team developed three versions of their solution:First Draft, Second Draft, and Team Final. Between the versions, the students received peerfeedback, experienced problem updates in the form of different/additional data, and received TAfeedback. From all student team work submitted, a sample of 50 teams was randomly selectedfor this study. Each of the teams’ three versions was rigorously scored and coded by anengineering expert using the previously developed evaluation tools which included a fourdimension generic MEA Rubric and JIT MEA specific assessment supports. The fourdimensions were: quality of the mathematical model, share-ability, re-usability, andmodifiability. The expert scores were then compared across the three versions, showing apositive trend of improvement in student performance across all dimensions. The largest gainswere seen in the aspect of share-ability which refers to the clarity of the actual procedure, suchthat the user can easily replicate results. These findings have implications for instruction alongeach dimension. These findings also provide opportunities to investigate the nature of peer andTA feedback that may (or may not) have resulted in change on a given iteration.

Citation
Format

Carnes, M. T., & Diefes-Dux, H. A., & Cardella, M. E. (2011, June), Evaluating Student Responses in Open-Ended Problems Involving Iterative Solution Development in Model-Eliciting Activities Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. 10.18260/1-2--17928

TY - CPAPER
AB - Evaluating Student Responses in Open-Ended Problems Involving Iterative Solution DevelopmentOpen-ended problems are an important part of the engineering curriculum because, when welldesigned, they closely resemble problem-solving situations students will encounter asprofessional engineers. However, valid and reliable evaluation of student performance on open-ended problems is a challenge given that numerous reasonable responses are likely to exist for agiven problem and multiple instructors and peers may be evaluating student work. In previouswork, evaluation tools (including a rubric, task-specific supports, and scorer training) for open-ended problems, specifically Model-Eliciting Activities (MEAs), were developed using threeeducational research perspectives: the models and modeling perspective, design researchmethodology, and multi-tiered teaching experiment methodology. This ensured that theevaluation tools evolve with fidelity to the characteristics of high performance that professionalscare about and with increased reliability.One of the features of the MEA implementation that models real-world engineering problemsolving is the iterative nature of solution development. This study uses the previously developedevaluation tools to examine the changes the student teams made between solution iterations thatwould indicate that they were making progress towards higher quality solutions over the courseof these iterations. The Just-in-Time Manufacturing MEA was implemented in spring 2009 in alarge first-year engineering course. Each student team developed three versions of their solution:First Draft, Second Draft, and Team Final. Between the versions, the students received peerfeedback, experienced problem updates in the form of different/additional data, and received TAfeedback. From all student team work submitted, a sample of 50 teams was randomly selectedfor this study. Each of the teams’ three versions was rigorously scored and coded by anengineering expert using the previously developed evaluation tools which included a fourdimension generic MEA Rubric and JIT MEA specific assessment supports. The fourdimensions were: quality of the mathematical model, share-ability, re-usability, andmodifiability. The expert scores were then compared across the three versions, showing apositive trend of improvement in student performance across all dimensions. The largest gainswere seen in the aspect of share-ability which refers to the clarity of the actual procedure, suchthat the user can easily replicate results. These findings have implications for instruction alongeach dimension. These findings also provide opportunities to investigate the nature of peer andTA feedback that may (or may not) have resulted in change on a given iteration.
AU - Mark T. Carnes
AU - Heidi A. Diefes-Dux
AU - Monica E. Cardella
CY - Vancouver, BC
DA - 2011/06/26
PB - ASEE Conferences
TI - Evaluating Student Responses in Open-Ended Problems Involving Iterative Solution Development in Model-Eliciting Activities
UR - https://peer.asee.org/17928
DO - 10.18260/1-2--17928
ER -