Asee peer logo

Process Evaluation: The Vital (and Usually) Missing Piece in Educational Research

Download Paper |

Conference

2012 ASEE Annual Conference & Exposition

Location

San Antonio, Texas

Publication Date

June 10, 2012

Start Date

June 10, 2012

End Date

June 13, 2012

ISSN

2153-5965

Conference Session

Research in Engineering Education I

Tagged Division

Educational Research and Methods

Page Count

11

Page Numbers

25.1068.1 - 25.1068.11

DOI

10.18260/1-2--21825

Permanent URL

https://peer.asee.org/21825

Download Count

377

Paper Authors

biography

Rebecca Brent Education Designs, Inc.

visit author page

Rebecca Brent is President of Education Designs, Inc., a consulting firm located in Cary, N.C. She is a faculty development and evaluation consultant for the College of Engineering at North Carolina State University and Co-director of the National Effective Teaching Institute sponsored by the American Society for Engineering Education. Brent received her B.A. from Millsaps College in Jackson, Miss., her M.Ed. from Mississippi State University, and her Ed.D. from Auburn University. She was an Associate Professor of education at East Carolina University before starting her consulting firm in 1996.

visit author page

Download Paper |

Abstract

Process Evaluation: The Vital (and Usually) Missing Piece in Educational ResearchMost large educational research projects are challenging to manage. Activities of many peoplemust be coordinated, and unanticipated problems commonly arise at every turn. Project directorsnormally deal with the challenges by trial-and-error. Unfortunately, it can take half or more ofthe term of the grant or contract for them to figure out how to make things work as proposed, andsometimes they never figure it out.There is a better way, known as process or implementation evaluation, in which a trainedevaluator is brought into a project from its inception, continually assesses the projectimplementation, and provides feedback that can keep the project from going off the rails andgreatly improve the chances that it will achieve its goals. Most researchers are familiar with theneed to determine the impact of a completed project, which in the evaluation community istermed impact or outcome evaluation, but the existence of process evaluation and the benefits itcan provide are still relatively unknown in the engineering education research community.Process evaluation can have many different structures, with the most appropriate one dependingon the goals and nature of the research project, but several elements are commonly involved: • Project planning. When the grant proposal is being written and/or at the start of the funding period, a process evaluator can work with the project leadership team to develop a logic model that includes major activities, personnel, and short-, medium-, and long- term objectives. The model helps the team uncover the underlying model of change that underlies the research. Development of the logic model can help to get all the leaders on the same page regarding the project implementation and assessment, and the model provides a framework for quickly identifying causes and remedies when things do not go as planned. • Leadership team functioning. The evaluator participates in team meetings as an observer, gathers data about patterns of communication, problem-solving, and conflict resolution, and provides feedback on how things are going and ideas for helping the team to work more effectively. • Implementation. The evaluator gathers data from leaders and participants in project events and activities and directly observes the activities, and provides feedback to the leadership team about how effectively the activities are leading to the desired goals and outcomes and how they might be improved. • Outcome/Impact Evaluation. The evaluator works closely with whoever conducts outcome or impact evaluation, sharing information gathered in the course of the three preceding functions. Such information often provides answers to the questions of why and how the project results came about.This paper defines and describes process evaluation, reviews different forms it can take,summarizes its potential benefits (which include increasing the likelihood of getting funding fora proposal by including a process evaluation component in it), and recommends qualities to lookfor in a process evaluator and a structure for identifying and working with the evaluator to getthe maximum benefit from the evaluation. A case study of a three-year ADVANCE PAID grantprogram will be used to illustrate every aspect of process evaluation.

Brent, R. (2012, June), Process Evaluation: The Vital (and Usually) Missing Piece in Educational Research Paper presented at 2012 ASEE Annual Conference & Exposition, San Antonio, Texas. 10.18260/1-2--21825

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2012 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015