Process Evaluation: The Vital (and Usually) Missing Piece in Educational Research

Rebecca Brent

Download Paper | Permalink

Conference: 2012 ASEE Annual Conference & Exposition
Location: San Antonio, Texas
Publication Date: June 10, 2012
Start Date: June 10, 2012
End Date: June 13, 2012
ISSN: 2153-5965
Conference Session: Research in Engineering Education I
Tagged Division: Educational Research and Methods
Page Count: 11
Page Numbers: 25.1068.1 - 25.1068.11
DOI: 10.18260/1-2--21825
Permanent URL: https://peer.asee.org/21825
Download Count: 324

Request a correction

Paper Authors

biography

Rebecca Brent Education Designs, Inc.

visit author page

Rebecca Brent is President of Education Designs, Inc., a consulting firm located in Cary, N.C. She is a faculty development and evaluation consultant for the College of Engineering at North Carolina State University and Co-director of the National Effective Teaching Institute sponsored by the American Society for Engineering Education. Brent received her B.A. from Millsaps College in Jackson, Miss., her M.Ed. from Mississippi State University, and her Ed.D. from Auburn University. She was an Associate Professor of education at East Carolina University before starting her consulting firm in 1996.

visit author page

Download Paper | Permalink

Abstract

Process Evaluation: The Vital (and Usually) Missing Piece in Educational ResearchMost large educational research projects are challenging to manage. Activities of many peoplemust be coordinated, and unanticipated problems commonly arise at every turn. Project directorsnormally deal with the challenges by trial-and-error. Unfortunately, it can take half or more ofthe term of the grant or contract for them to figure out how to make things work as proposed, andsometimes they never figure it out.There is a better way, known as process or implementation evaluation, in which a trainedevaluator is brought into a project from its inception, continually assesses the projectimplementation, and provides feedback that can keep the project from going off the rails andgreatly improve the chances that it will achieve its goals. Most researchers are familiar with theneed to determine the impact of a completed project, which in the evaluation community istermed impact or outcome evaluation, but the existence of process evaluation and the benefits itcan provide are still relatively unknown in the engineering education research community.Process evaluation can have many different structures, with the most appropriate one dependingon the goals and nature of the research project, but several elements are commonly involved: • Project planning. When the grant proposal is being written and/or at the start of the funding period, a process evaluator can work with the project leadership team to develop a logic model that includes major activities, personnel, and short-, medium-, and long- term objectives. The model helps the team uncover the underlying model of change that underlies the research. Development of the logic model can help to get all the leaders on the same page regarding the project implementation and assessment, and the model provides a framework for quickly identifying causes and remedies when things do not go as planned. • Leadership team functioning. The evaluator participates in team meetings as an observer, gathers data about patterns of communication, problem-solving, and conflict resolution, and provides feedback on how things are going and ideas for helping the team to work more effectively. • Implementation. The evaluator gathers data from leaders and participants in project events and activities and directly observes the activities, and provides feedback to the leadership team about how effectively the activities are leading to the desired goals and outcomes and how they might be improved. • Outcome/Impact Evaluation. The evaluator works closely with whoever conducts outcome or impact evaluation, sharing information gathered in the course of the three preceding functions. Such information often provides answers to the questions of why and how the project results came about.This paper defines and describes process evaluation, reviews different forms it can take,summarizes its potential benefits (which include increasing the likelihood of getting funding fora proposal by including a process evaluation component in it), and recommends qualities to lookfor in a process evaluator and a structure for identifying and working with the evaluator to getthe maximum benefit from the evaluation. A case study of a three-year ADVANCE PAID grantprogram will be used to illustrate every aspect of process evaluation.

Citation
Format

Brent, R. (2012, June), Process Evaluation: The Vital (and Usually) Missing Piece in Educational Research Paper presented at 2012 ASEE Annual Conference & Exposition, San Antonio, Texas. 10.18260/1-2--21825

TY  - CPAPER
AB  -     Process Evaluation: The Vital (and Usually) Missing Piece in Educational ResearchMost large educational research projects are challenging to manage. Activities of many peoplemust be coordinated, and unanticipated problems commonly arise at every turn. Project directorsnormally deal with the challenges by trial-and-error. Unfortunately, it can take half or more ofthe term of the grant or contract for them to figure out how to make things work as proposed, andsometimes they never figure it out.There is a better way, known as process or implementation evaluation, in which a trainedevaluator is brought into a project from its inception, continually assesses the projectimplementation, and provides feedback that can keep the project from going off the rails andgreatly improve the chances that it will achieve its goals. Most researchers are familiar with theneed to determine the impact of a completed project, which in the evaluation community istermed impact or outcome evaluation, but the existence of process evaluation and the benefits itcan provide are still relatively unknown in the engineering education research community.Process evaluation can have many different structures, with the most appropriate one dependingon the goals and nature of the research project, but several elements are commonly involved:   •   Project planning. When the grant proposal is being written and/or at the start of the       funding period, a process evaluator can work with the project leadership team to develop       a logic model that includes major activities, personnel, and short-, medium-, and long-       term objectives. The model helps the team uncover the underlying model of change that       underlies the research. Development of the logic model can help to get all the leaders on       the same page regarding the project implementation and assessment, and the model       provides a framework for quickly identifying causes and remedies when things do not go       as planned.   •   Leadership team functioning. The evaluator participates in team meetings as an       observer, gathers data about patterns of communication, problem-solving, and conflict       resolution, and provides feedback on how things are going and ideas for helping the team       to work more effectively.   •   Implementation. The evaluator gathers data from leaders and participants in project       events and activities and directly observes the activities, and provides feedback to the       leadership team about how effectively the activities are leading to the desired goals and       outcomes and how they might be improved.   •   Outcome/Impact Evaluation. The evaluator works closely with whoever conducts       outcome or impact evaluation, sharing information gathered in the course of the three       preceding functions. Such information often provides answers to the questions of why       and how the project results came about.This paper defines and describes process evaluation, reviews different forms it can take,summarizes its potential benefits (which include increasing the likelihood of getting funding fora proposal by including a process evaluation component in it), and recommends qualities to lookfor in a process evaluator and a structure for identifying and working with the evaluator to getthe maximum benefit from the evaluation. A case study of a three-year ADVANCE PAID grantprogram will be used to illustrate every aspect of process evaluation.
AU  - Rebecca Brent
CY  - San Antonio, Texas
DA  - 2012/06/10
PB  - ASEE Conferences
TI  - Process Evaluation: The Vital (and Usually) Missing Piece in Educational Research
UR  - https://peer.asee.org/21825
DO  - 10.18260/1-2--21825
ER  -