Evaluating the Pre-Professional Engineer: Exploring the Peer Review Process

Joy M. Adams; Mical D. DeGraaff; Gail S Hohner

Download Paper | Permalink

Conference: 2015 ASEE Annual Conference & Exposition
Location: Seattle, Washington
Publication Date: June 14, 2015
Start Date: June 14, 2015
End Date: June 17, 2015
ISBN: 978-0-692-50180-1
ISSN: 2153-5965
Conference Session: Design as a Social Process: Teams and Organizations
Tagged Division: Design in Engineering Education
Page Count: 23
Page Numbers: 26.692.1 - 26.692.23
DOI: 10.18260/p.24029
Permanent URL: https://peer.asee.org/24029
Download Count: 793

Paper Authors

biography

Joy M. Adams University of Michigan

visit author page

Joy Adams is the Program Manager for the Multidisciplinary Design Program at the University of Michigan. In this role, she focuses on Corporate Sponsored Projects, Communications and Student Performance Appraisals. She has seven years of diverse professional Human Resources experience, including prior roles in Training & Development, Campus Recruiting and Talent Management/Leadership Development at various Fortune 500 firms.

visit author page

biography

Mical D. DeGraaff University of Michigan

visit author page

Mical DeGraaff is a recent graduate of the University of Michigan, where she received her Master of Arts in Higher Education Administration. As a student, Mical worked as a Research Assistant for the University of Michigan College of Engineering's Multidisciplinary Design Program. Her research interests include student involvement and engaged learning.

visit author page

biography

Gail S Hohner University of Michigan

visit author page

Gail Hohner is the Managing Director of the Multidisciplinary Design Program in the College of Engineering at the University of Michigan Ann Arbor, where she develops multidisciplinary engaged learning experiences in the engineering design process. She teaches the seminar in Leadership/Mentorship in Multidisciplinary Engineering Design and her research focuses on the improvements in the pedagogy of engineering design process instruction. She is the 2016 program chair of the DEED division of ASEE. She has a background of 17 years of industrial experience and holds B.S.E in Chemical Engineering and a M.S. in Food Science/Chemical Engineering from Cornell.

visit author page

Download Paper | Permalink

Abstract

Evaluating the Pre-Professional Engineer: Project Team and Individual PerformanceAt a large midwestern research university, teams of 5-7 engineers from multiple disciplines worktogether on multi-term engineering design projects. Annually, the program enrolls approximately220 students on engineering design project teams. As we prepare to enter the fifth year of thisprogram, we are studying the effectiveness of scalable administration of effective peer reviewfeedback. Currently, we collect, anonymize and disseminate performance feedback utilizing theCATME team peer evaluation tool (Loughry, Ohland, & DeWayne, 2007) twice during thetenure of each team’s project. We observed that our students were unable to sufficientlyinterpret the feedback data, and their reactions to the feedback from their peers ranged fromcomplete dismissal to an inability to integrate the feedback into their performance. As a result,this has not generated our desired outcomes: improved project-specific engineering design skills,professional behavior, or evidence of self-reflection. To address this, we have created andimplemented an additional avenue for peer-to-peer anonymized feedback: a qualitative surveyutilizing coded competencies drawn from various industry-based best practices. In thissupplemental qualitative survey, students select two performance competencies (one positive andone developmental) from a predefined list for each of their peers. Students then support theselected competencies by citing specific behavioral examples.We coded these statements into three skill constructs, professionalism, teamwork, and core skills,using a unique combination of structural, attribute, provisional, and holistic qualitative codingtechniques. The coded data is analyzed against semi-anonymized demographic data in order todetermine how various identities affect the ways in which students evaluate and are evaluated bytheir peers. Utilizing action research methodology, we made relevant changes to the surveysprior to the second round of evaluations to improve students’ understanding of the surveyquestions.Student growth is measured via peers’ quantitative (CATME) and qualitative evaluations ofengineering design skills and professional behavior, as well as the students’ own ability to craftconstructive feedback statements. Because specific instances of behavior are cited in thequalitative survey, students are better equipped to leverage their strengths and addressshortcomings. We identify trends in peer evaluations with regard to self-identified gender,residency status, academic year, and other categories. The research team documents the ways inwhich students evaluate one another in an experiential team setting, the roles that certainacademic and social identities play in that evaluation process, and how students’ evaluationabilities change over time.

Citation
Format

Adams, J. M., & DeGraaff, M. D., & Hohner, G. S. (2015, June), Evaluating the Pre-Professional Engineer: Exploring the Peer Review Process Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24029

TY  - CPAPER
AB  -            Evaluating the Pre-Professional Engineer: Project Team and                             Individual PerformanceAt a large midwestern research university, teams of 5-7 engineers from multiple disciplines worktogether on multi-term engineering design projects. Annually, the program enrolls approximately220 students on engineering design project teams. As we prepare to enter the fifth year of thisprogram, we are studying the effectiveness of scalable administration of effective peer reviewfeedback. Currently, we collect, anonymize and disseminate performance feedback utilizing theCATME team peer evaluation tool (Loughry, Ohland, &amp; DeWayne, 2007) twice during thetenure of each team’s project. We observed that our students were unable to sufficientlyinterpret the feedback data, and their reactions to the feedback from their peers ranged fromcomplete dismissal to an inability to integrate the feedback into their performance. As a result,this has not generated our desired outcomes: improved project-specific engineering design skills,professional behavior, or evidence of self-reflection. To address this, we have created andimplemented an additional avenue for peer-to-peer anonymized feedback: a qualitative surveyutilizing coded competencies drawn from various industry-based best practices. In thissupplemental qualitative survey, students select two performance competencies (one positive andone developmental) from a predefined list for each of their peers. Students then support theselected competencies by citing specific behavioral examples.We coded these statements into three skill constructs, professionalism, teamwork, and core skills,using a unique combination of structural, attribute, provisional, and holistic qualitative codingtechniques. The coded data is analyzed against semi-anonymized demographic data in order todetermine how various identities affect the ways in which students evaluate and are evaluated bytheir peers. Utilizing action research methodology, we made relevant changes to the surveysprior to the second round of evaluations to improve students’ understanding of the surveyquestions.Student growth is measured via peers’ quantitative (CATME) and qualitative evaluations ofengineering design skills and professional behavior, as well as the students’ own ability to craftconstructive feedback statements. Because specific instances of behavior are cited in thequalitative survey, students are better equipped to leverage their strengths and addressshortcomings. We identify trends in peer evaluations with regard to self-identified gender,residency status, academic year, and other categories. The research team documents the ways inwhich students evaluate one another in an experiential team setting, the roles that certainacademic and social identities play in that evaluation process, and how students’ evaluationabilities change over time.
AU  - Joy M.  Adams
AU  - Mical D. DeGraaff
AU  - Gail S Hohner
CY  - Seattle, Washington
DA  - 2015/06/14
PB  - ASEE Conferences
TI  - Evaluating the Pre-Professional Engineer: Exploring the Peer Review Process
UR  - https://peer.asee.org/24029
DO  - 10.18260/p.24029
ER  -