Asee peer logo

The Comprehensive Assessment Of Team Member Effectiveness: A New Peer Evaluation Instrument

Download Paper |

Conference

2006 Annual Conference & Exposition

Location

Chicago, Illinois

Publication Date

June 18, 2006

Start Date

June 18, 2006

End Date

June 21, 2006

ISSN

2153-5965

Conference Session

NSF Grantees Poster Session

Tagged Division

Division Experimentation & Lab-Oriented Studies

Page Count

6

Page Numbers

11.1262.1 - 11.1262.6

DOI

10.18260/1-2--751

Permanent URL

https://peer.asee.org/751

Download Count

397

Request a correction

Paper Authors

biography

Hal R. Pomeranz Deer Run Associates, Inc.

visit author page

Hal R. Pomeranz is a computer network security and database programming consultant. He is a co-founder of Deer Run Associates, Inc., currently operating in Eugene, Oregon.

visit author page

biography

Harlan W. Feinstein Deer Run Associates, Inc.

visit author page

Harlan W. Feinstein is a consultant specializing a interface design for web pages. He is a subcontractor of Deer Run Associates, Inc.

visit author page

biography

Matthew Ohland Clemson University Orcid 16x16 orcid.org/0000-0003-4052-1452

visit author page

Matthew W. Ohland is an Associate Professor in Clemson University’s General Engineering program and is the President of Tau Beta Pi, the national engineering honor society. He received his Ph.D. in Civil Engineering with a minor in Education from the University of Florida in 1996. Previously, he served as Assistant Director of the NSF-sponsored SUCCEED Engineering Education Coalition. His research is primarily in freshman programs and educational assessment.

visit author page

Download Paper |

Abstract
NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract

The Comprehensive Assessment of Team Member Effectiveness: A New Peer Evaluation Instrument

Abstract

A multi-university research team has designed a peer evaluation instrument that is simple to use. The system gathers data from students through a web interface, ensuring the confidentiality of the peer ratings. The system analyzes the data to calculate suggested grade adjustments for equitably distributing a team’s grade among the team’s members. The system also provides extensive feedback to faculty as to certain dynamics of student teams that can be discerned from the peer evaluation data. Reliability and validity studies are under way. This paper includes a description of the behaviorally anchored rating scales, the electronic interface, and the feedback it provides.

The design of a new instrument

The instrument was developed from a broad base of teamwork research. The identification of potential items from the teamwork literature, the creation of a Likert-scale instrument, and the use of exploratory factor analysis and confirmatory factor on a large survey sample to reduce the instrument and identify the factor structure are in press elsewhere.1 Our earlier published work described the importance of assessing teamwork in the engineering classroom and the challenges it represents and laid out the ambitious assessment plan that would help develop an instrument that is easy to use and yet meaningful for both faculty and students, 2 described and demonstrated the benefit of a behaviorally anchored rating system,3 detailed the process of creating a new behaviorally anchored rating scale to simplify administration, data analysis, and reporting, and make feedback more understandable.4 This paper shows how the behaviorally anchored rating scales are incorporated into an electronic interface, how the database is designed, and the feedback it provides.

Designing a complicated database to keep the administration and reporting simple

To make the peer evaluation simple for students and faculty, the database schema became quite complicated, summarized in a 20-page functional requirements document. The system includes views for administrators, faculty, and students, and has modules for password protection and consent and reporting. The instrument is called the Comprehensive Assessment of Team Member Effectiveness, referred to as CATME for short.

Administrator interface: The administrator grants access to the system after confirming that the request has come from a faculty member. The interface shows active and pending faculty accounts, keeps track of each faculty’s last login date and time. This interface also provides access to the raw data for surveys released for research purposes.

Faculty interface: Participating faculty enter information about classes using the system, populate a class with students, populate teams of students within a class, and set up surveys for team activities. The faculty interface also controls the instructions given to the students, the

Pomeranz, H. R., & Feinstein, H. W., & Ohland, M. (2006, June), The Comprehensive Assessment Of Team Member Effectiveness: A New Peer Evaluation Instrument Paper presented at 2006 Annual Conference & Exposition, Chicago, Illinois. 10.18260/1-2--751

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2006 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015