The Comprehensive Assessment Of Team Member Effectiveness: A New Peer Evaluation Instrument
- Conference
- Location
-
Chicago, Illinois
- Publication Date
-
June 18, 2006
- Start Date
-
June 18, 2006
- End Date
-
June 21, 2006
- ISSN
-
2153-5965
- Conference Session
- Tagged Division
-
Division Experimentation & Lab-Oriented Studies
- Page Count
-
6
- Page Numbers
-
11.1262.1 - 11.1262.6
- DOI
-
10.18260/1-2--751
- Permanent URL
-
https://peer.asee.org/751
- Download Count
-
1000
Paper Authors
Hal R. Pomeranz Deer Run Associates, Inc.
Hal R. Pomeranz is a computer network security and database programming consultant. He is a co-founder of Deer Run Associates, Inc., currently operating in Eugene, Oregon.
Harlan W. Feinstein Deer Run Associates, Inc.
Harlan W. Feinstein is a consultant specializing a interface design for web pages. He is a subcontractor of Deer Run Associates, Inc.
Matthew Ohland
Clemson University
orcid.org/0000-0003-4052-1452
Matthew W. Ohland is an Associate Professor in Clemson University’s General Engineering program and is the President of Tau Beta Pi, the national engineering honor society. He received his Ph.D. in Civil Engineering with a minor in Education from the University of Florida in 1996. Previously, he served as Assistant Director of the NSF-sponsored SUCCEED Engineering Education Coalition. His research is primarily in freshman programs and educational assessment.
Abstract
NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract
The Comprehensive Assessment of Team Member Effectiveness: A New Peer Evaluation Instrument
Abstract
A multi-university research team has designed a peer evaluation instrument that is simple to use. The system gathers data from students through a web interface, ensuring the confidentiality of the peer ratings. The system analyzes the data to calculate suggested grade adjustments for equitably distributing a team’s grade among the team’s members. The system also provides extensive feedback to faculty as to certain dynamics of student teams that can be discerned from the peer evaluation data. Reliability and validity studies are under way. This paper includes a description of the behaviorally anchored rating scales, the electronic interface, and the feedback it provides.
The design of a new instrument
The instrument was developed from a broad base of teamwork research. The identification of potential items from the teamwork literature, the creation of a Likert-scale instrument, and the use of exploratory factor analysis and confirmatory factor on a large survey sample to reduce the instrument and identify the factor structure are in press elsewhere.1 Our earlier published work described the importance of assessing teamwork in the engineering classroom and the challenges it represents and laid out the ambitious assessment plan that would help develop an instrument that is easy to use and yet meaningful for both faculty and students, 2 described and demonstrated the benefit of a behaviorally anchored rating system,3 detailed the process of creating a new behaviorally anchored rating scale to simplify administration, data analysis, and reporting, and make feedback more understandable.4 This paper shows how the behaviorally anchored rating scales are incorporated into an electronic interface, how the database is designed, and the feedback it provides.
Designing a complicated database to keep the administration and reporting simple
To make the peer evaluation simple for students and faculty, the database schema became quite complicated, summarized in a 20-page functional requirements document. The system includes views for administrators, faculty, and students, and has modules for password protection and consent and reporting. The instrument is called the Comprehensive Assessment of Team Member Effectiveness, referred to as CATME for short.
Administrator interface: The administrator grants access to the system after confirming that the request has come from a faculty member. The interface shows active and pending faculty accounts, keeps track of each faculty’s last login date and time. This interface also provides access to the raw data for surveys released for research purposes.
Faculty interface: Participating faculty enter information about classes using the system, populate a class with students, populate teams of students within a class, and set up surveys for team activities. The faculty interface also controls the instructions given to the students, the
Pomeranz, H. R., & Feinstein, H. W., & Ohland, M. (2006, June), The Comprehensive Assessment Of Team Member Effectiveness: A New Peer Evaluation Instrument Paper presented at 2006 Annual Conference & Exposition, Chicago, Illinois. 10.18260/1-2--751
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2006 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015