Asee peer logo

Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments

Download Paper |

Conference

2011 ASEE Annual Conference & Exposition

Location

Vancouver, BC

Publication Date

June 26, 2011

Start Date

June 26, 2011

End Date

June 29, 2011

ISSN

2153-5965

Conference Session

Assessment Instruments

Tagged Division

Educational Research and Methods

Page Count

14

Page Numbers

22.639.1 - 22.639.14

Permanent URL

https://peer.asee.org/17920

Download Count

18

Request a correction

Paper Authors

biography

Robert Gerlick Pittsburg State University

visit author page

Dr. Robert Gerlick is Assistant Professor of Mechanical Engineering Technology at Pittsburg State University.

visit author page

biography

Denny C. Davis Washington State University

visit author page

Dr. Davis is Professor of Bioengineering and Director of the Engineering Education Research Center at Washington State University. He has led numerous multidisciplinary research projects to enhance engineering education. He currently leads projects creating and testing assessments and curriculum materials for engineering design and professional skills, especially for use in capstone engineering design courses. He has been a Fellow of the American Society for Engineering Education since 2002.

visit author page

biography

Michael S. Trevisan Washington State University

visit author page

Dr. Michael S. Trevisan is Professor of Educational Psychology and Associate Dean for Research and External Funding in the College of Education. Dr. Trevisan is published widely in the fields of educational measurement and evaluation. In recent years, he has collaborated with Dr. Denny Davis to develop assessments for engineering education design courses.

visit author page

biography

Shane A. Brown Washington State University Orcid 16x16 orcid.org/0000-0003-3669-8407

visit author page

Shane Brown is an Assistant Professor in the Department of Civil and Environmental Engineering at Washington State University. His research focuses on conceptual understanding of engineering students and practitioners and conceptual change processes that lead to differences in understanding.

visit author page

Download Paper |

Abstract

Establishing Inter-rater Agreement for TIDEE’s Teamwork and Professional Development AssessmentsABSTRACTSenior capstone design courses in engineering programs provide an opportunity to addressimportant curricular objectives related to teamwork and professional development. In this course,students work within a team environment and are challenged with non-technical issues, such ascommunication, organization, self-directed learning, etc. By the end of their capstone experienceit is hoped that students are prepared for the professional working environment. Capstonefaculty, often with technical expertise in a specific branch of engineering, have expresseddifficulty in teaching and assessing the types of knowledge, skills, and affective behaviorsassociated with these non-technical performance areas. When assessing teamwork, for example,the approach of “I know it when I see it” is not uncommon for an assessment process. Valid andreliable assessment instruments are needed for capstones which define expected performancecriteria, and therefore offer guidance for teaching and learning. In addition to this formative use,summative assessments are also needed to document achievement of student growth with regardsto these outcomes. To this end, collaborators from the Transferable Integrated DesignEngineering Education consortium (TIDEE) have developed a suite of assessments for use incapstone courses, comprising four common performance areas: teamwork, professionaldevelopment, design processes, and solution assets. For each of these areas of performance,multiple assessments have been developed and testing for validity and reliability has beenongoing. The purpose of this paper is to present results from a reliability study conducted withseven TIDEE assessments from the teamwork and professional development performance areas.For each of the assessments tested, the degree of inter-rater reliability was determined,representing an estimate of the consistency of scoring between multiple raters. This type ofreliability is significant for the TIDEE assessments as essay-type responses are elicited fromstudents and, therefore, requires professional judgments by faculty to assess achievement . Eachassessment was tested by having two faculty raters and two teaching assistant raters score asubset of student work with corresponding scoring rubrics. Percent agreement calculations andcorrelations were used to interpret the level of rater agreement. Interpretations of the results weremade in light of the intended uses of each assessment: formative and/or summative. In general,the assessments were found to have scoring agreement of 85% to 100% within a one-pointvariation. Exact agreement ranged from a high of 60% to a low of 20%. Overall, the resultsindicated sufficient agreement for use with formative assessment (for enhancing teaching andlearning). For summative use, five of the assessments should prove adequate in documentingstudent growth, including the Team Contract, Team Member Citizenship, Growth Planning,Growth Progress, and Professional Development assessments. The remaining two, TeamProcesses and Growth Achieved, may need to be revised to improve agreement. Suggestions forimprovement include revisions to rubric descriptors for each level of performance, improvedFrame-of-Reference rater training to decrease rater errors and increase accuracy, and, lastly,incorporation of Behavior-Observation-Training in the training protocol.

Gerlick, R., & Davis, D. C., & Trevisan, M. S., & Brown, S. A. (2011, June), Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. https://peer.asee.org/17920

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2011 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015