Gerlick, R., & Davis, D. C., & Trevisan, M. S., & Brown, S. A. (2011, June), Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments  Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. 10.18260/1-2--17920

\bibitem{asee_peer_17920}
  Gerlick, R., \& Davis, D. C., \& Trevisan, M. S., \& Brown, S. A. (2011, June), \emph{Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments}
  Paper presented at 2011 ASEE Annual Conference \& Exposition, Vancouver, BC.
  10.18260/1-2--17920

Robert Gerlick, Denny C. Davis, Michael S. Trevisan, and Shane A. Brown.  "Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments".  2011 ASEE Annual Conference & Exposition, Vancouver, BC, 2011, June.  ASEE Conferences, 2011.  https://peer.asee.org/17920 Internet.  23 Apr, 2024

\bibitem{asee_peer_17920}
  Robert Gerlick, Denny C. Davis, Michael S. Trevisan, and Shane A. Brown.
  "Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments".
  \emph{2011 ASEE Annual Conference \& Exposition, Vancouver, BC, 2011, June}.
  ASEE Conferences, 2011.
  https://peer.asee.org/17920 Internet.  23 Apr, 2024

@INPROCEEDINGS{asee_peer_17920
author = "Robert Gerlick, Denny C. Davis, Michael S. Trevisan, and Shane A. Brown"
title = "Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments"
booktitle = "2011 ASEE Annual Conference \& Exposition"
year = "2011"
month = "June"
address = "Vancouver, BC"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/17920}
number = {10.18260/1-2--17920}
}

TY  - CPAPER
AB  -                  Establishing Inter-rater Agreement for TIDEE’s               Teamwork and Professional Development AssessmentsABSTRACTSenior capstone design courses in engineering programs provide an opportunity to addressimportant curricular objectives related to teamwork and professional development. In this course,students work within a team environment and are challenged with non-technical issues, such ascommunication, organization, self-directed learning, etc. By the end of their capstone experienceit is hoped that students are prepared for the professional working environment. Capstonefaculty, often with technical expertise in a specific branch of engineering, have expresseddifficulty in teaching and assessing the types of knowledge, skills, and affective behaviorsassociated with these non-technical performance areas. When assessing teamwork, for example,the approach of “I know it when I see it” is not uncommon for an assessment process. Valid andreliable assessment instruments are needed for capstones which define expected performancecriteria, and therefore offer guidance for teaching and learning. In addition to this formative use,summative assessments are also needed to document achievement of student growth with regardsto these outcomes. To this end, collaborators from the Transferable Integrated DesignEngineering Education consortium (TIDEE) have developed a suite of assessments for use incapstone courses, comprising four common performance areas: teamwork, professionaldevelopment, design processes, and solution assets. For each of these areas of performance,multiple assessments have been developed and testing for validity and reliability has beenongoing. The purpose of this paper is to present results from a reliability study conducted withseven TIDEE assessments from the teamwork and professional development performance areas.For each of the assessments tested, the degree of inter-rater reliability was determined,representing an estimate of the consistency of scoring between multiple raters. This type ofreliability is significant for the TIDEE assessments as essay-type responses are elicited fromstudents and, therefore, requires professional judgments by faculty to assess achievement . Eachassessment was tested by having two faculty raters and two teaching assistant raters score asubset of student work with corresponding scoring rubrics. Percent agreement calculations andcorrelations were used to interpret the level of rater agreement. Interpretations of the results weremade in light of the intended uses of each assessment: formative and/or summative. In general,the assessments were found to have scoring agreement of 85% to 100% within a one-pointvariation. Exact agreement ranged from a high of 60% to a low of 20%. Overall, the resultsindicated sufficient agreement for use with formative assessment (for enhancing teaching andlearning). For summative use, five of the assessments should prove adequate in documentingstudent growth, including the Team Contract, Team Member Citizenship, Growth Planning,Growth Progress, and Professional Development assessments. The remaining two, TeamProcesses and Growth Achieved, may need to be revised to improve agreement. Suggestions forimprovement include revisions to rubric descriptors for each level of performance, improvedFrame-of-Reference rater training to decrease rater errors and increase accuracy, and, lastly,incorporation of Behavior-Observation-Training in the training protocol.
AU  - Robert Gerlick
AU  - Denny C. Davis
AU  - Michael S. Trevisan
AU  - Shane A. Brown
CY  - Vancouver, BC
DA  - 2011/06/26
PB  - ASEE Conferences
TI  - Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments
UR  - https://peer.asee.org/17920
DO  - 10.18260/1-2--17920
ER  -