June 26, 2011
June 26, 2011
June 29, 2011
Educational Research and Methods
22.639.1 - 22.639.14
Establishing Inter-rater Agreement for TIDEE’s Teamwork and Professional Development AssessmentsABSTRACTSenior capstone design courses in engineering programs provide an opportunity to addressimportant curricular objectives related to teamwork and professional development. In this course,students work within a team environment and are challenged with non-technical issues, such ascommunication, organization, self-directed learning, etc. By the end of their capstone experienceit is hoped that students are prepared for the professional working environment. Capstonefaculty, often with technical expertise in a specific branch of engineering, have expresseddifficulty in teaching and assessing the types of knowledge, skills, and affective behaviorsassociated with these non-technical performance areas. When assessing teamwork, for example,the approach of “I know it when I see it” is not uncommon for an assessment process. Valid andreliable assessment instruments are needed for capstones which define expected performancecriteria, and therefore offer guidance for teaching and learning. In addition to this formative use,summative assessments are also needed to document achievement of student growth with regardsto these outcomes. To this end, collaborators from the Transferable Integrated DesignEngineering Education consortium (TIDEE) have developed a suite of assessments for use incapstone courses, comprising four common performance areas: teamwork, professionaldevelopment, design processes, and solution assets. For each of these areas of performance,multiple assessments have been developed and testing for validity and reliability has beenongoing. The purpose of this paper is to present results from a reliability study conducted withseven TIDEE assessments from the teamwork and professional development performance areas.For each of the assessments tested, the degree of inter-rater reliability was determined,representing an estimate of the consistency of scoring between multiple raters. This type ofreliability is significant for the TIDEE assessments as essay-type responses are elicited fromstudents and, therefore, requires professional judgments by faculty to assess achievement . Eachassessment was tested by having two faculty raters and two teaching assistant raters score asubset of student work with corresponding scoring rubrics. Percent agreement calculations andcorrelations were used to interpret the level of rater agreement. Interpretations of the results weremade in light of the intended uses of each assessment: formative and/or summative. In general,the assessments were found to have scoring agreement of 85% to 100% within a one-pointvariation. Exact agreement ranged from a high of 60% to a low of 20%. Overall, the resultsindicated sufficient agreement for use with formative assessment (for enhancing teaching andlearning). For summative use, five of the assessments should prove adequate in documentingstudent growth, including the Team Contract, Team Member Citizenship, Growth Planning,Growth Progress, and Professional Development assessments. The remaining two, TeamProcesses and Growth Achieved, may need to be revised to improve agreement. Suggestions forimprovement include revisions to rubric descriptors for each level of performance, improvedFrame-of-Reference rater training to decrease rater errors and increase accuracy, and, lastly,incorporation of Behavior-Observation-Training in the training protocol.
Gerlick, R., & Davis, D. C., & Trevisan, M. S., & Brown, S. A. (2011, June), Establishing Inter-rater Agreement for TIDEE's Teamwork and Professional Development Assessments Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. https://peer.asee.org/17920
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2011 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015