Thompson, M. K., & Clemmensen, L. H., & Rosas, H. (2013, June), Statistical Outlier Detection for Jury Based Grading Systems  Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--22467

\bibitem{asee_peer_22467}
  Thompson, M. K., \& Clemmensen, L. H., \& Rosas, H. (2013, June), \emph{Statistical Outlier Detection for Jury Based Grading Systems}
  Paper presented at 2013 ASEE Annual Conference \& Exposition, Atlanta, Georgia.
  10.18260/1-2--22467

Mary Kathryn Thompson, Line H Clemmensen, and Harvey Rosas.  "Statistical Outlier Detection for Jury Based Grading Systems".  2013 ASEE Annual Conference & Exposition, Atlanta, Georgia, 2013, June.  ASEE Conferences, 2013.  https://peer.asee.org/22467 Internet.  15 Dec, 2024

\bibitem{asee_peer_22467}
  Mary Kathryn Thompson, Line H Clemmensen, and Harvey Rosas.
  "Statistical Outlier Detection for Jury Based Grading Systems".
  \emph{2013 ASEE Annual Conference \& Exposition, Atlanta, Georgia, 2013, June}.
  ASEE Conferences, 2013.
  https://peer.asee.org/22467 Internet.  15 Dec, 2024

@INPROCEEDINGS{asee_peer_22467
author = "Mary Kathryn Thompson, Line H Clemmensen, and Harvey Rosas"
title = "Statistical Outlier Detection for Jury Based Grading Systems"
booktitle = "2013 ASEE Annual Conference \& Exposition"
year = "2013"
month = "June"
address = "Atlanta, Georgia"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/22467}
number = {10.18260/1-2--22467}
}

TY  - CPAPER
AB  -            Statistical Outlier Detection for Jury Based Grading SystemsAbstractJury based grading systems are common in project-based design courses where the inclusion ofmultiple raters helps to balance out differences of opinion in subjective scoring and leads to morestable marks (Cook and Beckman 2009). However, even expert jurors do not always reach agood consensus about the appropriate score for a given project. For this reason, jury base gradingsystems, especially those where rating is not followed by a group discussion, should be pairedwith a system to identify and remove inappropriate or invalid scores produced by individual jurymembers.This paper presents an algorithm that was developed to identify statistical outliers from thescores of grading jury members in a large project-based first year design course. The functionalrequirements of the algorithm and the follow-up procedures for score validation and appeals arepresented. The effect of outlier removal on the scores and the overall grading distributions isdemonstrated. The performance of the algorithm in terms of the percentage of scores flagged andremoved is evaluated and common failure modes of the algorithm are presented. Finally,improvements to the current algorithm are suggested.It will be shown that the outlier detection system is successful in identifying almost allinappropriate scores without producing an unreasonable amount of work for the individuals whomust hand-check the grades based on the results. Flagged scores that are not removed from thedata set are usually due to a failure of the grading jury to reach a consensus or the appropriatedissent of an expert grader. The system is quick and easy to use and is well accepted by both thestudents and the teaching staff. Its flexibility makes it a good candidate for adoption in the jurybased evaluation of other subjective deliverables in project-based engineering courses andbeyond.
AU  - Mary Kathryn Thompson
AU  - Line H Clemmensen
AU  - Harvey Rosas
CY  - Atlanta, Georgia
DA  - 2013/06/23
PB  - ASEE Conferences
TI  - Statistical Outlier Detection for Jury Based Grading Systems
UR  - https://peer.asee.org/22467
DO  - 10.18260/1-2--22467
ER  -