June 23, 2013
June 23, 2013
June 26, 2013
Design in Engineering Education
23.1082.1 - 23.1082.13
Statistical Outlier Detection for Jury Based Grading SystemsAbstractJury based grading systems are common in project-based design courses where the inclusion ofmultiple raters helps to balance out differences of opinion in subjective scoring and leads to morestable marks (Cook and Beckman 2009). However, even expert jurors do not always reach agood consensus about the appropriate score for a given project. For this reason, jury base gradingsystems, especially those where rating is not followed by a group discussion, should be pairedwith a system to identify and remove inappropriate or invalid scores produced by individual jurymembers.This paper presents an algorithm that was developed to identify statistical outliers from thescores of grading jury members in a large project-based first year design course. The functionalrequirements of the algorithm and the follow-up procedures for score validation and appeals arepresented. The effect of outlier removal on the scores and the overall grading distributions isdemonstrated. The performance of the algorithm in terms of the percentage of scores flagged andremoved is evaluated and common failure modes of the algorithm are presented. Finally,improvements to the current algorithm are suggested.It will be shown that the outlier detection system is successful in identifying almost allinappropriate scores without producing an unreasonable amount of work for the individuals whomust hand-check the grades based on the results. Flagged scores that are not removed from thedata set are usually due to a failure of the grading jury to reach a consensus or the appropriatedissent of an expert grader. The system is quick and easy to use and is well accepted by both thestudents and the teaching staff. Its flexibility makes it a good candidate for adoption in the jurybased evaluation of other subjective deliverables in project-based engineering courses andbeyond.
Thompson, M. K., & Clemmensen, L. H., & Rosas, H. (2013, June), Statistical Outlier Detection for Jury Based Grading Systems Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. https://peer.asee.org/22467
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2013 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015