Asee peer logo

Statistical Outlier Detection for Jury Based Grading Systems

Download Paper |


2013 ASEE Annual Conference & Exposition


Atlanta, Georgia

Publication Date

June 23, 2013

Start Date

June 23, 2013

End Date

June 26, 2013



Conference Session

Design Methodology and Evaluation 1

Tagged Division

Design in Engineering Education

Page Count


Page Numbers

23.1082.1 - 23.1082.13



Permanent URL

Download Count


Request a correction

Paper Authors


Mary Kathryn Thompson Technical University of Denmark

visit author page

Mary Kathryn Thompson is an Associate Professor in the Department of Mechanical Engineering at the Technical University of Denmark. Her research interests include the development, improvement, and integration of formal design theories and methodologies; assessment in project-based engineering design courses; and numerical modeling of micro scale surface phenomena. From 2008 - 2011, Prof. Thompson was the Director of the KAIST Freshman Design Program, which earned her both the KAIST Grand Prize for Creative Teaching and the Republic of Korea Ministry of Education, Science and Technology Award for Innovation in Engineering Education in 2009. She earned her B.S., M.S., and Ph.D. from the Massachusetts Institute of Technology, Department of Mechanical Engineering.

visit author page


Line H Clemmensen Technical University of Denmark

visit author page

Line H. Clemmensen is an Assistant Professor in the Department of
Applied Mathematics and Computer Science at the Technical University
of Denmark. She is engaged in statistical research of models for high
dimensional data analysis including regularized statistics and machine
learning. She is also interested in educational research and is
involved in various projects on teaching and learning assessment at
the Technical University of Denmark. She earned her M.S. and Ph.D.
from the Technical University of Denmark, Department of Informatics
and Mathematical Modeling.

visit author page


Harvey Rosas Valparaiso University

visit author page

Harvey Rosas is an Associate Professor and Director of Undergraduate Studies in the Department of Statistics at the Valparaiso University, Chile. He is currently working on the application of machine learning techniques to the dimensionality reduction problem; Multidimensional Item Response Theory; and text classification. He received his BS in mathematics with an emphasis on the theory of computation from the National University of Colombia and his MS and PhD from the Department of Mathematical Sciences at the Korea Advanced Institute of Science and Technology (KAIST).

visit author page

Download Paper |


Statistical Outlier Detection for Jury Based Grading SystemsAbstractJury based grading systems are common in project-based design courses where the inclusion ofmultiple raters helps to balance out differences of opinion in subjective scoring and leads to morestable marks (Cook and Beckman 2009). However, even expert jurors do not always reach agood consensus about the appropriate score for a given project. For this reason, jury base gradingsystems, especially those where rating is not followed by a group discussion, should be pairedwith a system to identify and remove inappropriate or invalid scores produced by individual jurymembers.This paper presents an algorithm that was developed to identify statistical outliers from thescores of grading jury members in a large project-based first year design course. The functionalrequirements of the algorithm and the follow-up procedures for score validation and appeals arepresented. The effect of outlier removal on the scores and the overall grading distributions isdemonstrated. The performance of the algorithm in terms of the percentage of scores flagged andremoved is evaluated and common failure modes of the algorithm are presented. Finally,improvements to the current algorithm are suggested.It will be shown that the outlier detection system is successful in identifying almost allinappropriate scores without producing an unreasonable amount of work for the individuals whomust hand-check the grades based on the results. Flagged scores that are not removed from thedata set are usually due to a failure of the grading jury to reach a consensus or the appropriatedissent of an expert grader. The system is quick and easy to use and is well accepted by both thestudents and the teaching staff. Its flexibility makes it a good candidate for adoption in the jurybased evaluation of other subjective deliverables in project-based engineering courses andbeyond.

Thompson, M. K., & Clemmensen, L. H., & Rosas, H. (2013, June), Statistical Outlier Detection for Jury Based Grading Systems Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--22467

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2013 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015