Austin, Texas
June 14, 2009
June 14, 2009
June 17, 2009
2153-5965
Industrial Engineering
15
14.607.1 - 14.607.15
10.18260/1-2--5762
https://peer.asee.org/5762
529
Experiential Learning and Strictly Proper Scoring Rules
Abstract
Experiential learning is perhaps the most effective way to teach. One example is the scoring procedure used for exams in some decision analysis programs. Under this grading scheme, students take a multiple-choice exam, but rather than simply marking which answer they think is correct, they must assign a probability to each possible answer. The exam is then scored with a special scoring rule, under which students’ best strategy is to avoid guessing and instead assign their true beliefs. Such a scoring function is known as a strictly proper scoring rule. In this paper, we discuss several different scoring rules and demonstrate how their use in testing situations provides insights for both students and instructors.
Background
In several graduate industrial engineering / operations research programs (e.g., Stanford University, The University of Illinois at Urbana-Champaign, and The University of Texas at Austin), students face a unique grading system on their midterm exams, which are multiple choice. This grading scheme is also used in an undergraduate decision analysis course at Stanford University. Rather than simply marking the answer that they think is correct (or most likely to be correct), they must assign a probability to each possible answer. In theory, such an exam should better reveal the students’ mastery of the subject, but how should the instructor assign scores in this situation?
Formally, consider the assessment of a probability distribution by a student over n mutually exclusive and collectively exhaustive answers, where n > 1. Let p = (p1,…,pn) be an n- vector of probabilities representing the student’s private beliefs, where pi is the probability the student assigns to answer i being correct, and the sum of these probabilities is equal to one. These beliefs represent the student’s “true” state of knowledge, but are not directly observable to the instructor. Let the student’s public assessment or response be given by r = (r1,…,rn), where ri is the stated probability (the student’s answer) that answer i is correct, and the sum of these responses is equal to one.
Students are likely to have many different objectives in such a situation, ranging from learning the material to getting a good grade. We assume that letter grades are a strictly increasing function of the total points earned on the exam and that students seek to maximize their points. This simplification seems reasonable, particularly in programs that fractionalize letter grades (e.g., B+, A-, A).
If the student is scored according to some function R, then her expected score when she assigns r and believes p is R (r | p) p R (r ) , where Ri is the score received for assigning r i i i
when statement i is correct. If the student seeks to maximize her expected score then the optimal response is
r* arg max R (r | p) . (1) r
Bickel, J. E. (2009, June), Experiential Learning And Strictly Proper Scoring Rules Paper presented at 2009 Annual Conference & Exposition, Austin, Texas. 10.18260/1-2--5762
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2009 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015