Paper Authors

Abstract

NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract

Experiential Learning and Strictly Proper Scoring Rules

Abstract

Experiential learning is perhaps the most effective way to teach. One example is the scoring procedure used for exams in some decision analysis programs. Under this grading scheme, students take a multiple-choice exam, but rather than simply marking which answer they think is correct, they must assign a probability to each possible answer. The exam is then scored with a special scoring rule, under which students’ best strategy is to avoid guessing and instead assign their true beliefs. Such a scoring function is known as a strictly proper scoring rule. In this paper, we discuss several different scoring rules and demonstrate how their use in testing situations provides insights for both students and instructors.

Background

In several graduate industrial engineering / operations research programs (e.g., Stanford University, The University of Illinois at Urbana-Champaign, and The University of Texas at Austin), students face a unique grading system on their midterm exams, which are multiple choice. This grading scheme is also used in an undergraduate decision analysis course at Stanford University. Rather than simply marking the answer that they think is correct (or most likely to be correct), they must assign a probability to each possible answer. In theory, such an exam should better reveal the students’ mastery of the subject, but how should the instructor assign scores in this situation?

Formally, consider the assessment of a probability distribution by a student over n mutually exclusive and collectively exhaustive answers, where n > 1. Let p = (p1,…,pn) be an n- vector of probabilities representing the student’s private beliefs, where pi is the probability the student assigns to answer i being correct, and the sum of these probabilities is equal to one. These beliefs represent the student’s “true” state of knowledge, but are not directly observable to the instructor. Let the student’s public assessment or response be given by r = (r1,…,rn), where ri is the stated probability (the student’s answer) that answer i is correct, and the sum of these responses is equal to one.

Students are likely to have many different objectives in such a situation, ranging from learning the material to getting a good grade. We assume that letter grades are a strictly increasing function of the total points earned on the exam and that students seek to maximize their points. This simplification seems reasonable, particularly in programs that fractionalize letter grades (e.g., B+, A-, A).

If the student is scored according to some function R, then her expected score when she assigns r and believes p is R (r | p) p R (r ) , where Ri is the score received for assigning r i i i

when statement i is correct. If the student seeks to maximize her expected score then the optimal response is

r* arg max R (r | p) . (1) r

• Citation

• Format
  • APA
  • APA - LaTeX bibitem
  • MLA
  • MLA - LaTeX bibitem
  • Bibtex
  • EndNote - RIS

Bickel, J. E. (2009, June), Experiential Learning And Strictly Proper Scoring Rules Paper presented at 2009 Annual Conference & Exposition, Austin, Texas. 10.18260/1-2--5762