The Justification Effect on Two-Tier Multiple-Choice Exams

Pablo Frank Bolton; Rahul Simha; Liberty Rose Lehr; Michelle Lawson

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: June 26, 2024
Conference Session: Educational Research and Methods Division (ERM) Technical Session 8
Tagged Division: Educational Research and Methods Division (ERM)
Page Count: 22
DOI: 10.18260/1-2--48121
Permanent URL: https://peer.asee.org/48121
Download Count: 62

Paper Authors

biography

Pablo Frank Bolton Smith College orcid.org/0000-0002-1095-9245

visit author page

I am a Lecturer in the Computer Science department at Smith College. I received my PhD. from the George Washington University under the direction of Professor Rahul Simha. I currently teach a variety of undergraduate courses and have taught graduate courses in the past.

My research is currently focused on STEM, especially on the areas of identifying misconceptions, creating scalable and informative assessments, and in the use of active learning techniques such as learning-by-teaching, and peer learning.

In addition, I work on Human-Computer Interaction and how it might allow us to interact with virtual worlds and robots.

I enjoy collaborating with colleagues in other fields where I get to combine CS with Biology or Physics and play with their data.

Topics of interest include:

Flipped Classroom techniques to teach programming
The benefits of games and puzzles in learning
Construction of fair, scalable assessments
Multimodal teaching with an emphasis on getting students to articulate their understanding
3D-Shape reconstruction and analysis
The use of Embedded Systems and Machine Learning to automate (Biology) Laboratory tasks.

visit author page

author page

Abstract

Booming enrollment in computer science has raised the need for efficiently gradable assessments, among which are Multiple-Choice Question (MCQ) assessments. MCQs have drawbacks, among which are: random guessing, limited instructor insight into conceptual misunderstandings, student frustration at not being able to explain for partial credit. Recent years have seen several MCQ refinements that feature a two-tier structure that elicits justifications in addition to a correct-choice answer. These justifications can themselves be structured as choices to enable rapid grading. This paper introduces an additional refinement to the two-tier MCQ to acquire more information about a student's understanding by also eliciting explanations for why the wrong answers are wrong. In addition, this paper introduces a simple new metric, the Justification effect, or J-effect, that is easy to extract and apply in order to detect students that need help and identify questions that have design issues. The use of this approach allows instructors to easily provide automated yet rich feedback to students and to pinpoint issues with test implementations, all while remaining easy to design, implement, answer and grade. The study explores the testing and implementation of three different trials over two semesters.

Citation
Format

Frank Bolton, P., & Lehr, L. R., & Simha, R., & Lawson, M. (2024, June), The Justification Effect on Two-Tier Multiple-Choice Exams Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--48121