Formative Feedback For Improved Student Performance Through Adaptive Comparative Judgment

Scott R. Bartholomew; Greg J. Strimel; Esteban Garcia; Emily Yoshikawa; Liwei Zhang

Download Paper | Permalink

Conference: 2018 ASEE Annual Conference & Exposition
Location: Salt Lake City, Utah
Publication Date: June 23, 2018
Start Date: June 23, 2018
End Date: July 27, 2018
Conference Session: EDGD: Assessment & Student Learning
Tagged Division: Engineering Design Graphics
Page Count: 13
DOI: 10.18260/1-2--30531
Permanent URL: https://peer.asee.org/30531
Download Count: 744

Paper Authors

biography

Scott R. Bartholomew Purdue University, West Lafayette

visit author page

Scott R. Bartholomew, PhD. is an assistant professor of Engineering/Technology Teacher Education at Purdue University. Previously he taught Technology and Engineering classes at the middle school and university level. Dr. Bartholomew’s current work revolves around Adaptive Comparative Judgment (ACJ) assessment techniques, student design portfolios, and Technology & Engineering teacher preparation.

visit author page

biography

Greg J. Strimel Purdue University, West Lafayette orcid.org/0000-0002-4847-4526

visit author page

Dr. Greg J. Strimel is an assistant professor of engineering/technology teacher education in the Purdue Polytechnic Institute at Purdue University in West Lafayette, Indiana. His prior teaching experience includes serving as a high school engineering/technology teacher and a teaching assistant professor within the College of Engineering & Mineral Resources at West Virginia University.

visit author page

author page

Liwei Zhang is a master student at the Department of Technology Leadership and Innovation at Purdue University. In 2016, she received her Bachelor's degree in Industrial Engineering from Purdue University, West Lafayette, IN. Her research focuses on assessment in STEM education.

visit author page

Download Paper | Permalink

Abstract

The creative and ill-structured nature of open-ended design problems has contributed to notoriously unreliable and difficult to implement methods of assessment (Bartholomew, 2017; Pollitt, 2004, 2012). However, an innovative form of assessment called adaptive comparative judgment (ACJ) has recently been developed and validated (Pollitt, 2012). Rather than traditional forms of assessment such as rubrics, ACJ relies on a judge simply looking at two pieces of student work and picking the better of the two. Herein resides the strength of ACJ, as judgments between two items are significantly more reliable than rubric or criterion-based judgments (Pollitt, 2004, 2012). The judges are not asked to provide a grade for each piece of work but rather asked to provide a holistic decision as to which artifact is better based on their own professional opinion. In ACJ, this comparative judgment process is repeated, and, utilizing a devised algorithm, each piece of student work is compared with others until a rank-order is produced for all items. This method of assessment has proven more reliable than traditional forms of assessment (Kimbell, 2012; Pollitt, 2004; 2012) and has been used in a variety of settings and content-areas (Bartholomew & Yoshikawa, 2017). In addition to improved reliability through the ACJ assessment process, judges also have the opportunity to provide feedback, or justification, in conjunction with each judgment. This feedback becomes a valuable source of feedback and guidance for students and, although ACJ was originally designed as a tool for professional assessors and teachers, this paper will examine a new approach to ACJ with students acting as the judges of their peer work in the ACJ process. Situated in a university-level engineering graphics course this research will study the impact on student performance when students are engaged in ACJ as judges—both providing and receiving feedback to and from their peers—on an open-ended design problem. Additionally, this paper will examine the impacts of resituating the ACJ process in the midst of an open-ended design project, as opposed to the conclusion, with a specific emphasis on using ACJ as a formative assessment and feedback tool.

Citation
Format

Bartholomew, S. R., & Strimel, G. J., & Garcia, E., & Yoshikawa, E., & Zhang, L. (2018, June), Formative Feedback For Improved Student Performance Through Adaptive Comparative Judgment Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--30531

TY  - CPAPER
AB  - The creative and ill-structured nature of open-ended design problems has contributed to notoriously unreliable and difficult to implement methods of assessment (Bartholomew, 2017; Pollitt, 2004, 2012). However, an innovative form of assessment called adaptive comparative judgment (ACJ) has recently been developed and validated (Pollitt, 2012). Rather than traditional forms of assessment such as rubrics, ACJ relies on a judge simply looking at two pieces of student work and picking the better of the two.  Herein resides the strength of ACJ, as judgments between two items are significantly more reliable than rubric or criterion-based judgments (Pollitt, 2004, 2012). The judges are not asked to provide a grade for each piece of work but rather asked to provide a holistic decision as to which artifact is better based on their own professional opinion. In ACJ, this comparative judgment process is repeated, and, utilizing a devised algorithm, each piece of student work is compared with others until a rank-order is produced for all items. This method of assessment has proven more reliable than traditional forms of assessment (Kimbell, 2012; Pollitt, 2004; 2012) and has been used in a variety of settings and content-areas (Bartholomew &amp; Yoshikawa, 2017).  
In addition to improved reliability through the ACJ assessment process, judges also have the opportunity to provide feedback, or justification, in conjunction with each judgment.  This feedback becomes a valuable source of feedback and guidance for students and, although ACJ was originally designed as a tool for professional assessors and teachers, this paper will examine a new approach to ACJ with students acting as the judges of their peer work in the ACJ process.  Situated in a university-level engineering graphics course this research will study the impact on student performance when students are engaged in ACJ as judges—both providing and receiving feedback to and from their peers—on an open-ended design problem.  Additionally, this paper will examine the impacts of resituating the ACJ process in the midst of an open-ended design project, as opposed to the conclusion, with a specific emphasis on using ACJ as a formative assessment and feedback tool.
AU  - Scott R. Bartholomew
AU  - Greg J. Strimel
AU  - Esteban Garcia
AU  - Emily Yoshikawa
AU  - Liwei Zhang
CY  - Salt Lake City, Utah
DA  - 2018/06/23
PB  - ASEE Conferences
TI  - Formative Feedback For Improved Student Performance Through Adaptive Comparative Judgment
UR  - https://peer.asee.org/30531
DO  - 10.18260/1-2--30531
ER  -