Asee peer logo

Formative Feedback For Improved Student Performance Through Adaptive Comparative Judgment

Download Paper |


2018 ASEE Annual Conference & Exposition


Salt Lake City, Utah

Publication Date

June 23, 2018

Start Date

June 23, 2018

End Date

July 27, 2018

Conference Session

EDGD: Assessment & Student Learning

Tagged Division

Engineering Design Graphics

Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Scott R. Bartholomew Purdue University, West Lafayette

visit author page

Scott R. Bartholomew, PhD. is an assistant professor of Engineering/Technology Teacher Education at Purdue University. Previously he taught Technology and Engineering classes at the middle school and university level. Dr. Bartholomew’s current work revolves around Adaptive Comparative Judgment (ACJ) assessment techniques, student design portfolios, and Technology & Engineering teacher preparation.

visit author page


Greg J. Strimel Purdue University, West Lafayette Orcid 16x16

visit author page

Dr. Greg J. Strimel is an assistant professor of engineering/technology teacher education in the Purdue Polytechnic Institute at Purdue University in West Lafayette, Indiana. His prior teaching experience includes serving as a high school engineering/technology teacher and a teaching assistant professor within the College of Engineering & Mineral Resources at West Virginia University.

visit author page

author page

Esteban Garcia

author page

Emily Yoshikawa Purdue University, West Lafayette


Liwei Zhang

visit author page

Liwei Zhang is a master student at the Department of Technology Leadership and Innovation at Purdue University. In 2016, she received her Bachelor's degree in Industrial Engineering from Purdue University, West Lafayette, IN. Her research focuses on assessment in STEM education.

visit author page

Download Paper |


The creative and ill-structured nature of open-ended design problems has contributed to notoriously unreliable and difficult to implement methods of assessment (Bartholomew, 2017; Pollitt, 2004, 2012). However, an innovative form of assessment called adaptive comparative judgment (ACJ) has recently been developed and validated (Pollitt, 2012). Rather than traditional forms of assessment such as rubrics, ACJ relies on a judge simply looking at two pieces of student work and picking the better of the two. Herein resides the strength of ACJ, as judgments between two items are significantly more reliable than rubric or criterion-based judgments (Pollitt, 2004, 2012). The judges are not asked to provide a grade for each piece of work but rather asked to provide a holistic decision as to which artifact is better based on their own professional opinion. In ACJ, this comparative judgment process is repeated, and, utilizing a devised algorithm, each piece of student work is compared with others until a rank-order is produced for all items. This method of assessment has proven more reliable than traditional forms of assessment (Kimbell, 2012; Pollitt, 2004; 2012) and has been used in a variety of settings and content-areas (Bartholomew & Yoshikawa, 2017). In addition to improved reliability through the ACJ assessment process, judges also have the opportunity to provide feedback, or justification, in conjunction with each judgment. This feedback becomes a valuable source of feedback and guidance for students and, although ACJ was originally designed as a tool for professional assessors and teachers, this paper will examine a new approach to ACJ with students acting as the judges of their peer work in the ACJ process. Situated in a university-level engineering graphics course this research will study the impact on student performance when students are engaged in ACJ as judges—both providing and receiving feedback to and from their peers—on an open-ended design problem. Additionally, this paper will examine the impacts of resituating the ACJ process in the midst of an open-ended design project, as opposed to the conclusion, with a specific emphasis on using ACJ as a formative assessment and feedback tool.

Bartholomew, S. R., & Strimel, G. J., & Garcia, E., & Yoshikawa, E., & Zhang, L. (2018, June), Formative Feedback For Improved Student Performance Through Adaptive Comparative Judgment Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--30531

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2018 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015