Asee peer logo

Differences and Similarities in Student, Instructor, and Professional Perceptions of "Good Engineering Design" through Adaptive Comparative Judgment

Download Paper |


2018 ASEE Annual Conference & Exposition


Salt Lake City, Utah

Publication Date

June 23, 2018

Start Date

June 23, 2018

End Date

July 27, 2018

Conference Session

Design Assessment

Tagged Division

Design in Engineering Education

Tagged Topic


Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Scott R. Bartholomew Purdue Polytechnic Institute

visit author page

Scott R. Bartholomew, PhD. is an assistant professor of Engineering/Technology Teacher Education at Purdue University. Previously he taught Technology and Engineering classes at the middle school and university level. Dr. Bartholomew’s current work revolves around Adaptive Comparative Judgment (ACJ) assessment techniques, student design portfolios, and Technology & Engineering teacher preparation.

visit author page


Greg J. Strimel Purdue Polytechnic Institute Orcid 16x16

visit author page

Dr. Greg J. Strimel is an assistant professor of engineering/technology teacher education in the Purdue Polytechnic Institute at Purdue University in West Lafayette, Indiana. His prior teaching experience includes serving as a high school engineering/technology teacher and a teaching assistant professor within the College of Engineering & Mineral Resources at West Virginia University.

visit author page


Senay Purzer Purdue University, West Lafayette (College of Engineering) Orcid 16x16

visit author page

Ṣenay Purzer is an Associate Professor in the School of Engineering Education. She is the recipient of a 2012 NSF CAREER award, which examines how engineering students approach innovation. She serves on the editorial boards of Science Education and the Journal of Pre-College Engineering Education (JPEER). She received a B.S.E with distinction in Engineering in 2009 and a B.S. degree in Physics Education in 1999. Her M.A. and Ph.D. degrees are in Science Education from Arizona State University earned in 2002 and 2008, respectively.

visit author page


Liwei Zhang Purdue University, West Lafayette

visit author page

Liwei Zhang is a master student at the Department of Technology Leadership and Innovation at Purdue University. In 2016, she received her Bachelor's degree in Industrial Engineering from Purdue University, West Lafayette, IN. Her research focuses on assessment in STEM education.

visit author page

author page

Emily Yoshikawa Purdue University, West Lafayette

Download Paper |


Adequately preparing students for future careers, learning, and civic opportunities is at the forefront of discussions around best practices for education (Sharples, de Roock, Ferguson, Gaved, Herodotou, Koh,...& Weller, 2016). However, despite an emphasis on this preparation, research shows there is not always alignment between student, teacher, and industry perceptions of skills, traits, and competencies necessary for this preparation (Adecco, 2014; Deloitte, 2015). Further, much of this preparation revolves around difficult-to-assess-and-teach 21st century skills such as critical thinking, problem-solving, creativity, innovative thinking, and collaboration (NRC, 2011). Despite the challenges associated with these issues, the adequate preparation of students for future success is paramount (Partnership, 2017). This research will investigate the use of a relatively-new form of assessment, Adaptive Comparative Judgment, as a tool for not only improving the assessment practices of educators in open-ended situations, but also identifying the potential disconnects between student, teacher, and industry perceptions of quality. Rather than traditional forms of assessment such as rubrics, ACJ is an assessment approach which relies on a judge simply looking at two pieces of student work and picking the better of the two. The judges are not asked to provide a grade for each piece of work but rather asked to provide a holistic decision as to which artifact is better based on their own professional opinion. In ACJ, this comparative judgment process is repeated, and, using a devised algorithm, each piece of student work is compared with others until a rank-order is produced for all items. This method of assessment has proven more reliable than traditional forms of assessment (Kimbell, 2012; Pollitt, 2004; 2012) and has been used in a variety of settings and content-areas (Bartholomew & Yoshikawa, 2017). Significantly, researchers (Bartholomew, Hartell, & Strimel, 2017) have also used ACJ assessment to identify similarities and differences in design values amongst diverse groups with success. This project entails first-year university engineering students, engineering instructors, and industry professionals collaborating to assess student work. Each group (students, instructors, and industry professionals) used ACJ to rank the final projects from a first-year engineering course at a public land-grant institution. The similarities, differences, and rationale statements for assessing the students work were examined to identify the assessment values and perceptions of quality of each group. A better understanding of these similarities and differences can inform education practice and may assist educators in ensuring alignment between current practice and industry-needs towards future student success.

Bartholomew, S. R., & Strimel, G. J., & Purzer, S., & Zhang, L., & Yoshikawa, E. (2018, June), Differences and Similarities in Student, Instructor, and Professional Perceptions of "Good Engineering Design" through Adaptive Comparative Judgment Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--30333

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2018 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015