WIP: Adaptive Comparative Judgement as a Tool for Assessing First-Year Engineering Design Projects

Clodagh Reid; Gibin Raju; Sheryl A. Sorby; Niall Seery

Download Paper | Permalink

Conference: 2022 First-Year Engineering Experience
Location: East Lansing, Michigan
Publication Date: July 31, 2022
Start Date: July 31, 2022
End Date: August 2, 2022
Conference Session: Technical Session M1
Tagged Topic: Works In Progress
Page Count: 5
DOI: 10.18260/1-2--42252
Permanent URL: https://peer.asee.org/42252
Download Count: 285

Paper Authors

biography

Clodagh Reid Technological University of the Shannon: Midlands Midwest orcid.org/0000-0002-8593-1730

visit author page

PhD in spatial ability and problem solving in engineering education from Technological University of the Shannon: Midlands Midwest, Ireland. Graduated in 2017 from the University of Limerick, Ireland with a B. Tech (Ed.). Member of Technology Education Research Group (TERG).

visit author page

biography

Gibin Raju University of Cincinnati orcid.org/0000-0003-2559-6931

visit author page

Gibin Raju is a doctoral student in Engineering Education at the College of Engineering and Applied Science at the University of Cincinnati. His research interests are focused on spatial visualization, problem-solving, cognitive load, engineering design, spatial ability, ID/ODD, stem accessibility issues, workforce development, STEM education, and education practices.

visit author page

biography

Sheryl A. Sorby University of Cincinnati

visit author page

Dr. Sheryl Sorby is currently a Professor of Engineering Education at the University of Cincinnati and was recently a Fulbright Scholar at the Dublin Institute of Technology in Dublin, Ireland. She is a professor emerita of Mechanical Engineering-Engineering Mechanics at Michigan Technological University and the P.I. or co-P.I. on more than $14M in grant funding, most for educational projects. She is the former Associate Dean for Academic Programs in the College of Engineering at Michigan Tech and she served at the National Science Foundation as a Program Director in the Division of Undergraduate Education from January 2007 through August 2009. Prior to her appointment as Associate Dean, Dr. Sorby served as chair of the Engineering Fundamentals Department at Michigan Tech. She received a B.S. in Civil Engineering, an M.S. in Engineering Mechanics, and a Ph.D. in Mechanical Engineering-Engineering Mechanics, all from Michigan Tech. Dr. Sorby has a well-established research program in spatial visualization and is actively involved in the development of various educational programs.

visit author page

biography

Niall Seery Technological University of the Shannon orcid.org/0000-0003-4199-4753

visit author page

Dr. Niall Seery is the Director of the Technology Education Research Group (TERG)

visit author page

Download Paper | Permalink

Abstract

Design projects are an important part of engineering education and are included in many first-year programs. In assessing these projects, educators most often use rubrics where points are given for meeting specific criteria and grades determined through adding up these points which can be time consuming and restrictive. Seldom is a holistic approach taken to assessing student design projects. The desire to employ holistic assessment strategies to student work with open-ended and divergent responses has been widely noted in the literature. Holistic strategies can provide insight into the role of qualities, such as creativity, which are not typically incorporated into standard assessment rubrics. Adaptive Comparative Judgement (ACJ) is an assessment approach developed to assess student performance in a holistic manner. The ACJ system is composed of three elements; a set of portfolios produced by students in response to an open-ended assignment, a set of judges which may be made up of students or experts in the area, and a pairs system. ACJ is supported by a software solution that adaptively selects pairs of portfolios and presents them to the judges, who judge the work based on professional constructs of performance and capability e.g., creativity. At any one time the judges are making comparisons between only two portfolios, and they select the one of the two that they believe is more creative (in this example). Each portfolio is judged on multiple occasions by various judges, in various pair combinations, providing a broader consensus of the creativity of the work based on multiple perspectives. The output of the judges is a rank-ordered list of the least to most creative portfolios. In studies using the ACJ system to measure various design qualities, high reliability levels have been achieved among judges (~0.9). Given the dynamic nature of the ACJ assessment tool, there are various benefits this approach can offer engineering educators. The demands placed on an educator to assess many portfolios may be reduced as students can act as the judges and reliably assess their peer’s work. In addition, the ACJ approach can support students in gaining feedback on the standard of their own work relative to that of their peers which is a valuable experience for first-year engineering students. This paper will explore the use and benefits of ACJ for assessing first-year engineering design projects. Further, conference attendees will be provided the opportunity throughout the conference to engage with the ACJ software as judges to experience how this system can work in practice for assessing student design projects.

Citation
Format

Reid, C., & Raju, G., & Sorby, S. A., & Seery, N. (2022, July), WIP: Adaptive Comparative Judgement as a Tool for Assessing First-Year Engineering Design Projects Paper presented at 2022 First-Year Engineering Experience, East Lansing, Michigan. 10.18260/1-2--42252

TY - CPAPER
AB - Design projects are an important part of engineering education and are included in many first-year programs. In assessing these projects, educators most often use rubrics where points are given for meeting specific criteria and grades determined through adding up these points which can be time consuming and restrictive. Seldom is a holistic approach taken to assessing student design projects. The desire to employ holistic assessment strategies to student work with open-ended and divergent responses has been widely noted in the literature. Holistic strategies can provide insight into the role of qualities, such as creativity, which are not typically incorporated into standard assessment rubrics.
Adaptive Comparative Judgement (ACJ) is an assessment approach developed to assess student performance in a holistic manner. The ACJ system is composed of three elements; a set of portfolios produced by students in response to an open-ended assignment, a set of judges which may be made up of students or experts in the area, and a pairs system. ACJ is supported by a software solution that adaptively selects pairs of portfolios and presents them to the judges, who judge the work based on professional constructs of performance and capability e.g., creativity. At any one time the judges are making comparisons between only two portfolios, and they select the one of the two that they believe is more creative (in this example). Each portfolio is judged on multiple occasions by various judges, in various pair combinations, providing a broader consensus of the creativity of the work based on multiple perspectives. The output of the judges is a rank-ordered list of the least to most creative portfolios. In studies using the ACJ system to measure various design qualities, high reliability levels have been achieved among judges (~0.9).
Given the dynamic nature of the ACJ assessment tool, there are various benefits this approach can offer engineering educators. The demands placed on an educator to assess many portfolios may be reduced as students can act as the judges and reliably assess their peer’s work. In addition, the ACJ approach can support students in gaining feedback on the standard of their own work relative to that of their peers which is a valuable experience for first-year engineering students. This paper will explore the use and benefits of ACJ for assessing first-year engineering design projects. Further, conference attendees will be provided the opportunity throughout the conference to engage with the ACJ software as judges to experience how this system can work in practice for assessing student design projects.
AU - Clodagh Reid
AU - Gibin Raju
AU - Sheryl A. Sorby
AU - Niall Seery
CY - East Lansing, Michigan
DA - 2022/07/31
PB - ASEE Conferences
TI - WIP: Adaptive Comparative Judgement as a Tool for Assessing First-Year Engineering Design Projects
UR - https://peer.asee.org/42252
DO - 10.18260/1-2--42252
ER -