Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions: A Comparison of First-Year Students and Educators' Judgments

Clodagh Reid; Sheryl A. Sorby; Gibin Raju; Jeffrey Buckley; Niall Seery

Download Paper | Permalink

Conference: 2023 ASEE Annual Conference & Exposition
Location: Baltimore , Maryland
Publication Date: June 25, 2023
Start Date: June 25, 2023
End Date: June 28, 2023
Conference Session: First-Year Programs Division (FYP) - Technical Session 3: Evaluation & Assessment
Tagged Division: First-Year Programs Division (FYP)
Page Count: 14
DOI: 10.18260/1-2--44564
Permanent URL: https://peer.asee.org/44564
Download Count: 211

Paper Authors

biography

Clodagh Reid Technological University of the Shannon: Midlands Midwest orcid.org/0000-0002-8593-1730

visit author page

PhD in spatial ability and problem solving in engineering education from Technological University of the Shannon: Midlands Midwest. Graduated in 2017 from the University of Limerick with a B. Tech (Ed.). Member of Technology Education Research Group (TERG).

visit author page

biography

Sheryl A. Sorby University of Cincinnati

visit author page

Dr. Sheryl Sorby is currently a Professor of STEM Education at the University of Cincinnati and was recently a Fulbright Scholar at the Dublin Institute of Technology in Dublin, Ireland. She is a professor emerita of Mechanical Engineering-Engineering Mec

visit author page

biography

Gibin Raju University of Cincinnati orcid.org/0000-0003-2559-6931

visit author page

Gibin Raju is a second-year doctoral student pursuing Engineering Education in the Department of Engineering and Computing Education at the College of Engineering and Applied Sciences. His research interests are focused on spatial skills, cognitive stress, cognitive load, STEM accessibility issues, workforce development, engineering pathways, STEM education, ID/ODD, and education practices.

visit author page

biography

Jeffrey Buckley Royal Institute of Technology (KTH) orcid.org/0000-0002-8292-5642

visit author page

Dr Jeffrey Buckley received his PhD from KTH Royal Institute of Technology, Sweden, in the area of spatial ability and learning in technology education. He is a qualified post-primary teacher of Design and Communication Graphics and Construction Studies.

visit author page

biography

Niall Seery Technological University of the Shannon orcid.org/0000-0003-4199-4753

visit author page

Dr. Niall Seery is also the Director of the Technology Education Research Group (TERG) and is a Guest Professor in Technology Education at the Royal Institute of Technology (KTH) in Sweden.

visit author page

Download Paper | Permalink

Abstract

This Complete Research paper investigates the holistic assessment of creativity in design in engineering education. Design is a key element of engineering education given the emphasis on its development through the ABET criteria. As such, design projects play a central role in many first-year engineering courses. Creativity is a vital component of design capability which can influence design performance; however, it is difficult to measure using traditional assessment rubrics and holistic assessment approaches may be more suitable to assess these solutions. One such holistic assessment approach is Adaptive Comparative Judgement (ACJ). In this system, student designs are presented to judges in pairs, and they are asked to select the design that they deem to have demonstrated the greatest level of specified criteria. Each judge is asked to make multiple judgements where the work they are presented with is adaptively paired in order to create a ranked order of all solutions in the sample. The use of this approach in technology education has demonstrated high levels of reliability among judges (~0.9) irrespective of whether the judges are students or faculty. This research aimed to investigate the use of ACJ to holistically assess the creativity of design solutions in engineering education. The research also sought to explore the differences, if any, that would exist between the rank order produced by first-year engineering students and faculty who regularly teach first-year students. Forty-six first-year engineering students and 23 faculty participated in this research. A separate ACJ session was carried out with each of these groups; however, both groups were asked to assess the same items of work. Participants were instructed to assess the creativity of 101 solutions to a design task, a “Ping Pong problem,” where undergraduate engineering students had been asked to design a ping pong ball launcher. In both ACJ sessions each item of work (solution) was included in at least 11 pairwise comparisons, with the maximum number of comparisons for a single item being 29 in the faculty ACJ session and 50 in the student session. The data from the ACJ sessions were analyzed to determine the reliability of using ACJ to assess creativity of design solutions in first-year engineering education, and to explore whether the rankings produced from the first-year engineering students ACJ session differed significantly from those of the faculty. The results indicate a reasonably high level of reliability in both sessions as measured by the Scale Separation Reliability (SSR) coefficient, SSRfaculty = 0.65 ± 0.02, SSRstudents = 0.71 ± 0.02. Further a strong correlation was observed between the ACJ ranks produced by the students and faculty both when considered in terms of the relative differences between items of work, r = .533, p < .001, and their absolute rank position, σ = .553, p < .001. These findings indicate that ACJ is a promising tool for holistically assessing design solutions in engineering education. Additionally, given the strong correlation between ranks, ACJ could be used to include students in their own assessment to reduce the faculty grading burden or to develop a shared construct of capability to increase the alignment of teaching and learning.

Citation
Format

Reid, C., & Sorby, S. A., & Raju, G., & Buckley, J., & Seery, N. (2023, June), Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions: A Comparison of First-Year Students and Educators' Judgments Paper presented at 2023 ASEE Annual Conference & Exposition, Baltimore , Maryland. 10.18260/1-2--44564

TY - CPAPER
AB - This Complete Research paper investigates the holistic assessment of creativity in design in engineering education. Design is a key element of engineering education given the emphasis on its development through the ABET criteria. As such, design projects play a central role in many first-year engineering courses. Creativity is a vital component of design capability which can influence design performance; however, it is difficult to measure using traditional assessment rubrics and holistic assessment approaches may be more suitable to assess these solutions. One such holistic assessment approach is Adaptive Comparative Judgement (ACJ). In this system, student designs are presented to judges in pairs, and they are asked to select the design that they deem to have demonstrated the greatest level of specified criteria. Each judge is asked to make multiple judgements where the work they are presented with is adaptively paired in order to create a ranked order of all solutions in the sample. The use of this approach in technology education has demonstrated high levels of reliability among judges (~0.9) irrespective of whether the judges are students or faculty. This research aimed to investigate the use of ACJ to holistically assess the creativity of design solutions in engineering education. The research also sought to explore the differences, if any, that would exist between the rank order produced by first-year engineering students and faculty who regularly teach first-year students.
Forty-six first-year engineering students and 23 faculty participated in this research. A separate ACJ session was carried out with each of these groups; however, both groups were asked to assess the same items of work. Participants were instructed to assess the creativity of 101 solutions to a design task, a “Ping Pong problem,” where undergraduate engineering students had been asked to design a ping pong ball launcher. In both ACJ sessions each item of work (solution) was included in at least 11 pairwise comparisons, with the maximum number of comparisons for a single item being 29 in the faculty ACJ session and 50 in the student session. The data from the ACJ sessions were analyzed to determine the reliability of using ACJ to assess creativity of design solutions in first-year engineering education, and to explore whether the rankings produced from the first-year engineering students ACJ session differed significantly from those of the faculty. The results indicate a reasonably high level of reliability in both sessions as measured by the Scale Separation Reliability (SSR) coefficient, SSRfaculty = 0.65 ± 0.02, SSRstudents = 0.71 ± 0.02. Further a strong correlation was observed between the ACJ ranks produced by the students and faculty both when considered in terms of the relative differences between items of work, r = .533, p &amp;lt; .001, and their absolute rank position, σ = .553, p &amp;lt; .001. These findings indicate that ACJ is a promising tool for holistically assessing design solutions in engineering education. Additionally, given the strong correlation between ranks, ACJ could be used to include students in their own assessment to reduce the faculty grading burden or to develop a shared construct of capability to increase the alignment of teaching and learning.
AU - Clodagh Reid
AU - Sheryl A. Sorby
AU - Gibin Raju
AU - Jeffrey Buckley
AU - Niall Seery
CY - Baltimore , Maryland
DA - 2023/06/25
PB - ASEE Conferences
TI - Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions: A Comparison of First-Year Students and Educators' Judgments
UR - https://peer.asee.org/44564
DO - 10.18260/1-2--44564
ER -