Dawe, N., & Romkey, L., & McCahan, S., & Lesmond, G. (2016, June), User Testing with Assessors to Develop Universal Rubric Rows for Assessing Engineering Design  Paper presented at 2016 ASEE Annual Conference & Exposition, New Orleans, Louisiana. 10.18260/p.27118

\bibitem{asee_peer_27118}
  Dawe, N., \& Romkey, L., \& McCahan, S., \& Lesmond, G. (2016, June), \emph{User Testing with Assessors to Develop Universal Rubric Rows for Assessing Engineering Design}
  Paper presented at 2016 ASEE Annual Conference \& Exposition, New Orleans, Louisiana.
  10.18260/p.27118

Nikita Dawe, Lisa Romkey, Susan McCahan, and Gayle Lesmond.  "User Testing with Assessors to Develop Universal Rubric Rows for Assessing Engineering Design".  2016 ASEE Annual Conference & Exposition, New Orleans, Louisiana, 2016, June.  ASEE Conferences, 2016.  https://peer.asee.org/27118 Internet.  25 Apr, 2024

\bibitem{asee_peer_27118}
  Nikita Dawe, Lisa Romkey, Susan McCahan, and Gayle Lesmond.
  "User Testing with Assessors to Develop Universal Rubric Rows for Assessing Engineering Design".
  \emph{2016 ASEE Annual Conference \& Exposition, New Orleans, Louisiana, 2016, June}.
  ASEE Conferences, 2016.
  https://peer.asee.org/27118 Internet.  25 Apr, 2024

@INPROCEEDINGS{asee_peer_27118
author = "Nikita Dawe, Lisa Romkey, Susan McCahan, and Gayle Lesmond"
title = "User Testing with Assessors to Develop Universal Rubric Rows for Assessing Engineering Design"
booktitle = "2016 ASEE Annual Conference \& Exposition"
year = "2016"
month = "June"
address = "New Orleans, Louisiana"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/27118}
number = {10.18260/p.27118}
}

TY  - CPAPER
AB  - This paper describes the process of testing and refining modular rubric rows developed for the assessment of engineering design activities. This is one component of a larger project to develop universal analytic rubrics for valid and reliable competency assessment across different academic disciplines and years of study. The project is being undertaken by researchers based in the faculty of applied science and engineering at a large research-intensive public university. From January 2014 to June 2015, we defined and validated indicators (criteria) for design and communication learning outcomes, then created descriptors for each indicator to discriminate between four levels of performance: Fails, Below, Meets, and Exceeds graduate expectations. From this bank of modular rubric items, applicable rows can be selected and compiled to produce a rubric tailored to a particular assessment activity. Here we discuss these rubrics within the larger context of the assessment of engineering design.
We tested draft rubrics in focus group sessions with assessors. We followed the testing with structured discussions to elicit feedback on the quality and usability of these rubrics, and to investigate how the assessors interpreted the language used in the indicators and descriptors. We asked participants to identify indicators they believed were irrelevant, redundant, or missing from the rubric. We also asked them to identify and discuss indicators and descriptors that were confusing. Finally, we asked them what changes they would recommend and what training materials they would find useful when using rubrics of this design.
By transcribing, coding, and analyzing recordings of these discussions, we identified rubric content that is unclear, ambiguous, or confusing to assessors and synthesized their recommendations for making the rubrics more usable. While some rubric rows received similar criticism from most participants, we identified many differences in assessors' rubric design preferences and in how they apply rubrics to evaluate student work. For example, some participants stated that the level of specificity in the indicators and descriptors made it more difficult to select a performance level. This feedback is surprising as it seems to contradict claims in the literature that providing more specific descriptions of quality makes rating more consistent. It also emerged that assessors have different conceptions of engineering design and the design process, and are confused when presented with unfamiliar terminology.
We will be applying this information to refine the rubrics and develop accompanying training materials. The improved rubric items will be evaluated for inter-rater reliability and deployed in academic courses. We will also perform user testing with undergraduate students.
AU  - Nikita Dawe
AU  - Lisa Romkey
AU  - Susan McCahan
AU  - Gayle Lesmond
CY  - New Orleans, Louisiana
DA  - 2016/06/26
PB  - ASEE Conferences
TI  - User Testing with Assessors to Develop Universal Rubric Rows for Assessing Engineering Design
UR  - https://peer.asee.org/27118
DO  - 10.18260/p.27118
ER  -