BOARD # 313: [NSF:EEC CAREER]Fair Assessment: Year Three of Examining Cultural Familiarity to Decrease Bias in Engineering Classroom Assessments

Kerrie A Douglas; Tiantian Li; Shauna Adams

Download Paper | Permalink

Conference: 2025 ASEE Annual Conference & Exposition
Location: Montreal, Quebec, Canada
Publication Date: June 22, 2025
Start Date: June 22, 2025
End Date: August 15, 2025
Conference Session: NSF Grantees Poster Session II
Tagged Topics: Diversity and NSF Grantees Poster Session
Page Count: 6
Permanent URL: https://peer.asee.org/55679

Paper Authors

biography

Kerrie A Douglas Purdue University – West Lafayette (College of Engineering) orcid.org/0000-0002-2693-5272

visit author page

Dr. Douglas is an Associate Professor in the Purdue School of Engineering Education. Her research is focused on improving methods of assessment in engineering learning environments and supporting engineering students.

visit author page

biography

Tiantian Li Purdue University at West Lafayette (COE) orcid.org/0009-0001-4870-6008

visit author page

Tiantian Li (Olivia) is a dedicated Ph.D. student in Engineering Education at Purdue University. She received her Bachelor's degree in Biological Engineering, with a concentration in Pharmaceutical Processing Engineering. Currently, she is pursuing her Master's degree in Industrial Engineering. Tiantian has significant research experience in instrument development and validation analysis. Her research interests lie in developing reliable and valid measures for assessing complex engineering competencies, such as systems thinking skills. More particularly, she’s interested in assessing engineering students’ socio-technical systems thinking skills during their design process. In addition to her work on instrument development, Tiantian is also passionate about exploring the experiences of international scholars in the United States

visit author page

biography

Shauna Adams Purdue University – West Lafayette (College of Engineering)

visit author page

Shauna is currently a PhD student in Engineering Education at Purdue University. She previously worked for 10 years as a Systems Surety Engineer and 15 years as a coordinator/advisor of various pre-college engineering programs. Shauna earned her BS in Mechanical Engineering from North Carolina Agricultural and Technical State University and her MS in Mechanical Engineering from The Ohio State University. Her research interest includes understanding the impact of community-based engineering programs on student major choice for underrepresented groups in engineering, with a specific focus on Black/African American youth and inclusive program evaluation methods. Shauna is engaged in Purdue’s Engineering Education Graduate Student Association and Black Graduate Student Association.

visit author page

Download Paper | Permalink

Abstract

Background This CAREER project focuses on examining how cultural familiarity plays a role in racially minoritized students’ experience with engineering classroom assessments to advance knowledge on the fairness of such assessments. There are continual achievement gaps between racially minoritized students and racial majority students in engineering education assessments. In engineering classrooms, exams and quizzes make up a large percent of students’ course grades. Thus, student performances differences on these assessments impact their GPAs, which impact vital educational and career decisions such as selections of students to participate in internships, co-ops, research opportunities and even hiring. It is currently unknown how much of the achievement gaps reflects assessment bias as little attention has been paid on evaluating fairness of engineering education assessments. Unless more discourse on engineering education assessments’ fairness emerge to reveal underlying bias towards racially minoritized students, the field of engineering will continue to turn talented students away and its current issue of unequal representation will remain.

Purpose This CAREER project specifically examines commonly used concept inventories (CI) in engineering classrooms and how the CI items function for racially minoritized and racial majority students. The purpose of this poster paper is to provide an overview of this CAREER research project and plan. More specifically, this project investigates three research questions: 1) To what extent do items from commonly used engineering CIs demonstrate acceptable functioning (in terms of difficulty and discrimination) for racially minoritized students when compared to racial majorities? 2) What are patterns of cultural familiarity and content of problematic items and items that show acceptable functioning? And 3) How do racially minoritized students experience testing in engineering classrooms?

Methodology/approach This project adopts a mixed-method research design, consisting of first a quantitative phase followed by a qualitative phase. In the quantitative phase, we use Classical Test Theory to evaluate CI item functioning (in terms of difficulty and discrimination) for different racial/ethnicity groups and identify problematic items that consistently perform poorly to assess racially minoritized students. In the qualitative phase, we investigate the context of the problematic items to reveal underlying cultural familiarity bias. Also in the qualitative phase, we conduct interviews with racially minoritized students to explore their experiences with engineering classroom tests and their cognitive processes while answering the problematic CI items.

Implication Examining the fairness of engineering education assessments addresses an overlooked research area. Revealing underlying biases in commonly used CIs promotes the fair and equitable assessment for racially minoritized students. Ultimately, this project complements other efforts of increasing diversity and inclusion in engineering education, contributing to fixing systemic issues in this field that hinders the success of underrepresented minority students.

Citation
Format

Douglas, K. A., & Li, T., & Adams, S. (2025, June), BOARD # 313: [NSF:EEC CAREER]Fair Assessment: Year Three of Examining Cultural Familiarity to Decrease Bias in Engineering Classroom Assessments Paper presented at 2025 ASEE Annual Conference & Exposition , Montreal, Quebec, Canada . https://peer.asee.org/55679

TY  - CPAPER
AB  - Background
This CAREER project focuses on examining how cultural familiarity plays a role in racially minoritized students’ experience with engineering classroom assessments to advance knowledge on the fairness of such assessments. There are continual achievement gaps between racially minoritized students and racial majority students in engineering education assessments. In engineering classrooms, exams and quizzes make up a large percent of students’ course grades. Thus, student performances differences on these assessments impact their GPAs, which impact vital educational and career decisions such as selections of students to participate in internships, co-ops, research opportunities and even hiring. It is currently unknown how much of the achievement gaps reflects assessment bias as little attention has been paid on evaluating fairness of engineering education assessments. Unless more discourse on engineering education assessments’ fairness emerge to reveal underlying bias towards racially minoritized students, the field of engineering will continue to turn talented students away and its current issue of unequal representation will remain.

Purpose
This CAREER project specifically examines commonly used concept inventories (CI) in engineering classrooms and how the CI items function for racially minoritized and racial majority students. The purpose of this poster paper is to provide an overview of this CAREER research project and plan. More specifically, this project investigates three research questions: 1) To what extent do items from commonly used engineering CIs demonstrate acceptable functioning (in terms of difficulty and discrimination) for racially minoritized students when compared to racial majorities? 2) What are patterns of cultural familiarity and content of problematic items and items that show acceptable functioning? And 3) How do racially minoritized students experience testing in engineering classrooms?

Methodology/approach
This project adopts a mixed-method research design, consisting of first a quantitative phase followed by a qualitative phase. In the quantitative phase, we use Classical Test Theory to evaluate CI item functioning (in terms of difficulty and discrimination) for different racial/ethnicity groups and identify problematic items that consistently perform poorly to assess racially minoritized students. In the qualitative phase, we investigate the context of the problematic items to reveal underlying cultural familiarity bias. Also in the qualitative phase, we conduct interviews with racially minoritized students to explore their experiences with engineering classroom tests and their cognitive processes while answering the problematic CI items.

Implication
Examining the fairness of engineering education assessments addresses an overlooked research area. Revealing underlying biases in commonly used CIs promotes the fair and equitable assessment for racially minoritized students. Ultimately, this project complements other efforts of increasing diversity and inclusion in engineering education, contributing to fixing systemic issues in this field that hinders the success of underrepresented minority students.

AU  - Kerrie A Douglas
AU  - Tiantian Li
AU  - Shauna Adams
CY  - Montreal, Quebec, Canada 
DA  - 2025/06/22
PB  - ASEE Conferences
TI  - BOARD # 313: [NSF:EEC CAREER]Fair Assessment: Year Three of Examining Cultural Familiarity to Decrease Bias in Engineering Classroom Assessments
UR  - https://peer.asee.org/55679
ER  -