A Hybrid Approach to Natural Language Processing for Analyzing Student Feedback about Faculty Support

Neha Kardam; Denise Wilson

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: June 26, 2024
Conference Session: DSA Technical Session 8
Tagged Topic: Data Science & Analytics Constituent Committee (DSA)
Page Count: 18
DOI: 10.18260/1-2--46447
Permanent URL: https://peer.asee.org/46447
Download Count: 145

Paper Authors

biography

Neha Kardam University of Washington

visit author page

Neha Kardam is a fourth-year Ph.D. student in Electrical and Computer Engineering at the University of Washington, Seattle. She is an interdisciplinary researcher with experience in statistics, predictive analytics, mixed methods research, and machine learning techniques in data-driven research.

visit author page

biography

Denise Wilson University of Washington orcid.org/0000-0002-2367-8602

visit author page

Denise Wilson is a professor and associate chair of diversity, equity, and inclusion in electrical and computer engineering at the University of Washington, Seattle. Her research interests in engineering education focus on the role of self-efficacy, belonging, and instructional support on engagement and motivation in the classroom while her engineering workplace research focuses on the role of relatedness, autonomy, and competence needs on persistence and fulfillment.

visit author page

Download Paper | Permalink

Abstract

Short-answer questions in surveys serve as a valuable educational tool, used for evaluating student learning and exploring the perspectives of various stakeholders in educational research. However, it is essential to distinguish between the objectives of automated short answer scoring systems (ASAS) and automated short answer coding (ASAC) systems. ASAS aims to achieve high accuracy primarily for fair assessment, while ASAC systems can accommodate slightly lower accuracy without compromising the validity of conclusions drawn from code analysis in the context of survey responses.

This study focuses on a dataset comprising responses from 1857 undergraduate students who were asked to express their views on how faculty could enhance their learning experiences. The dataset encompasses students from different engineering majors and various learning environments, presenting a challenge due to its intricacy and variability. To address this challenge, the study introduces a novel approach by integrating domain expert interaction and unsupervised learning into the ASAC process, specifically tailored for complex and heterogeneous datasets. Unlike previous research, this study emphasizes the iterative process of code refinement by domain experts, which is a significant departure from fully automated methods.

By combining domain expert insights with non-negative matrix factorization (NMF), the research advances the field by demonstrating that unsupervised learning can achieve accuracy levels on par with supervised learning for complex qualitative data. This is a key contribution, as it suggests a new paradigm in which the nuanced understanding of domain experts can significantly enhance the performance of machine learning techniques in educational research.

The study's findings underscore the importance of domain expert interaction, which led to a robustly labeled dataset for the faculty support survey question. This interaction was critical in achieving high precision, recall, and F1 scores (91.3% to 99.3%) in supervised learning models, indicating a successful prediction of themes in student responses.

The research provides evidence that the integration of expert knowledge can improve the efficiency and effectiveness of ASAC systems. It also demonstrates the potential for replacing manual coding with automated NLP coding, as evidenced by a moderate to substantial agreement (Cohen's kappa) between expert raters and between expert and NLP coding.

This study not only presents a methodological innovation by merging expert interaction with unsupervised learning but also provides empirical evidence of its effectiveness, offering a valuable contribution to the field of educational research and the development of ASAC systems.

Citation
Format

Kardam, N., & Wilson, D. (2024, June), A Hybrid Approach to Natural Language Processing for Analyzing Student Feedback about Faculty Support Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--46447

TY  - CPAPER
AB  - Short-answer questions in surveys serve as a valuable educational tool, used for evaluating student learning and exploring the perspectives of various stakeholders in educational research. However, it is essential to distinguish between the objectives of automated short answer scoring systems (ASAS) and automated short answer coding (ASAC) systems. ASAS aims to achieve high accuracy primarily for fair assessment, while ASAC systems can accommodate slightly lower accuracy without compromising the validity of conclusions drawn from code analysis in the context of survey responses.

This study focuses on a dataset comprising responses from 1857 undergraduate students who were asked to express their views on how faculty could enhance their learning experiences. The dataset encompasses students from different engineering majors and various learning environments, presenting a challenge due to its intricacy and variability. To address this challenge, the study introduces a novel approach by integrating domain expert interaction and unsupervised learning into the ASAC process, specifically tailored for complex and heterogeneous datasets. Unlike previous research, this study emphasizes the iterative process of code refinement by domain experts, which is a significant departure from fully automated methods.

By combining domain expert insights with non-negative matrix factorization (NMF), the research advances the field by demonstrating that unsupervised learning can achieve accuracy levels on par with supervised learning for complex qualitative data. This is a key contribution, as it suggests a new paradigm in which the nuanced understanding of domain experts can significantly enhance the performance of machine learning techniques in educational research.

The study&#39;s findings underscore the importance of domain expert interaction, which led to a robustly labeled dataset for the faculty support survey question. This interaction was critical in achieving high precision, recall, and F1 scores (91.3% to 99.3%) in supervised learning models, indicating a successful prediction of themes in student responses.

The research provides evidence that the integration of expert knowledge can improve the efficiency and effectiveness of ASAC systems. It also demonstrates the potential for replacing manual coding with automated NLP coding, as evidenced by a moderate to substantial agreement (Cohen&#39;s kappa) between expert raters and between expert and NLP coding.

This study not only presents a methodological innovation by merging expert interaction with unsupervised learning but also provides empirical evidence of its effectiveness, offering a valuable contribution to the field of educational research and the development of ASAC systems.

AU  - Neha Kardam
AU  - Denise Wilson
CY  - Portland, Oregon
DA  - 2024/06/23
PB  - ASEE Conferences
TI  - A Hybrid Approach to Natural Language Processing for Analyzing Student Feedback about Faculty Support
UR  - https://peer.asee.org/46447
DO  - 10.18260/1-2--46447
ER  -