Asee peer logo

Using Natural Language Processing to Facilitate Student Feedback Analysis

Download Paper |

Conference

2021 ASEE Virtual Annual Conference Content Access

Location

Virtual Conference

Publication Date

July 26, 2021

Start Date

July 26, 2021

End Date

July 19, 2022

Conference Session

Research Methods and Studies on Engineering Education Research

Tagged Division

Educational Research and Methods

Page Count

14

Permanent URL

https://peer.asee.org/37994

Download Count

98

Request a correction

Paper Authors

biography

Andrew Katz Virginia Polytechnic Institute and State University

visit author page

Andrew Katz is an assistant professor in the Department of Engineering Education at Virginia Tech. He leads the Improving Decisions in Engineering Education Agents and Systems (IDEEAS) Lab, a group that uses multi-modal data to characterize, understand, and improve decisions made throughout engineering education ecosystems.

visit author page

biography

Matthew Norris Virginia Polytechnic Institute and State University Orcid 16x16 orcid.org/0000-0002-6587-7001

visit author page

Matthew Norris is a PhD student and Graduate Research Assistant in the Department of Engineering Education at Virginia Tech.

visit author page

biography

Abdulrahman M. Alsharif Virginia Polytechnic Institute and State University

visit author page

Abdulrahman M. Alsharif is a Ph.D. student and a research assistant in the Engineering Education department at Virginia Tech. He has received the Saudi Arabia Ministry of Higher Education and Scientific Research scholarship to pursue his Bachelor's and Master’s degrees in Industrial and Systems Engineering. His research interests are teaching and learning, policy and guidelines, and assessments. He hopes to work as a social scientist in engineering in higher education.

visit author page

author page

Michelle D. Klopfer Virginia Polytechnic Institute and State University

biography

David B. Knight Virginia Polytechnic Institute and State University Orcid 16x16 orcid.org/0000-0003-4576-2490

visit author page

David B. Knight is an Associate Professor in the Department of Engineering Education and Special Assistant to the Dean for Strategic Plan Implementation at Virginia Tech. He is also Director of Research of the Academy for Global Engineering at Virginia Tech and is affiliate faculty with the Higher Education Program. His research tends to be at the macro-scale, focused on a systems-level perspective of how engineering education can become more effective, efficient, and inclusive, tends to leverage large-scale institutional, state, or national data sets, and considers the intersection between policy and organizational contexts. He has B.S., M.S., and M.U.E.P. degrees from the University of Virginia and a Ph.D. in Higher Education from Pennsylvania State University.

visit author page

biography

Jacob R. Grohs Virginia Polytechnic Institute and State University

visit author page

Jacob Grohs is an Assistant Professor in Engineering Education at Virginia Tech with Affiliate Faculty status in Biomedical Engineering and Mechanics and the Learning Sciences and Technologies at Virginia Tech. He holds degrees in Engineering Mechanics (BS, MS) and in Educational Psychology (MAEd, PhD).

visit author page

Download Paper |

Abstract

In engineering education ecosystems, community members produce text through myriad activities both inside and outside of the classroom in teaching and research settings. In many of these cases, there is an abundance of text available to educators and researchers that could provide insight into various phenomena of interest within the system - student conceptual understanding, student experiences outside the classroom, how instructors can improve their teaching, or even shifts in collective conversations. Unfortunately, while these bodies of text have the potential to provide novel insights to educators and researchers, traditional analysis techniques do not scale well. For example, analyzing larger amounts of text can take one grader or researcher significantly more time than grading a small set of text responses. A larger body of text also creates more challenges for intrarater reliability. Likewise, expanding the size of the grading or research team can create interrater reliability challenges and the possibility of bias.

To address this opportunity, we have created a natural language processing system that augments human analysis so as to facilitate and enhance the work of one person (or team). Specifically, we take minimally pre-processed text, embed them using a pre-trained transformer (a specific kind of neural network architecture trained to encode inputs and decode outputs), and perform a sequence of dimension reduction techniques capped with a final clustering step. Such a system can help reduce the amount of time needed to analyze the text by effectively running a first pass on the text to group similar responses together. The human user can utilize these groupings to perform further analysis to fine tune and identify meanings in ways that only a human could. The system also can help improve consistency by analyzing across the entire collection of texts simultaneously and grouping similar items together. This is in contrast with a single person or a team that would have to work in series, analyzing responses sequentially and thereby creating the potential for inconsistencies across time.

In this paper we describe the system’s architecture and data processing steps. We demonstrate the utility of this approach by applying the method on a pair of questions from an end-of-semester feedback survey in a large, required introductory engineering course. The data were collected in spring 2020. The survey questions were part of a general feedback survey and asked students about their experiences in the transition to online learning subsequent to the SARS-CoV-2 outbreak. Given the variation in students’ experiences, this presented a challenging dataset.

Our results suggest that the pre-analysis text clustering can improve speed and accuracy of coding when compared with unassisted human coding. As natural language processing techniques continue to develop, the engineering education research community should continue to explore potential applications to improve understanding and sensemaking from large volumes of underutilized text data from both within and outside of classroom settings.

Katz, A., & Norris, M., & Alsharif, A. M., & Klopfer, M. D., & Knight, D. B., & Grohs, J. R. (2021, July), Using Natural Language Processing to Facilitate Student Feedback Analysis Paper presented at 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference. https://peer.asee.org/37994

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2021 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015