Using Machine Learning to Analyze Short-Answer Responses to Conceptually Challenging Chemical Engineering Thermodynamics Questions

Harpreet Auby; Namrata Shivagunde; Anna Rumshisky; Milo Koretsky

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: July 12, 2024
Conference Session: DSA Technical Session 7
Tagged Topic: Data Science & Analytics Constituent Committee (DSA)
Permanent URL: https://peer.asee.org/48236

Request a correction

Paper Authors

biography

Harpreet Auby Tufts University orcid.org/0000-0002-0117-6097

visit author page

Harpreet is a graduate student in Chemical Engineering and STEM Education. He works with Dr. Milo Koretsky and helps study the role of learning assistants in the classroom as well as machine learning applications within educational research and evaluation. He is also involved in projects studying the uptake of the Concept Warehouse. His research interests include chemical engineering education, learning sciences, and social justice.

visit author page

author page

Milo Koretsky is the McDonnell Family Bridge Professor in the Department of Chemical and Biological Engineering and in the Department of Education at Tufts University. He is also co-Director of the Institute for Research on Learning and Instruction (IRLI). He received his B.S. and M.S. degrees from UC San Diego and his Ph.D. from UC Berkeley, all in chemical engineering.

visit author page

Download Paper | Permalink

Abstract

This work builds off a collaboration between machine learning and engineering education researchers to build a human-computer partnership utilizing machine learning to analyze student narratives of understanding in written short-answer responses to conceptually challenging questions [1], [2]. This study investigates student thinking in written short-answer responses justifying their answer choice to conceptually challenging multiple-choice concept questions often used in peer instruction or other active learning practices [3], [4]. The study then uses Large Language Models (LLMs) to automate the analysis of these responses. Students writing short answer responses to justify their reasoning to multiple choice concept questions has been shown to improve engagement, conceptual understanding, and has an overall positive effect on students’ answers [5], [6]. We conceptualize the responses that students write as “narratives of understanding,” as students use a combination of everyday and disciplinary language to construct a short-answer response around their understanding.

We utilized written responses available from consenting students in the Concept Warehouse (CW) [7], a web-based online tool for active learning. Two related questions from chemical engineering thermodynamics were coded by humans using emergent and inductive coding approaches [8], [9]: an enthalpy of mixing question (1396 responses) and an entropy of mixing question (1387 responses).

The written responses were then analyzed using LLMs-based coding methods. We used in-context learning for GPT-4 [10], where we prompted the model with the question, four in-context examples of answers, and the corresponding codes and instructed it to generate the code(s) for the new answer instance. We evaluated the model on a test set of 140 samples for each of the thermodynamics questions. Using both manual and language model-based coding, we aim to answer two research questions:

1. What aspects of student thinking are present in narratives of understanding constructed to justify conceptual questions about the enthalpy and entropy of mixing ideal gases? 2. To what extent can we use Large Language Models to automate qualitative coding of student narratives of understanding? Through this coding process, elements of student thinking were identified to describe the narratives students created to convey their understanding, including identification, comparison, and inference. In this paper, we conceptualize and discuss these three aspects of student thinking and how they convey an understanding of these concepts. When compared against manually coded responses, GPT-4 can generate codes with an F1 score of 54%, precision of 61%, and recall of 49% for the enthalpy of mixing question. For the entropy of mixing question, the model has an F1 score of 47%, precision of 48%, and recall of 47%. This study aims to contribute to the body of work that investigates applications of natural language processing to education research and, more generally, to qualitative coding processes.

Citation
Format

Auby, H., & Shivagunde, N., & Rumshisky, A., & Koretsky, M. (2024, June), Using Machine Learning to Analyze Short-Answer Responses to Conceptually Challenging Chemical Engineering Thermodynamics Questions Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. https://peer.asee.org/48236

TY - CPAPER
AB - This work builds off a collaboration between machine learning and engineering education researchers to build a human-computer partnership utilizing machine learning to analyze student narratives of understanding in written short-answer responses to conceptually challenging questions [1], [2]. This study investigates student thinking in written short-answer responses justifying their answer choice to conceptually challenging multiple-choice concept questions often used in peer instruction or other active learning practices [3], [4]. The study then uses Large Language Models (LLMs) to automate the analysis of these responses.

Students writing short answer responses to justify their reasoning to multiple choice concept questions has been shown to improve engagement, conceptual understanding, and has an overall positive effect on students’ answers [5], [6]. We conceptualize the responses that students write as “narratives of understanding,” as students use a combination of everyday and disciplinary language to construct a short-answer response around their understanding.

We utilized written responses available from consenting students in the Concept Warehouse (CW) [7], a web-based online tool for active learning. Two related questions from chemical engineering thermodynamics were coded by humans using emergent and inductive coding approaches [8], [9]: an enthalpy of mixing question (1396 responses) and an entropy of mixing question (1387 responses).

The written responses were then analyzed using LLMs-based coding methods. We used in-context learning for GPT-4 [10], where we prompted the model with the question, four in-context examples of answers, and the corresponding codes and instructed it to generate the code(s) for the new answer instance. We evaluated the model on a test set of 140 samples for each of the thermodynamics questions. Using both manual and language model-based coding, we aim to answer two research questions:

1. What aspects of student thinking are present in narratives of understanding constructed to justify conceptual questions about the enthalpy and entropy of mixing ideal gases?
2. To what extent can we use Large Language Models to automate qualitative coding of student narratives of understanding?

Through this coding process, elements of student thinking were identified to describe the narratives students created to convey their understanding, including identification, comparison, and inference. In this paper, we conceptualize and discuss these three aspects of student thinking and how they convey an understanding of these concepts. When compared against manually coded responses, GPT-4 can generate codes with an F1 score of 54%, precision of 61%, and recall of 49% for the enthalpy of mixing question. For the entropy of mixing question, the model has an F1 score of 47%, precision of 48%, and recall of 47%. This study aims to contribute to the body of work that investigates applications of natural language processing to education research and, more generally, to qualitative coding processes.

AU - Harpreet Auby
AU - Namrata Shivagunde
AU - Anna Rumshisky
AU - Milo Koretsky
CY - Portland, Oregon
DA - 2024/06/23
PB - ASEE Conferences
TI - Using Machine Learning to Analyze Short-Answer Responses to Conceptually Challenging Chemical Engineering Thermodynamics Questions
UR - https://peer.asee.org/48236
ER -