WIP: Using Machine Learning to Automate Coding of Student Explanations to Challenging Mechanics Concept Questions

Harpreet Auby; Milo Koretsky; Namrata Shivagunde; Anna Rumshisky

Download Paper | Permalink

Conference: 2022 ASEE Annual Conference & Exposition
Location: Minneapolis, MN
Publication Date: August 23, 2022
Start Date: June 26, 2022
End Date: June 29, 2022
Conference Session: ERM: Exploring Educational Technology in Engineering
Page Count: 12
DOI: 10.18260/1-2--40507
Permanent URL: https://peer.asee.org/40507
Download Count: 777

Paper Authors

biography

Milo Koretsky Tufts University

visit author page

Milo Koretsky (he/him/his) is the McDonnell Family Bridge Professor holding a joint appointment in Chemical and Biological Engineering and Education at Tufts University. He received his BS and MS degrees from UC San Diego and his PhD from UC Berkeley, all in chemical engineering. He is interested in integrating technology into effective educational practices and in promoting the use of higher-level cognitive and social skills in engineering problem solving.

visit author page

biography

Harpreet Auby Tufts University

visit author page

Harpreet Auby is a STEM Education MS and Chemical Engineering PhD student at Tufts University. He is a graduate research assistant working with Dr. Milo Koretsky within the Institute for Research on Learning and Instruction (IRLI). Harpreet received his BS in Chemical and Biomolecular Engineering at the University of Illinois at Urbana-Champaign. His current work focuses on machine learning applications in educational research and evaluation, learning assistants, and uptake of an online technology tool emphasizing concept-based learning called the Concept Warehouse. His broad research interests include engineering education, learning and sensemaking in STEM, and liberatory pedagogies in STEM Ed.

visit author page

biography

Namrata Shivagunde University of Massachusetts Lowell

visit author page

Namrata Shivagunde is a phd student in computer science at UMass Lowell. She is working with Prof. Anna Rumshisky at Text Machine Lab. Her research is in application of deep learning techniques in natural language processing. Previously she did MS in Applied and Computational Mathematics from UMass Lowell.

visit author page

biography

Anna Rumshisky University of Massachusetts Lowell

visit author page

Anna Rumshisky is an Associate Professor of Computer Science at the University of Massachusetts Lowell, where she leads the Text Machine Lab for NLP. She is also a Visiting Academic at Amazon Alexa AI. Her primary research area is machine learning for natural language processing, with a focus on deep learning techniques.

visit author page

Download Paper | Permalink

Abstract

This work-in-progress paper describes a collaborative effort between engineering education and machine learning researchers to automate analysis of written responses conceptually challenging questions. These qualitative questions are often used in large STEM classes to support active learning pedagogies; they require minimum calculations and focus on the application of underlying physical phenomena to various situations. To emphasize reasoning and sense-making, we have developed an audience response system where students provide written justifications to concept questions. Written justifications better prepare students for discussions with peers and in the whole class and can also improve students’ answer choices. However, expository prose also presents a daunting amount of information for instructors to process.

For instructors, developing new machine learning algorithms will enable processing of large amounts of data regarding student explanations to provide information on patterns, trends, and general ideas of student thinking that they could utilize in their instructional practices and pedagogical decision-making processes. For educational researchers, these algorithms allow for ways to determine the narrative of understanding students have.

The participants in this study are part of a larger project in the propagation of an educational technology tool, the Concept Warehouse, to mechanics courses in a diverse set of two- and four- year institutions. Such service-oriented mechanics courses build foundational skills for upper-level engineering courses and develop students’ problem-solving capabilities. Educators describe these courses as focused on developing students' understanding of turning conceptual ideas like forces into physical representations of interactions between bodies. Thus, conceptualizing and developing standard mechanisms for problem solving may not be as concrete for students as compared to earlier courses.

Questions were selected from those commonly used by mechanics instructors in class. Following Creswell & Poth (2018), a coding scheme was developed to classify elements of student explanations and provide categories to train the machine learning algorithms. A combination of a priori and emergent approaches allowed for an initial set of concepts and ideas related to that specific question. These were then iteratively refined to create a narrative of students’ reasoning processes that included three main cognitive processes: identification, comparison, and inference.

For this task, we evaluated the capabilities of large pre-trained generative sequence-to-sequence language models (Sanh 2021; Brown 2020) leveraging coded data in a few-shot transfer learning setting. Such models are pre-trained on large amounts of narrative text in a self-supervised manner, using a language modeling objective, i.e., the task of predicting the next word in a natural language text. The few-shot transfer learning setting enables a model to learn the task when there is insufficient training data for training a supervised sequence tagging model. In order to identify and categorize the relevant concepts in the student responses, the model is prompted with an annotated example and asked to generate annotations for unseen text. This process leverages linguistic sequence patterns learned by the model during pre-training. We evaluate this approach using a held-out set of coded data, using a mapping based on the similarity between system outputs and coded concepts in the text embedding space.

Citation
Format

Koretsky, M., & Auby, H., & Shivagunde, N., & Rumshisky, A. (2022, August), WIP: Using Machine Learning to Automate Coding of Student Explanations to Challenging Mechanics Concept Questions Paper presented at 2022 ASEE Annual Conference & Exposition, Minneapolis, MN. 10.18260/1-2--40507

TY - CPAPER
AB - This work-in-progress paper describes a collaborative effort between engineering education and machine learning researchers to automate analysis of written responses conceptually challenging questions. These qualitative questions are often used in large STEM classes to support active learning pedagogies; they require minimum calculations and focus on the application of underlying physical phenomena to various situations. To emphasize reasoning and sense-making, we have developed an audience response system where students provide written justifications to concept questions. Written justifications better prepare students for discussions with peers and in the whole class and can also improve students’ answer choices. However, expository prose also presents a daunting amount of information for instructors to process.

For instructors, developing new machine learning algorithms will enable processing of large amounts of data regarding student explanations to provide information on patterns, trends, and general ideas of student thinking that they could utilize in their instructional practices and pedagogical decision-making processes. For educational researchers, these algorithms allow for ways to determine the narrative of understanding students have.

The participants in this study are part of a larger project in the propagation of an educational technology tool, the Concept Warehouse, to mechanics courses in a diverse set of two- and four- year institutions. Such service-oriented mechanics courses build foundational skills for upper-level engineering courses and develop students’ problem-solving capabilities. Educators describe these courses as focused on developing students' understanding of turning conceptual ideas like forces into physical representations of interactions between bodies. Thus, conceptualizing and developing standard mechanisms for problem solving may not be as concrete for students as compared to earlier courses.

Questions were selected from those commonly used by mechanics instructors in class. Following Creswell & Poth (2018), a coding scheme was developed to classify elements of student explanations and provide categories to train the machine learning algorithms. A combination of a priori and emergent approaches allowed for an initial set of concepts and ideas related to that specific question. These were then iteratively refined to create a narrative of students’ reasoning processes that included three main cognitive processes: identification, comparison, and inference.

For this task, we evaluated the capabilities of large pre-trained generative sequence-to-sequence language models (Sanh 2021; Brown 2020) leveraging coded data in a few-shot transfer learning setting. Such models are pre-trained on large amounts of narrative text in a self-supervised manner, using a language modeling objective, i.e., the task of predicting the next word in a natural language text. The few-shot transfer learning setting enables a model to learn the task when there is insufficient training data for training a supervised sequence tagging model. In order to identify and categorize the relevant concepts in the student responses, the model is prompted with an annotated example and asked to generate annotations for unseen text. This process leverages linguistic sequence patterns learned by the model during pre-training. We evaluate this approach using a held-out set of coded data, using a mapping based on the similarity between system outputs and coded concepts in the text embedding space.
AU - Milo Koretsky
AU - Harpreet Auby
AU - Namrata Shivagunde
AU - Anna Rumshisky
CY - Minneapolis, MN
DA - 2022/08/23
PB - ASEE Conferences
TI - WIP: Using Machine Learning to Automate Coding of Student Explanations to Challenging Mechanics Concept Questions
UR - https://peer.asee.org/40507
DO - 10.18260/1-2--40507
ER -