WIP: Detection of Student Misconceptions of Electrical Circuit Concepts in a Short Answer Question Using NLP

James P Becker; Indika Kahanda; Nazmul H. Kazi

Download Paper | Permalink

Conference: 2021 ASEE Virtual Annual Conference Content Access
Location: Virtual Conference
Publication Date: July 26, 2021
Start Date: July 26, 2021
End Date: July 19, 2022
Conference Session: Electrical and Computer Division Poster Session
Tagged Division: Electrical and Computer
Page Count: 13
DOI: 10.18260/1-2--38076
Permanent URL: https://peer.asee.org/38076
Download Count: 393

Request a correction

Paper Authors

biography

James P Becker Montana State University, Bozeman

visit author page

James Becker is a Professor of electrical and computer engineering at Montana State University. His professional interests include microwave circuits, radio frequency electronics, nanoelectronics, pedagogical research, and distance education.

visit author page

biography

Indika Kahanda University of North Florida orcid.org/0000-0002-4536-6917

visit author page

Dr. Indika Kahanda is an Assistant Professor in the School of Computing at the University of North Florida, where he directs the bioinformatics, biomedical informatics and medical informatics lab. Prior to that Dr. Kahanda worked as an Assistant Professor in the Gianforte School of Computing at Montana State University. He received his Ph.D. in Computer Science from Colorado State University in 2016 in the area of Bioinformatics, a Master of Science in Computer Engineering from Purdue University in 2010, and a Bachelor of Science in Computer Engineering from University of Peradeniya, Sri Lanka in 2007.

visit author page

biography

Nazmul H. Kazi Montana State University

visit author page

Nazmul Kazi is a master's student of Computer Science at Montana State University. His research interests include the application of Artificial Intelligence, Deep Learning, Natural Language Processing, and Parallel Computing.

visit author page

Download Paper | Permalink

Abstract

While the use of writing exercises in gateway STEM courses that focus on solving numeric problems is not widespread, there is evidence that students could benefit from the addition of such exercises [1]. Writing exercises may be effective in both uncovering student misconceptions that are not necessarily apparent with typical computation problems, and as tools to foster conceptual change and metacognitive skill.

In this paper, pilot studies of the use of two Natural Language Processing (NLP) techniques to identify common misconceptions in the writing of students in a course on electric circuit analysis are described. Performance on the writing exercise in question has been shown to correlate with a student’s performance in the course [2]. This is of particular interest as the writing exercise has been administered during the fifth class period, sufficiently early to direct additional resources to the success of students appearing to be at-risk for failing the course. Realizing an automated software solution to analyze the responses to this exercise would remove burden on instructor time and open the door to immediate and personalized feedback to the student.

The first pilot study was run to determine how successful a simplistic rule-based approach would be in identifying the most common misconceptions found in a writing exercise requiring a student to speculate on the change in the power in the elements of a resistive circuit with a change to a single resistor value. An open-source NLP rule-based matching engine within spaCy [3] was used. The corpus consisted of one hundred and eighty-five unique responses to the question. Precision, recall, and F1-score [4] were used to assess the effectiveness of the rule-based NLP pipeline in comparison to that of a subject matter expert in identifying responses exemplifying seven misconceptions. Should this NLP pipeline be used in a system in which feedback is to be given to the student, a Directed Line of Reasoning (DLR) approach [5] would be beneficial in cases in which identification of a given misconception is in doubt. Considering this pilot study employed an extremely simplistic purely lexical-level rule-based classifier, the results are very promising and suggest the planned approach of developing a highly accurate, advanced rule-based classifier encompassing lexical/syntax/semantic driven rules is viable. As a compliment to the rule-based approach, this paper also describes a pilot study of the use of BERT (Bidirectional Encoder Representations from Transformers) [6], a machine learning approach that has shown tremendous promise in short-answer grading [7].

Citation
Format

Becker, J. P., & Kahanda, I., & Kazi, N. H. (2021, July), WIP: Detection of Student Misconceptions of Electrical Circuit Concepts in a Short Answer Question Using NLP Paper presented at 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference. 10.18260/1-2--38076

TY  - CPAPER
AB  - While the use of writing exercises in gateway STEM courses that focus on solving numeric problems is not widespread, there is evidence that students could benefit from the addition of such exercises [1].  Writing exercises may be effective in both uncovering student misconceptions that are not necessarily apparent with typical computation problems, and as tools to foster conceptual change and metacognitive skill.  

In this paper, pilot studies of the use of two Natural Language Processing (NLP) techniques to identify common misconceptions in the writing of students in a course on electric circuit analysis are described.  Performance on the writing exercise in question has been shown to correlate with a student’s performance in the course [2].  This is of particular interest as the writing exercise has been administered during the fifth class period, sufficiently early to direct additional resources to the success of students appearing to be at-risk for failing the course.  Realizing an automated software solution to analyze the responses to this exercise would remove burden on instructor time and open the door to immediate and personalized feedback to the student.  

The first pilot study was run to determine how successful a simplistic rule-based approach would be in identifying the most common misconceptions found in a writing exercise requiring a student to speculate on the change in the power in the elements of a resistive circuit with a change to a single resistor value.  An open-source NLP rule-based matching engine within spaCy [3] was used.  The corpus consisted of one hundred and eighty-five unique responses to the question. Precision, recall, and F1-score [4] were used to assess the effectiveness of the rule-based NLP pipeline in comparison to that of a subject matter expert in identifying responses exemplifying seven misconceptions.  Should this NLP pipeline be used in a system in which feedback is to be given to the student, a Directed Line of Reasoning (DLR) approach [5] would be beneficial in cases in which identification of a given misconception is in doubt.  Considering this pilot study employed an extremely simplistic purely lexical-level rule-based classifier, the results are very promising and suggest the planned approach of developing a highly accurate, advanced rule-based classifier encompassing lexical/syntax/semantic driven rules is viable.  As a compliment to the rule-based approach, this paper also describes a pilot study of the use of BERT (Bidirectional Encoder Representations from Transformers) [6], a machine learning approach that has shown tremendous promise in short-answer grading [7].
AU  - James P Becker
AU  - Indika Kahanda
AU  - Nazmul H. Kazi
CY  - Virtual Conference
DA  - 2021/07/26
PB  - ASEE Conferences
TI  - WIP: Detection of Student Misconceptions of Electrical Circuit Concepts in a Short Answer Question Using NLP 
UR  - https://peer.asee.org/38076
DO  - 10.18260/1-2--38076
ER  -