Asee peer logo

Deciphering Student Ideas on Thermodynamics Using Computerized Lexical Analysis of Student Writing

Download Paper |

Conference

2012 ASEE Annual Conference & Exposition

Location

San Antonio, Texas

Publication Date

June 10, 2012

Start Date

June 10, 2012

End Date

June 13, 2012

ISSN

2153-5965

Conference Session

Research in Assessment

Tagged Division

Educational Research and Methods

Page Count

12

Page Numbers

25.379.1 - 25.379.12

DOI

10.18260/1-2--21137

Permanent URL

https://peer.asee.org/21137

Download Count

505

Paper Authors

biography

Luanna B. Prevost Michigan State University

visit author page

Luanna Prevost is a Postdoctoral Research Associate at the Center for Engineering Education Research (CEER) at Michigan State University. She is a member of the Automated Analysis of Constructed Responses program, an NSF-funded cross-institutional collaboration of interdisciplinary science, technology, engineering, and mathematics (STEM) education researchers interested in exploring the use of automated text analysis to evaluate constructed-response assessments. Her research activities focus on instructional material development, learning assessment, and investigating student cognition STEM disciplines.

visit author page

author page

Kevin C. Haudek Michigan State University

biography

John E. Merrill Michigan State University

visit author page

Biological Sciences

visit author page

author page

Mark Urban-Lurain Michigan State University

Download Paper |

Abstract

Deciphering student ideas on thermodynamics using computerized lexical analysis of student writingConstructed responses, in which students describe their understanding in their own language,provide better insight into their thinking than do multiple-choice assessments. However,constructed responses are not often employed in large enrollment courses due to the time andfinancial constraints involved in grading these assessments. In this study, we examined studentunderstanding of thermodynamics using computerized lexical analysis of constructed responsesin a large enrollment course (N=384). Students were asked to interpret a graph depicting changesin free energy during the course of a reaction using both multiple-choice and open-endedresponses. Opened ended responses were analyzed using SPSS Text Analytics for Surveys(TAFS). The software extracts scientific terms from the students’ writing and places them intocategories using custom dictionaries of science terms. We validated the automated lexicalanalysis by using the categories created by TAFS as independent variables in discriminantanalysis to predict expert scoring of the students’ writing. Our findings to date reveal i) thatstudents hold a mix of correct and incorrect ideas about thermodynamics, and ii) that this resultis masked by multiple-choice testing. Over 50% of the students answering multiple-choicecorrectly displayed incorrect, or both correct and incorrect conceptualizations in their open-ended responses. Our results support previous studies that have uncovered students’heterogeneous ideas about matter and energy conservation and acid-base chemistry using lexicalanalysis. These findings suggest that computerized lexical analysis can improve instructors’understanding of the heterogeneity of ideas that student harbor about key concepts in STEMdisciplines and inform assessment practices.

Prevost, L. B., & Haudek, K. C., & Merrill, J. E., & Urban-Lurain, M. (2012, June), Deciphering Student Ideas on Thermodynamics Using Computerized Lexical Analysis of Student Writing Paper presented at 2012 ASEE Annual Conference & Exposition, San Antonio, Texas. 10.18260/1-2--21137

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2012 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015