Asee peer logo

Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment Courses

Download Paper |


2013 ASEE Annual Conference & Exposition


Atlanta, Georgia

Publication Date

June 23, 2013

Start Date

June 23, 2013

End Date

June 26, 2013



Conference Session

Teaching with Technology

Tagged Division

Educational Research and Methods

Page Count


Page Numbers

23.236.1 - 23.236.15



Permanent URL

Download Count


Request a correction

Paper Authors


Luanna B Prevost Michigan State University

visit author page

Dr. Prevost is a postdoctoral research associate with the Center of Engineering Education at Michigan State University. Her research interests are in student writing, problem solving, and technologies that can be used to assess and teach these skills.

visit author page

author page

Kevin C Haudek Michigan State University

author page

Emily Norton Henry Michigan State University

author page

Matthew C Berry Michigan State University


Mark Urban-Lurain Michigan State University

visit author page

Associate Professor
Center for Engineering Education Research
Undergraduate Studies Office
College of Engineering
Michigan State University

Dr. Urban-Lurain is responsible for teaching, research and curriculum development, with emphasis on engineering education and, more broadly, STEM education.

His research interests are in theories of cognition, how these theories inform the design of instruction, how we might best design instructional technology within those frameworks, and how the research and development of instructional technologies can inform our theories of cognition. He is also interested in preparing future STEM faculty for teaching, incorporating instructional technology as part of instructional design, and STEM education improvement and reform.

visit author page

Download Paper |


Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment CoursesWritten formative assessments can provide instructors with rich insight into students’ thinkingabout scientific concepts. However, the time and effort involved in grading deter instructors fromhaving students write in large courses. As large-enrollment introductory STEM courses becomeincreasingly common, the need for innovations that facilitate the use of written assessmentscontinues to grow.We explored the use of automated text analysis to overcome these obstacles and facilitate the useof written formative assessment in a large-enrollment introductory biology course. Studentresponses to online homework on reaction thermodynamics were collected in two 300-personcourse sections one day before the next class meeting. We performed a two step analysis ofstudents’ written responses. First we used automated text analysis to extract and categorizethermodynamics concepts from student writing. Second, we used K-means cluster analysis toaggregate responses into distinct groups. We used IBM SPSS Modeler software to create“streams” which automate the complex series of analyses. We developed and piloted a rapidfeedback system to provide instructors feedback on students’ responses before the next classperiod ( less than one working day), so that they could use this feedback to inform theirinstruction.Rapid feedback reports presented the instructors with the number of clusters of responses, andthe percentage of students in each cluster. The reports describe the thermodynamics conceptsexpressed in student responses as captured by lexical categories, and provide the number ofresponses assigned to each category. The reports also summarize the categories which wereimportant to the clustering model and the category means for each cluster. For each cluster asample of responses closest to the cluster centroid, and therefore most representative of thecluster, was included. Additionally, instructors were provided with web diagrams - visualrepresentations of students’ ideas and the connections made among these ideas.Using this method in two course sections, we observed differences in students’ writing aboutreaction thermodynamics at the start of the semester. For example, our analysis isolated a clusterof students that identified the effect of increased temperature on reaction rate, but failed topropose a mechanism. This information was presented to the instructor in the rapid feedbackreport, so that mechanisms could be addressed during the next class meeting. Instructors alsoreceived examples of student writing that demonstrated student difficulties with thermodynamicconcepts, such as confusing reaction free energy and activation energy, or conflating bondstrength with reaction rate. The instructor incorporated students’ misconceptions into clickerquestions to tackle these misconceptions in class the following day. Our results suggest thatautomated text analysis paired with statistical analyses facilitates student writing in largeenrollment courses and provides instructors with feedback for just-in-time teaching.The full paper will include examples of the reports and more details about how the instructorsused the reports in their teaching.

Prevost, L. B., & Haudek, K. C., & Henry, E. N., & Berry, M. C., & Urban-Lurain, M. (2013, June), Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment Courses Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--19250

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2013 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015