June 23, 2013
June 23, 2013
June 26, 2013
Educational Research and Methods
23.236.1 - 23.236.15
Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment CoursesWritten formative assessments can provide instructors with rich insight into students’ thinkingabout scientific concepts. However, the time and effort involved in grading deter instructors fromhaving students write in large courses. As large-enrollment introductory STEM courses becomeincreasingly common, the need for innovations that facilitate the use of written assessmentscontinues to grow.We explored the use of automated text analysis to overcome these obstacles and facilitate the useof written formative assessment in a large-enrollment introductory biology course. Studentresponses to online homework on reaction thermodynamics were collected in two 300-personcourse sections one day before the next class meeting. We performed a two step analysis ofstudents’ written responses. First we used automated text analysis to extract and categorizethermodynamics concepts from student writing. Second, we used K-means cluster analysis toaggregate responses into distinct groups. We used IBM SPSS Modeler software to create“streams” which automate the complex series of analyses. We developed and piloted a rapidfeedback system to provide instructors feedback on students’ responses before the next classperiod ( less than one working day), so that they could use this feedback to inform theirinstruction.Rapid feedback reports presented the instructors with the number of clusters of responses, andthe percentage of students in each cluster. The reports describe the thermodynamics conceptsexpressed in student responses as captured by lexical categories, and provide the number ofresponses assigned to each category. The reports also summarize the categories which wereimportant to the clustering model and the category means for each cluster. For each cluster asample of responses closest to the cluster centroid, and therefore most representative of thecluster, was included. Additionally, instructors were provided with web diagrams - visualrepresentations of students’ ideas and the connections made among these ideas.Using this method in two course sections, we observed differences in students’ writing aboutreaction thermodynamics at the start of the semester. For example, our analysis isolated a clusterof students that identified the effect of increased temperature on reaction rate, but failed topropose a mechanism. This information was presented to the instructor in the rapid feedbackreport, so that mechanisms could be addressed during the next class meeting. Instructors alsoreceived examples of student writing that demonstrated student difficulties with thermodynamicconcepts, such as confusing reaction free energy and activation energy, or conflating bondstrength with reaction rate. The instructor incorporated students’ misconceptions into clickerquestions to tackle these misconceptions in class the following day. Our results suggest thatautomated text analysis paired with statistical analyses facilitates student writing in largeenrollment courses and provides instructors with feedback for just-in-time teaching.The full paper will include examples of the reports and more details about how the instructorsused the reports in their teaching.
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2013 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015