Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment Courses

Luanna B Prevost; Kevin C Haudek; Emily Norton Henry; Matthew C Berry; Mark Urban-Lurain

Download Paper | Permalink

Conference: 2013 ASEE Annual Conference & Exposition
Location: Atlanta, Georgia
Publication Date: June 23, 2013
Start Date: June 23, 2013
End Date: June 26, 2013
ISSN: 2153-5965
Conference Session: Teaching with Technology
Tagged Division: Educational Research and Methods
Page Count: 15
Page Numbers: 23.236.1 - 23.236.15
DOI: 10.18260/1-2--19250
Permanent URL: https://peer.asee.org/19250
Download Count: 557

Paper Authors

biography

Luanna B Prevost Michigan State University

visit author page

Dr. Prevost is a postdoctoral research associate with the Center of Engineering Education at Michigan State University. Her research interests are in student writing, problem solving, and technologies that can be used to assess and teach these skills.

visit author page

author page

Associate Professor
Center for Engineering Education Research
Undergraduate Studies Office
College of Engineering
Michigan State University

Dr. Urban-Lurain is responsible for teaching, research and curriculum development, with emphasis on engineering education and, more broadly, STEM education.

His research interests are in theories of cognition, how these theories inform the design of instruction, how we might best design instructional technology within those frameworks, and how the research and development of instructional technologies can inform our theories of cognition. He is also interested in preparing future STEM faculty for teaching, incorporating instructional technology as part of instructional design, and STEM education improvement and reform.

visit author page

Download Paper | Permalink

Abstract

Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment CoursesWritten formative assessments can provide instructors with rich insight into students’ thinkingabout scientific concepts. However, the time and effort involved in grading deter instructors fromhaving students write in large courses. As large-enrollment introductory STEM courses becomeincreasingly common, the need for innovations that facilitate the use of written assessmentscontinues to grow.We explored the use of automated text analysis to overcome these obstacles and facilitate the useof written formative assessment in a large-enrollment introductory biology course. Studentresponses to online homework on reaction thermodynamics were collected in two 300-personcourse sections one day before the next class meeting. We performed a two step analysis ofstudents’ written responses. First we used automated text analysis to extract and categorizethermodynamics concepts from student writing. Second, we used K-means cluster analysis toaggregate responses into distinct groups. We used IBM SPSS Modeler software to create“streams” which automate the complex series of analyses. We developed and piloted a rapidfeedback system to provide instructors feedback on students’ responses before the next classperiod ( less than one working day), so that they could use this feedback to inform theirinstruction.Rapid feedback reports presented the instructors with the number of clusters of responses, andthe percentage of students in each cluster. The reports describe the thermodynamics conceptsexpressed in student responses as captured by lexical categories, and provide the number ofresponses assigned to each category. The reports also summarize the categories which wereimportant to the clustering model and the category means for each cluster. For each cluster asample of responses closest to the cluster centroid, and therefore most representative of thecluster, was included. Additionally, instructors were provided with web diagrams - visualrepresentations of students’ ideas and the connections made among these ideas.Using this method in two course sections, we observed differences in students’ writing aboutreaction thermodynamics at the start of the semester. For example, our analysis isolated a clusterof students that identified the effect of increased temperature on reaction rate, but failed topropose a mechanism. This information was presented to the instructor in the rapid feedbackreport, so that mechanisms could be addressed during the next class meeting. Instructors alsoreceived examples of student writing that demonstrated student difficulties with thermodynamicconcepts, such as confusing reaction free energy and activation energy, or conflating bondstrength with reaction rate. The instructor incorporated students’ misconceptions into clickerquestions to tackle these misconceptions in class the following day. Our results suggest thatautomated text analysis paired with statistical analyses facilitates student writing in largeenrollment courses and provides instructors with feedback for just-in-time teaching.The full paper will include examples of the reports and more details about how the instructorsused the reports in their teaching.

Citation
Format

Prevost, L. B., & Haudek, K. C., & Henry, E. N., & Berry, M. C., & Urban-Lurain, M. (2013, June), Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment Courses Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--19250

TY  - CPAPER
AB  -            Automated Text Analysis Facilitates Using Written Formative        Assessments for Just-in-Time Teaching in Large Enrollment CoursesWritten formative assessments can provide instructors with rich insight into students’ thinkingabout scientific concepts. However, the time and effort involved in grading deter instructors fromhaving students write in large courses. As large-enrollment introductory STEM courses becomeincreasingly common, the need for innovations that facilitate the use of written assessmentscontinues to grow.We explored the use of automated text analysis to overcome these obstacles and facilitate the useof written formative assessment in a large-enrollment introductory biology course. Studentresponses to online homework on reaction thermodynamics were collected in two 300-personcourse sections one day before the next class meeting. We performed a two step analysis ofstudents’ written responses. First we used automated text analysis to extract and categorizethermodynamics concepts from student writing. Second, we used K-means cluster analysis toaggregate responses into distinct groups. We used IBM SPSS Modeler software to create“streams” which automate the complex series of analyses. We developed and piloted a rapidfeedback system to provide instructors feedback on students’ responses before the next classperiod ( less than one working day), so that they could use this feedback to inform theirinstruction.Rapid feedback reports presented the instructors with the number of clusters of responses, andthe percentage of students in each cluster. The reports describe the thermodynamics conceptsexpressed in student responses as captured by lexical categories, and provide the number ofresponses assigned to each category. The reports also summarize the categories which wereimportant to the clustering model and the category means for each cluster. For each cluster asample of responses closest to the cluster centroid, and therefore most representative of thecluster, was included. Additionally, instructors were provided with web diagrams - visualrepresentations of students’ ideas and the connections made among these ideas.Using this method in two course sections, we observed differences in students’ writing aboutreaction thermodynamics at the start of the semester. For example, our analysis isolated a clusterof students that identified the effect of increased temperature on reaction rate, but failed topropose a mechanism. This information was presented to the instructor in the rapid feedbackreport, so that mechanisms could be addressed during the next class meeting. Instructors alsoreceived examples of student writing that demonstrated student difficulties with thermodynamicconcepts, such as confusing reaction free energy and activation energy, or conflating bondstrength with reaction rate. The instructor incorporated students’ misconceptions into clickerquestions to tackle these misconceptions in class the following day. Our results suggest thatautomated text analysis paired with statistical analyses facilitates student writing in largeenrollment courses and provides instructors with feedback for just-in-time teaching.The full paper will include examples of the reports and more details about how the instructorsused the reports in their teaching.
AU  - Luanna B Prevost
AU  - Kevin C Haudek
AU  - Emily Norton Henry
AU  - Matthew C Berry
AU  - Mark Urban-Lurain
CY  - Atlanta, Georgia
DA  - 2013/06/23
PB  - ASEE Conferences
TI  - Automated Text Analysis Facilitates Using Written Formative Assessments for Just-in-Time Teaching in Large Enrollment Courses
UR  - https://peer.asee.org/19250
DO  - 10.18260/1-2--19250
ER  -