June 15, 2014
June 15, 2014
June 18, 2014
Computers in Education
24.565.1 - 24.565.17
Work-in-Progress: Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEMImproving STEM education requires valid and reliable instruments for providing insight intostudent thinking. Constructed response (CR) assessments reveal more about student thinkingand the persistence of misconceptions than do multiple-choice questions, but require moreanalysis on the part of educators.We have developed constructed response versions of well-established conceptual assessmentinventories and created computer automated analysis resources that predict human ratings ofstudent writing about these topics in introductory STEM courses. The research uses a two-stage,feature-based approach to automated analysis of constructed response assessments. First, wedesign items to identify important disciplinary constructs based on prior research. The items areadministered via online course management systems where students enter responses. We uselexical analysis software to extract key terms and scientific concepts from the students’ writing.These terms and concepts are used as variables for statistical classification techniques to predictexpert ratings of student responses. The inter-rater reliability (IRR) between automatedpredictions and expert human raters is as high as IRR between human experts.We recently received another round of funding to extend our work to provide an onlinecommunity where instructors may obtain, score and contribute to the library of items andresources necessary for their analyses.The specific goals of this project are to:1 Create a community web portal for the assessments to expand and deepen collaborations among STEM education researchers. This portal will provide the infrastructure for expanding the community of researchers and supporting the adoption and implementation of the innovative instructional materials by instructors at other institutions.2 Transport the innovations by providing instructors professional development and support to use the assessments. This includes information about common student conceptions revealed by the questions, instructional materials for addressing conceptual barriers, and the opportunity to join a community of practitioners who are using the questions and exchanging materials, and long-term, ongoing support.3 Expand the basic research to create and validate items in introductory chemistry, chemical engineering, and statistics.4 Engage in ongoing project evaluation for continuous quality improvement and to document the challenges and successes the project encounters.5 Lay the foundation for sustainability by providing interfaces for e-text publishers, Learning Management System (LMS) vendors and Massively Open Online Courses (MOOCs) as potential revenue streams to operate and maintain the online infrastructure.This paper will provide an overview of the goals of the project and introduce the opportunities toparticipate in the development of a national network of faculty using these techniques.
Urban-Lurain, M., & Cooper, M. M., & Haudek, K. C., & Kaplan, J. J., & Knight, J. K., & Lemons, P. P., & Lira, C. T., & Merrill, J. E., & Nehm, R., & Prevost, L. B., & Smith, M. K., & Sydlik, M. (2014, June), Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM Paper presented at 2014 ASEE Annual Conference & Exposition, Indianapolis, Indiana. https://peer.asee.org/20456
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2014 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015