WORK IN PROGRESS: Data Explorer – Assessment Data Integration, Analytics, and Visualization for STEM Education Research

Joshua Levi Weese; William H. Hsu

Download Paper | Permalink

Conference: 2016 ASEE Annual Conference & Exposition
Location: New Orleans, Louisiana
Publication Date: June 26, 2016
Start Date: June 26, 2016
End Date: June 29, 2016
ISBN: 978-0-692-68565-5
ISSN: 2153-5965
Conference Session: Computers in Education Division Poster Session
Tagged Division: Computers in Education
Tagged Topic: Diversity
Page Count: 15
DOI: 10.18260/p.27215
Permanent URL: https://216.185.13.174/27215
Download Count: 565

Request a correction

Paper Authors

biography

Joshua Levi Weese Kansas State University

visit author page

Josh Weese is a PhD candidate in the department of Computer Science at Kansas State University. Focusing on education research, this experience comes from work as a graduate teaching assistant, various outreach programs, and time spent as a NSF GK-12 fellow. His downtime is spent in outreach programs aimed toward enriching local K-12 students' experience in STEM, especially in computer science and sensor technologies.

visit author page

biography

William H. Hsu Kansas State University

visit author page

William H. Hsu is an associate professor of Computing and Information Sciences at Kansas State University. He received a B.S. in Mathematical Sciences and Computer Science and an M.S.Eng. in Computer Science from Johns Hopkins University in 1993, and a Ph.D. in Computer Science from the University of Illinois at Urbana-Champaign in 1998. His dissertation explored the optimization of inductive bias in supervised machine learning for predictive analytics. At the National Center for Supercomputing Applications (NCSA), he was a co-recipient of an Industrial Grand Challenge Award for visual analytics of text corpora. His research interests include machine learning, probabilistic reasoning, and information visualization, with applications to geoinformatics, cybersecurity, education, digital humanities, and biomedical informatics. Published applications of his research include structured information extraction; spatiotemporal event detection for veterinary epidemiology, crime mapping, and opinion mining; and analysis of heterogeneous information networks. Current work in his lab deals with: deep learning and spatiotemporal pattern recognition; data mining and visualization in education research; graphical models of probability and utility for data science; and developing domain-adaptive models of large natural language corpora and social media for text mining, network science, sentiment analysis, and recommender systems. Dr. Hsu has over 50 refereed publications in conferences, journals, and books, plus over 40 additional publications.

visit author page

Download Paper | Permalink

Abstract

We describe two primary components of an analytics system for STEM education research developed for a physics education research portal. The purpose of this data exploration system is to allow instructors to comparatively assess student performance in intraclass, longitudinal, and interinstitutional contexts. The interface allows instructors to upload course data including student demographics, exams, and grading rubrics to a secure site, then retrieve descriptive statistics and detailed visualizations of this data. The first component consists of a rule-based system for pattern analysis that allows multiple common assessment formats to be inferred without metadata, and in some cases without headers. This paper describes the incremental development of a priority-based inference mechanism with matching heuristics, based on real and synthetic sample data, and further discusses the application of machine learning and data mining algorithms to the adaptation of probabilistic pattern analyzers. Early results indicate potential for user modeling and adaptive personalized recognition of document types and abstract type definitions. The second component is an information retrieval and information visualization module for comparative evaluation of uploaded and preprocessed data. Views are provided for inspection of aggregate statistics about student scores, comparison over time within one course, or comparison across multiple years. The design of this system includes a search facility for retrieving anonymized data from classes similar to the uploader’s own. These visualizations include tracking of student performance on a range of standardized assessments including Halloun et al.’s Force Concept Inventory (FCI, 1995), Thornton and Sokoloff’s Force and Motion Conceptual Evaluation (FMCE, 1998), and Chabay & Sherwood's Brief Electricity and Magnetism Assessment (BEMA, 2006). Assessments can be viewed as pre- and post-tests with comparative statistics (e.g., normalized gain), decomposed by answer in the case of multiple-choice questions, and manipulated using prespecified data transformations such as aggregation and refinement (drill down and roll up). Furthermore, the system is designed to incorporate a scalable framework for machine learning-based analytics, including clustering and similarity-based retrieval, time series prediction, and probabilistic reasoning. Both informal assessment of the system and intensive user testing on a pre-release version have yielded positive feedback. This feedback is instrumental in feature revision, both to improve system functionality and to plan the adaptation of the design of these two data exploration components to other STEM disciplines, such as computer science and mathematics. Lessons learned from visualization design and user experience feedback are reported in the context of usability criteria such as desired functionality of the pattern inference system. The paper concludes with a discussion of the system as an emerging technology, the schedule for its deployment and continued augmentation, and the design rationale for user-centered intelligent systems components. The focal point of future work in this area is on facilitating meaningful interactive exploration of the data by multiple types of stakeholders who have been identified for this type of education research portal. This is being achieved using a synthesis of data-driven approaches towards information extraction, retrieval, transformation, and visualization.

Citation
Format

Weese, J. L., & Hsu, W. H. (2016, June), WORK IN PROGRESS: Data Explorer – Assessment Data Integration, Analytics, and Visualization for STEM Education Research Paper presented at 2016 ASEE Annual Conference & Exposition, New Orleans, Louisiana. 10.18260/p.27215

TY  - CPAPER
AB  - We describe two primary components of an analytics system for STEM education research developed for a physics education research portal.  The purpose of this data exploration system is to allow instructors to comparatively assess student performance in intraclass, longitudinal, and interinstitutional contexts.  The interface allows instructors to upload course data including student demographics, exams, and grading rubrics to a secure site, then retrieve descriptive statistics and detailed visualizations of this data.
The first component consists of a rule-based system for pattern analysis that allows multiple common assessment formats to be inferred without metadata, and in some cases without headers.  This paper describes the incremental development of a priority-based inference mechanism with matching heuristics, based on real and synthetic sample data, and further discusses the application of machine learning and data mining algorithms to the adaptation of probabilistic pattern analyzers.  Early results indicate potential for user modeling and adaptive personalized recognition of document types and abstract type definitions.
The second component is an information retrieval and information visualization module for comparative evaluation of uploaded and preprocessed data.  Views are provided for inspection of aggregate statistics about student scores, comparison over time within one course, or comparison across multiple years. The design of this system includes a search facility for retrieving anonymized data from classes similar to the uploader’s own. These visualizations include tracking of student performance on a range of standardized assessments including Halloun et al.’s Force Concept Inventory (FCI, 1995), Thornton and Sokoloff’s Force and Motion Conceptual Evaluation (FMCE, 1998), and Chabay &amp; Sherwood&#39;s Brief Electricity and Magnetism Assessment (BEMA, 2006).  Assessments can be viewed as pre- and post-tests with comparative statistics (e.g., normalized gain), decomposed by answer in the case of multiple-choice questions, and manipulated using prespecified data transformations such as aggregation and refinement (drill down and roll up).  Furthermore, the system is designed to incorporate a scalable framework for machine learning-based analytics, including clustering and similarity-based retrieval, time series prediction, and probabilistic reasoning.  
Both informal assessment of the system and intensive user testing on a pre-release version have yielded positive feedback.  This feedback is instrumental in feature revision, both to improve system functionality and to plan the adaptation of the design of these two data exploration components to other STEM disciplines, such as computer science and mathematics.  Lessons learned from visualization design and user experience feedback are reported in the context of usability criteria such as desired functionality of the pattern inference system.
The paper concludes with a discussion of the system as an emerging technology, the schedule for its deployment and continued augmentation, and the design rationale for user-centered intelligent systems components.  The focal point of future work in this area is on facilitating meaningful interactive exploration of the data by multiple types of stakeholders who have been identified for this type of education research portal.  This is being achieved using a synthesis of data-driven approaches towards information extraction, retrieval, transformation, and visualization.

AU  - Joshua Levi Weese
AU  - William H. Hsu
CY  - New Orleans, Louisiana
DA  - 2016/06/26
PB  - ASEE Conferences
TI  - WORK IN PROGRESS: Data Explorer – Assessment Data Integration, Analytics, and Visualization for STEM Education Research
UR  - https://216.185.13.174/27215
DO  - 10.18260/p.27215
ER  -