Daytona Beach, Florida
August 6, 2017
August 6, 2017
August 8, 2017
Diversity and FYEE Division - Paper Submission
The academic demands of college curricula often expose poor time management and study skills of freshmen students. Interventional advising can help get struggling students on track. Mid-term grades traditionally provide the first opportunity to identify those students; unfortunately, by the time mid-term grades are posted and interventional advising can occur, it is difficult for struggling students to significantly change their final grades. This project maximizes the utility of a Learning Management System (LMS) to improve the efficacy of early warnings by providing timely alerts to struggling students before they accrue poor performances.
An LMS can serve as a comprehensive platform for delivering rich multimedia content to learners, managing discussions, organizing collaborative and problem-based learning activities, and conducting assessments. In many engineering courses, the LMS is a barren space that provides only a syllabus, a few handouts, and - maybe - an online gradebook. We used the LMS to provide students with a rich digital environment for learning by creating and hosting all course materials within the LMS. A data management and visualization tool called Splunk was used to model usage of LMS resources and provide a timely picture of students’ learning progress. In an initial correlational study, students’ usage of course resources were found to correlate to performance in the first year engineering course (rLMSevents = .44, rfolders_accessed = .42, rLectureNoteDownloads = .39).
The traditional model of having mid-semester grades prompt meetings with an advisor is inherently flawed. First, it comes after mid-semester (i.e., week 9), which limits the time remaining for students who receive support to get themselves on track. Second, it relies on early poor performance, which once achieved diminish the potential to recover from early failure. Early, behavior-based prediction modeling and intervention avoids both these weaknesses.
A second study utilized educational data mining methods to produce a prediction algorithm based on digital course material usage. A logistic regression model was estimated using LMS behavioral data from the first five weeks of the course to predict student performance: whether a student obtained a C or better, or a D or worse. The cross-validated prediction model accurately classified 79% of students as C or Better vs. D or worse learners (Kappa = .57) based upon LMS access patterns. The model identified learners likely to perform poorly well before mid-semester grades. It accurately identified 54 of the 79 of students who ultimately failed to obtain a C or Better during the training and testing phase of prediction model development. This degree of specificity (68.39%) provided sufficient accuracy that the prediction algorithm was programmed back into Splunk to provide real time predictions of students’ success projections. An initial intervention study is ongoing to 1) identify students likely to struggle in the course, and 2) alert these students and provide them additional learning resources. The full paper will include a detailed account of prediction model features and additional results on the number of students identified, the percent of whom responded to alerts and learning supports, and the effects of the intervention.
Hayes, D. F., & Hong, W., & BERNACKI, M. L., & Voorhees, N. (2017, August), Using LMS Data to Provide Early Alerts to Struggling Students Paper presented at 2017 FYEE Conference, Daytona Beach, Florida. https://peer.asee.org/29442
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2017 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015