Asee peer logo

Using LMS Data to Provide Early Alerts to Struggling Students

Download Paper |


2017 FYEE Conference


Daytona Beach, Florida

Publication Date

August 6, 2017

Start Date

August 6, 2017

End Date

August 8, 2017

Conference Session

Student Success & Development - Focus on Academic Support

Tagged Topics

Diversity and FYEE Division - Paper Submission

Page Count


Permanent URL

Download Count


Request a correction

Paper Authors


Donald F. Hayes PE University of Nevada, Las Vegas

visit author page

Dr. Hayes is currently the Chair of Civil and Environmental Engineering and Construction at UNLV. He has over 25 years of academic experience and 12 years of industrial experience. He has been teaching a First Year Engineering Experience course since 2014.

visit author page

author page

Wonjoon Hong University of Nevada, Las Vegas


MATTHEW L BERNACKI University of Nevada, Las Vegas

visit author page

Matthew Bernacki is an Assistant Professor of Educational Psychology. He earned his Ph.D. in educational psychology in 2010 from Temple University in Philadelphia and also holds master’s degrees in Experimental Psychology from Saint Joseph’s University and Social Work from Temple University. Prior to arriving at UNLV, Matt worked at the Learning Research & Development Center (University of Pittsburgh) and the Human Computer Interaction Institute (Carnegie Mellon University) as a postdoctoral researcher at LearnLab.
Matt’s research focuses on (1) the roles that motivations and metacognitive processes play when learners use technologies like hypertext, intelligent tutoring systems, and learning management systems, (2) the development of interventions and software to promote effective learning strategies, and motivation to learn, and (3) the development of learning materials and environments that personalize learning to students’ interests.

visit author page

author page

Nicholas Voorhees

Download Paper |


The academic demands of college curricula often expose poor time management and study skills of freshmen students. Interventional advising can help get struggling students on track. Mid-term grades traditionally provide the first opportunity to identify those students; unfortunately, by the time mid-term grades are posted and interventional advising can occur, it is difficult for struggling students to significantly change their final grades. This project maximizes the utility of a Learning Management System (LMS) to improve the efficacy of early warnings by providing timely alerts to struggling students before they accrue poor performances.

An LMS can serve as a comprehensive platform for delivering rich multimedia content to learners, managing discussions, organizing collaborative and problem-based learning activities, and conducting assessments. In many engineering courses, the LMS is a barren space that provides only a syllabus, a few handouts, and - maybe - an online gradebook. We used the LMS to provide students with a rich digital environment for learning by creating and hosting all course materials within the LMS. A data management and visualization tool called Splunk was used to model usage of LMS resources and provide a timely picture of students’ learning progress. In an initial correlational study, students’ usage of course resources were found to correlate to performance in the first year engineering course (rLMSevents = .44, rfolders_accessed = .42, rLectureNoteDownloads = .39).

The traditional model of having mid-semester grades prompt meetings with an advisor is inherently flawed. First, it comes after mid-semester (i.e., week 9), which limits the time remaining for students who receive support to get themselves on track. Second, it relies on early poor performance, which once achieved diminish the potential to recover from early failure. Early, behavior-based prediction modeling and intervention avoids both these weaknesses.

A second study utilized educational data mining methods to produce a prediction algorithm based on digital course material usage. A logistic regression model was estimated using LMS behavioral data from the first five weeks of the course to predict student performance: whether a student obtained a C or better, or a D or worse. The cross-validated prediction model accurately classified 79% of students as C or Better vs. D or worse learners (Kappa = .57) based upon LMS access patterns. The model identified learners likely to perform poorly well before mid-semester grades. It accurately identified 54 of the 79 of students who ultimately failed to obtain a C or Better during the training and testing phase of prediction model development. This degree of specificity (68.39%) provided sufficient accuracy that the prediction algorithm was programmed back into Splunk to provide real time predictions of students’ success projections. An initial intervention study is ongoing to 1) identify students likely to struggle in the course, and 2) alert these students and provide them additional learning resources. The full paper will include a detailed account of prediction model features and additional results on the number of students identified, the percent of whom responded to alerts and learning supports, and the effects of the intervention.

Hayes, D. F., & Hong, W., & BERNACKI, M. L., & Voorhees, N. (2017, August), Using LMS Data to Provide Early Alerts to Struggling Students Paper presented at 2017 FYEE Conference, Daytona Beach, Florida.

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2017 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015