Asee peer logo

PrairieLearn: Mastery-based Online Problem Solving with Adaptive Scoring and Recommendations Driven by Machine Learning

Download Paper |

Conference

2015 ASEE Annual Conference & Exposition

Location

Seattle, Washington

Publication Date

June 14, 2015

Start Date

June 14, 2015

End Date

June 17, 2015

ISBN

978-0-692-50180-1

ISSN

2153-5965

Conference Session

Computer-Based Tests, Problems, and Other Instructional Materials

Tagged Division

Computers in Education

Page Count

14

Page Numbers

26.1238.1 - 26.1238.14

DOI

10.18260/p.24575

Permanent URL

https://peer.asee.org/24575

Download Count

1420

Paper Authors

biography

Matthew West University of Illinois, Urbana-Champaign Orcid 16x16 orcid.org/0000-0002-7605-0050

visit author page

Matthew West is an Associate Professor in the Department of Mechanical Science and Engineering at the University of Illinois at Urbana-Champaign. Prior to joining Illinois he was on the faculties of the Department of Aeronautics and Astronautics at Stanford University and the Department of Mathematics at the University of California, Davis. Prof. West holds a Ph.D. in Control and Dynamical Systems from the California Institute of Technology and a B.Sc. in Pure and Applied Mathematics from the University of Western Australia. His research is in the field of scientific computing and numerical analysis, where he works on computational algorithms for simulating complex stochastic systems such as atmospheric aerosols and feedback control. Prof. West is the recipient of the NSF CAREER award and is a University of Illinois Distinguished Teacher-Scholar and College of Engineering Education Innovation Fellow.

visit author page

biography

Geoffrey L. Herman University of Illinois, Urbana-Champaign Orcid 16x16 orcid.org/0000-0002-9501-2295

visit author page

Dr. Geoffrey L. Herman is a visiting assistant professor with the Illinois Foundry for Innovation in Engineering Education at the University of Illinois at Urbana-Champaign and a research assistant professor with the Department of Curriculum & Instruction. He earned his Ph.D. in Electrical and Computer Engineering from the University of Illinois at Urbana-Champaign as a Mavis Future Faculty Fellow and conducted postdoctoral research with Ruth Streveler in the School of Engineering Education at Purdue University. His research interests include creating systems for sustainable improvement in engineering education, promoting intrinsic motivation in the classroom, conceptual change and development in engineering students, and change in faculty beliefs about teaching and learning. He serves as the webmaster for the ASEE Educational Research and Methods Division.

visit author page

biography

Craig Zilles University of Illinois, Urbana-Champaign

visit author page

Craig Zilles is an Associate Professor in the Computer Science department at the University of Illinois at Urbana-Champaign. His current research focuses on computer science education and computer architecture. His research has been recognized by two best paper awards from ASPLOS (2010 and 2013) and by selection for inclusion in the IEEE Micro Top Picks from the 2007 Computer Architecture Conferences. He received the IEEE Education Society's Mac Van Valkenburg Early Career Teaching Award in 2010, a (campus-wise) Illinois Student Senate Teaching Excellence award in 2013, the NSF CAREER award, and the Univerisity of Illinois College of Engineering's Rose Award and Everitt Award for Teaching Excellence. Prior to his work on education and computer architecture, he developed the first algorithm that allowed rendering arbitrary three-dimensional polygonal shapes for haptic interfaces (force-feedback human-computer interfaces). He holds 6 patents.

visit author page

Download Paper |

Abstract

PrairieLearn: Mastery-based Online Problem Solving with Adaptive Scoring and Recommendations Driven by Machine LearningWe present an online problem-solving system (PrairieLearn) that is designed to facilitatelearning to mastery. The objectives of this system are to: (1) enable students to practicesolving randomized problem variants repeatedly until mastery, (2) incentivize students torepeat questions until mastery is achieved, and (3) provide immediate feedback abouttheir current mastery level to the student.To achieve these objectives, we implemented an open-source web-based online systemcalled PrairieLearn, which consists of a Node.js server and a JavaScript web-app topresent randomized question variants to students. As students attempt questions, thesystem uses Bayesian estimation on a four-parameter item-response model to computethe real-time maximum-likelihood estimate of the student’s ability on the current topic.This estimate is shown to the student as a “mastery score”, which they can increase bysolving questions correctly. Because the mastery score is based on an estimate of studentproblem-solving ability, solving each question will result in a different change inmastery. For example, if a student has a high mastery, then successfully solving an easyquestion will only increase their estimated mastery by a small amount. These per-question mastery changes from answering questions are dynamically pre-calculated foreach question and shown to students as “question scores”, which thus adaptively changein response to the student’s performance. Additionally, the expected value of masteryincrease for each question is computed and reflected to the student as a recommendationfor which question they should attempt next. Finally, the system recoreds all questionattempts by students and processes this offline to learn improved models for predictingstudent mastery via maximum likelihood optimization.The results of using PrairieLearn over several semesters in a large engineering course(Introductory Dynamics) include: (1) significant gains in student mastery, as measured byexam results and concept inventory questions, (2) improved student satisfaction whencompared to existing online problem-solving systems, and (3) high instructor satisfaction.We present data derived from students’ usage of PrairieLearn, as well as from studentsurveys and focus groups.

West, M., & Herman, G. L., & Zilles, C. (2015, June), PrairieLearn: Mastery-based Online Problem Solving with Adaptive Scoring and Recommendations Driven by Machine Learning Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24575

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015