Asee peer logo

Board 39: Designing Intelligent Review Forms for Peer Assessment: A Data-driven Approach

Download Paper |

Conference

2019 ASEE Annual Conference & Exposition

Location

Tampa, Florida

Publication Date

June 15, 2019

Start Date

June 15, 2019

End Date

June 19, 2019

Conference Session

Educational Research and Methods Division Poster Session

Tagged Division

Educational Research and Methods

Page Count

14

Permanent URL

https://peer.asee.org/32337

Download Count

7

Request a correction

Paper Authors

biography

Zachariah J. Beasley University of South Florida Orcid 16x16 orcid.org/0000-0002-0146-2739

visit author page

Zachariah J. Beasley is a Ph.D. candidate, teaching assistant, and instructor at the University of South Florida in the Department of Computer Science and Engineering. He received his M.S. in Computer Science from USF in May 2017 and a B.A. in Computer Science and a B.A. in Applied Mathematics from Franklin College in May 2015. His teaching and research interests include Data Mining, Natural Language Processing (sentiment analysis, text processing), Crowd sourcing, Cyberlearning, Software Engineering, and Data Structures. He is a musician (guitar and bass) and was a collegiate athlete (soccer and tennis).

visit author page

biography

Les A. Piegl University of South Florida

visit author page

Les A. Piegl is a professor of computer science at the University of South Florida.

visit author page

biography

Paul Rosen University of South Florida

visit author page

Paul Rosen is an Assistant Professor at the University of South Florida in the Department of Computer Science and Engineering. He received his PhD from the Computer Science Department of Purdue University. His research interests include data visualization, topological data analysis, and computer science education. Along with his collaborators, he has received awards for best paper at PacificVis 2016, IVAPP 2016, PacificVis 2014, and SIBGRAPI 2013 and honorable mentions at the VAST Challenge 2017 and CG&A 2011 best paper.

visit author page

Download Paper |

Abstract

Designing Intelligent Review Forms for Peer Assessment: A Data-driven Approach

This research paper employs a data-driven, explainable, and scalable approach to the development and application of an online peer review form in computer science and engineering courses. Crowd-sourced grading through peer review is an effective evaluation methodology that 1) allows the use of meaningful assignments in large or online classes (e.g. assignments other than true/false, multiple choice, or short answer), 2) fosters learning and critical thinking in a student evaluating another’s work, and 3) provides a defendable and non-biased score through the wisdom of the crowd. Although peer review is widely utilized, to the authors’ best knowledge, the form itself and associated grading process have never been subjected to data-driven analysis and design. We present a novel, iterative algorithm by first gathering the most appropriate review form questions through intelligent data mining of past student reviews. During this process, key words and ideas are gathered for the positive and negative sentiment, flag word, and negate word dictionaries. Next, we revise our grading algorithm using simulations and perturbation to determine robustness (measured by standard deviation within a section). Using the dictionaries, we leverage sentiment gathered from reviewer detailed comments as a quality assurance mechanism to generate a crowd comment “grade”. This grade supplements the weighted average of the other review form sections. This result of this innovative process is a peer assessment package (robust algorithm leveraging crowd sentiment) based on actual student work that provides a semi-automated grader that can be used by a professor to confidently assign and grade any assignment in any size class in any environment.

Beasley, Z. J., & Piegl, L. A., & Rosen, P. (2019, June), Board 39: Designing Intelligent Review Forms for Peer Assessment: A Data-driven Approach Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. https://peer.asee.org/32337

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2019 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015