June 15, 2019
June 15, 2019
October 19, 2019
Educational Research and Methods
Designing Intelligent Review Forms for Peer Assessment: A Data-driven Approach
This research paper employs a data-driven, explainable, and scalable approach to the development and application of an online peer review form in computer science and engineering courses. Crowd-sourced grading through peer review is an effective evaluation methodology that 1) allows the use of meaningful assignments in large or online classes (e.g. assignments other than true/false, multiple choice, or short answer), 2) fosters learning and critical thinking in a student evaluating another’s work, and 3) provides a defendable and non-biased score through the wisdom of the crowd. Although peer review is widely utilized, to the authors’ best knowledge, the form itself and associated grading process have never been subjected to data-driven analysis and design. We present a novel, iterative algorithm by first gathering the most appropriate review form questions through intelligent data mining of past student reviews. During this process, key words and ideas are gathered for the positive and negative sentiment, flag word, and negate word dictionaries. Next, we revise our grading algorithm using simulations and perturbation to determine robustness (measured by standard deviation within a section). Using the dictionaries, we leverage sentiment gathered from reviewer detailed comments as a quality assurance mechanism to generate a crowd comment “grade”. This grade supplements the weighted average of the other review form sections. This result of this innovative process is a peer assessment package (robust algorithm leveraging crowd sentiment) based on actual student work that provides a semi-automated grader that can be used by a professor to confidently assign and grade any assignment in any size class in any environment.
Beasley, Z. J., & Piegl, L. A., & Rosen, P. (2019, June), Board 39: Designing Intelligent Review Forms for Peer Assessment: A Data-driven Approach Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. 10.18260/1-2--32337
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2019 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015