Asee peer logo

Board # 147 : Go With Your Gut! – Using Low-Time-Investment Evaluations of Student Work for Identifying High versus Low Quality Responses

Download Paper |


2017 ASEE Annual Conference & Exposition


Columbus, Ohio

Publication Date

June 24, 2017

Start Date

June 24, 2017

End Date

June 28, 2017

Conference Session

NSF Grantees Poster Session

Tagged Topic

NSF Grantees Poster Session

Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Matthew A. Verleger Ph.D. (He/His/Him) Embry-Riddle Aeronautical Univ., Daytona Beach

visit author page

Matthew Verleger is an Associate Professor of Engineering Fundamentals at Embry-Riddle Aeronautical University in Daytona Beach, Florida. His research interests are focused on using action research methodologies to develop immediate, measurable improvements in classroom instruction and the use of Model-Eliciting Activities (MEAs) in teaching students about engineering problem solving. Dr. Verleger is an active member of ASEE. He also serves as the developer and site manager for the Model-Eliciting Activities Learning System (, a site designed for implementing, managing, and researching MEAs in large classes.

visit author page

Download Paper |


Peer review can be a beneficial pedagogical tool for providing students both feedback and varied perspectives on their work. Despite being a valuable tool, the best mechanism for assigning reviewers to reviewees is still often blind random assignment. While better mechanisms must exist, they necessarily rely on having some prior knowledge about the work being reviewed. Prior research by the second author has found it difficult to accurately categorize the quality of student’s written work using autonomous mechanisms. This paper explores the effectiveness of a new approach for such categorization, with the intent of identifying if it is a viable technique at providing some of the prior knowledge needed to improve the peer review matching process.

The purpose of this paper is to present the findings from an effort to classify student team performance on Model-Eliciting Activities (MEAs) using a trained reviewer’s gut instinct about the quality of the work. MEAs are realistic, open-ended, client-driven engineering problems where teams of students produce a written document describing the steps of how to solve the problem. Using an archival data set from a large first-year engineering course at a mid-western, public land grant RU/VH institution, nearly 450 MEA solutions were evaluated by 2 researchers.

As part of his dissertation research, the second author evaluated all three drafts of 147 teams, investing approximately one hour per draft producing a detailed evaluation of each sample. Intra-rater reliability was calculated on 10% of that data and found to be acceptable on all the rubric items.

The first author was trained using the same methodology used for training the course’s graders, as documented in <AUTHOR’S PAPER>, but with the knowledge of the task they would be completing once they were trained. Once the first author was sufficiently trained, they were tasked with reviewing the same dataset as the second author with a goal of producing each review in 2 minutes or less. The goal of the research is to explore if the reviewer is able to develop a sufficient heuristic for classifying general MEA quality based on the existing successful training model. While the reviewer is unlikely to be as precise as the expert, they may be sufficiently accurate enough to use their gut-feeling reviews as baseline data for making peer review matching decisions with a comparatively minuscule investment in time.

While results are still being analyzed, and thus, not available for the abstract at this time, the focus of this paper will be on: • how the data was obtained and prepared, • the process by which both reviewers were trained and how their review processes differed, • how well (or poorly) the gut-feeling based reviews compared to the detailed expert’s reviews, • how accurate the new reviews were compared to the autonomous mechanisms tested in prior research, • what the results indicate about our implementation of MEAs and peer review, • what the results indicate about using gut-feeling evaluations for classification, and • how the results will be informing the next stages of the research project.

Verleger, M. A. (2017, June), Board # 147 : Go With Your Gut! – Using Low-Time-Investment Evaluations of Student Work for Identifying High versus Low Quality Responses Paper presented at 2017 ASEE Annual Conference & Exposition, Columbus, Ohio. 10.18260/1-2--27766

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2017 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015