Grade-a-thons and Divide-and-Conquer: Effective Assessment at Scale

Brittany Ann Kos; Sarah Miller

Download Paper | Permalink

Conference: 2017 ASEE Annual Conference & Exposition
Location: Columbus, Ohio
Publication Date: June 24, 2017
Start Date: June 24, 2017
End Date: June 28, 2017
Conference Session: First-Year Programs: Assessment
Tagged Division: First-Year Programs
Page Count: 21
DOI: 10.18260/1-2--28415
Permanent URL: https://peer.asee.org/28415
Download Count: 677

Paper Authors

biography

Brittany Ann Kos University of Colorado, Boulder

visit author page

Brittany Kos is a PhD student at the ATLAS Institute at the University of Colorado Boulder. Her primary work is in undergraduate Computer Science Education and studying student hackathons from a feminist lens.

visit author page

biography

Sarah Miller University of Colorado, Boulder

visit author page

Sarah Miller provides vision and leadership for the recruitment, retention, and success of outstanding and diverse students, faculty, and staff to the University of Colorado Boulder’s College of Engineering and Applied Science. As Assistant Dean for Inclusive Excellence, she leads the Broadening Opportunity through Leadership and Diversity (BOLD) Center, overseeing efforts to attract and prepare students for the rigors of engineering study and careers, and to improve student performance and graduation rates. Appointed in January 2014, Miller comes to CU-Boulder from the National Science Foundation, where she worked in STEM education as a American Association for the Advancement of Science Fellow.

Sarah believes that every child deserves an excellent education. She has worked in inner-city public schools, both as a teacher and as an administrator, and in the admissions office of Amherst College, where she earned a B.A. in Chemistry. She holds a PhD from Yale University in chemical and environmental engineering, where her doctoral research produced a bio-based water purification system for removing arsenic from developing world water supplies.

visit author page

Download Paper | Permalink

Abstract

This complete evidence-based practice paper will describe our successful grading and assessment practices of a large freshmen engineering course. In the Fall of 2016 we taught “Introduction to Engineering”, a course designed to help students transition from high school to college and learn strategies to help them become successful engineering students. Over 70% of the students had not yet declared an engineering major but had intentions to transfer to an engineering major the following spring semester. This was a 1-credit hour, online and in-person hybrid class, technologically managed by a Learning Management Software (LMS).

Over 700 students enrolled in the course, and our instructional team consisted of one Instructor, one graduate TA, and two undergraduate TAs. This paper reports evidence-based practice of two assessment methods, Divide-and-Conquer and Grade-a-thons, that we used to successfully evaluate a large-enrollment course with small grading staff. The coursework was divided into two types of assignments: weekly homework and a final report.

The design of the course was based on content that had been previously implemented at this large, midwestern institution, as well as best practices learned from introduction to engineering courses at other institutions. In particular, the final project was based on Ray Landis’ work (Landis 2013). The weekly assignments were 1-page essay assignments. We asked students to reflect on the course’s assigned in-person activities, reading, and videos and to create a personal plan that would set themselves up to becoming a successful engineering student. To the best of our understanding, this is the largest implementation of “Design Your Successful Engineering Path” that has been able to grade final reports at this scale.

Weekly assignments were assessed with Divide-and-Conquer style grading. Student assignments were divided by last name into three even sections; each TA was responsible for one section and performed a combination of “hand-grading” and “mass-grading”. Hand-grading involved looking at each student's assignment, making an assessment based on a rubric, writing comments, and assigning a grade. Mass-grading consisted of giving all students full credit for the assignment. Every week TAs would hand-grade between 80-100 assignments and mass-grade the remaining 130-150 assignments. They would change which students were selected for hand-grading every week, so no student went more than two weeks without being hand-graded. This strategy allowed three TAs to give select students meaningful feedback on their assignments and monitor student progress. The TAs would first complete the hand-grading, to monitor the quality of student work for that assignment. Through this hand-grading assessment, the TAs found that there was a consistent rate of 90-95% of students turning in completed work that followed all requirements, earning the students full-credit. Due to the overall quality of student submissions, we justified mass-grading: giving the remaining students full-credit would be a reasonable strategy, since an overwhelming majority of submissions would have been assigned full-credit with the hand-grading strategy.

The second type of assignment that required assessment was a final report. This report was an accumulation of the previous weekly assignments, where students were expected to create a cohesive strategy to becoming a successful student engineer. These reports were 9-11 pages in length, not including appendices. Since this assignment was significantly longer than the weekly assignments and required that every student be hand-graded, we created a “Grade-a-thon” event that enabled us to grade all student reports in two eight-hour sessions. The grade-a-thon augmented practices seen in hackathons and standardized AP grading assessment. We hired an additional 16 Graders for this event, paying each Grader $100 for the entire day. The grade-a-thon was hosted on a Saturday from 9-5 in a large conference and workspace on campus. Before the event, the TAs graded four student reports to create a standardized grading practice. At the beginning of the event, we performed a calibration session where the Graders could see these examples and practice grading reports themselves; through the calibration session, we created regulated grading practices for all grading staff. We catered lunch and provided snacks for the grading team as well as organized breaks and activities every two hours. This prevented burn-out and kept all graders engaged in grading. In the end, this format allowed all students to receive individual assessment and feedback, while simultaneously expediting grading time and at a relatively low financial cost to the university.

We developed the Divide-and-Conquer and Grade-a-thon strategies as alternatives to automated assessment or hiring full-time appointment-based TAs. Our experiences with this introduction to engineering course show us that we can achieve effective assessment for a large enrollment course with a small instructional staff.

Citation
Format

Kos, B. A., & Miller, S. (2017, June), Grade-a-thons and Divide-and-Conquer: Effective Assessment at Scale Paper presented at 2017 ASEE Annual Conference & Exposition, Columbus, Ohio. 10.18260/1-2--28415

TY - CPAPER
AB - This complete evidence-based practice paper will describe our successful grading and assessment practices of a large freshmen engineering course. In the Fall of 2016 we taught “Introduction to Engineering”, a course designed to help students transition from high school to college and learn strategies to help them become successful engineering students. Over 70% of the students had not yet declared an engineering major but had intentions to transfer to an engineering major the following spring semester. This was a 1-credit hour, online and in-person hybrid class, technologically managed by a Learning Management Software (LMS).

Over 700 students enrolled in the course, and our instructional team consisted of one Instructor, one graduate TA, and two undergraduate TAs. This paper reports evidence-based practice of two assessment methods, Divide-and-Conquer and Grade-a-thons, that we used to successfully evaluate a large-enrollment course with small grading staff. The coursework was divided into two types of assignments: weekly homework and a final report.

The design of the course was based on content that had been previously implemented at this large, midwestern institution, as well as best practices learned from introduction to engineering courses at other institutions. In particular, the final project was based on Ray Landis’ work (Landis 2013). The weekly assignments were 1-page essay assignments. We asked students to reflect on the course’s assigned in-person activities, reading, and videos and to create a personal plan that would set themselves up to becoming a successful engineering student. To the best of our understanding, this is the largest implementation of “Design Your Successful Engineering Path” that has been able to grade final reports at this scale.

Weekly assignments were assessed with Divide-and-Conquer style grading. Student assignments were divided by last name into three even sections; each TA was responsible for one section and performed a combination of “hand-grading” and “mass-grading”. Hand-grading involved looking at each student&#39;s assignment, making an assessment based on a rubric, writing comments, and assigning a grade. Mass-grading consisted of giving all students full credit for the assignment. Every week TAs would hand-grade between 80-100 assignments and mass-grade the remaining 130-150 assignments. They would change which students were selected for hand-grading every week, so no student went more than two weeks without being hand-graded. This strategy allowed three TAs to give select students meaningful feedback on their assignments and monitor student progress. The TAs would first complete the hand-grading, to monitor the quality of student work for that assignment. Through this hand-grading assessment, the TAs found that there was a consistent rate of 90-95% of students turning in completed work that followed all requirements, earning the students full-credit. Due to the overall quality of student submissions, we justified mass-grading: giving the remaining students full-credit would be a reasonable strategy, since an overwhelming majority of submissions would have been assigned full-credit with the hand-grading strategy.

The second type of assignment that required assessment was a final report. This report was an accumulation of the previous weekly assignments, where students were expected to create a cohesive strategy to becoming a successful student engineer. These reports were 9-11 pages in length, not including appendices. Since this assignment was significantly longer than the weekly assignments and required that every student be hand-graded, we created a “Grade-a-thon” event that enabled us to grade all student reports in two eight-hour sessions. The grade-a-thon augmented practices seen in hackathons and standardized AP grading assessment. We hired an additional 16 Graders for this event, paying each Grader $100 for the entire day. The grade-a-thon was hosted on a Saturday from 9-5 in a large conference and workspace on campus. Before the event, the TAs graded four student reports to create a standardized grading practice. At the beginning of the event, we performed a calibration session where the Graders could see these examples and practice grading reports themselves; through the calibration session, we created regulated grading practices for all grading staff. We catered lunch and provided snacks for the grading team as well as organized breaks and activities every two hours. This prevented burn-out and kept all graders engaged in grading. In the end, this format allowed all students to receive individual assessment and feedback, while simultaneously expediting grading time and at a relatively low financial cost to the university.

We developed the Divide-and-Conquer and Grade-a-thon strategies as alternatives to automated assessment or hiring full-time appointment-based TAs. Our experiences with this introduction to engineering course show us that we can achieve effective assessment for a large enrollment course with a small instructional staff.

AU - Brittany Ann Kos
AU - Sarah Miller
CY - Columbus, Ohio
DA - 2017/06/24
PB - ASEE Conferences
TI - Grade-a-thons and Divide-and-Conquer: Effective Assessment at Scale
UR - https://peer.asee.org/28415
DO - 10.18260/1-2--28415
ER -