When is Automated Feedback a Barrier to Timely Feedback?

Andrew Deorio; Christina Keefer

Download Paper | Permalink

Conference: 2022 ASEE Annual Conference & Exposition
Location: Minneapolis, MN
Publication Date: August 23, 2022
Start Date: June 26, 2022
End Date: June 29, 2022
Conference Session: Computers in Education 3 - Modulus I
Page Count: 15
DOI: 10.18260/1-2--40405
Permanent URL: https://peer.asee.org/40405
Download Count: 206

Paper Authors

biography

Andrew Deorio University of Michigan

visit author page

Andrew DeOrio is a teaching faculty member at the University of Michigan and a consultant for web and machine learning projects. His research interests are in engineering education and interdisciplinary computing. His teaching has been recognized with the Provost's Teaching Innovation Prize, and he has twice been named Professor of the Year by the students in his department. Andrew is trying to visit every U.S. National Park.

visit author page

author page

Christina Keefer University of Michigan

Download Paper | Permalink

Abstract

@inproceedings{baer-deorio, author = {Baer, Amy and DeOrio, Andrew}, title = {A Longitudinal View of Gender Balance in a Large Computer Science Program}, year = {2020}, isbn = {9781450367936}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3328778.3366806}, doi = {10.1145/3328778.3366806}, booktitle = {Proceedings of the 51st ACM Technical Symposium on Computer Science Education}, pages = {23–29}, numpages = {7}, keywords = {cs2, cs3, gender diversity, women in computer science, cs1}, location = {Portland, OR, USA}, series = {SIGCSE ’20} }

@article{perretta_deorio, title={Teaching Software Testing with Automated Feedback}, url={https://www.asee.org/public/conferences/106/papers/21636/view}, DOI={10.18260/1-2--31062}, journal={2018 ASEE Annual Conference & Exposition Proceedings}, author={Perretta, James and DeOrio, Andrew}, year={2018}}

@article{timely_feedback, author = { Richard Higgins and Peter Hartley and Alan Skelton }, title = {The Conscientious Consumer: Reconsidering the role of assessment feedback in student learning}, journal = {Studies in Higher Education}, volume = {27}, number = {1}, pages = {53-64}, year = {2002}, publisher = {Routledge}, doi = {10.1080/03075070120099368},

URL = { https://doi.org/10.1080/03075070120099368 }, eprint = { https://doi.org/10.1080/03075070120099368 } }

@book{ramsden_2003, place={London}, edition={2}, title={Learning to teach in higher education}, publisher={Routledge}, author={Ramsden, Paul}, year={2003}}

@inproceedings{its_sys_review, author = {Crow, Tyne and Luxton-Reilly, Andrew and Wuensche, Burkhard}, title = {Intelligent Tutoring Systems for Programming Education: A Systematic Review}, year = {2018}, isbn = {9781450363402}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3160489.3160492}, doi = {10.1145/3160489.3160492}, abstract = {A variety of intelligent tutoring systems have been created for the purpose of teaching computer programming. Most published literature focuses on systems that have been developed to teach programming within tertiary courses. A majority of systems have been developed to teach introductory programming concepts; other systems tutor more specific aspects of programming like scope or recursion. Literature reports that these systems address many of the difficulties associated with teaching programming to novices; however, individual systems vary greatly, and there is a large range of supplementary features developed in these systems. Most intelligent programming tutors involve some form of interactive programming exercises, but the use of supplementary features like plans, quizzes and worked solutions vary greatly between different systems. This systematic review reports key information about existing systems and the prevalence of different features within them. An overview of how supplementary features are integrated into these systems is given, along with implications for how intelligent programming tutors could be improved by supporting a wider range of supplementary features.}, booktitle = {Proceedings of the 20th Australasian Computing Education Conference}, pages = {53–62}, numpages = {10}, keywords = {computer science education, programming education, intelligent tutoring, novice programming}, location = {Brisbane, Queensland, Australia}, series = {ACE '18} }

@inproceedings{its_cs1, author = {Yoo, Jungsoon and Pettey, Chrisila and Yoo, Sung and Hankins, Judy and Li, Cen and Seo, Suk}, title = {Intelligent Tutoring System for CS-I and II Laboratory}, year = {2006}, isbn = {1595933158}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/1185448.1185482}, doi = {10.1145/1185448.1185482}, abstract = {A Web-based adaptive tutoring system which dynamically adapts to each student's needs and gives a student immediate feedback is being developed for our CS-I and CS-II closed laboratories. The system currently contains the question tutor, the program tutor, and the course management components. The tutoring components help students learn programming concepts through hands-on, self-paced exercises. The course management component helps teachers prepare and maintain the lab materials. Experiments have been conducted to evaluate the effectiveness of this new tutoring system and promising preliminary results were obtained.}, booktitle = {Proceedings of the 44th Annual Southeast Regional Conference}, pages = {146–151}, numpages = {6}, keywords = {web interfaces to databases, adaptive tutor, web-based laboratory, visualization, courseware}, location = {Melbourne, Florida}, series = {ACM-SE 44} }

@INPROCEEDINGS{its_effectiveness, author={J. C. {Nesbit} and O. O. {Adesope} and Q. {Liu} and W. {Ma}}, booktitle={2014 IEEE 14th International Conference on Advanced Learning Technologies}, title={How Effective are Intelligent Tutoring Systems in Computer Science Education?}, year={2014}, volume={}, number={}, pages={99-103}, doi={10.1109/ICALT.2014.38}}

@article{avoiding_help, author = {Midgley, Carol and Ryan, Allison and Pintrich, Paul}, year = {2001}, month = {06}, pages = {}, title = {Avoiding Seeking Help in the Classroom: Who and Why?}, volume = {13}, journal = {Educational Psychology Review}, doi = {10.1023/A:1009013420053} }

@article{help_motivation, author = {Ryan, Allison and Pintrich, P.R.}, year = {1997}, month = {01}, pages = {326-341}, title = {"Should I Ask for Help?" The Role of Motivation and Attitudes in Adolescents' Help Seeking in Math Class}, volume = {2}, journal = {Journal of Educational Psychology}, doi = {10.1037//0022-0663.89.2.329} }

@article{formative_feedback, author = {Valerie J. Shute}, title ={Focus on Formative Feedback}, journal = {Review of Educational Research}, volume = {78}, number = {1}, pages = {153-189}, year = {2008}, doi = {10.3102/0034654307313795},

URL = { https://doi.org/10.3102/0034654307313795 }, eprint = { https://doi.org/10.3102/0034654307313795 } , abstract = { This article reviews the corpus of research on feedback, with a focus on formative feedback—defined as information communicated to the learner that is intended to modify his or her thinking or behavior to improve learning. According to researchers, formative feedback should be nonevaluative, supportive, timely, and specific. Formative feedback is usually presented as information to a learner in response to some action on the learner’s part. It comes in a variety of types (e.g., verification of response accuracy, explanation of the correct answer, hints, worked examples) and can be administered at various times during the learning process (e.g., immediately following an answer, after some time has elapsed). Finally, several variables have been shown to interact with formative feedback’s success at promoting learning (e.g., individual characteristics of the learner and aspects of the task). All of these issues are discussed. This review concludes with guidelines for generating formative feedback. } }

@inproceedings{what_help_do_students_seek, author = {Ren, Yanyan and Krishnamurthi, Shriram and Fisler, Kathi}, title = {What Help Do Students Seek in TA Office Hours?}, year = {2019}, isbn = {9781450361859}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3291279.3339418}, doi = {10.1145/3291279.3339418}, abstract = {In many universities, Teaching Assistants (TAs) are an important part of students' educational experience. This is especially true in early courses, where students may suffer from inexperience and anxiety, and find fellow students more accessible than professors.Despite its importance, this learning channel has not been studied very much. Part of the difficulty lies in how to meaningfully evaluate it. Any intervention needs to be both unintrusive and lightweight, and yet yield useful data. As a result, to many faculty and researchers, TA office hours remain fairly opaque.This paper presents one approach to studying the technical component (but not the social dynamics) of TA office hours. We use a program-design methodology as a device to help track what students are asking about in hours, using a simple survey-based method to gather data. Data from TAs effectively summarize students' questions. In addition, contrasting data from both TAs and students provides insight into students' progress on program design help-seeking over the course of the semester.}, booktitle = {Proceedings of the 2019 ACM Conference on International Computing Education Research}, pages = {41–49}, numpages = {9}, keywords = {student self-assessment, program design method, teaching assistants}, location = {Toronto ON, Canada}, series = {ICER '19} }

@inproceedings{meaningful_feedback, author = {Haldeman, Georgiana and Tjang, Andrew and Babe\c{s}-Vroman, Monica and Bartos, Stephen and Shah, Jay and Yucht, Danielle and Nguyen, Thu D.}, title = {Providing Meaningful Feedback for Autograding of Programming Assignments}, year = {2018}, isbn = {9781450351034}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3159450.3159502}, doi = {10.1145/3159450.3159502}, abstract = {Autograding systems are increasingly being deployed to meet the challenge of teaching programming at scale. We propose a methodology for extending autograders to provide meaningful feedback for incorrect programs. Our methodology starts with the instructor identifying the concepts and skills important to each programming assignment, designing the assignment, and designing a comprehensive test suite. Tests are then applied to code submissions to learn classes of common errors and produce classifiers to automatically categorize errors in future submissions. The instructor maps the errors to concepts and skills and writes hints to help students find their misconceptions and mistakes. We have applied the methodology to two assignments from our Introduction to Computer Science course. We used submissions from one semester of the class to build classifiers and write hints for observed common errors. We manually validated the automatic error categorization and potential usefulness of the hints using submissions from a second semester. We found that the hints given for erroneous submissions should be helpful for 96% or more of the cases. Based on these promising results, we have deployed our hints and are currently collecting submissions and feedback from students and instructors.}, booktitle = {Proceedings of the 49th ACM Technical Symposium on Computer Science Education}, pages = {278–283}, numpages = {6}, keywords = {autograding, error categorization, concepts/skills-based hints}, location = {Baltimore, Maryland, USA}, series = {SIGCSE '18} }

@inproceedings{reusable_feedback, author = {Head, Andrew and Glassman, Elena and Soares, Gustavo and Suzuki, Ryo and Figueredo, Lucas and D'Antoni, Loris and Hartmann, Bj\"{o}rn}, title = {Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis}, year = {2017}, isbn = {9781450344500}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3051457.3051467}, doi = {10.1145/3051457.3051467}, abstract = {In large introductory programming classes, teacher feedback on individual incorrect student submissions is often infeasible. Program synthesis techniques are capable of fixing student bugs and generating hints automatically, but they lack the deep domain knowledge of a teacher and can generate functionally correct but stylistically poor fixes. We introduce a mixed-initiative approach which combines teacher expertise with data-driven program synthesis techniques. We demonstrate our novel approach in two systems that use different interaction mechanisms. Our systems use program synthesis to learn bug-fixing code transformations and then cluster incorrect submissions by the transformations that correct them. The MistakeBrowser system learns transformations from examples of students fixing bugs in their own submissions. The FixPropagator system learns transformations from teachers fixing bugs in incorrect student submissions. Teachers can write feedback about a single submission or a cluster of submissions and propagate the feedback to all other submissions that can be fixed by the same transformation. Two studies suggest this approach helps teachers better understand student bugs and write reusable feedback that scales to a massive introductory programming classroom.}, booktitle = {Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale}, pages = {89–98}, numpages = {10}, keywords = {program synthesis, programming education}, location = {Cambridge, Massachusetts, USA}, series = {L@S '17} }

@inproceedings{digital_hand, author = {Smith, Aaron J. and Boyer, Kristy Elizabeth and Forbes, Jeffrey and Heckman, Sarah and Mayer-Patel, Ketan}, title = {My Digital Hand: A Tool for Scaling Up One-to-One Peer Teaching in Support of Computer Science Learning}, year = {2017}, isbn = {9781450346986}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3017680.3017800}, doi = {10.1145/3017680.3017800}, abstract = {Increased enrollments in computer science programs presents a new challenge of quickly accommodating higher enrollment in computer science introductory courses. Because peer teaching scales with enrollment size, it is a promising solution for supporting computer science students in this setting. However, pedagogical and logistical challenges can arise when implementing a large peer teaching program. To study these challenges, we developed a transparent online tool, My Digital Hand, for tracking one-to-one peer teaching interactions. We deployed the tool across three universities in large CS2 computer science courses. The data gathered confirms the pedagogical and logistical challenges that exist at scale and gives insight into ways we might address them. Using this information, we developed the second iteration of My Digital Hand to better support peer teaching. This paper presents the modified tool for use by the computer science education community.}, booktitle = {Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education}, pages = {549–554}, numpages = {6}, keywords = {peer teaching, large scale instruction, learning management tools, computer science education}, location = {Seattle, Washington, USA}, series = {SIGCSE '17} }

@inproceedings{taskgrader, author = {Sharrock, R\'{e}mi and Bonfert-Taylor, Petra and Hiron, Mathias and Blockelet, Michel and Miller, Chris and Goudzwaard, Mike and Hamonic, Ella}, title = {Teaching C Programming Interactively at Scale Using Taskgrader: An Open-Source Autograder Tool}, year = {2019}, isbn = {9781450368049}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3330430.3333670}, doi = {10.1145/3330430.3333670}, abstract = {This demo paper introduces a tool and a method to provide a barriers-free, rich, interactive learning experience for students of all levels of preparation in programming courses. Taskgrader is an open-source autograding tool providing instant feedback in large-scale online programming classes. This in-browser tool offers extensive feedback to student code submissions right within any LMS and pass data back to the gradebook.}, booktitle = {Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale}, articleno = {56}, numpages = {2}, keywords = {Automated feedback tools, Automated grading tools, large-scale learning environments}, location = {Chicago, IL, USA}, series = {L@S '19} }

@article{feedback_assignments, author = {Singh, Rishabh and Gulwani, Sumit and Solar-Lezama, Armando}, title = {Automated Feedback Generation for Introductory Programming Assignments}, year = {2013}, issue_date = {June 2013}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {48}, number = {6}, issn = {0362-1340}, url = {https://doi.org/10.1145/2499370.2462195}, doi = {10.1145/2499370.2462195}, abstract = {We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong.We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to Programming course at MIT (6.00) and MITx (6.00x). Our results show that relatively simple error models can correct on average 64% of all incorrect submissions in our benchmark set.}, journal = {SIGPLAN Not.}, month = jun, pages = {15–26}, numpages = {12}, keywords = {program synthesis, automated grading, computer-aided education} }

@inproceedings{feedback_framework, author = {Gao, Jianxiong and Pang, Bei and Lumetta, Steven S.}, title = {Automated Feedback Framework for Introductory Programming Courses}, year = {2016}, isbn = {9781450342315}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/2899415.2899440}, doi = {10.1145/2899415.2899440}, abstract = {Using automated grading tools to provide feedback to students is common in Computer Science education. The first step of automated grading is to find defects in the student program. However, finding bugs in code has never been easy. Comparing computation results using a fixed set of test cases is still the most common way to determine correctness among current automated grading tools. It takes time and effort to design a good set of test cases that can test the student code thoroughly. In practice, tests used for grading are often insufficient for accurate diagnosis.In this paper, we present our utilization of industrial automated testing on student assignments in an introductory programming course. We implemented a framework to collect student codes and apply industrial automated testing to their codes. Then we interpreted the results obtained from testing in a way that students can understand easily. We deployed our framework on five different introductory C programming assignments here at the University of Illinois at Urbana-Champaign.The results show that the automated feedback generation framework can discover more errors inside student submissions and can provide timely and useful feedback to both instructors and students. A total of 142 missed bugs were found within 446 submissions. More than 50% of students received their feedback within 3 minutes of submission.}, booktitle = {Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education}, pages = {53–58}, numpages = {6}, keywords = {concolic testing, computer science education, auto grader}, location = {Arequipa, Peru}, series = {ITiCSE '16} } @article{feedback_programming, author = {Keuning, Hieke and Jeuring, Johan and Heeren, Bastiaan}, title = {A Systematic Literature Review of Automated Feedback Generation for Programming Exercises}, year = {2018}, issue_date = {January 2019}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {19}, number = {1}, url = {https://doi.org/10.1145/3231711}, doi = {10.1145/3231711}, abstract = {Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.}, journal = {ACM Trans. Comput. Educ.}, month = sep, articleno = {3}, numpages = {43}, keywords = {learning programming, programming tools, automated feedback, Systematic literature review} }

@article{feedback_power, author = {John Hattie and Helen Timperley}, title ={The Power of Feedback}, journal = {Review of Educational Research}, volume = {77}, number = {1}, pages = {81-112}, year = {2007}, doi = {10.3102/003465430298487},

URL = { https://doi.org/10.3102/003465430298487 }, eprint = { https://doi.org/10.3102/003465430298487 } , abstract = { Feedback is one of the most powerful influences on learning and achievement, but this impact can be either positive or negative. Its power is frequently mentioned in articles about learning and teaching, but surprisingly few recent studies have systematically investigated its meaning. This article provides a conceptual analysis of feedback and reviews the evidence related to its impact on learning and achievement. This evidence shows that although feedback is among the major influences, the type of feedback and the way it is given can be differentially effective. A model of feedback is then proposed that identifies the particular properties and circumstances that make it effective, and some typically thorny issues are discussed, including the timing of feedback and the effects of positive and negative feedback. Finally, this analysis is used to suggest ways in which feedback can be used to enhance its effectiveness in classrooms. } }

@article{feedback_cs1, author = {Ott, Claudia and Robins, Anthony and Shephard, Kerry}, title = {Translating Principles of Effective Feedback for Students into the CS1 Context}, year = {2016}, issue_date = {February 2016}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {16}, number = {1}, url = {https://doi.org/10.1145/2737596}, doi = {10.1145/2737596}, abstract = {Learning the first programming language is challenging for many students. High failure rates and bimodally distributed grades lead to a pedagogical interest in supporting students in first-year programming courses (CS1). In higher education, the important role of feedback for guiding the learning process and improving the learning outcome is widely acknowledged. This article introduces contemporary models of effective feedback practice as found in the higher education literature and offers an interpretation of those in the CS1 context. One particular CS1 course and typical course components are investigated to identify likely loci for feedback interventions and to connect related computer science education literature to these forms of feedback.}, journal = {ACM Trans. Comput. Educ.}, month = jan, articleno = {1}, numpages = {27}, keywords = {Effective feedback practice, CS1, higher education} }

@mastersthesis{lambda, Author = {Ball, Michael}, Editor = {Garcia, Dan}, Title = {Lambda: An Autograder for Snap!}, School = {EECS Department, University of California, Berkeley}, Year = {2018}, Month = {Jan}, URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2018/EECS-2018-2.html}, Number = {UCB/EECS-2018-2}, Abstract = {While visual programming languages are hardly a new development, recent pushes for increased equity and access to computer science have led to a renewed interest and use of visual programming languages. Autograders are a critical component of both in-person and online computer science courses. However, until this point, there haven't been any autograders documented for general-purpose, visual programming development environments. This is a particular shortcoming as introductory courses have scaled to larger numbers of students and online environments. While there are challenges to using autograders, we believe that the instant feedback capabilities, as well as potential time savings for course staff will help us teach a greater number of students.

Over the past year, we have built an autograder, named λ (Lambda), for Snap!, a visual blocks-based programming language inspired by MIT's Scratch. The primary motivation for developing the autograder was to run a series of Massive Open Online Courses (MOOCs) on edx.org throughout the 2015-2016 academic year. However, we also wish to use the autograder to better support in-person computer science courses. In Spring 2016, the Snap! autograder was used as a part of UC Berkeley's CS10, _The Beauty and Joy of Computing_, a "CS0" non-majors course.

This report describes λ, which consists of a "backend" Ruby on Rails webserver that allows us to use the autograder in a classroom setting, through a protocol called LTI. The backend web application contains a database of questions and test files, while the Snap! interface contains new features and a view to present the results of the autograder. Our initial results show the autograder successfully being used in CS10, where the autograder was used to supplement oral lab checkoffs, and on edx.org where the autograder was the primary method for students to receive credit for code, graded both for effort and correctness.} }

@book{NAP24926, author = "National Academies of Sciences, Engineering, and Medicine", title = "Assessing and Responding to the Growth of Computer Science Undergraduate Enrollments", isbn = "978-0-309-46702-5", doi = "10.17226/24926", abstract = "The field of computer science (CS) is currently experiencing a surge in undergraduate degree production and course enrollments, which is straining program resources at many institutions and causing concern among faculty and administrators about how best to respond to the rapidly growing demand. There is also significant interest about what this growth will mean for the future of CS programs, the role of computer science in academic institutions, the field as a whole, and U.S. society more broadly.\n\nAssessing and Responding to the Growth of Computer Science Undergraduate Enrollments seeks to provide a better understanding of the current trends in computing enrollments in the context of past trends. It examines drivers of the current enrollment surge, relationships between the surge and current and potential gains in diversity in the field, and the potential impacts of responses to the increased demand for computing in higher education, and it considers the likely effects of those responses on students, faculty, and institutions. This report provides recommendations for what institutions of higher education, government agencies, and the private sector can do to respond to the surge and plan for a strong and sustainable future for the field of CS in general, the health of the institutions of higher education, and the prosperity of the nation.", url = "https://www.nap.edu/catalog/24926/assessing-and-responding-to-the-growth-of-computer-science-undergraduate-enrollments", year = 2018, publisher = "The National Academies Press", address = "Washington, DC" }

@electronic{occupation, author = {{U.S. Bureau of Labor Statistics}}, title = {Computer and Information Technology Occupations : Occupational Outlook Handbook}, url={https://www.bls.gov/ooh/computer-and-information-technology/home.htm}, year={2020}, month={Apr} }

@misc{upenn_wait_times, title={Some CIS courses are so overloaded that students wait more than an hour for homework help}, url={https://www.thedp.com/article/2018/12/cis-120-office-hours-wait-time-penn-upenn-philadelphia}, journal={The Daily Pennsylvanian}, author={Basu, Kimaya}, year={2018}, month={Dec}}

@misc{umich_wait_times, title={Students, professors discuss finding resources for extra help in University's largest classes}, url={https://www.michigandaily.com/section/academics/students-professors-discuss-finding-resources-extra-help-university’s-largest}, journal={The Michigan Daily}, author={Weinstein, Liat}, year={2019}, month={Feb}}

@misc{stanford_wait_times, title={CS in Crisis: Is Stanford doing enough to respond to capacity and inclusion challenges?}, url={https://www.stanforddaily.com/2019/02/19/cs-in-crisis-is-stanford-doing-enough-to-respond-to-capacity-and-inclusion-challenges/}, journal={The Stanford Daily}, author={Liu, Jasmine}, year={2019}, month={Feb}}

Citation
Format

Deorio, A., & Keefer, C. (2022, August), When is Automated Feedback a Barrier to Timely Feedback? Paper presented at 2022 ASEE Annual Conference & Exposition, Minneapolis, MN. 10.18260/1-2--40405

TY  - CPAPER
AB  - 
@inproceedings{baer-deorio,
author = {Baer, Amy and DeOrio, Andrew},
title = {A Longitudinal View of Gender Balance in a Large Computer Science Program},
year = {2020},
isbn = {9781450367936},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3328778.3366806},
doi = {10.1145/3328778.3366806},
booktitle = {Proceedings of the 51st ACM Technical Symposium on Computer Science Education},
pages = {23–29},
numpages = {7},
keywords = {cs2, cs3, gender diversity, women in computer science, cs1},
location = {Portland, OR, USA},
series = {SIGCSE ’20}
}

 @article{perretta_deorio, title={Teaching Software Testing with Automated Feedback}, url={https://www.asee.org/public/conferences/106/papers/21636/view}, DOI={10.18260/1-2--31062}, journal={2018 ASEE Annual Conference &amp; Exposition  Proceedings}, author={Perretta, James and DeOrio, Andrew}, year={2018}} 

@article{timely_feedback,
author = { Richard   Higgins  and  Peter   Hartley  and  Alan   Skelton },
title = {The Conscientious Consumer: Reconsidering the role of assessment feedback in student learning},
journal = {Studies in Higher Education},
volume = {27},
number = {1},
pages = {53-64},
year  = {2002},
publisher = {Routledge},
doi = {10.1080/03075070120099368},

URL = { 
        https://doi.org/10.1080/03075070120099368
    
},
eprint = { 
        https://doi.org/10.1080/03075070120099368
    
}
}

 @book{ramsden_2003, place={London}, edition={2}, title={Learning to teach in higher education}, publisher={Routledge}, author={Ramsden, Paul}, year={2003}}

@inproceedings{its_sys_review,
author = {Crow, Tyne and Luxton-Reilly, Andrew and Wuensche, Burkhard},
title = {Intelligent Tutoring Systems for Programming Education: A Systematic Review},
year = {2018},
isbn = {9781450363402},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3160489.3160492},
doi = {10.1145/3160489.3160492},
abstract = {A variety of intelligent tutoring systems have been created for the purpose of teaching computer programming. Most published literature focuses on systems that have been developed to teach programming within tertiary courses. A majority of systems have been developed to teach introductory programming concepts; other systems tutor more specific aspects of programming like scope or recursion. Literature reports that these systems address many of the difficulties associated with teaching programming to novices; however, individual systems vary greatly, and there is a large range of supplementary features developed in these systems. Most intelligent programming tutors involve some form of interactive programming exercises, but the use of supplementary features like plans, quizzes and worked solutions vary greatly between different systems. This systematic review reports key information about existing systems and the prevalence of different features within them. An overview of how supplementary features are integrated into these systems is given, along with implications for how intelligent programming tutors could be improved by supporting a wider range of supplementary features.},
booktitle = {Proceedings of the 20th Australasian Computing Education Conference},
pages = {53–62},
numpages = {10},
keywords = {computer science education, programming education, intelligent tutoring, novice programming},
location = {Brisbane, Queensland, Australia},
series = {ACE '18}
}

@inproceedings{its_cs1,
author = {Yoo, Jungsoon and Pettey, Chrisila and Yoo, Sung and Hankins, Judy and Li, Cen and Seo, Suk},
title = {Intelligent Tutoring System for CS-I and II Laboratory},
year = {2006},
isbn = {1595933158},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/1185448.1185482},
doi = {10.1145/1185448.1185482},
abstract = {A Web-based adaptive tutoring system which dynamically adapts to each student's needs and gives a student immediate feedback is being developed for our CS-I and CS-II closed laboratories. The system currently contains the question tutor, the program tutor, and the course management components. The tutoring components help students learn programming concepts through hands-on, self-paced exercises. The course management component helps teachers prepare and maintain the lab materials. Experiments have been conducted to evaluate the effectiveness of this new tutoring system and promising preliminary results were obtained.},
booktitle = {Proceedings of the 44th Annual Southeast Regional Conference},
pages = {146–151},
numpages = {6},
keywords = {web interfaces to databases, adaptive tutor, web-based laboratory, visualization, courseware},
location = {Melbourne, Florida},
series = {ACM-SE 44}
}

@INPROCEEDINGS{its_effectiveness,  author={J. C. {Nesbit} and O. O. {Adesope} and Q. {Liu} and W. {Ma}},  booktitle={2014 IEEE 14th International Conference on Advanced Learning Technologies},   title={How Effective are Intelligent Tutoring Systems in Computer Science Education?},   year={2014},  volume={},  number={},  pages={99-103},  doi={10.1109/ICALT.2014.38}}

@article{avoiding_help,
author = {Midgley, Carol and Ryan, Allison and Pintrich, Paul},
year = {2001},
month = {06},
pages = {},
title = {Avoiding Seeking Help in the Classroom: Who and Why?},
volume = {13},
journal = {Educational Psychology Review},
doi = {10.1023/A:1009013420053}
}

@article{help_motivation,
author = {Ryan, Allison and Pintrich, P.R.},
year = {1997},
month = {01},
pages = {326-341},
title = {"Should I Ask for Help?" The Role of Motivation and Attitudes in Adolescents' Help Seeking in Math Class},
volume = {2},
journal = {Journal of Educational Psychology},
doi = {10.1037//0022-0663.89.2.329}
}

@article{formative_feedback,
author = {Valerie J. Shute},
title ={Focus on Formative Feedback},
journal = {Review of Educational Research},
volume = {78},
number = {1},
pages = {153-189},
year = {2008},
doi = {10.3102/0034654307313795},

URL = { 
        https://doi.org/10.3102/0034654307313795
    
},
eprint = { 
        https://doi.org/10.3102/0034654307313795
    
}
,
    abstract = { This article reviews the corpus of research on feedback, with a focus on formative feedback—defined as information communicated to the learner that is intended to modify his or her thinking or behavior to improve learning. According to researchers, formative feedback should be nonevaluative, supportive, timely, and specific. Formative feedback is usually presented as information to a learner in response to some action on the learner’s part. It comes in a variety of types (e.g., verification of response accuracy, explanation of the correct answer, hints, worked examples) and can be administered at various times during the learning process (e.g., immediately following an answer, after some time has elapsed). Finally, several variables have been shown to interact with formative feedback’s success at promoting learning (e.g., individual characteristics of the learner and aspects of the task). All of these issues are discussed. This review concludes with guidelines for generating formative feedback. }
}

@inproceedings{what_help_do_students_seek,
author = {Ren, Yanyan and Krishnamurthi, Shriram and Fisler, Kathi},
title = {What Help Do Students Seek in TA Office Hours?},
year = {2019},
isbn = {9781450361859},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3291279.3339418},
doi = {10.1145/3291279.3339418},
abstract = {In many universities, Teaching Assistants (TAs) are an important part of students'
educational experience. This is especially true in early courses, where students may
suffer from inexperience and anxiety, and find fellow students more accessible than
professors.Despite its importance, this learning channel has not been studied very
much. Part of the difficulty lies in how to meaningfully evaluate it. Any intervention
needs to be both unintrusive and lightweight, and yet yield useful data. As a result,
to many faculty and researchers, TA office hours remain fairly opaque.This paper presents
one approach to studying the technical component (but not the social dynamics) of
TA office hours. We use a program-design methodology as a device to help track what
students are asking about in hours, using a simple survey-based method to gather data.
Data from TAs effectively summarize students' questions. In addition, contrasting
data from both TAs and students provides insight into students' progress on program
design help-seeking over the course of the semester.},
booktitle = {Proceedings of the 2019 ACM Conference on International Computing Education Research},
pages = {41–49},
numpages = {9},
keywords = {student self-assessment, program design method, teaching assistants},
location = {Toronto ON, Canada},
series = {ICER '19}
}


@inproceedings{meaningful_feedback,
author = {Haldeman, Georgiana and Tjang, Andrew and Babe\c{s}-Vroman, Monica and Bartos, Stephen and Shah, Jay and Yucht, Danielle and Nguyen, Thu D.},
title = {Providing Meaningful Feedback for Autograding of Programming Assignments},
year = {2018},
isbn = {9781450351034},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3159450.3159502},
doi = {10.1145/3159450.3159502},
abstract = {Autograding systems are increasingly being deployed to meet the challenge of teaching programming at scale. We propose a methodology for extending autograders to provide meaningful feedback for incorrect programs. Our methodology starts with the instructor identifying the concepts and skills important to each programming assignment, designing the assignment, and designing a comprehensive test suite. Tests are then applied to code submissions to learn classes of common errors and produce classifiers to automatically categorize errors in future submissions. The instructor maps the errors to concepts and skills and writes hints to help students find their misconceptions and mistakes. We have applied the methodology to two assignments from our Introduction to Computer Science course. We used submissions from one semester of the class to build classifiers and write hints for observed common errors. We manually validated the automatic error categorization and potential usefulness of the hints using submissions from a second semester. We found that the hints given for erroneous submissions should be helpful for 96% or more of the cases. Based on these promising results, we have deployed our hints and are currently collecting submissions and feedback from students and instructors.},
booktitle = {Proceedings of the 49th ACM Technical Symposium on Computer Science Education},
pages = {278–283},
numpages = {6},
keywords = {autograding, error categorization, concepts/skills-based hints},
location = {Baltimore, Maryland, USA},
series = {SIGCSE '18}
}

@inproceedings{reusable_feedback,
author = {Head, Andrew and Glassman, Elena and Soares, Gustavo and Suzuki, Ryo and Figueredo, Lucas and D'Antoni, Loris and Hartmann, Bj\"{o}rn},
title = {Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis},
year = {2017},
isbn = {9781450344500},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3051457.3051467},
doi = {10.1145/3051457.3051467},
abstract = {In large introductory programming classes, teacher feedback on individual incorrect student submissions is often infeasible. Program synthesis techniques are capable of fixing student bugs and generating hints automatically, but they lack the deep domain knowledge of a teacher and can generate functionally correct but stylistically poor fixes. We introduce a mixed-initiative approach which combines teacher expertise with data-driven program synthesis techniques. We demonstrate our novel approach in two systems that use different interaction mechanisms. Our systems use program synthesis to learn bug-fixing code transformations and then cluster incorrect submissions by the transformations that correct them. The MistakeBrowser system learns transformations from examples of students fixing bugs in their own submissions. The FixPropagator system learns transformations from teachers fixing bugs in incorrect student submissions. Teachers can write feedback about a single submission or a cluster of submissions and propagate the feedback to all other submissions that can be fixed by the same transformation. Two studies suggest this approach helps teachers better understand student bugs and write reusable feedback that scales to a massive introductory programming classroom.},
booktitle = {Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale},
pages = {89–98},
numpages = {10},
keywords = {program synthesis, programming education},
location = {Cambridge, Massachusetts, USA},
series = {L@S '17}
}

@inproceedings{digital_hand,
author = {Smith, Aaron J. and Boyer, Kristy Elizabeth and Forbes, Jeffrey and Heckman, Sarah and Mayer-Patel, Ketan},
title = {My Digital Hand: A Tool for Scaling Up One-to-One Peer Teaching in Support of Computer Science Learning},
year = {2017},
isbn = {9781450346986},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3017680.3017800},
doi = {10.1145/3017680.3017800},
abstract = {Increased enrollments in computer science programs presents a new challenge of quickly accommodating higher enrollment in computer science introductory courses. Because peer teaching scales with enrollment size, it is a promising solution for supporting computer science students in this setting. However, pedagogical and logistical challenges can arise when implementing a large peer teaching program. To study these challenges, we developed a transparent online tool, My Digital Hand, for tracking one-to-one peer teaching interactions. We deployed the tool across three universities in large CS2 computer science courses. The data gathered confirms the pedagogical and logistical challenges that exist at scale and gives insight into ways we might address them. Using this information, we developed the second iteration of My Digital Hand to better support peer teaching. This paper presents the modified tool for use by the computer science education community.},
booktitle = {Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education},
pages = {549–554},
numpages = {6},
keywords = {peer teaching, large scale instruction, learning management tools, computer science education},
location = {Seattle, Washington, USA},
series = {SIGCSE '17}
}

@inproceedings{taskgrader,
author = {Sharrock, R\'{e}mi and Bonfert-Taylor, Petra and Hiron, Mathias and Blockelet, Michel and Miller, Chris and Goudzwaard, Mike and Hamonic, Ella},
title = {Teaching C Programming Interactively at Scale Using Taskgrader: An Open-Source Autograder Tool},
year = {2019},
isbn = {9781450368049},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3330430.3333670},
doi = {10.1145/3330430.3333670},
abstract = {This demo paper introduces a tool and a method to provide a barriers-free, rich, interactive learning experience for students of all levels of preparation in programming courses. Taskgrader is an open-source autograding tool providing instant feedback in large-scale online programming classes. This in-browser tool offers extensive feedback to student code submissions right within any LMS and pass data back to the gradebook.},
booktitle = {Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale},
articleno = {56},
numpages = {2},
keywords = {Automated feedback tools, Automated grading tools, large-scale learning environments},
location = {Chicago, IL, USA},
series = {L@S '19}
}

@article{feedback_assignments,
author = {Singh, Rishabh and Gulwani, Sumit and Solar-Lezama, Armando},
title = {Automated Feedback Generation for Introductory Programming Assignments},
year = {2013},
issue_date = {June 2013},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {48},
number = {6},
issn = {0362-1340},
url = {https://doi.org/10.1145/2499370.2462195},
doi = {10.1145/2499370.2462195},
abstract = {We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong.We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to Programming course at MIT (6.00) and MITx (6.00x). Our results show that relatively simple error models can correct on average 64% of all incorrect submissions in our benchmark set.},
journal = {SIGPLAN Not.},
month = jun,
pages = {15–26},
numpages = {12},
keywords = {program synthesis, automated grading, computer-aided education}
}

@inproceedings{feedback_framework,
author = {Gao, Jianxiong and Pang, Bei and Lumetta, Steven S.},
title = {Automated Feedback Framework for Introductory Programming Courses},
year = {2016},
isbn = {9781450342315},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/2899415.2899440},
doi = {10.1145/2899415.2899440},
abstract = {Using automated grading tools to provide feedback to students is common in Computer Science education. The first step of automated grading is to find defects in the student program. However, finding bugs in code has never been easy. Comparing computation results using a fixed set of test cases is still the most common way to determine correctness among current automated grading tools. It takes time and effort to design a good set of test cases that can test the student code thoroughly. In practice, tests used for grading are often insufficient for accurate diagnosis.In this paper, we present our utilization of industrial automated testing on student assignments in an introductory programming course. We implemented a framework to collect student codes and apply industrial automated testing to their codes. Then we interpreted the results obtained from testing in a way that students can understand easily. We deployed our framework on five different introductory C programming assignments here at the University of Illinois at Urbana-Champaign.The results show that the automated feedback generation framework can discover more errors inside student submissions and can provide timely and useful feedback to both instructors and students. A total of 142 missed bugs were found within 446 submissions. More than 50% of students received their feedback within 3 minutes of submission.},
booktitle = {Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education},
pages = {53–58},
numpages = {6},
keywords = {concolic testing, computer science education, auto grader},
location = {Arequipa, Peru},
series = {ITiCSE '16}
}
@article{feedback_programming,
author = {Keuning, Hieke and Jeuring, Johan and Heeren, Bastiaan},
title = {A Systematic Literature Review of Automated Feedback Generation for Programming Exercises},
year = {2018},
issue_date = {January 2019},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {19},
number = {1},
url = {https://doi.org/10.1145/3231711},
doi = {10.1145/3231711},
abstract = {Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.},
journal = {ACM Trans. Comput. Educ.},
month = sep,
articleno = {3},
numpages = {43},
keywords = {learning programming, programming tools, automated feedback, Systematic literature review}
}

@article{feedback_power,
author = {John Hattie and Helen Timperley},
title ={The Power of Feedback},
journal = {Review of Educational Research},
volume = {77},
number = {1},
pages = {81-112},
year = {2007},
doi = {10.3102/003465430298487},

URL = { 
        https://doi.org/10.3102/003465430298487
    
},
eprint = { 
        https://doi.org/10.3102/003465430298487
    
}
,
    abstract = { Feedback is one of the most powerful influences on learning and achievement, but this impact can be either positive or negative. Its power is frequently mentioned in articles about learning and teaching, but surprisingly few recent studies have systematically investigated its meaning. This article provides a conceptual analysis of feedback and reviews the evidence related to its impact on learning and achievement. This evidence shows that although feedback is among the major influences, the type of feedback and the way it is given can be differentially effective. A model of feedback is then proposed that identifies the particular properties and circumstances that make it effective, and some typically thorny issues are discussed, including the timing of feedback and the effects of positive and negative feedback. Finally, this analysis is used to suggest ways in which feedback can be used to enhance its effectiveness in classrooms. }
}

@article{feedback_cs1,
author = {Ott, Claudia and Robins, Anthony and Shephard, Kerry},
title = {Translating Principles of Effective Feedback for Students into the CS1 Context},
year = {2016},
issue_date = {February 2016},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {16},
number = {1},
url = {https://doi.org/10.1145/2737596},
doi = {10.1145/2737596},
abstract = {Learning the first programming language is challenging for many students. High failure rates and bimodally distributed grades lead to a pedagogical interest in supporting students in first-year programming courses (CS1). In higher education, the important role of feedback for guiding the learning process and improving the learning outcome is widely acknowledged. This article introduces contemporary models of effective feedback practice as found in the higher education literature and offers an interpretation of those in the CS1 context. One particular CS1 course and typical course components are investigated to identify likely loci for feedback interventions and to connect related computer science education literature to these forms of feedback.},
journal = {ACM Trans. Comput. Educ.},
month = jan,
articleno = {1},
numpages = {27},
keywords = {Effective feedback practice, CS1, higher education}
}


@mastersthesis{lambda,
    Author = {Ball, Michael},
    Editor = {Garcia, Dan},
    Title = {Lambda: An Autograder for Snap!},
    School = {EECS Department, University of California, Berkeley},
    Year = {2018},
    Month = {Jan},
    URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2018/EECS-2018-2.html},
    Number = {UCB/EECS-2018-2},
    Abstract = {While visual programming languages are hardly a new development, recent pushes for increased equity and access to computer science have led to a renewed interest and use of visual programming languages. Autograders are a critical component of both in-person and online computer science courses. However, until this point, there haven't been any autograders documented for general-purpose, visual programming development environments. This is a particular shortcoming as introductory courses have scaled to larger numbers of students and online environments. While there are challenges to using autograders, we believe that the instant feedback capabilities, as well as potential time savings for course staff will help us  teach a greater number of students.

Over the past year, we have built an autograder, named λ (Lambda), for Snap<em>!</em>, a visual blocks-based programming language inspired by MIT's Scratch. The primary motivation for developing the autograder was to run a series of Massive Open Online Courses (MOOCs) on edx.org throughout the 2015-2016 academic year. However, we also wish to use the autograder to better support in-person computer science courses. In Spring 2016, the Snap<em>!</em> autograder was used as a part of UC Berkeley's CS10, <i>_The Beauty and Joy of Computing_</i>, a "CS0" non-majors course.

This report describes λ, which consists of a "backend" Ruby on Rails webserver that allows us to use the autograder in a classroom setting, through a protocol called LTI. The backend web application contains a database of questions and test files, while the Snap<em>!</em> interface contains new features and a view to present the results of the autograder. Our initial results show the autograder successfully being used in CS10, where the autograder was used to supplement oral lab checkoffs, and on edx.org where the autograder was the primary method for students to receive credit for code, graded both for effort and correctness.}
}

@book{NAP24926,
  author    = "National Academies of Sciences, Engineering, and Medicine",
  title     = "Assessing and Responding to the Growth of Computer Science Undergraduate Enrollments",
  isbn      = "978-0-309-46702-5",
  doi       = "10.17226/24926",
  abstract  = "The field of computer science (CS) is currently experiencing a surge in undergraduate degree production and course enrollments, which is straining program resources at many institutions and causing concern among faculty and administrators about how best to respond to the rapidly growing demand. There is also significant interest about what this growth will mean for the future of CS programs, the role of computer science in academic institutions, the field as a whole, and U.S. society more broadly.\n\nAssessing and Responding to the Growth of Computer Science Undergraduate Enrollments seeks to provide a better understanding of the current trends in computing enrollments in the context of past trends. It examines drivers of the current enrollment surge, relationships between the surge and current and potential gains in diversity in the field, and the potential impacts of responses to the increased demand for computing in higher education, and it considers the likely effects of those responses on students, faculty, and institutions. This report provides recommendations for what institutions of higher education, government agencies, and the private sector can do to respond to the surge and plan for a strong and sustainable future for the field of CS in general, the health of the institutions of higher education, and the prosperity of the nation.",
  url       = "https://www.nap.edu/catalog/24926/assessing-and-responding-to-the-growth-of-computer-science-undergraduate-enrollments",
  year      = 2018,
  publisher = "The National Academies Press",
  address   = "Washington, DC"
}

@electronic{occupation,
  author = {{U.S. Bureau of Labor Statistics}},
  title = {Computer and Information Technology Occupations : Occupational Outlook Handbook},
  url={https://www.bls.gov/ooh/computer-and-information-technology/home.htm},
  year={2020}, month={Apr}
}

@misc{upenn_wait_times, title={Some CIS courses are so overloaded that students wait more than an hour for homework help}, url={https://www.thedp.com/article/2018/12/cis-120-office-hours-wait-time-penn-upenn-philadelphia}, journal={The Daily Pennsylvanian}, author={Basu, Kimaya}, year={2018}, month={Dec}}

@misc{umich_wait_times, title={Students, professors discuss finding resources for extra help in University's largest classes}, url={https://www.michigandaily.com/section/academics/students-professors-discuss-finding-resources-extra-help-university’s-largest}, journal={The Michigan Daily}, author={Weinstein, Liat}, year={2019}, month={Feb}}

@misc{stanford_wait_times, title={CS in Crisis: Is Stanford doing enough to respond to capacity and inclusion challenges?}, url={https://www.stanforddaily.com/2019/02/19/cs-in-crisis-is-stanford-doing-enough-to-respond-to-capacity-and-inclusion-challenges/}, journal={The Stanford Daily}, author={Liu, Jasmine}, year={2019}, month={Feb}}


AU  - Andrew Deorio
AU  - Christina Keefer
CY  - Minneapolis, MN
DA  - 2022/08/23
PB  - ASEE Conferences
TI  - When is Automated Feedback a Barrier to Timely Feedback?
UR  - https://peer.asee.org/40405
DO  - 10.18260/1-2--40405
ER  -