An Analysis of the Impact of Advances in Generative Artificial Intelligence on Programming Assignments and Competitions

Devang Jayachandran; Pranit Shrikrishna Maldikar; Jeremy Joseph Blum

Download Paper | Permalink

Conference: ASEE Zone 1 Conference - Spring 2023
Location: State College,, Pennsylvania
Publication Date: March 30, 2023
Start Date: March 30, 2023
End Date: April 12, 2023
Page Count: 11
DOI: 10.18260/1-2--45088
Permanent URL: https://peer.asee.org/45088
Download Count: 236

Paper Authors

biography

Devang Jayachandran Pennsylvania State University, Harrisburg

visit author page

Devang Jayachandran is currently a graduate student pursuing a Masters of Science in Computer Science at the Mathematics and Computer Science department in Penn State Harrisburg. Devang received his Bachelor's of Engineering in Information Science from the National Institute of Engineering, Mysuru, India and then worked at JP Morgan Chase and Co, Bengaluru, India in the field of Natural Language Processing and Document Extraction.

visit author page

biography

Pranit Shrikrishna Maldikar Pennsylvania State University

visit author page

Pranit Shrikrishna Maldikar is a computer science graduate student at Pennsylvania State University at Harrisburg. He completed his bachelor's of engineering in information technology from Mumbai University, India.

visit author page

biography

Jeremy Joseph Blum Pennsylvania State University, Harrisburg

visit author page

Dr. Jeremy Blum is an associate professor of Computer Science at the Pennsylvania State University, Harrisburg, PA, USA. Prior to joining Penn State Harrisburg, Dr. Blum worked as a research scientist at the Center for Intelligent Systems Research at the George Washington University. Dr. Blum received a D.Sc. in Computer Science and an M.S. in Computational Sciences, both from the George Washington University, as well as a B.A. in Economics from Washington University. His research interests include computer science education and transportation safety.

visit author page

Download Paper | Permalink

Abstract

Research in the field of artificial intelligence has led to the creation of a multitude of chat bots and conversational algorithms. Chat bots have become an integral part of the internet and can be found in a variety of domains ranging across healthcare, customer service, e-commerce, education, banking, and finance, etc. Nowadays chat bots can easily handle mundane tasks like setting up reminders and scheduling appointments as well as complicated actions like providing language translation or language learning assistance, and analyzing concepts, algorithms, and code solutions in specified programming language. OpenAI has been a leader in the innovation of artificial intelligence systems that can understand and process natural language and images. OpenAI recently revealed ChatGPT - a conversational AI based on Generative Pre-trained Transformer 3 (GPT3), which can handle a wide range of tasks such as providing answers and detailed explanations to complicated questions, analyzing data and statistics, holding contextual conversation. ChatGPT’s capabilities have led to multiple predictions of rendering homework obsolete, due to its ability to produce high-quality and unique responses to homework prompts that evade traditional plagiarism detection methods.

This manuscript evaluates the impact of ChatGPT on programming assignments via an analysis of its abilities and limitations to solve competitive programming challenges presented in the IEEEXtreme programming contest problem set, which contains problems that were not in ChatGPT’s training set. The difficulty of the problems in the problem set were determined based on the number of teams that were able to solve the problems during the competition. ChatGPT was able to solve only the easiest of the problems in the problem set. It did, however, identify the correct data structure or algorithmic approach for multiple problems. This analysis describes problem design features that limit ChatGPT’s ability to solve or partially solve a challenge. These features include problems for which optimization is required to efficiently solve a problem, problems that have nuanced differences from standard approaches, and problem descriptions that cannot easily be input into the ChatGPT system. The paper also examines ChatGPT’s ability to obfuscate plagiarized submissions. Based on these findings, we present guidelines for developing programming challenges that cannot be solved by ChatGPT and required adjustments to plagiarism detection approaches. These guidelines will be of interest not only to contest program authors, but also to educators seeking to develop homework assignments given the availability of this platform.

Citation
Format

Jayachandran, D., & Maldikar, P. S., & Blum, J. J. (2023, March), An Analysis of the Impact of Advances in Generative Artificial Intelligence on Programming Assignments and Competitions Paper presented at ASEE Zone 1 Conference - Spring 2023, State College,, Pennsylvania. 10.18260/1-2--45088

TY  - CPAPER
AB  - Research in the field of artificial intelligence has led to the creation of a multitude of chat bots and conversational algorithms. Chat bots have become an integral part of the internet and can be found in a variety of domains ranging across healthcare, customer service, e-commerce, education, banking, and finance, etc. Nowadays chat bots can easily handle mundane tasks like setting up reminders and scheduling appointments as well as complicated actions like providing language translation or language learning assistance, and analyzing concepts, algorithms, and code solutions in specified programming language. OpenAI has been a leader in the innovation of artificial intelligence systems that can understand and process natural language and images. OpenAI recently revealed ChatGPT  - a conversational AI based on Generative Pre-trained Transformer 3 (GPT3), which can handle a wide range of tasks such as providing answers and detailed explanations to complicated questions, analyzing data and statistics, holding contextual conversation. ChatGPT’s capabilities have led to multiple predictions of rendering homework obsolete, due to its ability to produce high-quality and unique responses to homework prompts that evade traditional plagiarism detection methods.

This manuscript evaluates the impact of ChatGPT on programming assignments via an analysis of its abilities and limitations to solve competitive programming challenges presented in the IEEEXtreme programming contest problem set, which contains problems that were not in ChatGPT’s training set.  The difficulty of the problems in the problem set were determined based on the number of teams that were able to solve the problems during the competition. ChatGPT was able to solve only the easiest of the problems in the problem set. It did, however, identify the correct data structure or algorithmic approach for multiple problems.  This analysis describes problem design features that limit ChatGPT’s ability to solve or partially solve a challenge. These features include problems for which optimization is required to efficiently solve a problem, problems that have nuanced differences from standard approaches, and problem descriptions that cannot easily be input into the ChatGPT system.  The paper also examines ChatGPT’s ability to obfuscate plagiarized submissions. Based on these findings, we present guidelines for developing programming challenges that cannot be solved by ChatGPT and required adjustments to plagiarism detection approaches. These guidelines will be of interest not only to contest program authors, but also to educators seeking to develop homework assignments given the availability of this platform.

AU  - Devang Jayachandran
AU  - Pranit Shrikrishna Maldikar
AU  - Jeremy Joseph Blum
CY  - State College,, Pennsylvania
DA  - 2023/03/30
PB  - ASEE Conferences
TI  - An Analysis of the Impact of Advances in Generative Artificial Intelligence on Programming Assignments and Competitions
UR  - https://peer.asee.org/45088
DO  - 10.18260/1-2--45088
ER  -