Tanay, B. A., & Arinze, L. C., & Joshi, S. S., & Davis, K. A., & Davis, J. C. (2024, June), An Exploratory Study on Upper-Level Computing Students’ Use of Large Language Models as Tools in a Semester-Long Project   Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--46557

\bibitem{asee_peer_46557}
  Tanay, B. A., \& Arinze, L. C., \& Joshi, S. S., \& Davis, K. A., \& Davis, J. C. (2024, June), \emph{An Exploratory Study on Upper-Level Computing Students’ Use of Large Language Models as Tools in a Semester-Long Project }
  Paper presented at 2024 ASEE Annual Conference \& Exposition, Portland, Oregon.
  10.18260/1-2--46557

Ben Arie Tanay, Lexy Chiwete Arinze, Siddhant Sanjay Joshi, Kirsten A. Davis, and James C Davis.  "An Exploratory Study on Upper-Level Computing Students’ Use of Large Language Models as Tools in a Semester-Long Project ".  2024 ASEE Annual Conference & Exposition, Portland, Oregon, 2024, June.  ASEE Conferences, 2024.  https://peer.asee.org/46557 Internet.  03 Dec, 2024

\bibitem{asee_peer_46557}
  Ben Arie Tanay, Lexy Chiwete Arinze, Siddhant Sanjay Joshi, Kirsten A. Davis, and James C Davis.
  "An Exploratory Study on Upper-Level Computing Students’ Use of Large Language Models as Tools in a Semester-Long Project ".
  \emph{2024 ASEE Annual Conference \& Exposition, Portland, Oregon, 2024, June}.
  ASEE Conferences, 2024.
  https://peer.asee.org/46557 Internet.  03 Dec, 2024

@INPROCEEDINGS{asee_peer_46557
author = "Ben Arie Tanay, Lexy Chiwete Arinze, Siddhant Sanjay Joshi, Kirsten A. Davis, and James C Davis"
title = "An Exploratory Study on Upper-Level Computing Students’ Use of Large Language Models as Tools in a Semester-Long Project "
booktitle = "2024 ASEE Annual Conference \& Exposition"
year = "2024"
month = "June"
address = "Portland, Oregon"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/46557}
number = {10.18260/1-2--46557}
}

TY  - CPAPER
AB  -  Background: Large Language Models (LLMs) have begun to influence software engineering practice since the public release of GitHub's CoPilot and OpenAI's ChatGPT in 2022. As an interactive “assistant” that can answer questions and prototype software, LLMs could potentially revolutionize the way software engineering is practiced – and thus may inform how software engineering is taught. LLMs offer varying experiences among users. 
While some schools have banned ChatGPT (Dibble, 2023), researchers have proposed strategies to address potential issues inherent in the use of LLMs (Lo, 2023; Gimpel et al., n.d.; de Fine Licht, 2023). The Association for Computing Machinery (ACM) identifies curriculum guidelines with essential competences for Computer Science undergraduate degree programs. The 2023 guidelines which incorporates the use of LLMs are still in beta version and soliciting feedback (CS2023 – ACM/IEEE-CS/AAAI Computer Science Curricula, n.d.). It is, therefore, important to evaluate students’ perception of LLMs and possible ways of adapting the computing curriculum to these shifting paradigms. 
 
Purpose: The purpose of this study is to explore computing students’ experiences and approaches to using LLMs during a semester-long software engineering project.  
 
Design/Method: In this paper we will present data collected from a senior-level software engineering course at a large public university in the Midwest. This course uses a project-based learning (PBL) design with a semester-long team project. In Fall 2023, the students were required to use LLMs such as ChatGPT and CoPilot as they completed their projects. A sample of these student teams were interviewed in the middle and at the end of the semester to understand (1) how they used LLMs in their projects, and (2) whether and how their perspectives on LLMs changed over the course of the semester. We are analyzing the data qualitatively to identify themes related to students’ usage patterns and learning outcomes. 
 
Results/Discussion: We will report on students’ thinking over the course of the semester and how they developed strategies to use LLMs. We will discuss observed trends and how the use of LLMs shaped the teams’ dynamics and deliverables. The results will help us characterize the impact that the incorporation of LLMs had on the students’ learning. Based on our findings, we will make recommendations for future software programming courses seeking to incorporate LLMs. Our results can also inform professional development programs and policies associated with the integration of AI-driven technology in education. 
 
References 
CS2023 – ACM/IEEE-CS/AAAI Computer Science Curricula. (n.d.). Retrieved October 30, 2023, from https://csed.acm.org/ 
de Fine Licht, K. (2023). Integrating Large Language Models into Higher Education: Guidelines for Effective Implementation. Computer Sciences & Mathematics Forum, 8(1), Article 1. https://doi.org/10.3390/cmsf2023008065 
Dibble, M. (Director). (2023, February 6). Schools Ban ChatGPT amid Fears of Artificial Intelligence-Assisted Cheating. https://www.voanews.com/a/schools-ban-chatgpt-amid-fears-of-artificial-intelligence-assisted-cheating/6949800.html 
Gimpel, H., Hall, K., Decker, S., Eymann, T., Lämmermann, L., Mädche, A., er, R., Maximilian, R., Caroline, S., Manfred, S., Mareike, U., Nils, V., & rik, S. (n.d.). Unlocking the power of generative AI models and systems such as GPT-4 and ChatGPT for higher education: A guide for students and lecturers. 
Lo, C. K. (2023). What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410 
 
AU  - Ben Arie Tanay
AU  - Lexy Chiwete Arinze
AU  - Siddhant Sanjay Joshi
AU  - Kirsten A. Davis
AU  - James C Davis
CY  - Portland, Oregon
DA  - 2024/06/23
PB  - ASEE Conferences
TI  - An Exploratory Study on Upper-Level Computing Students’ Use of Large Language Models as Tools in a Semester-Long Project 
UR  - https://peer.asee.org/46557
DO  - 10.18260/1-2--46557
ER  -