Full Paper: A Generative AI Approach to Better Teamwork in First-Year Engineering

Ethan E Danahy; Mehek Kunal Vora; Yume Menghe Xu; William Church

Download Paper | Permalink

Conference: 15th Annual First-Year Engineering Experience Conference (FYEE)
Location: Boston, Massachusetts
Publication Date: July 28, 2024
Start Date: July 28, 2024
End Date: July 30, 2024
Page Count: 10
DOI: 10.18260/1-2--48596
Permanent URL: https://peer.asee.org/48596
Download Count: 64

Paper Authors

biography

Ethan E Danahy Tufts University orcid.org/0000-0002-1431-8031

visit author page

Dr. Ethan Danahy is a Research Associate Professor at the Center for Engineering Education and Outreach (CEEO) with secondary appointment in the Department of Computer Science within the School of Engineering at Tufts University. Having received his graduate degrees in Computer Science and Electrical Engineering from Tufts University, he continues research in the design, implementation, and evaluation of different educational technologies. With particular attention to engaging students in the STEAM content areas, he focuses his investigations on enhancing creativity and innovation, supporting better documentation, and encouraging collaborative learning.

visit author page

biography

Mehek Kunal Vora Tufts University

visit author page

Mehek Vora is a rising sophomore at Tufts University, originally from Mumbai, India. She is pursuing a Bachelor’s degree in Psychology and Economics, maintaining a place on the Dean’s Honour List. She is currently a summer research intern at the Tufts Institute for Research on Learning and Instruction and student researcher at the Tufts Center for Engineering Education and Outreach. She has recently developed a deep appreciation for the potential and capacity of generative AI in the learning environment and is curious to explore more areas focused on the intersection between educational and inequity.

visit author page

biography

Yume Menghe Xu Tufts Center for Engineering Education and Outreach

visit author page

Menghe (Yume) is a PhD student in STEM Education at Tufts University. She holds a bachelor’s and a master’s degree in Chemical System Engineering from the University of Tokyo, Japan. Prior to pursuing a PhD at Tufts, she designed and developed educational apps for children, and worked with students, teachers, and makerspace in Japan to host making workshops using various materials and tools. Her research interest lies in youth's identity construction in STEM learning spaces.

visit author page

author page

William Church

Download Paper | Permalink

Abstract

This Full Paper will describe a new method of facilitating teamwork in a first-year engineering course using generative AI. This work is inspired by, and builds upon, the many existing techniques and tools currently supporting engineering instructors in the forming of teams, overseeing of team dynamics, supporting interpersonal dynamics within teams, and evaluation of team members (e.g. CATME from Purdue University, Tandem from University of Michigan, etc). This work discusses new enhancements to each step of the process via generative AI tools and technologies. Through student data collected at different stages of the project development cycle, and specific prompts used to interact with generative AI, it was possible to customize and personalize the teamwork groupings, recommendations, feedback mechanisms, and individual evaluations in a scalable way across the entire class.

Research data was collected from a single section of a first-semester introduction to engineering course at a small, private engineering school in the northeast part of the United States. While the semester-long course of 30 first-year engineering students (21 of which consented to IRB-approved research around their course activities and submissions) featured several individual and group projects in Fall 2023, two group projects in particular are the focus of this research. The first (the initial group project of the semester, occurring one month into the semester) was orchestrated leveraging generative AI to process student input and formulate the student groups; the second group project of the semester (the subsequent project in sequence) allowed students to self-pick collaborators and self-orchestrate their groups. A post-survey was issued after the two projects to collect data on students' perceptions of the experience. Artifacts used in this analysis include students' initial survey, chatbot logs from a generative AI system provided to the students for use during the projects ([REF REMOVED], 2024), students' group project assignment submissions, and a post-project reflection survey.

This research first explores the various strategies used by the instructor in this introduction to engineering design course to introduce the use of OpenAI's ChatGPT to the team facilitation process. Due to the capabilities of large language models (LLMs) to process, summarize, and identify thematic commonalities across large collections of rich text, traditional simplistic surveys (e.g. consisting of numeric self-reported Likert scales, already known to be problematic in this type of data collection) were replaced by open-ended short-answer questions where students reflected, across a variety of dimensions, on prior group work experiences in their lives. This provided the opportunity for deeper insights into the individual abilities, personalities, experiences, and needs of each student with regards to their engagement in group work. These anonymized class-wide trends were created and shared with students to form mutual class-wide understandings and develop common collaborative culture prior to the start of group work. The chatbot’s inherent clustering capabilities was then leveraged to form initial teammate groupings based on individual recommendations. For each of these teams, personalized strengths and weaknesses were identified and shared internally to each group. Teamwork strategy summaries and warnings, generated from both internal and external sources, were also shared.

The theory underlying this work is that more insightful (and transparent) formulation of groups would create successful collaborative experiences, and by leveraging generative AI technologies these tools and techniques could be both scaled across larger numbers of students and open to a wider range of instructors. This initial experiment, creating 10 groups of 3 students each, demonstrates the feasibility of this methodology. A subsequent group project where students "picked their own partners" allowed students to subsequently reflect directly on the two experiences (albeit on different assignments with different scopes and scales) to compare and contrast their perceptions of how the AI-generated groupings fared. A project postmortem analysis of both team-level interactions (e.g. chatlogs from the generative AI system supporting in-class work) and individual reflections were evaluated in order to give more meaningful summative feedback on the experience. This in turn was used to generate other formative suggestions for future implementations.

Analysis and evidence from this work demonstrates the following: the generative AI system was able to successfully process the input data and generate the requested reports (class-wide summaries, individual assessments and recommendations, and group formations with characterizations and suggestions). Implementation details in the procedure section of the paper document instructor techniques for chatbot interactions to facilitate higher-quality output (e.g. adjusting chatbot responses to protect student identities and deliver results in more appropriate ways). Student reflections after both assignments indicate overall student approval of the AI-formed groups, with multiple students highlighting the alleviation of social anxiety when the AI identified groups for them. However, student sentiment included wariness of a fully-automated system (multiple requests for "human in the loop") and concerns over the AI summaries and decisions being based on so little input information (which also was prone to having errors, false information included, or being manipulated by students since self-reported). After completing a project with groups selected by AI and then a project with groups formed via self-selection, students proposed a hybrid approach that allows some automation but both teacher-informed influence and includes more student choice. Details of range of ideas and suggestions are included in the full description of results. From the experiences and findings presented, several more generalized recommendations for use within other contexts are also included.

Citation
Format

Danahy, E. E., & Vora, M. K., & Xu, Y. M., & Church, W. (2024, July), Full Paper: A Generative AI Approach to Better Teamwork in First-Year Engineering Paper presented at 15th Annual First-Year Engineering Experience Conference (FYEE), Boston, Massachusetts. 10.18260/1-2--48596

TY  - CPAPER
AB  - This Full Paper will describe a new method of facilitating teamwork in a first-year engineering course using generative AI. This work is inspired by, and builds upon, the many existing techniques and tools currently supporting engineering instructors in the forming of teams, overseeing of team dynamics, supporting interpersonal dynamics within teams, and evaluation of team members (e.g. CATME from Purdue University, Tandem from University of Michigan, etc). This work discusses new enhancements to each step of the process via generative AI tools and technologies. Through student data collected at different stages of the project development cycle, and specific prompts used to interact with generative AI, it was possible to customize and personalize the teamwork groupings, recommendations, feedback mechanisms, and individual evaluations in a scalable way across the entire class.

Research data was collected from a single section of a first-semester introduction to engineering course at a small, private engineering school in the northeast part of the United States. While the semester-long course of 30 first-year engineering students (21 of which consented to IRB-approved research around their course activities and submissions) featured several individual and group projects in Fall 2023, two group projects in particular are the focus of this research. The first (the initial group project of the semester, occurring one month into the semester) was orchestrated leveraging generative AI to process student input and formulate the student groups; the second group project of the semester (the subsequent project in sequence) allowed students to self-pick collaborators and self-orchestrate their groups. A post-survey was issued after the two projects to collect data on students&#39; perceptions of the experience. Artifacts used in this analysis include students&#39; initial survey, chatbot logs from a generative AI system provided to the students for use during the projects ([REF REMOVED], 2024), students&#39; group project assignment submissions, and a post-project reflection survey.

This research first explores the various strategies used by the instructor in this introduction to engineering design course to introduce the use of OpenAI&#39;s ChatGPT to the team facilitation process. Due to the capabilities of large language models (LLMs) to process, summarize, and identify thematic commonalities across large collections of rich text, traditional simplistic surveys (e.g. consisting of numeric self-reported Likert scales, already known to be problematic in this type of data collection) were replaced by open-ended short-answer questions where students reflected, across a variety of dimensions, on prior group work experiences in their lives. This provided the opportunity for deeper insights into the individual abilities, personalities, experiences, and needs of each student with regards to their engagement in group work. These anonymized class-wide trends were created and shared with students to form mutual class-wide understandings and develop common collaborative culture prior to the start of group work. The chatbot’s inherent clustering capabilities was then leveraged to form initial teammate groupings based on individual recommendations. For each of these teams, personalized strengths and weaknesses were identified and shared internally to each group. Teamwork strategy summaries and warnings, generated from both internal and external sources, were also shared.

The theory underlying this work is that more insightful (and transparent) formulation of groups would create successful collaborative experiences, and by leveraging generative AI technologies these tools and techniques could be both scaled across larger numbers of students and open to a wider range of instructors. This initial experiment, creating 10 groups of 3 students each, demonstrates the feasibility of this methodology. A subsequent group project where students &quot;picked their own partners&quot; allowed students to subsequently reflect directly on the two experiences (albeit on different assignments with different scopes and scales) to compare and contrast their perceptions of how the AI-generated groupings fared. A project postmortem analysis of both team-level interactions (e.g. chatlogs from the generative AI system supporting in-class work) and individual reflections were evaluated in order to give more meaningful summative feedback on the experience. This in turn was used to generate other formative suggestions for future implementations.

Analysis and evidence from this work demonstrates the following: the generative AI system was able to successfully process the input data and generate the requested reports (class-wide summaries, individual assessments and recommendations, and group formations with characterizations and suggestions). Implementation details in the procedure section of the paper document instructor techniques for chatbot interactions to facilitate higher-quality output (e.g. adjusting chatbot responses to protect student identities and deliver results in more appropriate ways). Student reflections after both assignments indicate overall student approval of the AI-formed groups, with multiple students highlighting the alleviation of social anxiety when the AI identified groups for them. However, student sentiment included wariness of a fully-automated system (multiple requests for &quot;human in the loop&quot;) and concerns over the AI summaries and decisions being based on so little input information (which also was prone to having errors, false information included, or being manipulated by students since self-reported). After completing a project with groups selected by AI and then a project with groups formed via self-selection, students proposed a hybrid approach that allows some automation but both teacher-informed influence and includes more student choice. Details of range of ideas and suggestions are included in the full description of results. From the experiences and findings presented, several more generalized recommendations for use within other contexts are also included.
AU  - Ethan E Danahy
AU  - Mehek Kunal Vora
AU  - Yume Menghe Xu
AU  - William Church
CY  - Boston, Massachusetts
DA  - 2024/07/28
PB  - ASEE Conferences
TI  - Full Paper: A Generative AI Approach to Better Teamwork in First-Year Engineering
UR  - https://peer.asee.org/48596
DO  - 10.18260/1-2--48596
ER  -