Full Paper: Supporting Students' Educational Robotics Experiences through Generative AI Chatbots

Ethan E Danahy; Mehek Kunal Vora; Yume Menghe Xu; William Church

Download Paper | Permalink

Conference: 15th Annual First-Year Engineering Experience Conference (FYEE)
Location: Boston, Massachusetts
Publication Date: July 28, 2024
Start Date: July 28, 2024
End Date: July 30, 2024
Page Count: 9
DOI: 10.18260/1-2--48604
Permanent URL: https://peer.asee.org/48604
Download Count: 42

Paper Authors

biography

Ethan E Danahy Tufts University orcid.org/0000-0002-1431-8031

visit author page

Dr. Ethan Danahy is a Research Associate Professor at the Center for Engineering Education and Outreach (CEEO) with secondary appointment in the Department of Computer Science within the School of Engineering at Tufts University. Having received his graduate degrees in Computer Science and Electrical Engineering from Tufts University, he continues research in the design, implementation, and evaluation of different educational technologies. With particular attention to engaging students in the STEAM content areas, he focuses his investigations on enhancing creativity and innovation, supporting better documentation, and encouraging collaborative learning.

visit author page

biography

Mehek Kunal Vora Tufts University

visit author page

Mehek Vora is an undergraduate student at Tufts University majoring in Psychology and Economics. Through her positions at the Tufts Institute for Research on Learning and Instruction (IRLI) and at the Tufts Center for Engineering Education and Outreach (CEEO), her research spans across educational methodologies, learning strategies and integration of technologies. She has recently developed a deep appreciation for the potential and capacity of generative AI’s impact on educational environments and is curious to explore more areas focused on the intersection between education and inequity.

visit author page

biography

Yume Menghe Xu Tufts Center for Engineering Education and Outreach

visit author page

Menghe (Yume) is a PhD student in STEM Education at Tufts University. She holds a bachelor’s and a master’s degree in Chemical System Engineering from the University of Tokyo, Japan. Prior to pursuing a PhD at Tufts, she designed and developed educational apps for children, and worked with students, teachers, and makerspace in Japan to host making workshops using various materials and tools. Her research interest lies in youth's identity construction in STEM learning spaces.

visit author page

author page

William Church

Download Paper | Permalink

Abstract

This full paper proposal describes a first-year engineering course that leveraged Generative Artificial Intelligence (AI) chatbots to support students' educational robotics experiences. Analysis of surveys, course work, and chatbot logs resulted in a categorization of experiences and classification of students within the course, and a proposed set of recommendations for enhancing the experience in order to better utilize the technology across all types of students (specifically those with less prior engineering/coding experience who subsequently struggled to leverage the generative AI technology to their full benefit).

With the release of and widespread availability of generative artificial intelligence interfaces, education (in general across ages, and within engineering education at the university level more specifically) is starting to experiment with new teaching and learning paradigms around the incorporation of these tools and exploring the effect (both good and bad) they can have on the classroom experience. This paper presents how generative AI chatbots, specifically primed with particular domain knowledge, can be used to support various steps of the engineering design process within a university-level first-year introductory engineering class.

Research data was collected from a single section of a first-semester introduction to engineering course at a small, private engineering school in the northeast part of the United States. The semester-long course, consisting of 30 first-year engineering students (21 of which consented to IRB-approved research around their course activities and submissions), occurred during the 15-week Fall 2023 semester (September 6th through December 22nd, 2023). Artifacts used in this analysis include students' self-reported responses to class surveys (beginning and end of semester), assignment submissions to projects and other course work (14 different tasks), and logs of generative AI usage captured by the custom chatbot interface developed for use in the course. Throughout the semester, the students in the class initiated 1,014 different conversations with the generative AI chatbots and the system captured a total of 3,245 individual messages.

The course is themed around using robotics to solve a variety of engineering challenges, with focus on students’ introduction to and engagement in engineering design. In Fall 2023 the same set of robotics challenges was presented (as in previous years) but several different custom chatbots (built on OpenAI's ChatGPT-4 platform but primed with course-specific information and accessed via a course-specific interface) were created and made available to the students in order to enhance the experience and supplement, replace, or circumvent some of the traditional instructor provided content. For instance, as this particular course utilizes a LEGO-based robotics platform (LEGO Education's SPIKE Prime), two chatbots (coined "BuildBot" and "PrimeBot" respectively) were created to assist students in the construction and programming tasks associated with the platform. This shifted the need away from class instruction time to require focus specifically on details of the hardware kit or the Python-based coding language. Similarly, the following chatbots were also provided to students: "GeneralBot" which could be used to ask for general questions of the generative AI, "WebDeveloperBot" that assisted in updating HTML/CSS/JS code for creating web pages and web-based digital portfolios that were required for project documentation submissions, and "SyllabusBot" which was primed with course-specific logistical information direct from the course syllabus. Students were granted permission to leverage these new resources (the 5 custom chatbots) as needed to accomplish any of the semester tasks presented, and examination of the usage logs show that many did for things such as designing and building mechanical solutions, coding the robotic system, customizing their digital project documentation, and navigating class-specific logistical information.

This paper introduces the new technologies developed (detailing the platforms on which these systems were created and the interface design for chat interactions) and an analysis of the usage by the students (breakdown of use across the different chatbots, and a temporal analysis of when in the semester different resources were leveraged and in what way/to what extent). Previous work ([REF REMOVED], 2024) looked at pre-post changes in perceptions and attitudes of the students with regards to use of AI in education, and this work will build on that class-wide analysis to examine in more detail at the student-level, combining the pre-post survey data with the data from usage logs. A classification of the data has resulted in four characterizations of students across two dimensions (in-coming engineering experience and in-class utilization of the generative-AI platform): (1) students with high-levels of prior engineering (or coding) experience who successfully leveraged AI a lot in their work to enhancing their abilities, (2) students with high-levels of prior engineering (or coding) experience who had low utilization of AI and manually completed many of the tasks, (3) students with low-levels of prior engineering (or coding) experience who engaged with the AI a lot to successfully accomplish difficult tasks, and (4) students with low-levels of prior engineering (or coding) experience who had minimal usage or less successful patterns of use with the tools. This work theorizes that, for this fourth group, issues during early attempts at using the generative AI for tasks discouraged them from future use and, overall, they didn’t experience the potential for how the AI could augment their work and assist in completing the course tasks.

Implications of this chatbot usage analysis (and connections to individual student perceptions and overall class experience) indicate a need for thoughtful introduction and integration of generative AI tools in engineering education, especially in the first-year during formative engineering experiences. To address this, this work proposes early assignments that help students learn beneficial styles of interaction (e.g. prompt formulation strategies that produce higher quality output) and general experiences that facilitate students in understanding the value of using such tools, in the theory that these will provide skills and motivation to continue sustained successful usage throughout the semester across all categories of students.

Citation
Format

Danahy, E. E., & Vora, M. K., & Xu, Y. M., & Church, W. (2024, July), Full Paper: Supporting Students' Educational Robotics Experiences through Generative AI Chatbots Paper presented at 15th Annual First-Year Engineering Experience Conference (FYEE), Boston, Massachusetts. 10.18260/1-2--48604

TY  - CPAPER
AB  - This full paper proposal describes a first-year engineering course that leveraged Generative Artificial Intelligence (AI) chatbots to support students&#39; educational robotics experiences. Analysis of surveys, course work, and chatbot logs resulted in a categorization of experiences and classification of students within the course, and a proposed set of recommendations for enhancing the experience in order to better utilize the technology across all types of students (specifically those with less prior engineering/coding experience who subsequently struggled to leverage the generative AI technology to their full benefit).

With the release of and widespread availability of generative artificial intelligence interfaces, education (in general across ages, and within engineering education at the university level more specifically) is starting to experiment with new teaching and learning paradigms around the incorporation of these tools and exploring the effect (both good and bad) they can have on the classroom experience. This paper presents how generative AI chatbots, specifically primed with particular domain knowledge, can be used to support various steps of the engineering design process within a university-level first-year introductory engineering class.

Research data was collected from a single section of a first-semester introduction to engineering course at a small, private engineering school in the northeast part of the United States. The semester-long course, consisting of 30 first-year engineering students (21 of which consented to IRB-approved research around their course activities and submissions), occurred during the 15-week Fall 2023 semester (September 6th through December 22nd, 2023). Artifacts used in this analysis include students&#39; self-reported responses to class surveys (beginning and end of semester), assignment submissions to projects and other course work (14 different tasks), and logs of generative AI usage captured by the custom chatbot interface developed for use in the course. Throughout the semester, the students in the class initiated 1,014 different conversations with the generative AI chatbots and the system captured a total of 3,245 individual messages.

The course is themed around using robotics to solve a variety of engineering challenges, with focus on students’ introduction to and engagement in engineering design. In Fall 2023 the same set of robotics challenges was presented (as in previous years) but several different custom chatbots (built on OpenAI&#39;s ChatGPT-4 platform but primed with course-specific information and accessed via a course-specific interface) were created and made available to the students in order to enhance the experience and supplement, replace, or circumvent some of the traditional instructor provided content. For instance, as this particular course utilizes a LEGO-based robotics platform (LEGO Education&#39;s SPIKE Prime), two chatbots (coined &quot;BuildBot&quot; and &quot;PrimeBot&quot; respectively) were created to assist students in the construction and programming tasks associated with the platform. This shifted the need away from class instruction time to require focus specifically on details of the hardware kit or the Python-based coding language. Similarly, the following chatbots were also provided to students: &quot;GeneralBot&quot; which could be used to ask for general questions of the generative AI, &quot;WebDeveloperBot&quot; that assisted in updating HTML/CSS/JS code for creating web pages and web-based digital portfolios that were required for project documentation submissions, and &quot;SyllabusBot&quot; which was primed with course-specific logistical information direct from the course syllabus. Students were granted permission to leverage these new resources (the 5 custom chatbots) as needed to accomplish any of the semester tasks presented, and examination of the usage logs show that many did for things such as designing and building mechanical solutions, coding the robotic system, customizing their digital project documentation, and navigating class-specific logistical information.

This paper introduces the new technologies developed (detailing the platforms on which these systems were created and the interface design for chat interactions) and an analysis of the usage by the students (breakdown of use across the different chatbots, and a temporal analysis of when in the semester different resources were leveraged and in what way/to what extent). Previous work ([REF REMOVED], 2024) looked at pre-post changes in perceptions and attitudes of the students with regards to use of AI in education, and this work will build on that class-wide analysis to examine in more detail at the student-level, combining the pre-post survey data with the data from usage logs. A classification of the data has resulted in four characterizations of students across two dimensions (in-coming engineering experience and in-class utilization of the generative-AI platform): (1) students with high-levels of prior engineering (or coding) experience who successfully leveraged AI a lot in their work to enhancing their abilities, (2) students with high-levels of prior engineering (or coding) experience who had low utilization of AI and manually completed many of the tasks, (3) students with low-levels of prior engineering (or coding) experience who engaged with the AI a lot to successfully accomplish difficult tasks, and (4) students with low-levels of prior engineering (or coding) experience who had minimal usage or less successful patterns of use with the tools. This work theorizes that, for this fourth group, issues during early attempts at using the generative AI for tasks discouraged them from future use and, overall, they didn’t experience the potential for how the AI could augment their work and assist in completing the course tasks.

Implications of this chatbot usage analysis (and connections to individual student perceptions and overall class experience) indicate a need for thoughtful introduction and integration of generative AI tools in engineering education, especially in the first-year during formative engineering experiences. To address this, this work proposes early assignments that help students learn beneficial styles of interaction (e.g. prompt formulation strategies that produce higher quality output) and general experiences that facilitate students in understanding the value of using such tools, in the theory that these will provide skills and motivation to continue sustained successful usage throughout the semester across all categories of students.
AU  - Ethan E Danahy
AU  - Mehek Kunal Vora
AU  - Yume Menghe Xu
AU  - William Church
CY  - Boston, Massachusetts
DA  - 2024/07/28
PB  - ASEE Conferences
TI  - Full Paper: Supporting Students' Educational Robotics Experiences through Generative AI Chatbots
UR  - https://peer.asee.org/48604
DO  - 10.18260/1-2--48604
ER  -