Evaluating ChatGPT's Efficacy in Qualitative Analysis of Engineering Education Research

Xiaorong Zhang; Stephanie Claussen; Fatemeh Khalkhal; Yiyi Wang

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: June 26, 2024
Conference Session: Educational Research and Methods Division (ERM) Technical Session 22
Tagged Division: Educational Research and Methods Division (ERM)
Page Count: 16
DOI: 10.18260/1-2--47341
Permanent URL: https://peer.asee.org/47341
Download Count: 148

Paper Authors

biography

Xiaorong Zhang San Francisco State University

visit author page

Dr. Xiaorong Zhang is an Associate Professor in Computer Engineering in the School of Engineering at San Francisco State University (SFSU). She is the Director of the Intelligent Computing and Embedded Systems Laboratory (ICE Lab) at SFSU. She has broad research experience in human-machine interfaces, neural-controlled artificial limbs, embedded systems, and intelligent computing technologies. She is a recipient of the NSF CAREER Award to develop the next-generation neural-machine interfaces (NMI) for electromyography (EMG)-controlled neurorehabilitation. She is a senior member of the Institute of Electrical and Electronics Engineers (IEEE) and a member of the Society of Women Engineers (SWE). She has served in professional societies in various capacities including the Chair of the IEEE Engineering in Medicine and Biology Society (EMBS) San Francisco Chapter (2018-present), an Associate Editor of the IEEE Inside Signal Processing E-Newsletter (2016-2018), an Outreach Co-Chair of the Society of Women Engineers (SWE) Golden Gate Section (2017-2018), a Co-Chair of the Doctoral Consortium at 2014 IEEE Symposium Series on Computational Intelligence, a Program Committee Member of various international conferences, and a regular reviewer of a variety of journals and conferences in related fields.

visit author page

biography

Stephanie Claussen San Francisco State University

visit author page

Stephanie Claussen is an Assistant Professor in the School of Engineering at San Francisco State University. She previously spent eight years as a Teaching Professor in the Engineering, Design, and Society Division and the Electrical Engineering Departmen

visit author page

biography

Fatemeh Khalkhal San Francisco State University

visit author page

Dr. Khalkhal is an assistant professor in mechanical engineering at San Francisco State University. She has a PhD in Chemical Engineering from Ecole Polytechnique de Montreal. Her research experience and interest are in developing structure-property relationships in complex fluids and broadening the participation of women and underrepresented minorities in engineering.

visit author page

biography

Yiyi Wang San Francisco State University

visit author page

Yiyi Wang is an assistant professor of civil engineering at San Francisco State University. In addition to engineering education, her research also focuses on the nexus between mapping, information technology, and transportation and has published in Accident Analysis & Prevention, Journal of Transportation Geography, and Annuals of Regional Science. She served on the Transportation Research Board (TRB) ABJ80 Statistical Analysis committee and the National Cooperative Highway Research Program (NCHRP) panel. She advises the student chapter of the Society of Women Engineers (SWE) at SFSU.

visit author page

Download Paper | Permalink

Abstract

This study explores the potential of ChatGPT, a leading-edge language model-based chatbot, in crafting analytic research memos (ARMs) from student interview transcripts for use in qualitative data analysis. With a rising interest in harnessing artificial intelligence (AI) for qualitative research, our study aims to explore ChatGPT's capability to streamline and enhance this process.

The research is part of a mixed-methods project examining the relationships between engineering students' team experiences, team disagreements, and engineering identities. Our team had previously developed an interview protocol for collecting qualitative data and initiated analysis using coding methods and ARMs for individual transcripts. We designed an ARM Development Guidelines document to ensure consistency among four team members in the ARM creation process. The guidelines include a set of key questions that each ARM should aim to address.

Our objective is to assess ChatGPT's proficiency in creating ARMs based on our development guidelines and compare its outputs with human-written ARMs for accuracy and depth of insight. For this purpose, we selected two student interview transcripts. A structured analysis protocol for ChatGPT was devised in adherence to the ARM Development Guidelines.

Two team members, experienced in qualitative analysis and ARM composition, drafted ARMs for the chosen transcripts using the same guidelines, enabling a direct outcome comparison. Subsequently, a rigorous validation process was conducted, using rubrics to assess narratives from both methods. The manual ARM authors performed a self-assessment, while the other researchers conducted a blind evaluation of the human-generated and AI-generated ARMs. We used two rubrics for this comparison. A general rubric gauged accuracy, clarity, analysis time, and usefulness. A specialized rubric was used to determine if the ARMs address the topics laid out in the ARM guidelines, such as self-identification, perceptions of engineering, teamwork descriptions, connections between identity and team experiences, comparisons with other interviews, and reflections.

In this paper, we describe our research methodology, present our findings, evaluate the advantages and limitations of ChatGPT in qualitative analysis within engineering education research, and provide guidance for future research directions. We aim to shed light on the capabilities of ChatGPT in qualitative analysis and contribute to the ongoing dialogue on harnessing AI for research in engineering education. Our findings will inform researchers and practitioners about the benefits, challenges, and best practices associated with integrating AI-powered tools such as ChatGPT into qualitative research methods.

Citation
Format

Zhang, X., & Claussen, S., & Khalkhal, F., & Wang, Y. (2024, June), Evaluating ChatGPT's Efficacy in Qualitative Analysis of Engineering Education Research Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--47341

TY - CPAPER
AB - This study explores the potential of ChatGPT, a leading-edge language model-based chatbot, in crafting analytic research memos (ARMs) from student interview transcripts for use in qualitative data analysis. With a rising interest in harnessing artificial intelligence (AI) for qualitative research, our study aims to explore ChatGPT&#39;s capability to streamline and enhance this process.

The research is part of a mixed-methods project examining the relationships between engineering students&#39; team experiences, team disagreements, and engineering identities. Our team had previously developed an interview protocol for collecting qualitative data and initiated analysis using coding methods and ARMs for individual transcripts. We designed an ARM Development Guidelines document to ensure consistency among four team members in the ARM creation process. The guidelines include a set of key questions that each ARM should aim to address.

Our objective is to assess ChatGPT&#39;s proficiency in creating ARMs based on our development guidelines and compare its outputs with human-written ARMs for accuracy and depth of insight. For this purpose, we selected two student interview transcripts. A structured analysis protocol for ChatGPT was devised in adherence to the ARM Development Guidelines.

Two team members, experienced in qualitative analysis and ARM composition, drafted ARMs for the chosen transcripts using the same guidelines, enabling a direct outcome comparison. Subsequently, a rigorous validation process was conducted, using rubrics to assess narratives from both methods. The manual ARM authors performed a self-assessment, while the other researchers conducted a blind evaluation of the human-generated and AI-generated ARMs. We used two rubrics for this comparison. A general rubric gauged accuracy, clarity, analysis time, and usefulness. A specialized rubric was used to determine if the ARMs address the topics laid out in the ARM guidelines, such as self-identification, perceptions of engineering, teamwork descriptions, connections between identity and team experiences, comparisons with other interviews, and reflections.

In this paper, we describe our research methodology, present our findings, evaluate the advantages and limitations of ChatGPT in qualitative analysis within engineering education research, and provide guidance for future research directions. We aim to shed light on the capabilities of ChatGPT in qualitative analysis and contribute to the ongoing dialogue on harnessing AI for research in engineering education. Our findings will inform researchers and practitioners about the benefits, challenges, and best practices associated with integrating AI-powered tools such as ChatGPT into qualitative research methods.

AU - Xiaorong Zhang
AU - Stephanie Claussen
AU - Fatemeh Khalkhal
AU - Yiyi Wang
CY - Portland, Oregon
DA - 2024/06/23
PB - ASEE Conferences
TI - Evaluating ChatGPT's Efficacy in Qualitative Analysis of Engineering Education Research
UR - https://peer.asee.org/47341
DO - 10.18260/1-2--47341
ER -