ASEE PEER - Design and Development of Survey Instrument to Measure Engineering Students’ Perspectives on the Use of ChatGPT
Asee peer logo

Design and Development of Survey Instrument to Measure Engineering Students’ Perspectives on the Use of ChatGPT

Download Paper |

Conference

2024 ASEE Annual Conference & Exposition

Location

Portland, Oregon

Publication Date

June 23, 2024

Start Date

June 23, 2024

End Date

July 12, 2024

Conference Session

Educational Research and Methods Division (ERM) Technical Session 22

Tagged Division

Educational Research and Methods Division (ERM)

Permanent URL

https://peer.asee.org/47120

Request a correction

Paper Authors

author page

Mohammad Faraz Sajawal University of Oklahoma

biography

Javeed Kittur University of Oklahoma Orcid 16x16 orcid.org/0000-0001-6132-7304

visit author page

Dr. Kittur is an Assistant Professor in the Gallogly College of Engineering at The University of Oklahoma. He completed his Ph.D. in Engineering Education Systems and Design program from Arizona State University, 2022. He received a bachelor’s degree in Electrical and Electronics Engineering and a Master’s in Power Systems from India in 2011 and 2014, respectively. He has worked with Tata Consultancy Services as an Assistant Systems Engineer from 2011–2012 in India. He has worked as an Assistant Professor (2014–2018) in the department of Electrical and Electronics Engineering, KLE Technological University, India. He is a certified IUCEE International Engineering Educator. He was awarded the ’Ing.Paed.IGIP’ title at ICTIEE, 2018. He is serving as an Associate Editor of the Journal of Engineering Education Transformations (JEET).

He is interested in conducting engineering education research, and his interests include student retention in online and in-person engineering courses/programs, data mining and learning analytics in engineering education, broadening student participation in engineering, faculty preparedness in cognitive, affective, and psychomotor domains of learning, and faculty experiences in teaching online courses. He has published papers at several engineering education research conferences and journals. Particularly, his work is published in the International Conference on Transformations in Engineering Education (ICTIEE), American Society for Engineering Education (ASEE), Computer Applications in Engineering Education (CAEE), International Journal of Engineering Education (IJEE), Journal of Engineering Education Transformations (JEET), and IEEE Transactions on Education. He is also serving as a reviewer for a number of conferences and journals focused on engineering education research.

visit author page

Download Paper |

Abstract

Chat Generative Pre-Trained Transformer (ChatGPT) is a language model created by engineers working in Open Artificial Intelligence. It is a type of artificial intelligence system that generates human-like text responses to a wide range of prompts and questions. ChatGPT offers several advantages including 24/7 support, quick response to questions, finding research-related information, writing a coding program, etc. Despite these advantages, ChatGPT has limited contextual understanding of a certain topic, which can lead to incorrect/irrelevant responses. It can also be biased based on the data used to train the program, which can lead to unfair or inaccurate feedback. ChatGPT can unfortunately be vulnerable to causing security risks, which may lead to data breaches and sensitive information of students being leaked. With the rising popularity of ChatGPT, just like any other online resource, the over-reliance on it could lead to a decline in independent problem-solving skills and critical thinking in an academic setting.

This research project aims at understanding the students’ perspectives on the use of ChatGPT in engineering. This topic is relevant, timely, and important as ChatGPT as created sufficient stir in education. By exploring students’ experiences and perspectives, we aim to shed light on different aspects of usage of ChatGPT and glean critical insights. The objective of this study was to design, develop and validate a survey instrument that measures engineering students’ perceptions on the use of ChatGPT. To meet the objective of this research study, a survey instrument was designed which included five dimensions: learning tool (10 items), trustworthiness (5 items), ethical considerations (5 items), ease of access (6 items), and concerns with ChatGPT (6 items). To collect the evidence for content validity and face, the survey instrument was reviewed by three content experts and three potential participants. The survey instrument was revised/updated using the feedback from both content experts and potential participants. The data for this study was collected in summer and fall 2023, and 323 responses were included in the analysis. Exploratory factor analysis (EFA) revealed four factors learning tool, trustworthiness, ease of access and concerns with ChatGPT, and the dimension ‘ethical considerations’ was suggested to be removed after the EFA. The Cronbach’s alpha ranged between 0.62 to 0.82 suggesting good internal consistency reliability between the items.

Sajawal, M. F., & Kittur, J. (2024, June), Design and Development of Survey Instrument to Measure Engineering Students’ Perspectives on the Use of ChatGPT Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. https://peer.asee.org/47120

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2024 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015