Asee peer logo

Evaluating ChatGPT’s Engineering-Reasoning Capabilities and Constraints Through Examples from Mechanical-Engineering Education

Download Paper |

Conference

2024 ASEE Annual Conference & Exposition

Location

Portland, Oregon

Publication Date

June 23, 2024

Start Date

June 23, 2024

End Date

June 26, 2024

Conference Session

MECH - Technical Session 13: Technological Advancements and Applications

Tagged Division

Mechanical Engineering Division (MECH)

Page Count

14

DOI

10.18260/1-2--47342

Permanent URL

https://peer.asee.org/47342

Download Count

149

Paper Authors

author page

Bingling Huang California State University, Fullerton

biography

Chan Lu University of Georgia

visit author page

higher education researcher

visit author page

Download Paper |

Abstract

This paper evaluates the mechanical engineering reasoning capabilities of ChatGPT-4, an advanced Large Language Model (LLM), with the aim of enhancing mechanical engineering education. Mechanical engineering education extends beyond text comprehension, a domain for which ChatGPT is renowned. It aims to nurture future engineers to become critical thinkers, who are proficient in applying acquired knowledge to execute complicated engineering tasks. However, there’s an apparent gap in understanding how ChatGPT can be effectively integrated into educational practices within this specialized area due to a lack of detailed insights into its abilities and limitations. This research seeks to fill this void by exploring and assessing ChatGPT’s reasoning abilities and limitations within the context of mechanical engineering. It examines the capabilities and constraints of ChatGPT in engineering reasoning by analyzing two mechanical examples, which are drawn from machine design and dynamics. By comparing ChatGPT’s entire reasoning process and individual steps with human reasoning, this investigation unveils both its constraints and capacities. The results show that ChatGPT’s limited capability to understand the profound implications of text. It addresses the need for caution when employing it in reasoning tasks within the context of mechanical engineering education.

Huang, B., & Lu, C. (2024, June), Evaluating ChatGPT’s Engineering-Reasoning Capabilities and Constraints Through Examples from Mechanical-Engineering Education Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--47342

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2024 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015