Paper ID #49105An Assessment of ChatGPT 4o’s Performance on Mechanical EngineeringConcept InventoriesDr. Rujun Gao, Texas A&M University Dr. Rujun Gao has completed her Ph.D. in Mechanical Engineering at Texas A&M University and holds an M.S. in Mechanical Engineering from Zhejiang University, China. Her research focuses on Generative AI, Natural Language Processing (NLP), Large Language Models (LLMs), LLM Agents, and the development of educational technology products.Hillary E. Merzdorf, Cornell University STEM Instructional Design Associate, eCornell, Cornell UniversityXiaosu Guo, University of Texas at
Paper ID #48784BOARD # 185: Possibility of ChatGPT Application on Mechanical DesignEducation: A Comparative Study of Traditional and AI-Assisted Design ProcessesProf. Yingxiao Song, Muskingum University Assistant Professor in Physics and Engineering Department of Muskingum University ©American Society for Engineering Education, 2025 Possibility of ChatGPT Application on Mechanical DesignEducation: A Comparative study of traditional and AI-assisted Design ProcessAbstractMany educators are hesitant to integrate tools like ChatGPT into the classroomdue to concerns about plagiarism and copying. However
ThermodynamicsAbstractGenerative artificial intelligence (GenAI) has become ubiquitous. Convincing languagecomplemented by constant modifications and upgrades have made GenAI models, such asOpenAI’s ChatGPT, an appealing tool to address complex problems. According to a survey byIntelligent.com nearly a third of college students in AY 2022-2023 used ChatGPT for schoolworkand 77.4% of them were likely to recommend using it to study to another student. Despite theirappeal, these models have proven flawed in answering technical prompts. Their convincinglanguage may entice the user to trust the responses without verifying them. For example, theauthors failed to retrieve accurate thermodynamics properties of some common substances fromthree publicly available models (OpenAI’s
areas. F08 did well overall, but should beencouraged to improve his or her timeliness in reviewing team submissions. Figure 2. Capstone advisor survey results for F02 Figure 3. Capstone advisor survey results for F08Answers to the free response questions on advisor strengths and areas for improvement weresubmitted to ChatGPT for analysis and summary. The data was first anonymized by replacingstudent and faculty names with a random-ordered research ID, changing gendered pronouns to“they,” and redacting identifying comments that could not otherwise be removed.The prompt given to ChatGPT was as follows: “Assume the role of an experienced highereducation administrator. You are reviewing student
calculator of the 1970s, and then to computer-based software andcomputational methods. Wolfram Alpha made a large step forward in the ability to solve a varietyof problems and explain the steps to learners everywhere. 2 Now, AI, and specificallylarge-language models (LLMs) such as ChatGPT provide the next evolution in solving complexproblems while showing detailed commentary on every step and calculation made. But AI’sability to aid an engineers in their endeavor to solve the world’s technical challenges is muchmore broad. A brief review of AI’s definition, emergence, and varied types is appropriate.2.1 Artificial Intelligence DefinitionA term as broad as ‘artificial intelligence’ is bound to have many definitions, most with significantoverlap
. The session then transitions to the transformative role of Transformers in NLP, focusing on their improvements over RNNs without delving into advanced mathematical details. This leads to the discussion of Large Language Models (LLMs), such as ChatGPT [8], emphasizing their engineering applications. Students learn to use ChatGPT’s API to integrate NLP into workflows, with a Python example showing how to send prompts, receive responses, and maintain conversational context by including prior interactions. These hands-on examples help MET students understand the practical applications of NLP tools like LLMs in solving engineering problems. • Topic 8: Reinforcement Learning (RL) – The last topic
understanding and experience.Additionally, applying robust statistical methods is essential for tracking and analyzing studentperformance over time, ensuring that the effectiveness of the VR interventions can be measured andrefined for future improvement. IX. Suggested Survey QuestionsPlease rate your agreement with each statement using the following scale:*These questions were formatted and formulated with the help of ChatGPT: • 1 - Strongly Disagree • 2 - Disagree • 3 - Neutral • 4 - Agree • 5 - Strongly AgreeA. Learning and Understanding 1. The VR activities enhanced my understanding of complex mechanical concepts. 2. VR helped me visualize engineering problems better than traditional methods
., Kim, S.H., Lee, D. et al. Utilizing Generative AI for Instructional Design: Exploring Strengths, Weaknesses, Opportunities, and Threats. TechTrends 68, 832–844 (2024). https://doi.org/10.1007/s11528-024-00967-w [4] Nikolic, S., Sandison, C., Haque, R., Daniel, S., Grundy, S., Belkina, M., … Neal, P. (2024). ChatGPT, Copilot, Gemini, SciSpace and Wolfram versus higher education assessments: an updated multi-institutional study of the academic integrity impacts of Generative Artificial Intelligence (GenAI) on assessment, teaching and learning in engineering. Australasian Journal of Engineering Education, 29(2), 126–153. https://doi.org/10.1080/22054952.2024.2372154 [5] Subramanian, R., & Vidalis, S
in freshman information processing and the rise in Using academic resources both fall outside of the 2 sigma bands starting the Fall of 2022. Since ChatGPT was introduced in November 2022, this decrease is likely not due to AI usage. The COVID-19 pandemic effect, on the other hand, matches the timing. The national trends mentioned in sections IV.A.1 and 2 above support this theory. However, whether this is the true cause needs further research. Further, the rise in Using academic resources may be happening in compensation for the dropping Information Processing skill. Again, to establish if this is the case, will need further research. 2
students’learning gains in STEM education as in Arora et al.1 and Van den Broeck et al.2. However, a rapidchange in online landscape accelerated by COVID-19 pandemic has brought up serious academicmisconduct issues, as evidenced by the students’ frequent utilization of websites and AI tools suchChegg3, Quizlet4, and ChatGPT 4o5. The matter was compounded during COVID-19 when theisolated environments contributed to students’ lack of motivation to study and learn, Y. Terada6.The academic misbehaviors are further described by P. Charlesworth et al.7, M. M. Lanier8 as wellas by A. Fask et al.9. In effect, this creates grade inflation and possibly jeopardizes the academicintegrity of the institution’s program that could in turn dampen students’ motivation.One