Marshall University, Huntington, West Virginia
March 28, 2025
March 28, 2025
March 29, 2025
Diversity
17
https://peer.asee.org/54654
This study examines how large language models categorize sentences from scientific papers using prompt engineering. We use two advanced web-based models, OpenAI’s GPT-4o and DeepSeek R1, to classify sentences into predefined relationship categories. DeepSeek R1 has been tested on benchmark datasets in its technical report. However, its performance in scientific text categorization remains unexplored. To address this gap, we introduce a new evaluation method designed specifically for this task. We also compile a dataset of cleaned scientific papers from diverse domains. This dataset provides a platform for comparing the two models. Using this dataset, we analyze their effectiveness and consistency in categorization
Maiti, A., & Adewumi, S., & TIKURE, T. A., & Wang, Z., & Sengupta, N., & Sukhanova, A., & Jana, A. (2025, March), Comparative Analysis of OpenAI GPT-4o and DeepSeek R1 for Scientific Text Categorization Using Prompt Engineering Paper presented at 2025 ASEE North Central Section (NCS) Annual Conference, Marshall University, Huntington, West Virginia. https://peer.asee.org/54654
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2025 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015