Ding, X., & Gopannagari, M., & Sun, K., & Tao, A., & Zhao, D. L., & Varadhan, S., & Hardy, B. L. B., & Dalpiaz, D., & Vogiatzis, C., & Angrave, L., & Liu, H. (2024, June), Evaluation of LLMs and Other Machine Learning Methods in the Analysis of Qualitative Survey Responses for Accessible Engineering Education Research  Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--47360

\bibitem{asee_peer_47360}
  Ding, X., \& Gopannagari, M., \& Sun, K., \& Tao, A., \& Zhao, D. L., \& Varadhan, S., \& Hardy, B. L. B., \& Dalpiaz, D., \& Vogiatzis, C., \& Angrave, L., \& Liu, H. (2024, June), \emph{Evaluation of LLMs and Other Machine Learning Methods in the Analysis of Qualitative Survey Responses for Accessible Engineering Education Research}
  Paper presented at 2024 ASEE Annual Conference \& Exposition, Portland, Oregon.
  10.18260/1-2--47360

Xiuhao Ding, Meghana Gopannagari, Kang Sun, Alan Tao, Delu Louis Zhao, Sujit Varadhan, Bobbi Lee Battleson Hardy, David Dalpiaz, Chrysafis Vogiatzis, Lawrence Angrave, and Hongye Liu.  "Evaluation of LLMs and Other Machine Learning Methods in the Analysis of Qualitative Survey Responses for Accessible Engineering Education Research".  2024 ASEE Annual Conference & Exposition, Portland, Oregon, 2024, June.  ASEE Conferences, 2024.  https://peer.asee.org/47360 Internet.  15 Dec, 2024

\bibitem{asee_peer_47360}
  Xiuhao Ding, Meghana Gopannagari, Kang Sun, Alan Tao, Delu Louis Zhao, Sujit Varadhan, Bobbi Lee Battleson Hardy, David Dalpiaz, Chrysafis Vogiatzis, Lawrence Angrave, and Hongye Liu.
  "Evaluation of LLMs and Other Machine Learning Methods in the Analysis of Qualitative Survey Responses for Accessible Engineering Education Research".
  \emph{2024 ASEE Annual Conference \& Exposition, Portland, Oregon, 2024, June}.
  ASEE Conferences, 2024.
  https://peer.asee.org/47360 Internet.  15 Dec, 2024

@INPROCEEDINGS{asee_peer_47360
author = "Xiuhao Ding, Meghana Gopannagari, Kang Sun, Alan Tao, Delu Louis Zhao, Sujit Varadhan, Bobbi Lee Battleson Hardy, David Dalpiaz, Chrysafis Vogiatzis, Lawrence Angrave, and Hongye Liu"
title = "Evaluation of LLMs and Other Machine Learning Methods in the Analysis of Qualitative Survey Responses for Accessible Engineering Education Research"
booktitle = "2024 ASEE Annual Conference \& Exposition"
year = "2024"
month = "June"
address = "Portland, Oregon"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/47360}
number = {10.18260/1-2--47360}
}

TY  - CPAPER
AB  - This research paper provides insights and guidance for selecting appropriate analytical tools in engineering educational research. Currently, educators and researchers face difficulties in gaining insights effectively from free-response survey data. We evaluate the effectiveness and accuracy of Large Language Models (LLMs), in addition to the existing methods that employ topic modeling, document clustering coupled with Support Vector Machine (SVM) and Random Forest (RF) approaches, and the unsupervised Latent Dirichlet Allocation (LDA) method. Free responses to open-ended questions from student surveys in multiple courses at University of Illinois Urbana-Champaign were previously collected by engineering education accessibility researchers. The data (N=129 with seven free response questions per student) were previously analyzed to assess the effectiveness, satisfaction, and quality of adding accessible digital notes to multiple engineering courses and the students’ perceived belongingness, and self-efficacy. Manual codings for the seven open-ended questions were generated for qualitative tasks of sentiment analysis, topic modeling, and summarization and were used in this study as a gold standard to evaluate automated text analytic approaches. Raw text from open-ended questions was converted into numerical vectors using text vectorization and word embeddings and an unsupervised analysis using document clustering and topic modeling was performed using LDA and BERT methods. In addition to conventional machine learning models, multiple pre-trained open-sourced local LLMs were evaluated (BART and LLaMA) for summarization. The remote online ChatGPT closed-model services by OpenAI (ChatGPT-3.5 and ChatGPT-4) were excluded due to subject data privacy concerns. By comparing the accuracy, recall, and depth of thematic insights derived, we evaluated how effectively the method based on each model categorized and summarized students’ responses across educational research interests of effectiveness, satisfaction, and quality of education materials. The paper will present these results and discuss the implications of our findings and conclusions.

AU  - Xiuhao Ding
AU  - Meghana Gopannagari
AU  - Kang Sun
AU  - Alan Tao
AU  - Delu Louis Zhao
AU  - Sujit Varadhan
AU  - Bobbi Lee Battleson Hardy
AU  - David Dalpiaz
AU  - Chrysafis Vogiatzis
AU  - Lawrence Angrave
AU  - Hongye Liu
CY  - Portland, Oregon
DA  - 2024/06/23
PB  - ASEE Conferences
TI  - Evaluation of LLMs and Other Machine Learning Methods in the Analysis of Qualitative Survey Responses for Accessible Engineering Education Research
UR  - https://peer.asee.org/47360
DO  - 10.18260/1-2--47360
ER  -