WIP: AI-based Sentiment Analysis and Grader Enhancements

Bobby F Hodgkinson; Nathan Eric Whittenburg

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: June 26, 2024
Conference Session: Teaching with ML and Generative AI
Tagged Division: Computers in Education Division (COED)
Page Count: 15
DOI: 10.18260/1-2--48287
Permanent URL: https://peer.asee.org/48287
Download Count: 57

Paper Authors

biography

Bobby F Hodgkinson University of Colorado Boulder

visit author page

Bobby Hodgkinson is an Associate Teaching Professor in the Smead Aerospace Engineering Sciences Department (AES) and co-manages the educational electronics and instrumentation shop. He assists students and researchers in the department for sensor and data acquisition needs as well as manages several lab courses and experiments. He is a member of the Professional Advisory Board for the senior capstone projects course. Prior to joining Smead Aerospace department in 2012, he was the lab manager at the Institute of Networked Autonomous Systems at the University of Florida, Gainesville where he focused on the research and development of small, autonomous aerial and underwater vehicles, sensors and actuators. He received a BS and MS degree from the Aerospace Engineering Sciences department at CU Boulder in 2010 and 2011 respectively.

visit author page

biography

Nathan Eric Whittenburg University of Colorado Boulder

visit author page

Nathan Whittenburg is currently pursuing a degree in Aerospace Engineering with a minor in Computer Science at the University of Colorado Boulder. He serves as a Lab Assistant in the Aerospace department, where his responsibilities include employing Large Language Models and Natural Language Processing to enhance educational outcomes, managing and maintaining lab equipment, and assisting students with conducting experimental procedures.

visit author page

Download Paper | Permalink

Abstract

Peer feedback for group activities gives instructors valuable insight into the inner workings of a group. Peer assessment can also provide useful feedback to the individual participants. In experiential learning, particularly in disciplines like undergraduate Aerospace Engineering, group work plays a valuable role to prepare students for a career in collaborative environments and feedback on an individual’s performance can be a useful pedagogical tool. To enhance the peer review process for large (200+ students), this study implements sentiment analysis, specifically using the roBERTa sentiment analysis model [1], to provide a quantitative assessment of reviews received by individual students. Additionally, the work aims to evolve to include AI-based constructive criticism paraphrasing to allow for timely individualized feedback in a large-enrollment setting.

The primary objective of this ongoing work is to examine how sentiment analysis scores correlate with the numerical scores assigned by students to their peers. By leveraging natural language processing and machine learning, we aim to quantify the qualitative aspect of sentiment in peer reviews. The underlying hypothesis is that sentiment analysis can provide a complementary perspective on the quality of peer reviews, shedding light on aspects such as positivity, constructiveness, and critical feedback.

The study focuses on an Aerospace Engineering sophomore experiential learning course, where group work serves as a fundamental medium of student engagement. The course uses a basic peer review process, where students evaluate their peers' contributions. These evaluations form the basis for an individual’s grade based on their contributions to their group’s work. However, there may be subjectivity, bias, or other unaccounted factors in a simple numerical score which sentiment analysis can potentially reveal.

In our presentation, we will discuss the methodology used to implement sentiment analysis and the dataset involved. We will share preliminary results and insights from our ongoing investigation, analyzing the extent to which sentiment analysis scores align with the numeric peer review scores. This alignment can serve as a measure of the effectiveness of sentiment analysis in capturing the quality of peer reviews.

As a further development of this work, we plan to incorporate AI-based constructive criticism paraphrasing. This extension will involve the use of natural language generation models to offer suggestions for improving an individual’s performance in future groups. By providing students with constructive, actionable recommendations from their peers, this addition aims to enhance the usefulness of peer reviews and contribute to a more positive and collaborative learning environment.

Our study has the potential to contribute to the ongoing dialog on peer assessment in education and to inform educators on the benefits and challenges of implementing sentiment analysis and AI-based paraphrasing in the evaluation process. Ultimately, we aim to improve the fairness and objectivity of peer assessment, thereby enhancing the overall learning experience in the field of engineering

This presentation will be of interest to educators, researchers, and practitioners in the Computers in Education Division at ASEE 2024, as it explores innovative approaches to improving the quality and effectiveness of peer review processes in STEM education. We look forward to sharing our insights and engaging in discussions with fellow attendees to advance the understanding of the role of sentiment analysis and AI-based paraphrasing in education.

References: [1] Accessed from: https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment

Citation
Format

Hodgkinson, B. F., & Whittenburg, N. E. (2024, June), WIP: AI-based Sentiment Analysis and Grader Enhancements Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--48287

TY - CPAPER
AB - Peer feedback for group activities gives instructors valuable insight into the inner workings of a group. Peer assessment can also provide useful feedback to the individual participants. In experiential learning, particularly in disciplines like undergraduate Aerospace Engineering, group work plays a valuable role to prepare students for a career in collaborative environments and feedback on an individual’s performance can be a useful pedagogical tool. To enhance the peer review process for large (200+ students), this study implements sentiment analysis, specifically using the roBERTa sentiment analysis model [1], to provide a quantitative assessment of reviews received by individual students. Additionally, the work aims to evolve to include AI-based constructive criticism paraphrasing to allow for timely individualized feedback in a large-enrollment setting.

The primary objective of this ongoing work is to examine how sentiment analysis scores correlate with the numerical scores assigned by students to their peers. By leveraging natural language processing and machine learning, we aim to quantify the qualitative aspect of sentiment in peer reviews. The underlying hypothesis is that sentiment analysis can provide a complementary perspective on the quality of peer reviews, shedding light on aspects such as positivity, constructiveness, and critical feedback.

The study focuses on an Aerospace Engineering sophomore experiential learning course, where group work serves as a fundamental medium of student engagement. The course uses a basic peer review process, where students evaluate their peers&#39; contributions. These evaluations form the basis for an individual’s grade based on their contributions to their group’s work. However, there may be subjectivity, bias, or other unaccounted factors in a simple numerical score which sentiment analysis can potentially reveal.

In our presentation, we will discuss the methodology used to implement sentiment analysis and the dataset involved. We will share preliminary results and insights from our ongoing investigation, analyzing the extent to which sentiment analysis scores align with the numeric peer review scores. This alignment can serve as a measure of the effectiveness of sentiment analysis in capturing the quality of peer reviews.

As a further development of this work, we plan to incorporate AI-based constructive criticism paraphrasing. This extension will involve the use of natural language generation models to offer suggestions for improving an individual’s performance in future groups. By providing students with constructive, actionable recommendations from their peers, this addition aims to enhance the usefulness of peer reviews and contribute to a more positive and collaborative learning environment.

Our study has the potential to contribute to the ongoing dialog on peer assessment in education and to inform educators on the benefits and challenges of implementing sentiment analysis and AI-based paraphrasing in the evaluation process. Ultimately, we aim to improve the fairness and objectivity of peer assessment, thereby enhancing the overall learning experience in the field of engineering

This presentation will be of interest to educators, researchers, and practitioners in the Computers in Education Division at ASEE 2024, as it explores innovative approaches to improving the quality and effectiveness of peer review processes in STEM education. We look forward to sharing our insights and engaging in discussions with fellow attendees to advance the understanding of the role of sentiment analysis and AI-based paraphrasing in education.

References:
[1] Accessed from: https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment

AU - Bobby F Hodgkinson
AU - Nathan Eric Whittenburg
CY - Portland, Oregon
DA - 2024/06/23
PB - ASEE Conferences
TI - WIP: AI-based Sentiment Analysis and Grader Enhancements
UR - https://peer.asee.org/48287
DO - 10.18260/1-2--48287
ER -