Asee peer logo

Work-in-Progress: Using Latent Dirichlet Allocation to uncover themes in student comments from peer evaluations of teamwork

Download Paper |

Conference

2022 ASEE Annual Conference & Exposition

Location

Minneapolis, MN

Publication Date

August 23, 2022

Start Date

June 26, 2022

End Date

June 29, 2022

Conference Session

Educational Research and Methods (ERM) Division Poster Session

Page Count

8

DOI

10.18260/1-2--41066

Permanent URL

https://peer.asee.org/41066

Download Count

473

Request a correction

Paper Authors

biography

Gaurav Nanda Purdue University at West Lafayette (COE)

visit author page

Dr. Gaurav Nanda is an Assistant Professor in the School of Engineering Technology at Purdue University with focus in Industrial Engineering Technology. His research interests include text mining, collaborative information systems, and intelligent decision support systems combining AI and human expertise. Dr. Nanda has worked as a postdoctoral researcher at Purdue University for two years and in the software industry for five years. He obtained his Ph.D. in Industrial Engineering from Purdue University and his Bachelors (B.Tech.) and Masters (M.Tech.) from Indian Institute of Technology Kharagpur, India.

visit author page

biography

Siqing Wei Purdue University at West Lafayette (COE)

visit author page

Siqing Wei received B.S. and M.S. in Electrical Engineering from Purdue University. He is currently pursuing a Ph.D. degree in Engineering Education program at Purdue University. After years of experience serving as a peer teacher and a graduate teaching assistant in first-year engineering courses, he has been a research assistant at CATME research group studying multicultural team dynamics and outcomes. The research interests span how cultural diversity impacts teamwork and how to help students improve intercultural competency and teamwork competency by interventions, counseling, pedagogy, and tool selection (such as how to use CATME Team-Maker to form inclusive and diversified teams) to promote DEI. In addition, he also works on many research-to-practice projects to enhance educational technology usage in engineering classrooms and educational research by various methods, such as natural language processing. In addition, he is also interested in the learning experiences of international students. Siqing also works as the technical development and support manager at the CATME research group.

visit author page

biography

Matthew Ohland Purdue University at West Lafayette (COE)

visit author page

Matthew W. Ohland is Associate Head and the Dale and Suzi Gallagher of Professor of Engineering Education at Purdue University. He has degrees from Swarthmore College, Rensselaer Polytechnic Institute, and the University of Florida. He studies the longitudinal study of engineering students and forming and managing student teams and with collaborators has been recognized for the best paper published in the Journal of Engineering Education in 2008, 2011, and 2019 and from the IEEE Transactions on Education in 2011 and 2015. Dr. Ohland is an ABET Program Evaluator for ASEE. He was the 2002–2006 President of Tau Beta Pi and is a Fellow of the ASEE, IEEE, and AAAS.

visit author page

author page

Andrew Katz Virginia Polytechnic Institute and State University

author page

Christopher Brinton Purdue University at West Lafayette (COE)

Download Paper |

Abstract

This work-in-process research paper investigates common themes in peer-to-peer comments of teamwork behavior effectiveness collected with peer evaluations in engineering student teams in three time horizons – prior to COVID-19 pandemic, early phase of pandemic, and mature phase of pandemic. Constructive feedback is imperative to maintaining healthy team climate and dynamic, which facilitates positive individual and team learning outcomes. Asking engineering students to provide self- and peer-evaluation feedback in comments accomplishes multiple objectives. Students reflect on teammates’ behavior and performance rather than relying on (potentially biased) general perceptions to provide evidence-based comments for the assessment period. Repeated practice giving feedback also tends to improve students’ ability to provide constructive and insightful evaluations. To better understand what and how engineering students provide feedback in teamwork, the Comprehensive Assessment of Team-Member Effectiveness (CATME) peer evaluation tool suite was used to provide a framework to teach students about effective team behaviors using a behavioral-anchored rating scale. Using CATME also provided a mechanism for collecting self- and peer- evaluation survey data in both structured (the behavioral scale) and open-ended (comments) ways. Latent Dirichlet Allocation (LDA) was used as the classic method for topic modeling to analyze first-year engineering students’ self- and peer- comments in the introductory engineering foundation courses in a large Midwestern R1 university. Topic Coherence measure (c_v) for topic quality was used to determine the optimal number of topics to represent the comment data. The themes of each of the topics identified were interpreted by thematic analysis of the most commonly used words and responses associated with each topic identified by the LDA model. The preliminary results showed that pre-pandemic themes closely matched the five behavioral dimensions of the CATME instrument. Data collected in Spring 2020 required more themes to capture the complexity of the transition to online learning. Comments from Spring 2021 required an even larger number of themes to describe the experience of teamwork during a fully virtual class implementation.

Nanda, G., & Wei, S., & Ohland, M., & Katz, A., & Brinton, C. (2022, August), Work-in-Progress: Using Latent Dirichlet Allocation to uncover themes in student comments from peer evaluations of teamwork Paper presented at 2022 ASEE Annual Conference & Exposition, Minneapolis, MN. 10.18260/1-2--41066

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2022 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015