Evaluating Publications' Keywords in Computer Science Education Research: A Bibliometric NLP Approach

Jia Zhu; Leila Zahedi; Monique S. Ross

Download Paper | Permalink

Conference: 2021 ASEE Virtual Annual Conference Content Access
Location: Virtual Conference
Publication Date: July 26, 2021
Start Date: July 26, 2021
End Date: July 19, 2022
Conference Session: Computing and Information Technology Division Technical Session 3
Tagged Division: Computing and Information Technology
Page Count: 15
DOI: 10.18260/1-2--37104
Permanent URL: https://peer.asee.org/37104
Download Count: 471

Request a correction

Paper Authors

biography

Jia Zhu Florida International University orcid.org/0000-0001-9234-5919

visit author page

Jia Zhu is a Ph.D. student in the Knight Foundation School of Computing and Information Science at Florida International University (FIU). Her research interests include computer science education, educational data mining, and data science, with a focus on broadening participation in computing.

visit author page

biography

Leila Zahedi Florida International University orcid.org/0000-0002-7325-1025

visit author page

Leila Zahedi is a Ph.D. candidate in computer science at the School of Computing and Information Sciences (SCIS) at Florida International University (FIU). Her research interests span the fields of educational data mining, machine learning optimization, and data science. Leila also received an M.S. degree in Management of Advanced Information Systems, in addition to her B.S. and M.S. degrees in Computer Science.

visit author page

biography

Monique S. Ross Florida International University orcid.org/0000-0002-6320-636X

visit author page

Monique Ross, Assistant Professor in the School of Computing and Information Sciences and STEM Transformation Institute at Florida International University, designs research focused on broadening participation in computer science through the exploration of: 1) race, gender, and disciplinary identity; 2) discipline-based education research (with a focus on computer science and computer engineering courses) in order to inform pedagogical practices that garner interest and retain women (specifically Black and Hispanic women) in computer-related engineering fields.

visit author page

Download Paper | Permalink

Abstract

In the rapidly growing field of computer science education research (CSER), the collection of publication keywords represent the knowledge landscape of the research domain. This helps stakeholders to contextualize existing research topics, identify the strategic direction for future works, and shape the culture of this growing community. For the research studies published in CSER, the use of proper keywords will help to improve publication visibility in research networks. Publication visibility is defined as the share of traffic a study receives based on its ranking from Search Engine Result Pages (SERPs), which can be gained through indexing relevant keywords to help search engines to find the related studies. It is important to acknowledge that analyzing keyword selections in the context provided by the publication abstract, enables us to determine the keywords’ relevance to the research topics. However, evaluating the selection of publication keywords is an important but lesser-addressed issue.

In this work, we utilized a bibliometric approach to scrutinize publication keywords in CSER publications from 2015 to 2020. We gathered the titles, abstracts, and keywords collectively from studies published in two major CS education conferences and journals. We also extracted the strategic directions from their mission statements. By applying the Natural Language Processing (NLP) technique, known as Term Frequency-Inverse Document Frequency (tf-idf), we aimed to answer: 1) What are the most prevalent research foci represented in CSER through the publication keywords? 2) Are publication keywords in alignment with the CSER publications’ stated strategic directions? 3) Do selected keywords in CSER publications adequately represent the research topics presented via publication abstracts?

Our preliminary results suggest that, as expected, the most prevalent research foci represented by the most commonly used keywords of CSER were on teaching and learning and broadening participation, which is in alignment with the corresponding strategic directions. Our analysis suggests that the CSER community is putting efforts to formulate a diversified culture through teaching and learning for inclusivity. However, our results also suggest that there is a misalignment between keyword identification by authors and the topics presented in the abstracts. With our work, we hope to motivate scholars to carefully evaluate and select the keywords that will be used in the future for indexing publications to improve the research topic relevancy and publication visibility for broader impact.

Citation
Format

Zhu, J., & Zahedi, L., & Ross, M. S. (2021, July), Evaluating Publications' Keywords in Computer Science Education Research: A Bibliometric NLP Approach Paper presented at 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference. 10.18260/1-2--37104

TY - CPAPER
AB - In the rapidly growing field of computer science education research (CSER), the collection of publication keywords represent the knowledge landscape of the research domain. This helps stakeholders to contextualize existing research topics, identify the strategic direction for future works, and shape the culture of this growing community. For the research studies published in CSER, the use of proper keywords will help to improve publication visibility in research networks. Publication visibility is defined as the share of traffic a study receives based on its ranking from Search Engine Result Pages (SERPs), which can be gained through indexing relevant keywords to help search engines to find the related studies. It is important to acknowledge that analyzing keyword selections in the context provided by the publication abstract, enables us to determine the keywords’ relevance to the research topics. However, evaluating the selection of publication keywords is an important but lesser-addressed issue.

In this work, we utilized a bibliometric approach to scrutinize publication keywords in CSER publications from 2015 to 2020. We gathered the titles, abstracts, and keywords collectively from studies published in two major CS education conferences and journals. We also extracted the strategic directions from their mission statements. By applying the Natural Language Processing (NLP) technique, known as Term Frequency-Inverse Document Frequency (tf-idf), we aimed to answer: 1) What are the most prevalent research foci represented in CSER through the publication keywords? 2) Are publication keywords in alignment with the CSER publications’ stated strategic directions? 3) Do selected keywords in CSER publications adequately represent the research topics presented via publication abstracts?

Our preliminary results suggest that, as expected, the most prevalent research foci represented by the most commonly used keywords of CSER were on teaching and learning and broadening participation, which is in alignment with the corresponding strategic directions. Our analysis suggests that the CSER community is putting efforts to formulate a diversified culture through teaching and learning for inclusivity. However, our results also suggest that there is a misalignment between keyword identification by authors and the topics presented in the abstracts. With our work, we hope to motivate scholars to carefully evaluate and select the keywords that will be used in the future for indexing publications to improve the research topic relevancy and publication visibility for broader impact.

AU - Jia Zhu
AU - Leila Zahedi
AU - Monique S. Ross
CY - Virtual Conference
DA - 2021/07/26
PB - ASEE Conferences
TI - Evaluating Publications' Keywords in Computer Science Education Research: A Bibliometric NLP Approach
UR - https://peer.asee.org/37104
DO - 10.18260/1-2--37104
ER -