Utilizing Cluster Analysis of Close-Ended Survey Responses to Select Participants for Qualitative Data Collection

Katherine M. Ehlert; Courtney June Faber; Marian S. Kennedy; Lisa Benson

Download Paper | Permalink

Conference: 2017 ASEE Annual Conference & Exposition
Location: Columbus, Ohio
Publication Date: June 24, 2017
Start Date: June 24, 2017
End Date: June 28, 2017
Conference Session: Quantitative Research Methods
Tagged Division: Educational Research and Methods
Page Count: 25
DOI: 10.18260/1-2--29099
Permanent URL: https://peer.asee.org/29099
Download Count: 2917

Request a correction

Paper Authors

biography

Katherine M. Ehlert Clemson University

visit author page

Katherine M. Ehlert is a doctoral student in the Engineering and Science Education department in the College of Engineering, Computing, and Applied Sciences at Clemson University. She earned her BS in Mechanical Engineering from Case Western Reserve University and her MS in Mechanical Engineering focusing on Biomechanics from Cornell University. Prior to her enrollment at Clemson, Katherine worked as a Biomedical Engineering consultant in Philadelphia, PA. Her research interests include identity development through co and extra-curricular experiences for engineering students.

visit author page

biography

Courtney June Faber University of Tennessee

visit author page

Courtney is a Research Assistant Professor and Lecturer in the College of Engineering Honors Program at the University of Tennessee. She completed her Ph.D. in Engineering & Science Education at Clemson University. Prior to her Ph.D. work, she received her B.S. in Bioengineering at Clemson University and her M.S. in Biomedical Engineering at Cornell University. Courtney’s research interests include epistemic cognition in the context of problem solving, researcher identity, and mixed methods.

visit author page

biography

Marian S. Kennedy Clemson University

visit author page

Marian Kennedy is an Associate Professor within the Department of Materials Science & Engineering at Clemson University. Her research group focused on the mechanical and tribological characterization of thin films. She also contributes to the engineering education community through research related to undergraduate research programs and navigational capital needed for graduate school.

visit author page

biography

Lisa Benson Clemson University

visit author page

Lisa Benson is a Professor of Engineering and Science Education at Clemson University, with a joint appointment in Bioengineering. Her research focuses on the interactions between student motivation and their learning experiences. Her projects involve the study of student perceptions, beliefs and attitudes towards becoming engineers and scientists, and their problem solving processes. Other projects in the Benson group include effects of student-centered active learning, self-regulated learning, and incorporating engineering into secondary science and mathematics classrooms. Her education includes a B.S. in Bioengineering from the University of Vermont, and M.S. and Ph.D. in Bioengineering from Clemson University.

visit author page

Download Paper | Permalink

Abstract

The purpose of this research paper is to discuss the application of cluster analysis within engineering education research, the variations in cluster analysis methods, and the impact of this analysis on the process of selecting participants for interviews. Cluster analysis can be used to frame how research participants are selected for qualitative or mixed methods studies when saturation of viewpoints within the participant population is desired. Cluster analysis has been used successfully within many fields to group data based on similar attributes. Within engineering education, researchers have utilized clustering techniques to group participants that have similar traits, beliefs, ideas, backgrounds, demographics, and other measures.

As new methods are incorporated into engineering education research practices, it is important to understand their underlying assumptions, boundary conditions, and limitations. This is particularly important in cluster analysis since not all clustering techniques yield the same response from the same data set. This paper will describe commonly used techniques in data science, and discuss their suitableness for different data sets within the context of engineering education research. We will then describe in detail the three clustering techniques we have used for participant selection: k-means, Ward’s, and Complete-Link.

This work is situated within a larger, explanatory mixed methods project focused on understanding how undergraduates conceptualize their identities as researchers and their engineering epistemic beliefs. We will highlight the nature of the quantitative data, cluster analyses, and characteristics of the resulting clusters. Responses to anchored survey items were collected from participants from five institutions varying in size, type, and location within the United States. This data set includes 45 anchored items probing participants’ epistemic beliefs and need for cognitive closure. Average scores for six factors based on underlying theoretical constructs were calculated for each participant. Four of these factors had high reliability measures and were used for cluster analysis (Discomfort with Ambiguity, Closed-Mindedness, Certainty of Knowledge, and Sources of Knowledge). All three clustering techniques that were applied resulted in two distinct groups of participants. The number of participants in each cluster varied between techniques with k-means creating similarly sized clusters (n1 = 58, n2 = 51), Ward’s slightly less similarly sized (n1 = 63, n2 = 46), and Complete Link with least similar sizes (n1 = 83, n2 = 26). For all cluster solutions, the two clusters differed on the constructs of Closed-Mindedness and Certainty of Knowledge. Cluster solutions using k-means and Ward’s method showed differences between clusters on the Sources of Knowledge construct whereas Complete Link did not indicate this difference. Regardless of the technique, clusters did not differ on the Discomfort with Ambiguity construct with relatively high scores (average around 5 out of 7, where 7 represented “Strongly Agree”).

Results from these cluster analyses will be used to characterize participants for the qualitative phase of the project which will examine students’ perceptions of research and their epistemic beliefs within the context of research. Selecting participants based on cluster analyses will allow for theoretical sampling, which is appropriate for our grounded theory approach.

Citation
Format

Ehlert, K. M., & Faber, C. J., & Kennedy, M. S., & Benson, L. (2017, June), Utilizing Cluster Analysis of Close-Ended Survey Responses to Select Participants for Qualitative Data Collection Paper presented at 2017 ASEE Annual Conference & Exposition, Columbus, Ohio. 10.18260/1-2--29099

TY - CPAPER
AB - The purpose of this research paper is to discuss the application of cluster analysis within engineering education research, the variations in cluster analysis methods, and the impact of this analysis on the process of selecting participants for interviews. Cluster analysis can be used to frame how research participants are selected for qualitative or mixed methods studies when saturation of viewpoints within the participant population is desired. Cluster analysis has been used successfully within many fields to group data based on similar attributes. Within engineering education, researchers have utilized clustering techniques to group participants that have similar traits, beliefs, ideas, backgrounds, demographics, and other measures.

As new methods are incorporated into engineering education research practices, it is important to understand their underlying assumptions, boundary conditions, and limitations. This is particularly important in cluster analysis since not all clustering techniques yield the same response from the same data set. This paper will describe commonly used techniques in data science, and discuss their suitableness for different data sets within the context of engineering education research. We will then describe in detail the three clustering techniques we have used for participant selection: k-means, Ward’s, and Complete-Link.

This work is situated within a larger, explanatory mixed methods project focused on understanding how undergraduates conceptualize their identities as researchers and their engineering epistemic beliefs. We will highlight the nature of the quantitative data, cluster analyses, and characteristics of the resulting clusters. Responses to anchored survey items were collected from participants from five institutions varying in size, type, and location within the United States. This data set includes 45 anchored items probing participants’ epistemic beliefs and need for cognitive closure. Average scores for six factors based on underlying theoretical constructs were calculated for each participant. Four of these factors had high reliability measures and were used for cluster analysis (Discomfort with Ambiguity, Closed-Mindedness, Certainty of Knowledge, and Sources of Knowledge). All three clustering techniques that were applied resulted in two distinct groups of participants. The number of participants in each cluster varied between techniques with k-means creating similarly sized clusters (n1 = 58, n2 = 51), Ward’s slightly less similarly sized (n1 = 63, n2 = 46), and Complete Link with least similar sizes (n1 = 83, n2 = 26). For all cluster solutions, the two clusters differed on the constructs of Closed-Mindedness and Certainty of Knowledge. Cluster solutions using k-means and Ward’s method showed differences between clusters on the Sources of Knowledge construct whereas Complete Link did not indicate this difference. Regardless of the technique, clusters did not differ on the Discomfort with Ambiguity construct with relatively high scores (average around 5 out of 7, where 7 represented “Strongly Agree”).

Results from these cluster analyses will be used to characterize participants for the qualitative phase of the project which will examine students’ perceptions of research and their epistemic beliefs within the context of research. Selecting participants based on cluster analyses will allow for theoretical sampling, which is appropriate for our grounded theory approach.

AU - Katherine M. Ehlert
AU - Courtney June Faber
AU - Marian S. Kennedy
AU - Lisa Benson
CY - Columbus, Ohio
DA - 2017/06/24
PB - ASEE Conferences
TI - Utilizing Cluster Analysis of Close-Ended Survey Responses to Select Participants for Qualitative Data Collection
UR - https://peer.asee.org/29099
DO - 10.18260/1-2--29099
ER -