Designing a Survey for Engineering Undergraduates Using Free Listing - An Anthropological Structured Technique

Chrystal A. S. Smith; Hesborn Wao; Julie P Martin; George T. MacDonald; Reginald S Lee; Gladis Kersaint

Download Paper | Permalink

Conference: 2015 ASEE Annual Conference & Exposition
Location: Seattle, Washington
Publication Date: June 14, 2015
Start Date: June 14, 2015
End Date: June 17, 2015
ISBN: 978-0-692-50180-1
ISSN: 2153-5965
Conference Session: Survey and Assessment Development
Tagged Division: Educational Research and Methods
Tagged Topic: Diversity
Page Count: 11
Page Numbers: 26.476.1 - 26.476.11
DOI: 10.18260/p.23814
Permanent URL: https://peer.asee.org/23814
Download Count: 558

Paper Authors

biography

Chrystal A. S. Smith University of South Florida

visit author page

Chrystal A. S. Smith is a Research Assistant Professor in the Department of Anthropology at the University of South Florida, Tampa. She is the Co-Principal Investigator for the National Science Foundation (NSF) funded study, “The Effects of Social Capital and Cultural Models on the Retention and Degree Attainment of Women and Minority Engineering Undergraduates.” Her research uses anthropological and sociological theories and methodologies to investigate the implicit factors that contribute to the under representation of women and minorities in STEM education. She received her Ph.D. in Anthropology from the University of South Florida, her M.A.A. in Applied Anthropology from the University of Maryland, College Park, her M.P.H. in Epidemiology from the University of South Florida, and her B.A. in Anthropology from Howard University.

visit author page

biography

Hesborn Wao University of South Florida

visit author page

Hesborn Wao is a Research Assistant Professor in the Department of Internal Medicine at the University of South Florida, Tampa. He is the Co-Principal Investigator for the National Science Foundation (NSF) funded study, “The Effects of Social Capital and Cultural Models on the Retention and Degree Attainment of Women and Minority Engineering Undergraduates.” Hesborn’s research uses mixed methods approaches to investigate the implicit factors associated with the under representation of women and minorities in STEM education. He strives to improve K-20 STEM learning experiences and degree attainment. He received his Ph.D. in Measurement & Evaluation and M.Ed. in Curriculum & Instruction both from the University of South Florida, and his B. Ed in Mathematics from the University of Nairobi, Kenya.

visit author page

biography

Julie P Martin Clemson University

visit author page

Julie P. Martin is an assistant professor of Engineering and Science Education at Clemson University. Her research interests focus on social factors affecting the recruitment, retention, and career development of underrepresented students in engineering. Dr. Martin is a 2009 NSF CAREER awardee for her research entitled, “Influence of Social Capital on Under-Represented Engineering Students Academic and Career Decisions.” She held an American Association for the Advancement of Science (AAAS) Science and Technology Policy Fellowship in 2013-2014, with a placement at the National Science Foundation.

visit author page

biography

George T. MacDonald University of South Florida

visit author page

Dr. George MacDonald is the interim Director for the Center for Research, Evaluation, Assessment, and Measurement (CREAM) in the College of Education at the University of South Florida(USF). He is the Co-Principal Investigator for the National Science Foundation (NSF) funded study, “The Effects of Social Capital and Cultural Models on the Retention and Degree Attainment of Women and Minority Engineering Undergraduates.” George holds a Ph.D. in Educational Measurement and Evaluation. He is responsible for a number of program evaluations and conducts the Assisted Living Facilities certification exam for the Department of Elder Affairs. His research agenda is focused on cognitive diagnostic assessment particularly as it applies to mathematics education. George received his Ph.D. from USF, an M. Div. from The Atlantic School of Theology, a B.A. in psychology from St. Leo University, and a B.A. in philosophy from Mount Allison University.

visit author page

biography

Reginald S Lee University of South Florida

visit author page

Reginald S. Lee is a Senior Social-Behavioral Researcher at the David C. Anchin Center, College of Education, University of South Florida. His research interests include measurement, equity and access issues in public education with an emphasis on high school and college course taking. He is a Co-Principal Investigator on the NSF-funded study, The Effects of Social Capital and Cultural Models on the Retention and Degree Attainment of Women and Minority Engineering Undergraduates.

visit author page

biography

Gladis Kersaint University of South Florida

visit author page

Gladis Kersaint, Ph.D., is a Professor of Mathematics Education and Associate Dean at the University of South Florida’s College of Education. She is the principal investigator of several grants including the NSF- funded study, “The Effects of Social Capital and Cultural Models on the Retention and Degree Attainment of Women and Minority Engineering Undergraduates.” Her areas of professional interests include factors that influence STEM education, mathematics teaching, and learning of at-risk students, and use of technology for learning and teaching mathematics. She received her doctorate in mathematics education from Illinois State University and her Masters degree in Education and bachelors degree mathematics from the University of Miami.

visit author page

Download Paper | Permalink

Abstract

Free Listing: Using an Anthropological Structured Technique to Identify Social Capital and Cultural Model Items for Engineering Undergraduate SurveysThis paper describes free listing (FL), a cognitive anthropological structured technique that canbe used to improve the validity of survey instruments and the design of questions on interviewprotocols in engineering education research. Anthropologists use FL to systematically collectdata about specific cultural models. Cultural models are internalized cognitive schemas thatindividuals within a culture share to varying degrees and draw upon to form and organize theirbeliefs, meanings, and practices. In our National Science Foundation funded study, we used FLto understand the cultural model of “success” in undergraduate engineering programs. Our studyasks, “what are the effects of social capital and cultural models of engineering success on theretention and degree attainment of women and minorities in engineering?” In this paper, we willpresent our approach to using FL to design items for our survey instruments that measure socialcapital and cultural models of success among engineering undergraduates as the first step inanswering our research question.In FL, participants are asked a series of questions that represent the major conceptual areas in acultural model as identified by the researcher from previous studies and the literature. For eachquestion, participants are asked to list as many responses as possible. When the participantpauses, they are prompted for additional responses. FL assumes that individuals 1) withextensive knowledge give more responses than those with less knowledge, 2) list most familiarand meaningful responses first, and 3) give responses that reflect their local cultural knowledge.Frequency and rank of each response are reflected in the salience, Smith’s S, a number calculatedusing an anthropological qualitative software program, in our case, ANTHROPAC 4.98. Usingsalience, researchers decide a cutoff line to determine which responses should be examinedfurther and included in the survey or interview protocol.We also discuss the advantages and limitations of the FL technique. For example, the primaryvalue of FL is that participants, especially if they are from an understudied population, oftenidentify beliefs or attitudes about the cultural model that were previously unknown or interpreteddifferently in the literature. By including these responses in the survey or interview protocol, theresearcher can determine if these beliefs or attitudes are shared throughout the study population.One limitation to FL is that there is no definitive method to identify the appropriate sample size.A larger the sample size will likely lead to a greater variety of responses, lessening the likelihoodof a high salience. Hence, is that is important for the researcher to narrowly specify the culturalmodel or cognitive area that they are exploring so that participants can easily mentally unpacktheir knowledge.The rich data obtained from FL can improve the design and validity of survey instruments andinterview protocols. Specific examples of how we used FL in our research study are described indetail, along with implications and recommendations for adoption of the technique to otherengineering education research.

Citation
Format

Smith, C. A. S., & Wao, H., & Martin, J. P., & MacDonald, G. T., & Lee, R. S., & Kersaint, G. (2015, June), Designing a Survey for Engineering Undergraduates Using Free Listing - An Anthropological Structured Technique Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.23814

TY  - CPAPER
AB  -          Free Listing: Using an Anthropological Structured Technique to                 Identify Social Capital and Cultural Model Items                     for Engineering Undergraduate SurveysThis paper describes free listing (FL), a cognitive anthropological structured technique that canbe used to improve the validity of survey instruments and the design of questions on interviewprotocols in engineering education research. Anthropologists use FL to systematically collectdata about specific cultural models. Cultural models are internalized cognitive schemas thatindividuals within a culture share to varying degrees and draw upon to form and organize theirbeliefs, meanings, and practices. In our National Science Foundation funded study, we used FLto understand the cultural model of “success” in undergraduate engineering programs. Our studyasks, “what are the effects of social capital and cultural models of engineering success on theretention and degree attainment of women and minorities in engineering?” In this paper, we willpresent our approach to using FL to design items for our survey instruments that measure socialcapital and cultural models of success among engineering undergraduates as the first step inanswering our research question.In FL, participants are asked a series of questions that represent the major conceptual areas in acultural model as identified by the researcher from previous studies and the literature. For eachquestion, participants are asked to list as many responses as possible. When the participantpauses, they are prompted for additional responses. FL assumes that individuals 1) withextensive knowledge give more responses than those with less knowledge, 2) list most familiarand meaningful responses first, and 3) give responses that reflect their local cultural knowledge.Frequency and rank of each response are reflected in the salience, Smith’s S, a number calculatedusing an anthropological qualitative software program, in our case, ANTHROPAC 4.98. Usingsalience, researchers decide a cutoff line to determine which responses should be examinedfurther and included in the survey or interview protocol.We also discuss the advantages and limitations of the FL technique. For example, the primaryvalue of FL is that participants, especially if they are from an understudied population, oftenidentify beliefs or attitudes about the cultural model that were previously unknown or interpreteddifferently in the literature. By including these responses in the survey or interview protocol, theresearcher can determine if these beliefs or attitudes are shared throughout the study population.One limitation to FL is that there is no definitive method to identify the appropriate sample size.A larger the sample size will likely lead to a greater variety of responses, lessening the likelihoodof a high salience. Hence, is that is important for the researcher to narrowly specify the culturalmodel or cognitive area that they are exploring so that participants can easily mentally unpacktheir knowledge.The rich data obtained from FL can improve the design and validity of survey instruments andinterview protocols. Specific examples of how we used FL in our research study are described indetail, along with implications and recommendations for adoption of the technique to otherengineering education research.
AU  - Chrystal A. S. Smith
AU  - Hesborn Wao
AU  - Julie P Martin
AU  - George T. MacDonald
AU  - Reginald S Lee
AU  - Gladis Kersaint
CY  - Seattle, Washington
DA  - 2015/06/14
PB  - ASEE Conferences
TI  - Designing a Survey for Engineering Undergraduates Using Free Listing - An Anthropological Structured Technique
UR  - https://peer.asee.org/23814
DO  - 10.18260/p.23814
ER  -