Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering Undergraduates

Julie Martin Trenor; Matthew K. Miller; Kyle G. Gipson

Download Paper | Permalink

Conference: 2011 ASEE Annual Conference & Exposition
Location: Vancouver, BC
Publication Date: June 26, 2011
Start Date: June 26, 2011
End Date: June 29, 2011
ISSN: 2153-5965
Conference Session: Assessment Instruments
Tagged Division: Educational Research and Methods
Page Count: 15
Page Numbers: 22.1656.1 - 22.1656.15
DOI: 10.18260/1-2--18492
Permanent URL: https://peer.asee.org/18492
Download Count: 805

Paper Authors

biography

Julie Martin Trenor Clemson University

visit author page

Julie Martin Trenor, Ph.D. is an Assistant Professor of Engineering and Science Education with a joint appointment in the School of Materials Science and Engineering. Her research interests focus on social factors affecting the recruitment, retention, and career development of under-represented students in engineering. Dr. Trenor is a recent NSF CAREER award winner for her research entitled, “Influence of Social Capital on Under-Represented Engineering Students Academic and Career Decisions.”

visit author page

biography

Matthew K. Miller Clemson University

visit author page

Matthew K. Miller, M.S., is a doctoral research assistant in the Engineering & Science Education department at Clemson University. He received his undergraduate and Master's degrees in Industrial Engineering from Clemson University.

visit author page

biography

Kyle G. Gipson Clemson University

visit author page

Kyle Gipson is a doctoral candidate in the School of Materials Science and Engineering at Clemson University. His research is focused on polymer nanocomposites for optical materials and increasing diversity within STEM disciplines through the creation of inclusive learning environments. Mr. Gipson received a B.S. in Physics from Wofford College and he also received his Master’s of Science in Textile Technology from the Institute of Textile Technology.

visit author page

Download Paper | Permalink

Abstract

Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering UndergraduatesThe use of verbal protocol (e.g. “think-aloud”) techniques in developing a survey instrument canbe critical to establishing an instrument’s cognitive validity. The advantages of utilizing a verbalcognitive validation protocol include providing the researcher with evidence that survey itemsare interpreted by participants in the same way the researcher intended, as well as providing anopportunity for direct participant feedback before the instrument is administered to a largesample.Think-aloud protocols have been used to accomplish different goals in a variety of fields, e.g.measuring constructs such as motivation (educational psychology), improving the usability ofconsumer products (human factors engineering), and identifying students’ internal thoughtprocesses during problem solving (engineering education). Even though think-alouds have beenused in many different domains, the general protocol is quite similar. Typically, a participantfrom the target demographic interacts with the “system” under investigation, a survey instrumentin our case. While interacting with the instrument, participants constantly verbalize everythingthink about. Meanwhile, researchers observe the participant, taking note of issues encounteredand prompting the participant with questions to determine what was confusing and what led toactions or thoughts that did not match their intentions when designing the instrument.While think-alouds have been used by engineering education researchers, the literature includesfew resources for researchers regarding the use of these protocols with respect to surveydevelopment and refinement. In this paper, we present a protocol based on elements of think-alouds conducted inside and outside the engineering education domain for use in establishingcognitive validity of a survey instrument.While the purpose of our particular instrument was to identify and quantify engineeringundergraduates’ access to and activation of social capital in making decisions about entering andpersisting in engineering, our protocol and results can serve as a model for other engineeringeducation researchers. Specifically, our instrument was designed to address the followingresearch questions: “How do social network indicators vary with gender, ethnicity, andgenerational status in college?” and “What forms of social capital do students employ in makingdecisions about selecting and persisting in undergraduate-level engineering?”.By following an iterative cognitive validation model, participants verbalized several issues ofconcern when interacting with our web-based survey. These issues ranged from minorgrammatical errors to serious cognitive mismatches which caused participants to interpret and/orrespond to items differently than we intended. Participants were asked for suggestions to correctthese issues, and changes were made to the survey based on this feedback. The survey was re-tested in two phases during additional think-aloud sessions with new participants to ensure therevisions successfully remedied the issues previously encountered. Finally, the refined surveywas pilot tested before being administered at seven institutions. This paper includes evidence andspecific examples of how the cognitive validation model resulted in a refined survey instrument, as well as recommendations for other engineering education researchers wishing to employsimilar techniques in designing and validating survey instruments.

Citation
Format

Trenor, J. M., & Miller, M. K., & Gipson, K. G. (2011, June), Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering Undergraduates Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. 10.18260/1-2--18492

TY  - CPAPER
AB  -             Utilization of a Think-Aloud Protocol to Cognitively Validate a             Survey Instrument Identifying Social Capital Resources of                             Engineering UndergraduatesThe use of verbal protocol (e.g. “think-aloud”) techniques in developing a survey instrument canbe critical to establishing an instrument’s cognitive validity. The advantages of utilizing a verbalcognitive validation protocol include providing the researcher with evidence that survey itemsare interpreted by participants in the same way the researcher intended, as well as providing anopportunity for direct participant feedback before the instrument is administered to a largesample.Think-aloud protocols have been used to accomplish different goals in a variety of fields, e.g.measuring constructs such as motivation (educational psychology), improving the usability ofconsumer products (human factors engineering), and identifying students’ internal thoughtprocesses during problem solving (engineering education). Even though think-alouds have beenused in many different domains, the general protocol is quite similar. Typically, a participantfrom the target demographic interacts with the “system” under investigation, a survey instrumentin our case. While interacting with the instrument, participants constantly verbalize everythingthink about. Meanwhile, researchers observe the participant, taking note of issues encounteredand prompting the participant with questions to determine what was confusing and what led toactions or thoughts that did not match their intentions when designing the instrument.While think-alouds have been used by engineering education researchers, the literature includesfew resources for researchers regarding the use of these protocols with respect to surveydevelopment and refinement. In this paper, we present a protocol based on elements of think-alouds conducted inside and outside the engineering education domain for use in establishingcognitive validity of a survey instrument.While the purpose of our particular instrument was to identify and quantify engineeringundergraduates’ access to and activation of social capital in making decisions about entering andpersisting in engineering, our protocol and results can serve as a model for other engineeringeducation researchers. Specifically, our instrument was designed to address the followingresearch questions: “How do social network indicators vary with gender, ethnicity, andgenerational status in college?” and “What forms of social capital do students employ in makingdecisions about selecting and persisting in undergraduate-level engineering?”.By following an iterative cognitive validation model, participants verbalized several issues ofconcern when interacting with our web-based survey. These issues ranged from minorgrammatical errors to serious cognitive mismatches which caused participants to interpret and/orrespond to items differently than we intended. Participants were asked for suggestions to correctthese issues, and changes were made to the survey based on this feedback. The survey was re-tested in two phases during additional think-aloud sessions with new participants to ensure therevisions successfully remedied the issues previously encountered. Finally, the refined surveywas pilot tested before being administered at seven institutions. This paper includes evidence andspecific examples of how the cognitive validation model resulted in a refined survey instrument, as well as recommendations for other engineering education researchers wishing to employsimilar techniques in designing and validating survey instruments.
AU  - Julie Martin Trenor
AU  - Matthew K. Miller
AU  - Kyle G. Gipson
CY  - Vancouver, BC
DA  - 2011/06/26
PB  - ASEE Conferences
TI  - Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering Undergraduates 
UR  - https://peer.asee.org/18492
DO  - 10.18260/1-2--18492
ER  -