June 26, 2011
June 26, 2011
June 29, 2011
Educational Research and Methods
22.1656.1 - 22.1656.15
Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering UndergraduatesThe use of verbal protocol (e.g. “think-aloud”) techniques in developing a survey instrument canbe critical to establishing an instrument’s cognitive validity. The advantages of utilizing a verbalcognitive validation protocol include providing the researcher with evidence that survey itemsare interpreted by participants in the same way the researcher intended, as well as providing anopportunity for direct participant feedback before the instrument is administered to a largesample.Think-aloud protocols have been used to accomplish different goals in a variety of fields, e.g.measuring constructs such as motivation (educational psychology), improving the usability ofconsumer products (human factors engineering), and identifying students’ internal thoughtprocesses during problem solving (engineering education). Even though think-alouds have beenused in many different domains, the general protocol is quite similar. Typically, a participantfrom the target demographic interacts with the “system” under investigation, a survey instrumentin our case. While interacting with the instrument, participants constantly verbalize everythingthink about. Meanwhile, researchers observe the participant, taking note of issues encounteredand prompting the participant with questions to determine what was confusing and what led toactions or thoughts that did not match their intentions when designing the instrument.While think-alouds have been used by engineering education researchers, the literature includesfew resources for researchers regarding the use of these protocols with respect to surveydevelopment and refinement. In this paper, we present a protocol based on elements of think-alouds conducted inside and outside the engineering education domain for use in establishingcognitive validity of a survey instrument.While the purpose of our particular instrument was to identify and quantify engineeringundergraduates’ access to and activation of social capital in making decisions about entering andpersisting in engineering, our protocol and results can serve as a model for other engineeringeducation researchers. Specifically, our instrument was designed to address the followingresearch questions: “How do social network indicators vary with gender, ethnicity, andgenerational status in college?” and “What forms of social capital do students employ in makingdecisions about selecting and persisting in undergraduate-level engineering?”.By following an iterative cognitive validation model, participants verbalized several issues ofconcern when interacting with our web-based survey. These issues ranged from minorgrammatical errors to serious cognitive mismatches which caused participants to interpret and/orrespond to items differently than we intended. Participants were asked for suggestions to correctthese issues, and changes were made to the survey based on this feedback. The survey was re-tested in two phases during additional think-aloud sessions with new participants to ensure therevisions successfully remedied the issues previously encountered. Finally, the refined surveywas pilot tested before being administered at seven institutions. This paper includes evidence andspecific examples of how the cognitive validation model resulted in a refined survey instrument, as well as recommendations for other engineering education researchers wishing to employsimilar techniques in designing and validating survey instruments.
Trenor, J. M., & Miller, M. K., & Gipson, K. G. (2011, June), Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering Undergraduates Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. https://peer.asee.org/18492
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2011 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015