Asee peer logo

Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering Undergraduates

Download Paper |

Conference

2011 ASEE Annual Conference & Exposition

Location

Vancouver, BC

Publication Date

June 26, 2011

Start Date

June 26, 2011

End Date

June 29, 2011

ISSN

2153-5965

Conference Session

Assessment Instruments

Tagged Division

Educational Research and Methods

Page Count

15

Page Numbers

22.1656.1 - 22.1656.15

Permanent URL

https://peer.asee.org/18492

Download Count

28

Request a correction

Paper Authors

biography

Julie Martin Trenor Clemson University

visit author page

Julie Martin Trenor, Ph.D. is an Assistant Professor of Engineering and Science Education with a joint appointment in the School of Materials Science and Engineering. Her research interests focus on social factors affecting the recruitment, retention, and career development of under-represented students in engineering. Dr. Trenor is a recent NSF CAREER award winner for her research entitled, “Influence of Social Capital on Under-Represented Engineering Students Academic and Career Decisions.”

visit author page

biography

Matthew K. Miller Clemson University

visit author page

Matthew K. Miller, M.S., is a doctoral research assistant in the Engineering & Science Education department at Clemson University. He received his undergraduate and Master's degrees in Industrial Engineering from Clemson University.

visit author page

biography

Kyle G. Gipson Clemson University

visit author page

Kyle Gipson is a doctoral candidate in the School of Materials Science and Engineering at Clemson University. His research is focused on polymer nanocomposites for optical materials and increasing diversity within STEM disciplines through the creation of inclusive learning environments. Mr. Gipson received a B.S. in Physics from Wofford College and he also received his Master’s of Science in Textile Technology from the Institute of Textile Technology.

visit author page

Download Paper |

Abstract

  Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering UndergraduatesThe use of verbal protocol (e.g. “think-aloud”) techniques in developing a survey instrument canbe critical to establishing an instrument’s cognitive validity. The advantages of utilizing a verbalcognitive validation protocol include providing the researcher with evidence that survey itemsare interpreted by participants in the same way the researcher intended, as well as providing anopportunity for direct participant feedback before the instrument is administered to a largesample.Think-aloud protocols have been used to accomplish different goals in a variety of fields, e.g.measuring constructs such as motivation (educational psychology), improving the usability ofconsumer products (human factors engineering), and identifying students’ internal thoughtprocesses during problem solving (engineering education). Even though think-alouds have beenused in many different domains, the general protocol is quite similar. Typically, a participantfrom the target demographic interacts with the “system” under investigation, a survey instrumentin our case. While interacting with the instrument, participants constantly verbalize everythingthink about. Meanwhile, researchers observe the participant, taking note of issues encounteredand prompting the participant with questions to determine what was confusing and what led toactions or thoughts that did not match their intentions when designing the instrument.While think-alouds have been used by engineering education researchers, the literature includesfew resources for researchers regarding the use of these protocols with respect to surveydevelopment and refinement. In this paper, we present a protocol based on elements of think-alouds conducted inside and outside the engineering education domain for use in establishingcognitive validity of a survey instrument.While the purpose of our particular instrument was to identify and quantify engineeringundergraduates’ access to and activation of social capital in making decisions about entering andpersisting in engineering, our protocol and results can serve as a model for other engineeringeducation researchers. Specifically, our instrument was designed to address the followingresearch questions: “How do social network indicators vary with gender, ethnicity, andgenerational status in college?” and “What forms of social capital do students employ in makingdecisions about selecting and persisting in undergraduate-level engineering?”.By following an iterative cognitive validation model, participants verbalized several issues ofconcern when interacting with our web-based survey. These issues ranged from minorgrammatical errors to serious cognitive mismatches which caused participants to interpret and/orrespond to items differently than we intended. Participants were asked for suggestions to correctthese issues, and changes were made to the survey based on this feedback. The survey was re-tested in two phases during additional think-aloud sessions with new participants to ensure therevisions successfully remedied the issues previously encountered. Finally, the refined surveywas pilot tested before being administered at seven institutions. This paper includes evidence andspecific examples of how the cognitive validation model resulted in a refined survey instrument, as well as recommendations for other engineering education researchers wishing to employsimilar techniques in designing and validating survey instruments.

Trenor, J. M., & Miller, M. K., & Gipson, K. G. (2011, June), Utilization of a Think-Aloud Protocol to Cognitively Validate a Survey Instrument Identifying Social Capital Resources of Engineering Undergraduates Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. https://peer.asee.org/18492

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2011 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015