June 15, 2014
June 15, 2014
June 18, 2014
Educational Research and Methods
24.1355.1 - 24.1355.16
Utilizing a Think-Aloud Protocol to Assess the Usability of a Test for Ethical Sensitivity in ConstructionERM Paper Type: OtherProfessional societies such as the American Society of Civil Engineers (ASCE) and theConstruction Management Association of America have codes of ethics to guide constructionpractitioners on professional practice. Additionally, ABET accredited civil and constructionengineering programs are expected to provide professional ethics instruction to its students tomeet the standards established by the profession. However, no assessment tool measuring astudent’s ethical sensitivity to professional issues in construction exists to determine whether thisinstruction improves student recognition of ethical issues in construction. The purpose of thispaper is to outline the use of a think aloud protocol as a part of the instrument development of atest for ethical sensitivity in construction (TESC). The paper details the process taken to developand administer the think aloud protocol, the results of the think aloud protocol, and how theresults assisted refinement of the TESC.The TESC is a structured qualitative assessment instrument that requires student participants toread, recognize, and list ethical issues embedded in vignettes. Vignettes are helpful inunderstanding a person's attitudes, perceptions, and beliefs. The “think-aloud” protocol allowedfor usability evaluation of the TESC by requiring participants to verbalize their thoughts,processes, and reasoning as they complete the items of the instrument. Prior research hasassociated the think aloud administration process with cognitive validation however, it alsoallows researchers to evaluate the affective processing of students when completing the test.Different from other uses of the think aloud protocol as a tool of verbal cognitive validation, thethink aloud protocol is also used as a means to understanding both the cognitive response andaffective responses of student participants of the TESC, observing their moral imagination,cognitive recognition and affective processing.Results from administering the think aloud protocol assisted the TESC’s clarity, administration,and approach to scoring participants’ responses. Problems of the original TESC included itssimplicity to complete and ease of understanding vignettes. The think aloud providedunderstanding of administration parameters of the test, which included time ofadministration/completion of the TESC, and strain required of participants to complete theTESC. The think aloud added qualitative understanding of responses and assisted the authors indeveloping an argument for construct validity. Additionally, the think aloud protocol providedadditional means of items analysis. Lastly, which is considerably important to the researchers,the think aloud assisted in identifying what qualitative responses generated by participants‘actually’ meant, adding to refinement of the scoring manual for the TESC. The authorsanticipate that results will assist researchers in understanding the benefits of using think aloudprotocols for refining an instrument, while adding validity arguments.
Sands, K. S., & Simmons, D. R. (2014, June), Utilizing Think-Aloud Protocols to Assess the Usability of a Test for Ethical Sensitivity in Construction Paper presented at 2014 ASEE Annual Conference & Exposition, Indianapolis, Indiana. 10.18260/1-2--23288
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2014 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015