June 14, 2015
June 14, 2015
June 17, 2015
Educational Research and Methods
26.1149.1 - 26.1149.10
Methods for establishing validity and reliability of observation protocolsClassroom observations can be a useful tool in conducting research on a myriad ofinteractions and events that occur in an educational setting. To answer research questionsabout faculty/student interactions, faculty/student behaviors, instructor effectiveness, andteaching methods utilized by instructors, researchers in the social sciences and industrialmanagement fields typically use trained human observers as a primary source of datacollection. Engineering education researchers have similarly used classroom observationsto address these same questions, as well as to study relationship dynamics of students inteam design courses.Our research team has utilized classroom observations to examine students’ resistance tothe use of nontraditional teaching methods by engineering faculty. Specifically, we haveused observations to further understand the various ways in which students exhibitresistance to nontraditional teaching methods and the ways faculty respond to thischallenge. Since our observations are conducted by trained observers who are notparticipants in the classroom (i.e., neither students nor instructors), we have had tocontinually reflect on the precise detection, perception, recognition, and judgment ofcertain events to ensure our observations accurately capture what is occurring in theclassroom. This experience is much different than training observers to obtain inter-rateragreement, an approach used to ensure that observations are reliable across manydifferent observers. Instead, we have examined ways in which we can confirm that ourinterpretation of the events is a valid depiction of classroom behaviors.In this paper, we will discuss the essential steps we applied for confirming validobservations of classroom behaviors. We will begin by explaining the concepts ofreliability and validity as they pertain to classroom observations and presenting previoustechniques from the social science literature. Next, we will discuss our project on studentresistance and describe the steps we took to ensure reliability and validity in ourobservations. Although our classroom observation instrument was developed from otherpublished observation protocols, we used several approaches (e.g., videotaping classroombehaviors and conducting student focus groups) to ensure the accuracy of both ourinstrument and our observations. Finally, we will offer recommendations for otherengineering education researchers who wish to use observations as a way to assessclassroom behaviors.
DeMonbrun, R. M., & Finelli, C. J., & Shekhar, P. (2015, June), Methods for Establishing Validity and Reliability of Observation Protocols Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24486
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015