Methods for Establishing Validity and Reliability of Observation Protocols

Robert Matthew DeMonbrun; Cynthia J. Finelli; Prateek Shekhar

Download Paper | Permalink

Conference: 2015 ASEE Annual Conference & Exposition
Location: Seattle, Washington
Publication Date: June 14, 2015
Start Date: June 14, 2015
End Date: June 17, 2015
ISBN: 978-0-692-50180-1
ISSN: 2153-5965
Conference Session: Qualitative Methodologies
Tagged Division: Educational Research and Methods
Page Count: 10
Page Numbers: 26.1149.1 - 26.1149.10
DOI: 10.18260/p.24486
Permanent URL: https://peer.asee.org/24486
Download Count: 3655

Paper Authors

biography

Robert Matthew DeMonbrun University of Michigan

visit author page

Matt DeMonbrun is a Ph.D. student at the Center for the Study of Higher and Postsecondary Education (CSHPE) in the School of Education at the University of Michigan. His research interests include college student development theory and teaching practices and how they relate to student learning outcomes in engineering education.

visit author page

biography

Cynthia J. Finelli University of Michigan orcid.org/0000-0001-9148-1492

visit author page

Dr. Cynthia Finelli, Director of the Center for Research on Learning and Teaching in Engineering and research associate professor of engineering education at University of Michigan (U-M), earned B.S.E.E., M.S.E.E., and Ph.D. degrees from U-M in 1988, 1989, and 1993, respectively. Prior to joining U-M in 2003, she was the Richard L. Terrell Professor of Excellence in Teaching, founding director of the Center for Excellence in Teaching and Learning, and associate professor of electrical engineering at Kettering University. In her current role, she coordinates faculty and TA professional development in the College of Engineering, conducts rigorous engineering education research, and promotes the growth of engineering education both locally at UM and nationally. Dr. Finelli's current research interests include evaluating methods to improve teaching, studying faculty motivation to change classroom practices, and exploring ethical decision-making in engineering students. She also has established a national presence in engineering education; she is a fellow in the American Society of Engineering Education, is an Associate Editor of the IEEE Transactions on Education, and past chair of the Educational Research and Methods Division of ASEE.

visit author page

biography

Prateek Shekhar University of Texas, Austin orcid.org/0000-0001-6552-2887

visit author page

Prateek Shekhar is a PhD student in the Department of Mechanical Education at the University of Texas at Austin. His research is focused in understanding students’ and faculty’s reaction to adoption of active learning based teaching methods in engineering classrooms. He holds a M.S. in Electrical Engineering from University of Southern California and B.S. in Electronics and Communication Engineering from India.

visit author page

Download Paper | Permalink

Abstract

Methods for establishing validity and reliability of observation protocolsClassroom observations can be a useful tool in conducting research on a myriad ofinteractions and events that occur in an educational setting. To answer research questionsabout faculty/student interactions, faculty/student behaviors, instructor effectiveness, andteaching methods utilized by instructors, researchers in the social sciences and industrialmanagement fields typically use trained human observers as a primary source of datacollection. Engineering education researchers have similarly used classroom observationsto address these same questions, as well as to study relationship dynamics of students inteam design courses.Our research team has utilized classroom observations to examine students’ resistance tothe use of nontraditional teaching methods by engineering faculty. Specifically, we haveused observations to further understand the various ways in which students exhibitresistance to nontraditional teaching methods and the ways faculty respond to thischallenge. Since our observations are conducted by trained observers who are notparticipants in the classroom (i.e., neither students nor instructors), we have had tocontinually reflect on the precise detection, perception, recognition, and judgment ofcertain events to ensure our observations accurately capture what is occurring in theclassroom. This experience is much different than training observers to obtain inter-rateragreement, an approach used to ensure that observations are reliable across manydifferent observers. Instead, we have examined ways in which we can confirm that ourinterpretation of the events is a valid depiction of classroom behaviors.In this paper, we will discuss the essential steps we applied for confirming validobservations of classroom behaviors. We will begin by explaining the concepts ofreliability and validity as they pertain to classroom observations and presenting previoustechniques from the social science literature. Next, we will discuss our project on studentresistance and describe the steps we took to ensure reliability and validity in ourobservations. Although our classroom observation instrument was developed from otherpublished observation protocols, we used several approaches (e.g., videotaping classroombehaviors and conducting student focus groups) to ensure the accuracy of both ourinstrument and our observations. Finally, we will offer recommendations for otherengineering education researchers who wish to use observations as a way to assessclassroom behaviors.

Citation
Format

DeMonbrun, R. M., & Finelli, C. J., & Shekhar, P. (2015, June), Methods for Establishing Validity and Reliability of Observation Protocols Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24486

TY  - CPAPER
AB  - Methods for establishing validity and reliability of observation protocolsClassroom observations can be a useful tool in conducting research on a myriad ofinteractions and events that occur in an educational setting. To answer research questionsabout faculty/student interactions, faculty/student behaviors, instructor effectiveness, andteaching methods utilized by instructors, researchers in the social sciences and industrialmanagement fields typically use trained human observers as a primary source of datacollection. Engineering education researchers have similarly used classroom observationsto address these same questions, as well as to study relationship dynamics of students inteam design courses.Our research team has utilized classroom observations to examine students’ resistance tothe use of nontraditional teaching methods by engineering faculty. Specifically, we haveused observations to further understand the various ways in which students exhibitresistance to nontraditional teaching methods and the ways faculty respond to thischallenge. Since our observations are conducted by trained observers who are notparticipants in the classroom (i.e., neither students nor instructors), we have had tocontinually reflect on the precise detection, perception, recognition, and judgment ofcertain events to ensure our observations accurately capture what is occurring in theclassroom. This experience is much different than training observers to obtain inter-rateragreement, an approach used to ensure that observations are reliable across manydifferent observers. Instead, we have examined ways in which we can confirm that ourinterpretation of the events is a valid depiction of classroom behaviors.In this paper, we will discuss the essential steps we applied for confirming validobservations of classroom behaviors. We will begin by explaining the concepts ofreliability and validity as they pertain to classroom observations and presenting previoustechniques from the social science literature. Next, we will discuss our project on studentresistance and describe the steps we took to ensure reliability and validity in ourobservations. Although our classroom observation instrument was developed from otherpublished observation protocols, we used several approaches (e.g., videotaping classroombehaviors and conducting student focus groups) to ensure the accuracy of both ourinstrument and our observations. Finally, we will offer recommendations for otherengineering education researchers who wish to use observations as a way to assessclassroom behaviors.
AU  - Robert Matthew DeMonbrun
AU  - Cynthia J. Finelli
AU  - Prateek Shekhar
CY  - Seattle, Washington
DA  - 2015/06/14
PB  - ASEE Conferences
TI  - Methods for Establishing Validity and Reliability of Observation Protocols
UR  - https://peer.asee.org/24486
DO  - 10.18260/p.24486
ER  -