Asee peer logo

Methods for Establishing Validity and Reliability of Observation Protocols

Download Paper |

Conference

2015 ASEE Annual Conference & Exposition

Location

Seattle, Washington

Publication Date

June 14, 2015

Start Date

June 14, 2015

End Date

June 17, 2015

ISBN

978-0-692-50180-1

ISSN

2153-5965

Conference Session

Qualitative Methodologies

Tagged Division

Educational Research and Methods

Page Count

10

Page Numbers

26.1149.1 - 26.1149.10

DOI

10.18260/p.24486

Permanent URL

https://peer.asee.org/24486

Download Count

570

Request a correction

Paper Authors

biography

Robert Matthew DeMonbrun University of Michigan

visit author page

Matt DeMonbrun is a Ph.D. student at the Center for the Study of Higher and Postsecondary Education (CSHPE) in the School of Education at the University of Michigan. His research interests include college student development theory and teaching practices and how they relate to student learning outcomes in engineering education.

visit author page

biography

Cynthia J. Finelli University of Michigan Orcid 16x16 orcid.org/0000-0001-9148-1492

visit author page

Dr. Cynthia Finelli, Director of the Center for Research on Learning and Teaching in Engineering and research associate professor of engineering education at University of Michigan (U-M), earned B.S.E.E., M.S.E.E., and Ph.D. degrees from U-M in 1988, 1989, and 1993, respectively. Prior to joining U-M in 2003, she was the Richard L. Terrell Professor of Excellence in Teaching, founding director of the Center for Excellence in Teaching and Learning, and associate professor of electrical engineering at Kettering University. In her current role, she coordinates faculty and TA professional development in the College of Engineering, conducts rigorous engineering education research, and promotes the growth of engineering education both locally at UM and nationally. Dr. Finelli's current research interests include evaluating methods to improve teaching, studying faculty motivation to change classroom practices, and exploring ethical decision-making in engineering students. She also has established a national presence in engineering education; she is a fellow in the American Society of Engineering Education, is an Associate Editor of the IEEE Transactions on Education, and past chair of the Educational Research and Methods Division of ASEE.

visit author page

biography

Prateek Shekhar University of Texas, Austin Orcid 16x16 orcid.org/0000-0001-6552-2887

visit author page

Prateek Shekhar is a PhD student in the Department of Mechanical Education at the University of Texas at Austin. His research is focused in understanding students’ and faculty’s reaction to adoption of active learning based teaching methods in engineering classrooms. He holds a M.S. in Electrical Engineering from University of Southern California and B.S. in Electronics and Communication Engineering from India.

visit author page

Download Paper |

Abstract

Methods for establishing validity and reliability of observation protocolsClassroom observations can be a useful tool in conducting research on a myriad ofinteractions and events that occur in an educational setting. To answer research questionsabout faculty/student interactions, faculty/student behaviors, instructor effectiveness, andteaching methods utilized by instructors, researchers in the social sciences and industrialmanagement fields typically use trained human observers as a primary source of datacollection. Engineering education researchers have similarly used classroom observationsto address these same questions, as well as to study relationship dynamics of students inteam design courses.Our research team has utilized classroom observations to examine students’ resistance tothe use of nontraditional teaching methods by engineering faculty. Specifically, we haveused observations to further understand the various ways in which students exhibitresistance to nontraditional teaching methods and the ways faculty respond to thischallenge. Since our observations are conducted by trained observers who are notparticipants in the classroom (i.e., neither students nor instructors), we have had tocontinually reflect on the precise detection, perception, recognition, and judgment ofcertain events to ensure our observations accurately capture what is occurring in theclassroom. This experience is much different than training observers to obtain inter-rateragreement, an approach used to ensure that observations are reliable across manydifferent observers. Instead, we have examined ways in which we can confirm that ourinterpretation of the events is a valid depiction of classroom behaviors.In this paper, we will discuss the essential steps we applied for confirming validobservations of classroom behaviors. We will begin by explaining the concepts ofreliability and validity as they pertain to classroom observations and presenting previoustechniques from the social science literature. Next, we will discuss our project on studentresistance and describe the steps we took to ensure reliability and validity in ourobservations. Although our classroom observation instrument was developed from otherpublished observation protocols, we used several approaches (e.g., videotaping classroombehaviors and conducting student focus groups) to ensure the accuracy of both ourinstrument and our observations. Finally, we will offer recommendations for otherengineering education researchers who wish to use observations as a way to assessclassroom behaviors.

DeMonbrun, R. M., & Finelli, C. J., & Shekhar, P. (2015, June), Methods for Establishing Validity and Reliability of Observation Protocols Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24486

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015