Asee peer logo

Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns

Download Paper |


2019 ASEE Annual Conference & Exposition


Tampa, Florida

Publication Date

June 15, 2019

Start Date

June 15, 2019

End Date

June 19, 2019

Conference Session

ERM Technical Session 1: Methods Refresh: Approaches to Data Analysis in Engineering Education Research

Tagged Division

Educational Research and Methods

Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Manoj Malviya Pennsylvania State University


Catherine G.P. Berdanier Pennsylvania State University Orcid 16x16

visit author page

Catherine G.P. Berdanier is an Assistant Professor in the Department of Mechanical Engineering at Pennsylvania State University. She earned her B.S. in Chemistry from The University of South Dakota, her M.S. in Aeronautical and Astronautical Engineering and Ph.D. in Engineering Education from Purdue University. Her research interests include graduate-level engineering education, including inter- and multidisciplinary graduate education, online engineering cognition and learning, and engineering communication.

visit author page


Natascha Trellinger Buswell University of California, Irvine Orcid 16x16

visit author page

Natascha Trellinger Buswell is an assistant teaching professor in the department of mechanical and aerospace engineering at the University of California, Irvine. She earned her B.S. in aerospace engineering from Syracuse University and her Ph.D. in engineering education from the School of Engineering Education at Purdue University. She is particularly interested in teaching conceptions and methods and graduate level engineering education.

visit author page

Download Paper |


Traditionally, interrater reliability (IRR) is determined for easily defined events, such as deciding within which category a piece of qualitative data falls. However, for time-resolved or time-dependent observational data and other nontraditional data, complications arise due to the complexity of the data being interpreted and analyzed. In this paper, we present two promising new methods for calculating IRR based on visual representations of analyzed time-resolved data. We compare the IRR calculated using these two visual methods with five of the most common statistical measures for calculating IRR, finding excellent agreement between our new methods and existing statistical formulae. This methods development is exemplified using data for our ongoing research, in which we are working to analyze time-resolved engineering writing data recorded through screen capture technology. The process of developing methods of interrater reliability for our context can also be applied to other researchers who seek to analyze non-traditional data, such as those collected during eye-tracking, screen capture, or observational studies.

Malviya, M., & Berdanier, C. G., & Buswell, N. T. (2019, June), Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. 10.18260/1-2--33541

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2019 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015