Malviya, M., & Berdanier, C. G., & Buswell, N. T. (2019, June), Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns  Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. 10.18260/1-2--33541

\bibitem{asee_peer_33541}
  Malviya, M., \& Berdanier, C. G., \& Buswell, N. T. (2019, June), \emph{Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns}
  Paper presented at 2019 ASEE Annual Conference \& Exposition , Tampa, Florida.
  10.18260/1-2--33541

Manoj Malviya, Catherine G.P. Berdanier, and Natascha Trellinger Buswell.  "Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns".  2019 ASEE Annual Conference & Exposition , Tampa, Florida, 2019, June.  ASEE Conferences, 2019.  https://peer.asee.org/33541 Internet.  15 Dec, 2024

\bibitem{asee_peer_33541}
  Manoj Malviya, Catherine G.P. Berdanier, and Natascha Trellinger Buswell.
  "Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns".
  \emph{2019 ASEE Annual Conference \& Exposition , Tampa, Florida, 2019, June}.
  ASEE Conferences, 2019.
  https://peer.asee.org/33541 Internet.  15 Dec, 2024

@INPROCEEDINGS{asee_peer_33541
author = "Manoj Malviya, Catherine G.P. Berdanier, and Natascha Trellinger Buswell"
title = "Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns"
booktitle = "2019 ASEE Annual Conference \& Exposition "
year = "2019"
month = "June"
address = "Tampa, Florida"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/33541}
number = {10.18260/1-2--33541}
}

TY  - CPAPER
AB  - Traditionally, interrater reliability (IRR) is determined for easily defined events, such as deciding within which category a piece of qualitative data falls. However, for time-resolved or time-dependent observational data and other nontraditional data, complications arise due to the complexity of the data being interpreted and analyzed.  In this paper, we present two promising new methods for calculating IRR based on visual representations of analyzed time-resolved data. We compare the IRR calculated using these two visual methods with five of the most common statistical measures for calculating IRR, finding excellent agreement between our new methods and existing statistical formulae.  This methods development is exemplified using data for our ongoing research, in which we are working to analyze time-resolved engineering writing data recorded through screen capture technology. The process of developing methods of interrater reliability for our context can also be applied to other researchers who seek to analyze non-traditional data, such as those collected during eye-tracking, screen capture, or observational studies.  
AU  - Manoj Malviya
AU  - Catherine G.P. Berdanier
AU  - Natascha Trellinger Buswell
CY  - Tampa, Florida
DA  - 2019/06/15
PB  - ASEE Conferences
TI  - Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns
UR  - https://peer.asee.org/33541
DO  - 10.18260/1-2--33541
ER  -