June 15, 2019
June 15, 2019
June 19, 2019
Individual Item or Dimension Analysis (IDA) using the Social Relations Model (SRM) has the power to reveal subtle changes in behavior as recorded in a behaviorally anchored scale. These changes cannot be seen in Across Dimension Analysis (ADA). Analysis of mean and standard deviation across all 5 CATME dimensions does not uncover differences at the individual item (or dimension) basis. The Social Relations Model decomposes variance into components. In our case, the variance is identified as rater, target, or relationship. Rater variance measures how individuals tend to rate their teammates. A large rater variance would indicate that students give their teammates ratings that are idiosyncratic – they don’t relate to the ratings given by others. Target variance measures how consistently individuals are rated by their teammates. A larger target effect would indicate that students tend to receive similar ratings from various raters. Relationship variance measures how ratings are affected by unique dyadic relationships after other variances have been removed. A larger relationship effect would indicate that the ratings are based more on distinct relationship pairings within the team – that is, that the various team members display different behaviors to each of their teammates and are rated based on those behaviors. The individual dimensions can be defined as follows: Having (H) relevant KSAs, Contributing (C) to the team’s work, Interacting (I) with team members, Keeping (K) the team on track, and Expecting (E) quality. In this analysis, we are examining three different teamwork studies that were based on evaluating average changes in multiple behaviorally anchored items and found additional insights into behavior changes through IDA. All data included in this analysis was obtained through the CATME peer evaluation system and the studies focused on methods of improving the quality of peer evaluations. In an initial study of a large Midwestern university, IDA SRM analysis revealed significant results in dimensions C, I, K, and H that were not seen using the ADA analysis, the average rating scores across items (or dimensions). All IDA analyses will compare IDA and ADA and groups who received teamwork training interventions and groups who did not receive any teamwork training.
Benner, M., & Ferguson, D. M., & Ohland, M. W., & Beigpourian, B. (2019, June), Analyzing Changes in the Individual Dimensions of a Behaviorally Anchored Rating Scale Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. https://peer.asee.org/32091
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2019 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015