Asee peer logo

Analyzing Changes in the Individual Dimensions of a Behaviorally Anchored Rating Scale

Download Paper |

Conference

2019 ASEE Annual Conference & Exposition

Location

Tampa, Florida

Publication Date

June 15, 2019

Start Date

June 15, 2019

End Date

June 19, 2019

Conference Session

Mathematics Division Technical Session 3

Tagged Division

Mathematics

Page Count

13

Permanent URL

https://peer.asee.org/32091

Download Count

5

Request a correction

Paper Authors

author page

Maizey Benner Purdue University

biography

Daniel M. Ferguson Purdue University

visit author page

Daniel M. Ferguson is CATME Managing Director and the recipient of several NSF awards for research in engineering education and a research associate at Purdue University. Prior to coming to Purdue he was Assistant Professor of Entrepreneurship at Ohio Northern University. Before assuming that position he was Associate Director of the Inter-Professional Studies Program [IPRO] and Senior Lecturer at Illinois Institute of Technology and involved in research in service learning, assessment processes and interventions aimed at improving learning objective attainment. Prior to his University assignments he was the Founder and CEO of The EDI Group, Ltd. and The EDI Group Canada, Ltd, independent professional services companies specializing in B2B electronic commerce and electronic data interchange. The EDI Group companies conducted syndicated market research, offered educational seminars and conferences and published The Journal of Electronic Commerce. He was also a Vice President at the First National Bank of Chicago [now J.P. Morgan Chase], where he founded and managed the bank’s market leading professional Cash Management Consulting Group, initiated the bank’s non-credit service product management organization and profit center profitability programs and was instrumental in the breakthrough EDI/EFT payment system implemented by General Motors. Dr. Ferguson is a graduate of Notre Dame, Stanford and Purdue Universities, a special edition editor of the Journal of Engineering Entrepreneurship and a member of Tau Beta Pi.

visit author page

biography

Matthew W. Ohland Purdue University Orcid 16x16 orcid.org/0000-0003-4052-1452

visit author page

Matthew W. Ohland is Professor of Engineering Education at Purdue University. He has degrees from Swarthmore College, Rensselaer Polytechnic Institute, and the University of Florida. His research on the longitudinal study of engineering students, team assignment, peer evaluation, and active and collaborative teaching methods has been supported by the National Science Foundation and the Sloan Foundation and his team received Best Paper awards from the Journal of Engineering Education in 2008 and 2011 and from the IEEE Transactions on Education in 2011 and 2015. Dr. Ohland is an ABET Program Evaluator for ASEE. He was the 2002–2006 President of Tau Beta Pi and is a Fellow of the ASEE, IEEE, and AAAS.

visit author page

biography

Behzad Beigpourian Purdue University

visit author page

Behzad Beigpourian is a Ph.D. student and Research Assistant in Engineering Education at Purdue University. He earned his master’s in Structural Engineering from Shahid Chamran University in Iran, and his bachelor’s in Civil Technical Teacher from Shahid Rajaee Teacher Training University in Iran, Tehran. He has been official Technical Teacher at Ministry of Education in Iran from 2007 to 2018, and received many certificate in education such as Educational Planning, Developing Research Report, and Understanding School Culture. During these years, he has taught construction courses in several technical schools. Mr. Beigpourian currently works in the CATME project, which is NSF funding project, on optimizing teamwork skills and assessing the quality of Peer Evaluations.

visit author page

Download Paper |

Abstract

Individual Item or Dimension Analysis (IDA) using the Social Relations Model (SRM) has the power to reveal subtle changes in behavior as recorded in a behaviorally anchored scale. These changes cannot be seen in Across Dimension Analysis (ADA). Analysis of mean and standard deviation across all 5 CATME dimensions does not uncover differences at the individual item (or dimension) basis. The Social Relations Model decomposes variance into components. In our case, the variance is identified as rater, target, or relationship. Rater variance measures how individuals tend to rate their teammates. A large rater variance would indicate that students give their teammates ratings that are idiosyncratic – they don’t relate to the ratings given by others. Target variance measures how consistently individuals are rated by their teammates. A larger target effect would indicate that students tend to receive similar ratings from various raters. Relationship variance measures how ratings are affected by unique dyadic relationships after other variances have been removed. A larger relationship effect would indicate that the ratings are based more on distinct relationship pairings within the team – that is, that the various team members display different behaviors to each of their teammates and are rated based on those behaviors. The individual dimensions can be defined as follows: Having (H) relevant KSAs, Contributing (C) to the team’s work, Interacting (I) with team members, Keeping (K) the team on track, and Expecting (E) quality. In this analysis, we are examining three different teamwork studies that were based on evaluating average changes in multiple behaviorally anchored items and found additional insights into behavior changes through IDA. All data included in this analysis was obtained through the CATME peer evaluation system and the studies focused on methods of improving the quality of peer evaluations. In an initial study of a large Midwestern university, IDA SRM analysis revealed significant results in dimensions C, I, K, and H that were not seen using the ADA analysis, the average rating scores across items (or dimensions). All IDA analyses will compare IDA and ADA and groups who received teamwork training interventions and groups who did not receive any teamwork training.

Benner, M., & Ferguson, D. M., & Ohland, M. W., & Beigpourian, B. (2019, June), Analyzing Changes in the Individual Dimensions of a Behaviorally Anchored Rating Scale Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. https://peer.asee.org/32091

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2019 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015