Asee peer logo

Final Results of Reliability Testing for the Norback-Utschig Presentation Scoring System and Implications for Instruction

Download Paper |


2014 ASEE Annual Conference & Exposition


Indianapolis, Indiana

Publication Date

June 15, 2014

Start Date

June 15, 2014

End Date

June 18, 2014



Conference Session

Teaching Communication II

Tagged Division

Liberal Education/Engineering & Society

Page Count


Page Numbers

24.605.1 - 24.605.18



Permanent URL

Download Count


Request a correction

Paper Authors


Judith Shaul Norback Georgia Institute of Technology

visit author page

Judith Shaul Norback, Ph.D., is general faculty and Director of Workplace and Academic Communication in the Stewart School of Industrial and Systems Engineering at Georgia Tech. She applies her skills as a social psychologist to gather data from executives about stellar presentations and other oral communication skills and she conducts research on communication, in order to improve instruction. Dr. Norback has developed and provided instruction for students in industrial and biomedical engineering and has advised on oral communication instruction at other universities. Since she founded the Presentation Coaching Program in 2003, the coaching has had over 21,000 student visits. As of winter 2013, she shared her instructional materials with over 250 schools from the U.S., Australia, Germany, and South Korea.
Dr. Norback has studied communication and other basic skills in the workplace and developed curriculum over the past 30 years—first at Educational Testing Service; then as part of the Center for Skills Enhancement, Inc., which she founded, with clients including the U.S. Department of Labor, the National Skill Standards Board, and universities. Since 2000, when she arrived at Georgia Tech, her work has focused on oral communication for engineering students and engineers. Dr. Norback has published over 20 articles in the past decade alone, including articles in the ASEE Annual Conference Proceedings, IEEE Transactions on Professional Communication, INFORMS Transactions on Education, and the International Journal of Engineering Education. She authored the book Oral Communication Excellence for Engineers and Scientists, published in summer, 2013.
Over the past 12 years Dr. Norback has given over 40 conference presentations and workshops at nation-wide conferences such as ASEE, where she currently serves as chair of the Liberal Education/Engineering & Society (LEES) Division. She has been an officer for the Education Forum of the Institute for Operations Research and the Management Sciences (INFORMS) and has served as Associate Chair for the national Capstone Design Conference. Dr. Norback has a Bachelors’ degree from Cornell University and a Masters and PhD from Princeton University. Her current research interests include increasing the reliability of the Norback-Utschig Presentation Scoring System for Engineers and Scientists and identifying the mental models engineering students use when creating graphical representations.

visit author page


Tristan T. Utschig Georgia Institute of Technology Orcid 16x16

visit author page

Dr. Tristan T. Utschig is a Senior Academic Professional in the Center for the Enhancement of Teaching and Learning and is Assistant Director for the Scholarship and Assessment of Teaching and Learning at the Georgia Institute of Technology. Formerly, he was a tenured Associate Professor of Engineering Physics at Lewis-Clark State College. Dr. Utschig consults with faculty across the university about bringing scholarly teaching and learning innovations into their classroom and assessing their impact. He has regularly published and presented work on a variety of topics including assessment instruments and methodologies, using technology in the classroom, faculty development in instructional design, teaching diversity, and peer coaching. Dr. Utschig completed his PhD in Nuclear Engineering at the University of Wisconsin–Madison.

visit author page


Anthony Joseph Bonifonte Georgia Institute of Technology

visit author page

Anthony Joseph Bonifonte is currently in his 3rd year of Georgia Tech's PhD program in Operations Research in the Industrial and Systems Engineering Department. He attended Oberlin College as an undergraduate, majoring in math and biology. He has served as teaching assistant five times for math and industrial engineering courses. He currently works as a graduate research assistant in Georgia Tech's Center for the Enhancement of Teaching and Learning (CETL) where he assists with assessment and data analysis for ongoing CETL projects. His thesis research involves mathematical models and decision making in cardiology.

visit author page


Gloria J. Ross Georgia Institute of Technology

visit author page

Gloria Ross is currently a PhD candidate in History and Sociology of Science and Technology at Georgia Tech. Her research focuses on the spatial and demographic factors that shape urban food distribution systems. She currently works as a graduate research assistant in Georgia Tech's Center for the Enhancement of Teaching and Learning (CETL) where she assists with assessment and data analysis for ongoing CETL projects.

visit author page

Download Paper |


Final Results of Reliability Testing for the xx Presentation Scoring SystemOur team has completed our final step in the concerted effort to provide an analytical basis forthe reliability of the xx Presentation Scoring System. Previously we have reported on (1)reliability testing results on an earlier version of the presentation scoring system with 19presentation skills and (2) our use of the first stage of a modified Delphi Method to refine thescoring system to a set of 13 skills. Now we can report on the final results of the modifiedDelphi Method, which includes our final reliability testing. Based on our extensive literaturereview, we believe this work is rare for presentation scoring systems. In this final phase, weconducted a study including presentations by students in biomedical engineering and industrialengineering.We report the reliability levels for each of the 13 presentation skills--for example, “linking thematerial presented to the big picture." Approximately 20 presentations were rated by groups ofundergraduate peers, teaching assistants, and faculty members. We calculated the frequency ofexact matches and matches within one point on a 5-point scale. The matching frequency wasdone using both a pairwise approach (all raters are compared to each other equally) and acriterion referenced approach (raters are compared to expert referees). The reliability statisticswere divided into three categories: excellent reliability, good reliability, and reasonable but lowerreliability. The information demonstrates that the scoring system can be used reliably in multipleengineering education settings. We will demonstrate just how to implement the presentationscoring system in different engineering education settings.

Norback, J. S., & Utschig, T. T., & Bonifonte, A. J., & Ross, G. J. (2014, June), Final Results of Reliability Testing for the Norback-Utschig Presentation Scoring System and Implications for Instruction Paper presented at 2014 ASEE Annual Conference & Exposition, Indianapolis, Indiana. 10.18260/1-2--20496

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2014 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015