Asee peer logo

Rubric-Based Energy Literacy Assessment of Student Posters: Effects of Extended Calibration and Addition of Raters

Download Paper |


2016 ASEE Annual Conference & Exposition


New Orleans, Louisiana

Publication Date

June 26, 2016

Start Date

June 26, 2016

End Date

August 28, 2016





Conference Session

Assessment, Course, and Curricular Development

Tagged Division

Energy Conversion and Conservation

Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Quinn Langfitt Washington State University

visit author page

Quinn is a PhD candidate in the Department of Civil and Environmental Engineering at Washington State University. His research is mostly focused on sustainability, including work on life cycle assessment and energy literacy assessment.

visit author page


Liv Haselbach P.E. Washington State University

visit author page

Dr. Liv Haselbach is the author of the McGraw-Hill GreenSource book, The Engineering Guide to LEED- New Construction, Sustainable Construction for Engineers. She has authored numerous papers on sustainability related to topics such as low impact development and carbon sequestration, and is active in the sustainability education community. Dr. Haselbach is a licensed professional engineer and a LEED AP (BD+C). Prior to her academic career she founded an engineering consulting company in the New York – Connecticut area. Her degrees include a BS in Civil and Environmental Engineering from Cornell, an MS in Chemical Engineering from UC Berkeley, and a PhD in Environmental Engineering from the University of Connecticut. She is currently an Associate Professor in Civil and Environmental Engineering at Washington State University, an Associate Director of the USDOT Tier 1 UTC: Center for Environmentally Sustainable Transportation in Cold Climates (CESTiCC) and a Fulbright-ALCOA Distinguished Chair in the Environmental Sciences and Engineering.

visit author page

Download Paper |


Energy literacy encompasses knowledge of energy principles in technical, social, and economic realms, as well as the ability to critically apply that knowledge to solve problems and form opinions. Collective advancement of energy literacy among the general population is thought to be instrumental in implementing sustainable energy solutions in the near future. As efforts to improve energy literacy have advanced, so has the need to assess the outcomes of those efforts. This paper describes advancements in a recently developed approach of examining energy literacy in student projects through application of a rubric, and the results of a case study using the methodology on the Imagine Tomorrow high school energy competition. Changes made to the approach include a more detailed rater calibration session and a significant increase in the number of raters over a previous cases study which used the same rubric. Similar to the previous study, results show that raters exhibit moderate to substantial agreement when interrater reliability is measured by Kendall’s coefficient of concordance. As a component of this paper, group-wise comparisons of raters (pairs, triplets, and quadruplets) are examined to see if conclusions might have been different with different subsets of raters, both in terms of agreement statistics and in terms of energy literacy characteristics exhibited by various discrete groupings of students. No subset of raters would have resulted in significantly different conclusions in terms of scoring trends, though reliability statistics would be slightly altered. With respect to the competition energy literacy characteristics, it was found that posters created by students participating in more techno-centric challenges, with competition experience, or when mentored by returning advisors scored slightly higher than others. The energy literacy observed was unaffected by gender of the students or the teaching subject of advisors. Continual assessment, and improvement of assessment instruments, is vital as project-based learning continues to be a focal point for teaching about energy, and as organizers plan how to best shape future events to improve energy literacy of our current and future decision-makers.

Langfitt, Q., & Haselbach, L. (2016, June), Rubric-Based Energy Literacy Assessment of Student Posters: Effects of Extended Calibration and Addition of Raters Paper presented at 2016 ASEE Annual Conference & Exposition, New Orleans, Louisiana. 10.18260/p.26128

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2016 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015