Rubric-Based Energy Literacy Assessment of Student Posters: Effects of Extended Calibration and Addition of Raters

Liv Haselbach; Quinn Langfitt

Download Paper | Permalink

Conference: 2016 ASEE Annual Conference & Exposition
Location: New Orleans, Louisiana
Publication Date: June 26, 2016
Start Date: June 26, 2016
End Date: June 29, 2016
ISBN: 978-0-692-68565-5
ISSN: 2153-5965
Conference Session: Assessment, Course, and Curricular Development
Tagged Division: Energy Conversion and Conservation
Page Count: 16
DOI: 10.18260/p.26128
Permanent URL: https://peer.asee.org/26128
Download Count: 623

Paper Authors

biography

Quinn Langfitt Washington State University

visit author page

Quinn is a PhD candidate in the Department of Civil and Environmental Engineering at Washington State University. His research is mostly focused on sustainability, including work on life cycle assessment and energy literacy assessment.

visit author page

biography

Liv Haselbach P.E. Washington State University

visit author page

Dr. Liv Haselbach is the author of the McGraw-Hill GreenSource book, The Engineering Guide to LEED- New Construction, Sustainable Construction for Engineers. She has authored numerous papers on sustainability related to topics such as low impact development and carbon sequestration, and is active in the sustainability education community. Dr. Haselbach is a licensed professional engineer and a LEED AP (BD+C). Prior to her academic career she founded an engineering consulting company in the New York – Connecticut area. Her degrees include a BS in Civil and Environmental Engineering from Cornell, an MS in Chemical Engineering from UC Berkeley, and a PhD in Environmental Engineering from the University of Connecticut. She is currently an Associate Professor in Civil and Environmental Engineering at Washington State University, an Associate Director of the USDOT Tier 1 UTC: Center for Environmentally Sustainable Transportation in Cold Climates (CESTiCC) and a Fulbright-ALCOA Distinguished Chair in the Environmental Sciences and Engineering.

visit author page

Download Paper | Permalink

Abstract

Energy literacy encompasses knowledge of energy principles in technical, social, and economic realms, as well as the ability to critically apply that knowledge to solve problems and form opinions. Collective advancement of energy literacy among the general population is thought to be instrumental in implementing sustainable energy solutions in the near future. As efforts to improve energy literacy have advanced, so has the need to assess the outcomes of those efforts. This paper describes advancements in a recently developed approach of examining energy literacy in student projects through application of a rubric, and the results of a case study using the methodology on the Imagine Tomorrow high school energy competition. Changes made to the approach include a more detailed rater calibration session and a significant increase in the number of raters over a previous cases study which used the same rubric. Similar to the previous study, results show that raters exhibit moderate to substantial agreement when interrater reliability is measured by Kendall’s coefficient of concordance. As a component of this paper, group-wise comparisons of raters (pairs, triplets, and quadruplets) are examined to see if conclusions might have been different with different subsets of raters, both in terms of agreement statistics and in terms of energy literacy characteristics exhibited by various discrete groupings of students. No subset of raters would have resulted in significantly different conclusions in terms of scoring trends, though reliability statistics would be slightly altered. With respect to the competition energy literacy characteristics, it was found that posters created by students participating in more techno-centric challenges, with competition experience, or when mentored by returning advisors scored slightly higher than others. The energy literacy observed was unaffected by gender of the students or the teaching subject of advisors. Continual assessment, and improvement of assessment instruments, is vital as project-based learning continues to be a focal point for teaching about energy, and as organizers plan how to best shape future events to improve energy literacy of our current and future decision-makers.

Citation
Format

Langfitt, Q., & Haselbach, L. (2016, June), Rubric-Based Energy Literacy Assessment of Student Posters: Effects of Extended Calibration and Addition of Raters Paper presented at 2016 ASEE Annual Conference & Exposition, New Orleans, Louisiana. 10.18260/p.26128

TY  - CPAPER
AB  - Energy literacy encompasses knowledge of energy principles in technical, social, and economic realms, as well as the ability to critically apply that knowledge to solve problems and form opinions. Collective advancement of energy literacy among the general population is thought to be instrumental in implementing sustainable energy solutions in the near future. As efforts to improve energy literacy have advanced, so has the need to assess the outcomes of those efforts. This paper describes advancements in a recently developed approach of examining energy literacy in student projects through application of a rubric, and the results of a case study using the methodology on the Imagine Tomorrow high school energy competition. Changes made to the approach include a more detailed rater calibration session and a significant increase in the number of raters over a previous cases study which used the same rubric. Similar to the previous study, results show that raters exhibit moderate to substantial agreement when interrater reliability is measured by Kendall’s coefficient of concordance. As a component of this paper, group-wise comparisons of raters (pairs, triplets, and quadruplets) are examined to see if conclusions might have been different with different subsets of raters, both in terms of agreement statistics and in terms of energy literacy characteristics exhibited by various discrete groupings of students. No subset of raters would have resulted in significantly different conclusions in terms of scoring trends, though reliability statistics would be slightly altered. With respect to the competition energy literacy characteristics, it was found that posters created by students participating in more techno-centric challenges, with competition experience, or when mentored by returning advisors scored slightly higher than others. The energy literacy observed was unaffected by gender of the students or the teaching subject of advisors. Continual assessment, and improvement of assessment instruments, is vital as project-based learning continues to be a focal point for teaching about energy, and as organizers plan how to best shape future events to improve energy literacy of our current and future decision-makers.
AU  - Quinn Langfitt
AU  - Liv Haselbach P.E.
CY  - New Orleans, Louisiana
DA  - 2016/06/26
PB  - ASEE Conferences
TI  - Rubric-Based Energy Literacy Assessment of Student Posters: Effects of Extended Calibration and Addition of Raters
UR  - https://peer.asee.org/26128
DO  - 10.18260/p.26128
ER  -