Asee peer logo

Determining Reliability of Scores from an Energy Literacy Rubric

Download Paper |


2015 ASEE Annual Conference & Exposition


Seattle, Washington

Publication Date

June 14, 2015

Start Date

June 14, 2015

End Date

June 17, 2015





Conference Session

Communication and Literacy

Tagged Division

Educational Research and Methods

Page Count


Page Numbers

26.482.1 - 26.482.9



Permanent URL

Download Count


Request a correction

Paper Authors


Chad M Gotch Washington State University

visit author page

Chad Gotch's interests center on maximizing effective and proper use of educational and psychological measurements. To this end, he studies assessment/measurement literacy among teachers, score reporting, and building validity arguments from both technical and non-technical evidence. These complimentary lines of research inform the life cycle of assessment, from development to use and policy.

visit author page


Quinn Langfitt Washington State University

visit author page

Quinn is a PhD student in the School of Civil and Environmental Engineering at Washington State University. His research is mostly focused on sustainability, including work on life cycle assessment and energy literacy assessment.

visit author page


Brian F French Washington State University

visit author page

Brian F. French is a Professor of Educational Psychology with an emphasis in Psychometrics and Research Methods. He is the Director of the Learning and Performance Research Center at Washington State University.

visit author page

author page

Liv Haselbach Washington State University

Download Paper |


Determining Reliability of Scores from an Energy Literacy RubricThe purpose of this paper is to estimate the reliability of scores from an energy literacy rubricdeveloped on the framework of the U.S. Department of Energy’s “Essential Principles andFundamental Concepts for Energy Education”. Energy literacy holds a place of prominencewithin engineering education, as the planet experiences increased demand for sustainable energygeneration and as the public and lawmakers strive to create energy policy that accounts fortradeoffs between complex options. Given that energy literacy levels in the population have beenfound to be generally low, there is a need to develop educational activities to improve energyliteracy. Example activities include initiatives such as high school energy competitions,development of interdisciplinary curricula, and field experiences and internships.Assessment of student learning is a central function of educational endeavors. Rubrics are avaluable way to assess competencies, such as those associated with energy literacy. Objects ofassessment using a rubric may include a performance, portfolio, poster, or other educationalartifacts. While rubrics provide a means to rate performances and products across multipledimensions and levels of proficiency, a concern in their use is that the rubric ratings aredependent upon the individuals supplying the ratings. Error is introduced into the assessmentprocess through this inherent characteristic of use. A rubric that can demonstrate a high level ofconsistency across raters—suggesting a small amount of measurement error—is highly valuable.Reliability refers to the consistency in observed assessment scores, and represents one piece ofevidence needed to support valid score interpretations and uses. This paper utilizesgeneralizability theory to determine the consistency of scores from the energy literacy rubric.The present study represents a necessary effort to document performance of the rubric forrefinement and to advance the study of energy literacy.

Gotch, C. M., & Langfitt, Q., & French, B. F., & Haselbach, L. (2015, June), Determining Reliability of Scores from an Energy Literacy Rubric Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.23820

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015