Asee peer logo

Concept Inventories Meet Cognitive Psychology: Using Beta Testing As A Mechanism For Identifying Engineering Student Misconceptions

Download Paper |


2005 Annual Conference


Portland, Oregon

Publication Date

June 12, 2005

Start Date

June 12, 2005

End Date

June 15, 2005



Conference Session

Engineering Education Research and Assessment II

Page Count


Page Numbers

10.335.1 - 10.335.18



Permanent URL

Download Count


Request a correction

Paper Authors

author page

Mary Nelson

author page

Barbara Olds

author page

Monica Geist

author page

Ronald Miller

author page

Ruth Streveler

Download Paper |

NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract

Concept Inventories Meet Cognitive Psychology: Using Beta Testing as a Mechanism for Identifying Engineering Student Misconceptions

Ronald L. Miller, Ruth A. Streveler Colorado School of Mines

Mary A. Nelson, Monica R. Geist University of Colorado/University of Northern Colorado

Barbara M. Olds National Science Foundation and Colorado School of Mines


This paper summarizes our analysis of beta test results collected at four U.S. engineering schools using the Thermal and Transport Concept Inventory (TTCI) currently under development. The instrument consists of questions in heat transfer, fluid mechanics, and thermodynamics and is designed to elicit information about the nature of undergraduate engineering students’ misconceptions in these disciplines. In addition to using conventional methods for determining TTCI reliability with correlation coefficients, we have applied cross-tabulation data analysis for six heat transfer questions to identify robust student misconceptions. The results of these analyses identified two categories of misconceptions which persist across heat transfer problems and contexts: 1) confusion about differences between energy and temperature, and 2) confusion about differences in steady-state and equilibrium processes.


With funding from the National Science Foundation (DUE-0127806), a team of researchers at the Colorado School of Mines (CSM) is creating a concept inventory to measure engineering students’ understanding of difficult concepts in thermal and transport science, the Thermal and Transport Concept Inventory (TTCI). [1] Our project began with a Delphi survey of experts to determine the concepts that were the most difficult and important for engineering students studying thermodynamics, heat transfer, and fluid mechanics. [2] Once the key concepts were identified, we created open-ended questions related to each concept and asked students to think aloud as they answered the questions. Multiple-choice questions were then developed using students’ misunderstandings as documented in the think-alouds as the basis for writing distractors (incorrect but plausible answers). The resulting inventory was alpha tested at CSM [3], expanded, and beta tested at four other engineering institutions.

Standard psychometric procedures include testing to establish various forms of reliability (how repeatable are the results?) and validity (are we really testing the concepts that we intend to test?) for an instrument under development. We are currently analyzing beta test data from the TTCI to establish these parameters, and more detailed findings will be reported elsewhere. However, we are also using the beta test data to develop methods for identifying the kinds of misunderstandings or misconceptions students hold about concepts included in the TTCI. In

Proceedings of the 2005 American Society for Engineering Education Annual Conference & Exposition Copyright © 2005, American Society for Engineering Education

Nelson, M., & Olds, B., & Geist, M., & Miller, R., & Streveler, R. (2005, June), Concept Inventories Meet Cognitive Psychology: Using Beta Testing As A Mechanism For Identifying Engineering Student Misconceptions Paper presented at 2005 Annual Conference, Portland, Oregon. 10.18260/1-2--14620

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2005 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015