Jorion, N., & Gane, B. D., & DiBello, L. V., & Pellegrino, J. W. (2015, June), Developing and Validating a Concept Inventory  Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.23836

\bibitem{asee_peer_23836}
  Jorion, N., \& Gane, B. D., \& DiBello, L. V., \& Pellegrino, J. W. (2015, June), \emph{Developing and Validating a Concept Inventory}
  Paper presented at 2015 ASEE Annual Conference \& Exposition, Seattle, Washington.
  10.18260/p.23836

Natalie Jorion, Brian Douglas Gane, Louis V DiBello, and James W Pellegrino.  "Developing and Validating a Concept Inventory".  2015 ASEE Annual Conference & Exposition, Seattle, Washington, 2015, June.  ASEE Conferences, 2015.  https://peer.asee.org/23836 Internet.  11 Jan, 2025

\bibitem{asee_peer_23836}
  Natalie Jorion, Brian Douglas Gane, Louis V DiBello, and James W Pellegrino.
  "Developing and Validating a Concept Inventory".
  \emph{2015 ASEE Annual Conference \& Exposition, Seattle, Washington, 2015, June}.
  ASEE Conferences, 2015.
  https://peer.asee.org/23836 Internet.  11 Jan, 2025

@INPROCEEDINGS{asee_peer_23836
author = "Natalie Jorion, Brian Douglas Gane, Louis V DiBello, and James W Pellegrino"
title = "Developing and Validating a Concept Inventory"
booktitle = "2015 ASEE Annual Conference \& Exposition"
year = "2015"
month = "June"
address = "Seattle, Washington"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/23836}
number = {10.18260/p.23836}
}

TY  - CPAPER
AB  -                           Developing and Validating a Concept                                      InventoryConcept inventories (CIs) have been used to assess undergraduate students’ understanding ofimportant and difficult concepts in engineering disciplines. However, research has shown thateven meticulously designed CIs often fall short of measuring student conceptual understanding.CI developers’ intentions of measuring particular understandings are sometimes not congruent tothe skills and conceptual understandings students use to interpret and respond to items inpractice. This incongruity can occur despite developers’ expertise, perhaps because of the“expert blind spot”. Even when developers are able to create items that tap into the intendedconceptual understandings, the assessments may not indicate the extent to which students havemastered particular concepts. To create an inventory that is interpretable and meaningful requiresthat developers take great care in constructing their assessment and consider their validityarguments from the outset. They need to consider the extent to which an assessment measureswhat it was intended to measure as demonstrated through properties of the assessment thatinclude examinee response patterns; such considerations are part of an evidentiary argumentprocess.This paper outlines several actions developers can take to create high-quality inventories, usingan evidentiary approach to assessment design. Additionally, it presents an analytic frameworkthat can be used to evaluate validity arguments once the assessment has been developed. We usetwo case studies to illustrate application of these analyses to evaluate a CI’s validity: theConceptual Assessment Tool for Statics (CATS) and the Dynamics Concept Inventory (DCI).For example, the developers of these CIs make claims about overall conceptual understanding ofthe domain, understanding of specific concepts, and the presence of particular misconceptions orcommon errors. Our analyses found varying degrees of support for each claim, such as the use ofthe assessment as a measure of students’ overall understanding. Only CATS was able to provideevidence regarding student understanding of specific domain concepts. Neither assessmentshowed evidence for differentiating among student misconceptions. Overall, this framework canhelp to provide structure in evaluating the validity arguments of CIs, as well as help CIdevelopers look ahead when creating inventories so that the validity claims are better aligned tostudent reasoning.                                           ReferencesKane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational      Measurement, 50, 1-73.Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). Focus article: On the structure of      educational assessments. Measurement: Interdisciplinary research and perspectives,      1(1), 3-62.Nathan, M. J., & Koedinger, K. R. (2000). An investigation of teachers' beliefs of students'      algebra development. Cognition and Instruction, 18(2), 209-237.Nathan, M. J., & Petrosino, A. (2003). Expert blind spot among preservice teachers. American      Educational Research Journal, 40(4), 905-928.  
AU  - Natalie Jorion
AU  - Brian Douglas Gane
AU  - Louis V DiBello
AU  - James W Pellegrino
CY  - Seattle, Washington
DA  - 2015/06/14
PB  - ASEE Conferences
TI  - Developing and Validating a Concept Inventory
UR  - https://peer.asee.org/23836
DO  - 10.18260/p.23836
ER  -