June 14, 2015
June 14, 2015
June 17, 2015
Educational Research and Methods
26.497.1 - 26.497.12
Developing and Validating a Concept InventoryConcept inventories (CIs) have been used to assess undergraduate students’ understanding ofimportant and difficult concepts in engineering disciplines. However, research has shown thateven meticulously designed CIs often fall short of measuring student conceptual understanding.CI developers’ intentions of measuring particular understandings are sometimes not congruent tothe skills and conceptual understandings students use to interpret and respond to items inpractice. This incongruity can occur despite developers’ expertise, perhaps because of the“expert blind spot”. Even when developers are able to create items that tap into the intendedconceptual understandings, the assessments may not indicate the extent to which students havemastered particular concepts. To create an inventory that is interpretable and meaningful requiresthat developers take great care in constructing their assessment and consider their validityarguments from the outset. They need to consider the extent to which an assessment measureswhat it was intended to measure as demonstrated through properties of the assessment thatinclude examinee response patterns; such considerations are part of an evidentiary argumentprocess.This paper outlines several actions developers can take to create high-quality inventories, usingan evidentiary approach to assessment design. Additionally, it presents an analytic frameworkthat can be used to evaluate validity arguments once the assessment has been developed. We usetwo case studies to illustrate application of these analyses to evaluate a CI’s validity: theConceptual Assessment Tool for Statics (CATS) and the Dynamics Concept Inventory (DCI).For example, the developers of these CIs make claims about overall conceptual understanding ofthe domain, understanding of specific concepts, and the presence of particular misconceptions orcommon errors. Our analyses found varying degrees of support for each claim, such as the use ofthe assessment as a measure of students’ overall understanding. Only CATS was able to provideevidence regarding student understanding of specific domain concepts. Neither assessmentshowed evidence for differentiating among student misconceptions. Overall, this framework canhelp to provide structure in evaluating the validity arguments of CIs, as well as help CIdevelopers look ahead when creating inventories so that the validity claims are better aligned tostudent reasoning. ReferencesKane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1-73.Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary research and perspectives, 1(1), 3-62.Nathan, M. J., & Koedinger, K. R. (2000). An investigation of teachers' beliefs of students' algebra development. Cognition and Instruction, 18(2), 209-237.Nathan, M. J., & Petrosino, A. (2003). Expert blind spot among preservice teachers. American Educational Research Journal, 40(4), 905-928.
Jorion, N., & Gane, B. D., & DiBello, L. V., & Pellegrino, J. W. (2015, June), Developing and Validating a Concept Inventory Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.23836
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015