June 23, 2013
June 23, 2013
June 26, 2013
Educational Research and Methods
23.105.1 - 23.105.14
A Statistical Study of Concept Mapping MetricsBackground: The use of concept maps in engineering education research is growing, with applications inthe assessment of knowledge mastery and integration within courses, programs, and across multipledisciplines. Concept maps are also being used in the early stages of engineering problem solving anddesign to help teams gain a shared understanding of group tasks. Within these applications, a variety ofmetrics have been developed for assessing concept maps, including both “traditional” and “holistic”approaches to measuring the accuracy, breadth, and depth of students’ understanding. In general,traditional metrics rely on counting elements of a concept map (e.g., concepts, links, hierarchies) or thecomputation of map descriptors (e.g., map density, map complexity) as functions of these elements. Dueto their dependence on relatively clear-cut features, traditional metrics are generally considered to bequite objective (i.e., different evaluators are likely to derive the same results); nevertheless, holisticmetrics that focus on a more subjective “quality of understanding” represented in a concept map (ratherthan the “quantity” of its elements) have also emerged. These holistic scoring methods include structuralcomplexity approaches that assess the dominant structural patterns of concept maps (e.g., hub/spoke, tree,network), as well as integrated rubrics based on the organization, comprehensiveness, and correctness ofmap content.Motivation: A wide variety of concept mapping metrics exists, but very few studies have examined therelationships between them in detail. To address this need, we performed an exploratory statisticalanalysis to determine if and how the predominant traditional and holistic concept mapping metrics arecorrelated, with the future aim of identifying sets of metrics that are most effective for the evaluation ofstudents’ understanding in specific situations.Research Methods: Our samples included 73 undergraduate engineering students (first-year engineeringdesign) and 52 graduate engineering students (master’s-level systems engineering) at a large, publicuniversity. Each student completed at least one technical concept map of a course-related topic; thesemaps were assessed by two trained evaluators using eleven traditional and seven holistic mappingmetrics. Traditional metrics included number of concepts, number of links, map density, map complexity,and number of hierarchies, among others; holistic metrics included comprehensiveness, organization, andcompleteness, as well as dominant structural patterns (e.g., tree, network). Statistical analyses wereperformed to determine if and how these metrics were correlated, both among and between the traditionaland holistic metric subsets.Results: Our analyses revealed a range of statistically significant correlations among and between thetraditional and holistic map metrics. Some of the strongest correlations were found betweencomprehensiveness (a holistic metric) and numbers of concepts and links (traditional metrics),respectively, as well as between similarity and matching links (both traditional metrics), using an expertmap for reference. Equally interesting was the lack of significant correlations between metrics that appearto be related from a theoretical perspective, including (for example) number of hierarchies (traditional)and organization (holistic).Conclusions and Significance: While the small sample sizes used in this study limit our conclusions, theresults are encouraging and suggest further investigation. Understanding the relationships betweendifferent concept mapping metrics marks the first step in helping educators make informed choices aboutwhich metrics they use to assess student outcomes effectively.
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2013 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015