A Statistical Study of Concept Mapping Metrics

Kathryn W. Jablokow; Joanna F. DeFranco; Sally Sue Richmond

Download Paper | Permalink

Conference: 2013 ASEE Annual Conference & Exposition
Location: Atlanta, Georgia
Publication Date: June 23, 2013
Start Date: June 23, 2013
End Date: June 26, 2013
ISSN: 2153-5965
Conference Session: ERM Potpourri
Tagged Division: Educational Research and Methods
Page Count: 14
Page Numbers: 23.105.1 - 23.105.14
DOI: 10.18260/1-2--19119
Permanent URL: https://strategy.asee.org/19119
Download Count: 467

Request a correction

Paper Authors

biography

Kathryn W. Jablokow Pennsylvania State University

visit author page

Dr. Kathryn Jablokow is an Associate Professor of Mechanical Engineering and Engineering Design at Penn State University. A graduate of Ohio State University (Ph.D., Electrical Engineering), Dr. Jablokow's teaching and research interests include problem solving, invention, and creativity in science and engineering, as well as robotics and computational dynamics. In addition to her membership in ASEE, she is a Senior Member of IEEE and a Fellow of ASME. Dr. Jablokow is the architect of a unique 4-course module focused on creativity and problem solving leadership and is currently developing a new methodology for cognition-based design. She is one of three instructors for Penn State’s Massive Open Online Course (MOOC) on Creativity, Innovation, and Change, and she is the founding director of the Problem Solving Research Group, whose 50+ collaborating members include faculty and students from several universities, as well as industrial representatives, military leaders, and corporate consultants.

visit author page

biography

Joanna F. DeFranco Pennsylvania State University, Great Valley

visit author page

Joanna F. DeFranco is Assistant Professor of Software Engineering and a member of the Graduate Faculty at The Pennsylvania State University. Prior to joining Penn State, she held faculty positions at Cabrini College and the New Jersey Institute of Technology. She also held a number of positions in industry and government including an Electronics Engineer for the Naval Air Development Center in Warminster, PA and a Software Engineer at Motorola in Horsham, PA.
Dr. DeFranco received her B.S. in Electrical Engineering from Penn State University, M.S. in Computer Engineering from Villanova University, and Ph.D. in Computer and Information Science from the New Jersey Institute of Technology. She is a member of ASEE and has had numerous publications in journals and conference proceedings. She is also on the curriculum advisory board for a local technical high school.

visit author page

author page

Sally Sue Richmond Penn State Great Valley School of Graduate Professional Studies

Download Paper | Permalink

Abstract

A Statistical Study of Concept Mapping MetricsBackground: The use of concept maps in engineering education research is growing, with applications inthe assessment of knowledge mastery and integration within courses, programs, and across multipledisciplines. Concept maps are also being used in the early stages of engineering problem solving anddesign to help teams gain a shared understanding of group tasks. Within these applications, a variety ofmetrics have been developed for assessing concept maps, including both “traditional” and “holistic”approaches to measuring the accuracy, breadth, and depth of students’ understanding. In general,traditional metrics rely on counting elements of a concept map (e.g., concepts, links, hierarchies) or thecomputation of map descriptors (e.g., map density, map complexity) as functions of these elements. Dueto their dependence on relatively clear-cut features, traditional metrics are generally considered to bequite objective (i.e., different evaluators are likely to derive the same results); nevertheless, holisticmetrics that focus on a more subjective “quality of understanding” represented in a concept map (ratherthan the “quantity” of its elements) have also emerged. These holistic scoring methods include structuralcomplexity approaches that assess the dominant structural patterns of concept maps (e.g., hub/spoke, tree,network), as well as integrated rubrics based on the organization, comprehensiveness, and correctness ofmap content.Motivation: A wide variety of concept mapping metrics exists, but very few studies have examined therelationships between them in detail. To address this need, we performed an exploratory statisticalanalysis to determine if and how the predominant traditional and holistic concept mapping metrics arecorrelated, with the future aim of identifying sets of metrics that are most effective for the evaluation ofstudents’ understanding in specific situations.Research Methods: Our samples included 73 undergraduate engineering students (first-year engineeringdesign) and 52 graduate engineering students (master’s-level systems engineering) at a large, publicuniversity. Each student completed at least one technical concept map of a course-related topic; thesemaps were assessed by two trained evaluators using eleven traditional and seven holistic mappingmetrics. Traditional metrics included number of concepts, number of links, map density, map complexity,and number of hierarchies, among others; holistic metrics included comprehensiveness, organization, andcompleteness, as well as dominant structural patterns (e.g., tree, network). Statistical analyses wereperformed to determine if and how these metrics were correlated, both among and between the traditionaland holistic metric subsets.Results: Our analyses revealed a range of statistically significant correlations among and between thetraditional and holistic map metrics. Some of the strongest correlations were found betweencomprehensiveness (a holistic metric) and numbers of concepts and links (traditional metrics),respectively, as well as between similarity and matching links (both traditional metrics), using an expertmap for reference. Equally interesting was the lack of significant correlations between metrics that appearto be related from a theoretical perspective, including (for example) number of hierarchies (traditional)and organization (holistic).Conclusions and Significance: While the small sample sizes used in this study limit our conclusions, theresults are encouraging and suggest further investigation. Understanding the relationships betweendifferent concept mapping metrics marks the first step in helping educators make informed choices aboutwhich metrics they use to assess student outcomes effectively.

Citation
Format

Jablokow, K. W., & DeFranco, J. F., & Richmond, S. S. (2013, June), A Statistical Study of Concept Mapping Metrics Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--19119

TY  - CPAPER
AB  -                      A Statistical Study of Concept Mapping MetricsBackground: The use of concept maps in engineering education research is growing, with applications inthe assessment of knowledge mastery and integration within courses, programs, and across multipledisciplines. Concept maps are also being used in the early stages of engineering problem solving anddesign to help teams gain a shared understanding of group tasks. Within these applications, a variety ofmetrics have been developed for assessing concept maps, including both “traditional” and “holistic”approaches to measuring the accuracy, breadth, and depth of students’ understanding. In general,traditional metrics rely on counting elements of a concept map (e.g., concepts, links, hierarchies) or thecomputation of map descriptors (e.g., map density, map complexity) as functions of these elements. Dueto their dependence on relatively clear-cut features, traditional metrics are generally considered to bequite objective (i.e., different evaluators are likely to derive the same results); nevertheless, holisticmetrics that focus on a more subjective “quality of understanding” represented in a concept map (ratherthan the “quantity” of its elements) have also emerged. These holistic scoring methods include structuralcomplexity approaches that assess the dominant structural patterns of concept maps (e.g., hub/spoke, tree,network), as well as integrated rubrics based on the organization, comprehensiveness, and correctness ofmap content.Motivation: A wide variety of concept mapping metrics exists, but very few studies have examined therelationships between them in detail. To address this need, we performed an exploratory statisticalanalysis to determine if and how the predominant traditional and holistic concept mapping metrics arecorrelated, with the future aim of identifying sets of metrics that are most effective for the evaluation ofstudents’ understanding in specific situations.Research Methods: Our samples included 73 undergraduate engineering students (first-year engineeringdesign) and 52 graduate engineering students (master’s-level systems engineering) at a large, publicuniversity. Each student completed at least one technical concept map of a course-related topic; thesemaps were assessed by two trained evaluators using eleven traditional and seven holistic mappingmetrics. Traditional metrics included number of concepts, number of links, map density, map complexity,and number of hierarchies, among others; holistic metrics included comprehensiveness, organization, andcompleteness, as well as dominant structural patterns (e.g., tree, network). Statistical analyses wereperformed to determine if and how these metrics were correlated, both among and between the traditionaland holistic metric subsets.Results: Our analyses revealed a range of statistically significant correlations among and between thetraditional and holistic map metrics. Some of the strongest correlations were found betweencomprehensiveness (a holistic metric) and numbers of concepts and links (traditional metrics),respectively, as well as between similarity and matching links (both traditional metrics), using an expertmap for reference. Equally interesting was the lack of significant correlations between metrics that appearto be related from a theoretical perspective, including (for example) number of hierarchies (traditional)and organization (holistic).Conclusions and Significance: While the small sample sizes used in this study limit our conclusions, theresults are encouraging and suggest further investigation. Understanding the relationships betweendifferent concept mapping metrics marks the first step in helping educators make informed choices aboutwhich metrics they use to assess student outcomes effectively.
AU  - Kathryn W. Jablokow
AU  - Joanna F. DeFranco
AU  - Sally Sue Richmond
CY  - Atlanta, Georgia
DA  - 2013/06/23
PB  - ASEE Conferences
TI  - A Statistical Study of Concept Mapping Metrics
UR  - https://strategy.asee.org/19119
DO  - 10.18260/1-2--19119
ER  -