Asee peer logo

Methods for Measuring Systems Thinking: Differences Between Student Self-assessment, Concept Map Scores, and Cortical Activation During Tasks About Sustainability

Download Paper |


2018 ASEE Annual Conference & Exposition


Salt Lake City, Utah

Publication Date

June 23, 2018

Start Date

June 23, 2018

End Date

July 27, 2018

Conference Session

Sustainability in Civil Engineering

Tagged Division

Civil Engineering

Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Mo Hu Virginia Tech

visit author page

Mo Hu is a Ph.D. student in Civil and Environmental Engineering at Virginia Tech. Her research interest is applying neuroscience methods in engineering to provide better solutions for sustainability.

visit author page


Tripp Shealy Virginia Tech

visit author page

Tripp Shealy is an Assistant Professor of Civil and Environmental Engineering at Virginia Tech.

visit author page

Download Paper |


Systems thinking is a necessary skill towards solving complex civil engineering problems with interconnected environmental, social, and economic inputs and outputs. The dynamic relationship between systems components can act as a barrier for sustainability if decision makers work to reduce rather than understand complexities. To help advance methods for assessing and measuring students’ ability to think in systems, multiple methods were used and compared, including: a previously developed 15-item self-report survey named Systems Thinking Scale Revised (STSR), three scoring approaches to concept mapping and advanced cognitive neuro-imaging methods to measure physical changes in cognition. Engineering students (n=28) completed the STSR survey to report their capacity of systems thinking. They were outfitted to wear a functional near-infrared spectroscopy (fNIRS) system and asked to draw a concept map related to sustainability topics about energy, food, climate, and water. Their concept maps were scored using three approaches: the traditional, holistic, and categorical scoring method. The result shows that students’ self-evaluated systems thinking tendency is negatively (Spearman’s r = -0.50, p = 0.016) correlated with their concept map performance graded with the traditional scoring method, while positively (r = 0.39, p = 0.038) correlated to the sub-scores on the environment concepts using the categorical scoring method. Efficiency in brain connectivity, which is calculated using fNIRS data, is positively correlated with the complexity index using the categorical scoring approach (r = 0.45, p = 0.016) and the sub-scores of comprehensiveness in holistic scores (r = 0.42, p = 0.025). The results suggest students with higher performance of systems thinking were also more cognitively efficient. This study contributes to engineering education by demonstrating a new measurement tool to understand systems thinking and students’ cognitive abilities. The results also demonstrate possible discrepancies in previously developed surveys, concept map scoring techniques and cognition measured through changes in cortical activation. This trans-disciplinary approach bridges engineering education, sustainability, and neuroscience and begins to open new avenues of research helping measure the effectiveness of assessment techniques with physical responses of cognitive activation.

Hu, M., & Shealy, T. (2018, June), Methods for Measuring Systems Thinking: Differences Between Student Self-assessment, Concept Map Scores, and Cortical Activation During Tasks About Sustainability Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--30807

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2018 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015