Asee peer logo

Measuring Engineering Students’ Metacognition with a Think-Aloud Protocol

Download Paper |


2018 ASEE Annual Conference & Exposition


Salt Lake City, Utah

Publication Date

June 23, 2018

Start Date

June 23, 2018

End Date

July 27, 2018

Conference Session

Problem Solving, Adaptive Expertise, and Social Engagement

Tagged Division

Educational Research and Methods

Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Carolyn Plumb Montana State University

visit author page

Carolyn Plumb is the recently retired Director of Educational Innovation and Strategic Projects in the College of Engineering at Montana State University (MSU). Plumb has been involved in engineering education and program evaluation for 30 years, and she continues to work on externally funded projects relating to engineering education.

visit author page


Rose M. Marra University of Missouri

visit author page

Rose M. Marra is a Professor of Learning Technologies at the University of Missouri. She is PI of the NSF-funded Supporting Collaboration in Engineering Education, and has studied and published on engineering education, women and minorities in STEM, online learning and assessment. Marra holds a PhD. in Educational Leadership and Innovation and worked as a software engineer before entering academe.

visit author page


Douglas J. Hacker University of Utah

visit author page

Dr. Hacker is a full professor in the Department of Educational Psychology and participates in both the Learning Sciences Program and the Reading and Literacy Program. Prior to receiving his Ph. D. in educational psychology from the University of Washington in 1994, Dr. Hacker worked as a high school science and math teacher and then as a school counselor. From 1994 to 1999, Dr. Hacker was an assistant/associate professor in the Department of Counseling, Educational Psychology and Research at The University of Memphis. During those years, he worked in the areas of reading and writing processes, metacognition, self-regulated learning, teacher education, and school and program evaluation. Dr. Hacker moved to the University of Utah in 1999 and has continued his research in the previous areas and has added to them research in the area of the detection of deception. Also at the University of Utah, he served as chair of the Teaching and Learning Department. His publications have appeared in the Journal of Educational Psychology, Contemporary Educational Psychology, Journal of Experimental Psychology: Applied, and Journal of Experimental Education. At both universities, Dr. Hacker has maintained a strong commitment to work in elementary and middle schools, working directly with teachers by providing professional development in reading and writing instruction. Since 1994, Dr. Hacker has been either the principal investigator or co-principal investigator on grants totaling $2,548,960. He has served as an editorial board member for the Journal of Educational Psychology, Metacognition and Learning, and Frontiers of Educational Psychology. He is a former Associate Editor for the Journal of Educational Psychology.

visit author page

author page

John Dunlosky

Download Paper |


Measuring Engineering Students’ Metacognition with a Think-Aloud Protocol

This evidence-based practice paper describes one method of measuring engineering students’ use of metacognition while problem solving. The capacity for life-long learning is critical for engineering practice and according to the Accreditation Board for Engineering and Technology (ABET) is a required outcome for engineering accreditation. Metacognition, defined as cognition of one’s cognitions, is a higher-order thinking skill and provides the key to developing life-long learning skills necessary for ABET and for an effective work career, yet it is rarely integrated into engineering education. At Iron Range Engineering (IRE), an innovative, problem-based-learning (PBL) engineering program in Virginia, Minnesota, students learn about and apply metacognitive skills. In our NSF-funded project, we are (1) identifying and understanding the metacognitive skills students develop and use during their preparation as engineers in a PBL program and (2) examining whether the preparation of students in the PBL program (particularly in the area of metacognition) gives them a “leg up” in their transition to the engineering workforce. Our mixed-methods study consists of student interview data supplemented with interviews of recent IRE graduates employed as engineers and of employers of the IRE graduates to better understand the effect of this unique undergraduate program on student preparedness for the engineering workforce. In addition, we are conducting think-aloud protocols during student solving of an open-ended design problem.

In this paper, we focus on the think-aloud data we have gathered during this project. We describe our process of gathering think-aloud data from students, as well as our coding structure, which categorizes utterances as metacognitive or not, and, if metacognitive, whether the utterance is a type of metacognitive monitoring or metacognitive control. In addition, we detail our coding process and measures of inter-rater reliability. Finally, we share results to date, including comparisons of the think-aloud data from students about to graduate from the program with students entering the program. The think-alouds from entering students and graduating students are showing marked differences. Our intent is to demonstrate the presence of critical metacognitive processes during problem solving as students progress through their engineering education.

Plumb, C., & Marra, R. M., & Hacker, D. J., & Dunlosky, J. (2018, June), Measuring Engineering Students’ Metacognition with a Think-Aloud Protocol Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--30796

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2018 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015