Paper Authors

Abstract

NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract

Developing a Method to Measure the Metacognitive Effects Of a Course on Design, Engineering and Technology over Time

Dale Baker, Senay Yasar, & Sharon Robinson Kurpius: College of Education Steve Krause & Chell Roberts: Ira A. Fulton School of Engineering Arizona State University

Abstract

Measuring and tracking how individuals become aware of their own understanding (metacognition) cannot easily be measured by traditional tests or assessments. Consequently, this paper presents the development and application of a rubric to examine qualitative data that illustrates how graduate students in science education, who were enrolled in a Design, Engineering and Technology (DET) course, became aware of the changes in their understanding of DET. Weekly reflection papers, weekly written pre and post tests and lesson plans were used as data sources. A rubric linking the course outcomes with six major categories (engineering as a design process, gender and diversity, societal relevance of engineering, technical self-efficacy, tinkering self-efficacy and transfer to classroom teaching) was developed to code text. Several passes through the data led to the refinements for the six categories that allowed the coding of almost all of the text. We specifically looked for shifts in understanding over a 15-week period and an awareness that these shifts were taking place (e.g. “It’s not that I had a bad attitude about technology to begin with, rather this class as a whole and our group project has forced me to think about its appropriate applications at the K-12 level.” and “Both technology education papers addressed the difference between technology education and educational technology – two different concepts I had not thought of before”). Our technique allowed us to capture the subtleties of understanding and the progression of metacognition. The rubric demonstrated that the DET course had a strong impact on students thinking about and applying DET to teaching.

Introduction

Quantitative approaches to assessment can tell you how much, how many or whether group A outperforms group B and provide descriptive statistics for a data set. However, quantitative analysis is unable to tell us why a group of students did well or poorly on an assessment. And, although quantitative techniques can be applied to data sets that are products or artifacts (e.g. a robot designed to perform specific tasks) qualitative techniques provide additional descriptive information as to the quality of the performance. Furthermore qualitative techniques, when used to evaluate large amounts of complex written materials, provide the flavor of the data by identifying salient quotations to support conclusions. Qualitative data, once analyzed with a rubric, can be reduced to numbers by assigning a numerical value to particular types of responses. However, doing so reduces the information available to a researcher.

Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition Copyright © 2004, American Society for Engineering Education

• Citation

• Format
  • APA
  • APA - LaTeX bibitem
  • MLA
  • MLA - LaTeX bibitem
  • Bibtex
  • EndNote - RIS

Krause, S. (2004, June), Developing A Method To Measure The Metacognitive Effects Paper presented at 2004 Annual Conference, Salt Lake City, Utah. 10.18260/1-2--13118