Adaptive Expertise: The Development of a Measurement Instrument

Janna H. Ferguson; Jennifer Lehmann; Yevgeniya V. Zastavker; Susan Chang; Reid P. Higginson; Cigdem P. Talgar

Download Paper | Permalink

Conference: 2018 ASEE Annual Conference & Exposition
Location: Salt Lake City, Utah
Publication Date: June 23, 2018
Start Date: June 23, 2018
End Date: July 27, 2018
Conference Session: Problem Solving, Adaptive Expertise, and Social Engagement
Tagged Division: Educational Research and Methods
Tagged Topic: Diversity
Page Count: 19
DOI: 10.18260/1-2--29752
Permanent URL: https://peer.asee.org/29752
Download Count: 2437

Request a correction

Paper Authors

biography

Janna H. Ferguson Northeastern University

visit author page

Dr. Ferguson designs assessments and analyzes data related to student learning and its relevance to student success. Focusing on how experiential learning and co-curricular education works in conjunction with traditional academic environments, Dr. Ferguson works to develop, plan, implement, and evaluate meaningful assessments across multiple learning environments and provides support for projects related to institutional assessment.

visit author page

author page

Jennifer Lehmann Northeastern University

biography

Yevgeniya V. Zastavker Franklin W. Olin College of Engineering

visit author page

Yevgeniya V. Zastavker, Ph.D., is an Associate Professor of Physics at Franklin W. Olin College of Engineering. She earned her B.S. degree in Physics from Yale University in 1995 and her Ph. D. degree in Biological Physics from MIT in 2001. Dr. Zastavker's research interests lie in the field of STEM education with specific emphasis on innovative pedagogical and curricular practices at the intersection with the issues of gender and diversity. Dr. Zastavker is currently working with Dr. Stolk on an NSF-supported project to understand students’ motivational attitudes in a variety of educational environments with the goal of improving learning opportunities for students and equipping faculty with the knowledge and skills necessary to create such opportunities. One of the founding faculty at Olin College, Dr. Zastavker has been engaged in development and implementation of project-based experiences in fields ranging from science to engineering and design to social sciences (e.g., Critical Reflective Writing; Teaching and Learning in Undergraduate Science and Engineering, etc.) All of these activities share a common goal of creating curricular and pedagogical structures as well as academic cultures that facilitate students' interests, motivation, and desire to persist in engineering. Through this work, outreach, and involvement in the community, Dr. Zastavker continues to focus on the issues of women and minorities in science/engineering.

visit author page

author page

Susan Chang Northeastern University

biography

Reid P. Higginson Harvard University

visit author page

Reid P. Higginson is a PhD Candidate in Education at Harvard University.

visit author page

biography

Cigdem P. Talgar Northeastern University

visit author page

As Associate Vice Provost for Teaching and Learning at Northeastern University, Dr. Talgar oversees the Center for Advancing Teaching and Learning Through Research, the Research Institute for Experiential Learning Science, the The Self-Authored Integrated Learning, and the Academic Assessment Group in addition to leading special projects out of the Office of the Senior Vice Provost for Undergraduate Education and Experiential Education. Her work is focused on integrating the teaching and learning landscape at Northeastern University for educators and learners alike to ensure that the University fully integrates its signature experiential learning model to all facets of University life.

Dr. Talgar joined Northeastern University in 2013 to establish the Center for Advancing Teaching and Learning through Research at Northeastern University. As its Founding Director, she successfully created a Center that offered numerous faculty development programs to engage Northeastern educators in effective and impactful educational practices. With a focus on evidence-based teaching, Dr. Talgar initiated numerous institutional programs, including the Conference for Advancing Evidence-Based Teaching and the Faculty Scholars Cohort Programs. Today, the Center is recognized as a leader in evidence-based applied experiential learning science and works closely with the Research Institute for Experiential Learning Science to contribute to research on the mechanisms underlying experiential learning.

visit author page

Download Paper | Permalink

Abstract

Developing adaptive expertise in engineering students is of increasing importance as we strive to educate students who can respond to changes in the field by flexibly applying their theoretical knowledge and prior experience to new situations [Peng et al., 2014; Cantor et al., 2015; Bruenger, 2015]. To understand the extent to which different curricular and educational practices meet this aim, a means of measuring this complex skillset is necessary. In this paper, we introduce a revised Adaptive Expertise survey, building on Fisher and Peterson’s (2001) 42-item scale. Our 20-item instrument reflects advances in adaptive expertise research from both the learning sciences and human resources, literatures and has been developed through cognitive interviews, pilot testing, and exploratory factor analysis. Our revision of Fisher and Peterson’s 2001 scale followed our use of that instrument on a sample of 4,704 undergraduate students as a pre- and post-test measure to understand the effect of reflective writing exercises on their development of adaptive expertise. A Cronbach’s alpha of the scale produced reliability scores for the test’s sub-scales in the poor-acceptable range (multiple perspectives, 0.68; meta-cognition, 0.68; goals and beliefs, 0.73; epistemology, 0.5) placing them in the poor-acceptable range [Kline, 2000; Tavakol & Dennick, 2011]. A factor analysis failed to replicate the factor structure outlined by Fisher and Peterson. Using current theorizing and research on adaptive expertise, we designed a revised scale with items addressing innovative skills, domain skills, metacognition [Bohle Carbonell et al., 2016], self-efficacy, and resilience. The revised scale included modified items from Fisher and Peterson’s 2001 survey, as well as additional items of our own construction, and several items based on work by van der Heijden (2000), Charbonnier-Voiirin et al. (2012), and Bohle Carbonell et al. (2016). The first step of the process of refining the scale was to conduct nine cognitive interviews with undergraduate students who were currently on co-operative education placements. These interviews were then used to clarify the wording of scale items, reduce duplication, and remove or refine items which were interpreted ambiguously by students, resulting in a 39-item scale. We piloted that revised scale with 347 undergraduate students who were in co-operative education placements. Following an exploratory factor analysis which produced a four-factor model with strong face validity and significant goodness-of-fit for a scale including 20 items, we introduced the survey into a longitudinal research project on co-operative education. We are currently using this survey to collect pre- and post-test responses from 1,009 undergraduate students in all majors who are in co-operative education placements. Once this data has been collected, we will be able to provide additional insights into the reliability and validity of this new scale through additional factor analysis, including a confirmatory factor analysis. We also plan to present an analysis of how a students’ year of study, whether the student is an engineering major or not, the number of cooperative education placements, and gender may affect survey responses on this revised scale.

Citation
Format

Ferguson, J. H., & Lehmann, J., & Zastavker, Y. V., & Chang, S., & Higginson, R. P., & Talgar, C. P. (2018, June), Adaptive Expertise: The Development of a Measurement Instrument Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--29752

TY - CPAPER
AB - Developing adaptive expertise in engineering students is of increasing importance as we strive to educate students who can respond to changes in the field by flexibly applying their theoretical knowledge and prior experience to new situations [Peng et al., 2014; Cantor et al., 2015; Bruenger, 2015]. To understand the extent to which different curricular and educational practices meet this aim, a means of measuring this complex skillset is necessary. In this paper, we introduce a revised Adaptive Expertise survey, building on Fisher and Peterson’s (2001) 42-item scale. Our 20-item instrument reflects advances in adaptive expertise research from both the learning sciences and human resources, literatures and has been developed through cognitive interviews, pilot testing, and exploratory factor analysis.
Our revision of Fisher and Peterson’s 2001 scale followed our use of that instrument on a sample of 4,704 undergraduate students as a pre- and post-test measure to understand the effect of reflective writing exercises on their development of adaptive expertise. A Cronbach’s alpha of the scale produced reliability scores for the test’s sub-scales in the poor-acceptable range (multiple perspectives, 0.68; meta-cognition, 0.68; goals and beliefs, 0.73; epistemology, 0.5) placing them in the poor-acceptable range [Kline, 2000; Tavakol &amp; Dennick, 2011]. A factor analysis failed to replicate the factor structure outlined by Fisher and Peterson.
Using current theorizing and research on adaptive expertise, we designed a revised scale with items addressing innovative skills, domain skills, metacognition [Bohle Carbonell et al., 2016], self-efficacy, and resilience. The revised scale included modified items from Fisher and Peterson’s 2001 survey, as well as additional items of our own construction, and several items based on work by van der Heijden (2000), Charbonnier-Voiirin et al. (2012), and Bohle Carbonell et al. (2016).
The first step of the process of refining the scale was to conduct nine cognitive interviews with undergraduate students who were currently on co-operative education placements. These interviews were then used to clarify the wording of scale items, reduce duplication, and remove or refine items which were interpreted ambiguously by students, resulting in a 39-item scale.
We piloted that revised scale with 347 undergraduate students who were in co-operative education placements. Following an exploratory factor analysis which produced a four-factor model with strong face validity and significant goodness-of-fit for a scale including 20 items, we introduced the survey into a longitudinal research project on co-operative education. We are currently using this survey to collect pre- and post-test responses from 1,009 undergraduate students in all majors who are in co-operative education placements. Once this data has been collected, we will be able to provide additional insights into the reliability and validity of this new scale through additional factor analysis, including a confirmatory factor analysis. We also plan to present an analysis of how a students’ year of study, whether the student is an engineering major or not, the number of cooperative education placements, and gender may affect survey responses on this revised scale.

AU - Janna H. Ferguson
AU - Jennifer Lehmann
AU - Yevgeniya V. Zastavker
AU - Susan Chang
AU - Reid P. Higginson
AU - Cigdem P. Talgar
CY - Salt Lake City, Utah
DA - 2018/06/23
PB - ASEE Conferences
TI - Adaptive Expertise: The Development of a Measurement Instrument
UR - https://peer.asee.org/29752
DO - 10.18260/1-2--29752
ER -