Developing Direct Measures of Global Competence

Jennifer DeBoer; Glenda S Stump; Frances Carter-Johnson; Gayle Allen; Lori Breslow

Download Paper | Permalink

Conference: 2013 ASEE Annual Conference & Exposition
Location: Atlanta, Georgia
Publication Date: June 23, 2013
Start Date: June 23, 2013
End Date: June 26, 2013
ISSN: 2153-5965
Conference Session: Global Engineering Education Cross-Cultural Awareness and Social Impacts
Tagged Division: International
Page Count: 11
Page Numbers: 23.398.1 - 23.398.11
DOI: 10.18260/1-2--19412
Permanent URL: https://peer.asee.org/19412
Download Count: 529

Paper Authors

biography

Jennifer DeBoer MIT

visit author page

Jennifer DeBoer is currently a postdoctoral associate for education research at MIT's Teaching and Learning Laboratory. She completed her doctoral work at Vanderbilt University in international education policy studies, focusing on engineering student access, equity, and success, and she completed her bachelor's degrees in mechanical engineering and foreign languages and literatures at MIT. Her research interests include the use of technology in education in low-income contexts and the structure of engineering training for local capacity building, currently focusing on online learning.

visit author page

biography

Glenda S Stump Massachusetts Institute of Technology

visit author page

Dr. Stump is an Associate Director of Assessment and Evaluation in MIT's Teaching and Learning Laboratory. Her work includes assessment of student outcomes from teaching innovations, evaluation of larger programs within the institute, and educational research. Dr. Stump earned a PhD in Educational Psychology with a concentration in Learning along with a certificate in Educational Technology from Arizona State University. Her research interests encompass both the science of learning and its measurement, including understanding how factors such as students’ attitudes and beliefs relate to their choice and utilization of learning strategies, their metacognitive behaviors, and ultimately their learning outcomes.

visit author page

biography

Frances Carter-Johnson MIT, Teaching and Learning Laboratory

visit author page

Dr. Carter-Johnson is responsible for research and evaluating several undergraduate education initiatives at MIT in her role as a Postdoctoral Associate for Educational Research in the Teaching and Learning Laboratory. She completed her PhD in Public Policy with a concentration in evaluation and analytical methods from the University of Maryland Baltimore County. As a result of years of academic and research support from agencies such as the National Aeronautics and Space Administration (NASA), the National Science Foundation (NSF), the National Institutes of Health (NIH), and the U.S. Department of Education, she is also repeatedly invited to provide funding, proposal, and application preparation training for undergraduate and graduate students.

visit author page

author page

Gayle Allen Ed.D. BrightBytes

biography

Lori Breslow Massachusetts Institute of Technology

visit author page

Lori Breslow is the founding Director of the Teaching & Learning Laboratory (TLL) at the Massachusetts Institute of Technology. An internationally recognized expert in teaching and learning in higher education, she conducts research on the development, diffusion, and assessment of educational innovation, particularly in science and engineering. She is currently leading a joint MIT-Harvard research project to analyze data from edX’s first MOOC, “Circuits and Electronics.”

visit author page

Download Paper | Permalink

Abstract

Work in Progress: Developing Direct Measures of Global CompetenciesSession Topic Preferences: - preparing engineering students for international practice - definition of a global engineer - engineering education responses to global challenges/exposing engineering students to other culturesThe purpose of this study is to develop performance tasks and instruments to assess the globalcompetencies of college graduates more directly than is currently possible. This approach willallow us to determine more precisely the extent to which educational experiences, both locallyand abroad, contribute to students’ global competencies.As a first step, we defined global competency as “the ability to ‘work and communicateeffectively in national and international contexts.” We further described the construct as beingcomprised of intended learning outcomes, which are the skills that would describe a globallycompetent student. Finally, we identified performance indicators, or more specific behaviors thatwould be evident if students had mastered the skills. Adequate content coverage by theperformance indicators is supported in the literature as well as by other instruments that arecurrently used.We then proposed three performance tasks that each use different approaches to gatherinformation on some or all of the intended learning outcomes. Each of these tasks involves sometype of student participation in a scenario. Lastly, we are in the process of developing rubrics toscore student responses to those performance tasks.Two studies will follow rubric development to further the development of usable instruments.We will first pilot each performance task with its accompanying rubric on a sample of 10students in their final year of engineering coursework. We will examine inter-rater reliability ofthe scores produced by each rubric and the interpretation of the rubric’s stated levels ofcompetence by both the raters and the students.We will then conduct a larger study to gather validity evidence for the internal structure of theinstruments. This study will implement the instruments with 30 or more students. We willexamine internal consistency of ratings for performance indicators within each learning outcomeand the relationships between students’ scores on performance indicators that are not expected tobe related. We will compare students’ rubric scores (direct assessment) with their scores onanother established self-report instrument (indirect assessment) to provide further evidence tosupport the use of rubric scores as indication of global competence.

Citation
Format

DeBoer, J., & Stump, G. S., & Carter-Johnson, F., & Allen, G., & Breslow, L. (2013, June), Developing Direct Measures of Global Competence Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--19412

TY  - CPAPER
AB  -              Work in Progress: Developing Direct Measures of Global                              CompetenciesSession Topic Preferences:   -   preparing engineering students for international practice   -   definition of a global engineer   -   engineering education responses to global challenges/exposing engineering students to other       culturesThe purpose of this study is to develop performance tasks and instruments to assess the globalcompetencies of college graduates more directly than is currently possible. This approach willallow us to determine more precisely the extent to which educational experiences, both locallyand abroad, contribute to students’ global competencies.As a first step, we defined global competency as “the ability to ‘work and communicateeffectively in national and international contexts.” We further described the construct as beingcomprised of intended learning outcomes, which are the skills that would describe a globallycompetent student. Finally, we identified performance indicators, or more specific behaviors thatwould be evident if students had mastered the skills. Adequate content coverage by theperformance indicators is supported in the literature as well as by other instruments that arecurrently used.We then proposed three performance tasks that each use different approaches to gatherinformation on some or all of the intended learning outcomes. Each of these tasks involves sometype of student participation in a scenario. Lastly, we are in the process of developing rubrics toscore student responses to those performance tasks.Two studies will follow rubric development to further the development of usable instruments.We will first pilot each performance task with its accompanying rubric on a sample of 10students in their final year of engineering coursework. We will examine inter-rater reliability ofthe scores produced by each rubric and the interpretation of the rubric’s stated levels ofcompetence by both the raters and the students.We will then conduct a larger study to gather validity evidence for the internal structure of theinstruments. This study will implement the instruments with 30 or more students. We willexamine internal consistency of ratings for performance indicators within each learning outcomeand the relationships between students’ scores on performance indicators that are not expected tobe related. We will compare students’ rubric scores (direct assessment) with their scores onanother established self-report instrument (indirect assessment) to provide further evidence tosupport the use of rubric scores as indication of global competence.
AU  - Jennifer DeBoer
AU  - Glenda S Stump
AU  - Frances Carter-Johnson
AU  - Gayle Allen Ed.D.
AU  - Lori Breslow
CY  - Atlanta, Georgia
DA  - 2013/06/23
PB  - ASEE Conferences
TI  - Developing Direct Measures of Global Competence
UR  - https://peer.asee.org/19412
DO  - 10.18260/1-2--19412
ER  -