Asee peer logo

Developing Direct Measures of Global Competence

Download Paper |


2013 ASEE Annual Conference & Exposition


Atlanta, Georgia

Publication Date

June 23, 2013

Start Date

June 23, 2013

End Date

June 26, 2013



Conference Session

Global Engineering Education Cross-Cultural Awareness and Social Impacts

Tagged Division


Page Count


Page Numbers

23.398.1 - 23.398.11



Permanent URL

Download Count


Request a correction

Paper Authors


Jennifer DeBoer MIT

visit author page

Jennifer DeBoer is currently a postdoctoral associate for education research at MIT's Teaching and Learning Laboratory. She completed her doctoral work at Vanderbilt University in international education policy studies, focusing on engineering student access, equity, and success, and she completed her bachelor's degrees in mechanical engineering and foreign languages and literatures at MIT. Her research interests include the use of technology in education in low-income contexts and the structure of engineering training for local capacity building, currently focusing on online learning.

visit author page


Glenda S Stump Massachusetts Institute of Technology

visit author page

Dr. Stump is an Associate Director of Assessment and Evaluation in MIT's Teaching and Learning Laboratory. Her work includes assessment of student outcomes from teaching innovations, evaluation of larger programs within the institute, and educational research. Dr. Stump earned a PhD in Educational Psychology with a concentration in Learning along with a certificate in Educational Technology from Arizona State University. Her research interests encompass both the science of learning and its measurement, including understanding how factors such as students’ attitudes and beliefs relate to their choice and utilization of learning strategies, their metacognitive behaviors, and ultimately their learning outcomes.

visit author page


Frances Carter-Johnson MIT, Teaching and Learning Laboratory

visit author page

Dr. Carter-Johnson is responsible for research and evaluating several undergraduate education initiatives at MIT in her role as a Postdoctoral Associate for Educational Research in the Teaching and Learning Laboratory. She completed her PhD in Public Policy with a concentration in evaluation and analytical methods from the University of Maryland Baltimore County. As a result of years of academic and research support from agencies such as the National Aeronautics and Space Administration (NASA), the National Science Foundation (NSF), the National Institutes of Health (NIH), and the U.S. Department of Education, she is also repeatedly invited to provide funding, proposal, and application preparation training for undergraduate and graduate students.

visit author page

author page

Gayle Allen Ed.D. BrightBytes


Lori Breslow Massachusetts Institute of Technology

visit author page

Lori Breslow is the founding Director of the Teaching & Learning Laboratory (TLL) at the Massachusetts Institute of Technology. An internationally recognized expert in teaching and learning in higher education, she conducts research on the development, diffusion, and assessment of educational innovation, particularly in science and engineering. She is currently leading a joint MIT-Harvard research project to analyze data from edX’s first MOOC, “Circuits and Electronics.”

visit author page

Download Paper |


Work in Progress: Developing Direct Measures of Global CompetenciesSession Topic Preferences: - preparing engineering students for international practice - definition of a global engineer - engineering education responses to global challenges/exposing engineering students to other culturesThe purpose of this study is to develop performance tasks and instruments to assess the globalcompetencies of college graduates more directly than is currently possible. This approach willallow us to determine more precisely the extent to which educational experiences, both locallyand abroad, contribute to students’ global competencies.As a first step, we defined global competency as “the ability to ‘work and communicateeffectively in national and international contexts.” We further described the construct as beingcomprised of intended learning outcomes, which are the skills that would describe a globallycompetent student. Finally, we identified performance indicators, or more specific behaviors thatwould be evident if students had mastered the skills. Adequate content coverage by theperformance indicators is supported in the literature as well as by other instruments that arecurrently used.We then proposed three performance tasks that each use different approaches to gatherinformation on some or all of the intended learning outcomes. Each of these tasks involves sometype of student participation in a scenario. Lastly, we are in the process of developing rubrics toscore student responses to those performance tasks.Two studies will follow rubric development to further the development of usable instruments.We will first pilot each performance task with its accompanying rubric on a sample of 10students in their final year of engineering coursework. We will examine inter-rater reliability ofthe scores produced by each rubric and the interpretation of the rubric’s stated levels ofcompetence by both the raters and the students.We will then conduct a larger study to gather validity evidence for the internal structure of theinstruments. This study will implement the instruments with 30 or more students. We willexamine internal consistency of ratings for performance indicators within each learning outcomeand the relationships between students’ scores on performance indicators that are not expected tobe related. We will compare students’ rubric scores (direct assessment) with their scores onanother established self-report instrument (indirect assessment) to provide further evidence tosupport the use of rubric scores as indication of global competence.

DeBoer, J., & Stump, G. S., & Carter-Johnson, F., & Allen, G., & Breslow, L. (2013, June), Developing Direct Measures of Global Competence Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--19412

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2013 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015