A Preliminary Exploration of the Role of Surveys In Student Reflection and Behavior

Amber Dale Levine; Tua A. Björklund; Shannon Katherine Gilmartin; Sheri Sheppard

Download Paper | Permalink

Conference: 2017 ASEE Annual Conference & Exposition
Location: Columbus, Ohio
Publication Date: June 24, 2017
Start Date: June 24, 2017
End Date: June 28, 2017
Conference Session: Reflection
Tagged Division: Educational Research and Methods
Page Count: 15
DOI: 10.18260/1-2--27500
Permanent URL: https://peer.asee.org/27500
Download Count: 616

Paper Authors

biography

Amber Dale Levine Stanford University

visit author page

Amber Levine is pursuing her Bachelors degree in Engineering with a focus in Architectural Design and a minor in Dance at Stanford University. She is particularly interested in education and inclusiveness in engineering.

visit author page

biography

Tua A. Björklund Aalto University Design Factory

visit author page

Dr. Björklund focuses on supporting idea development efforts in product design, entrepreneurship and teaching in higher education. She has been a part of creating the Aalto University Design Factory, an experimentation platform for students, teachers, researchers and practitioners in Finland. Currently Dr. Björklund is a visiting Fulbright scholar at Stanford University, working at the Center for Design Research and Scandinavian Consortium for Organisational Research as a Fulbright Finland - Technology Industries of Finland Centennial Foundation and Tutkijat Maailmalle - KAUTE Foundation grantee.

visit author page

biography

Shannon Katherine Gilmartin Stanford University orcid.org/0000-0001-8925-3271

visit author page

Shannon K. Gilmartin, Ph.D., is a Senior Research Scholar at the Michelle R. Clayman Institute for Gender Research and Adjunct Professor in Mechanical Engineering at Stanford University. She is also Managing Director of SKG Analysis, a research consulting firm. Her expertise and interests focus on education and workforce development in engineering and science fields. Previous and current clients include the American Chemical Society, the Anita Borg Institute for Women and Technology, California Institute of Technology, the College of Natural Sciences and Mathematics at California State University Fullerton, the Office of the Vice Provost for Graduate Education at Stanford University, the School of Medicine at Stanford University, and the School of Fisheries and Ocean Sciences at the University of Alaska, Fairbanks.

visit author page

biography

Sheri Sheppard Stanford University

visit author page

Sheri D. Sheppard, Ph.D., P.E., is professor of Mechanical Engineering at Stanford University. Besides teaching both undergraduate and graduate design and education related classes at Stanford University, she conducts research on engineering education and work-practices, and applied finite element analysis. From 1999-2008 she served as a Senior Scholar at the Carnegie Foundation for the Advancement of Teaching, leading the Foundation’s engineering study (as reported in Educating Engineers: Designing for the Future of the Field). In addition, in 2011 Dr. Sheppard was named as co-PI of a national NSF innovation center (Epicenter), and leads an NSF program at Stanford on summer research experiences for high school teachers. Her industry experiences includes engineering positions at Detroit's "Big Three:" Ford Motor Company, General Motors Corporation, and Chrysler Corporation.

At Stanford she has served a chair of the faculty senate, and recently served as Associate Vice Provost for Graduate Education.

visit author page

Download Paper | Permalink

Abstract

This research paper explores the premise that surveys are not neutral data collection instruments. Surveys are often used in educational research to gather information about respondents without considering the effect that answering the questions may have on the survey-takers themselves. However, during longitudinal studies, where the same subjects complete multiple surveys or interviews, a phenomenon called ‘panel conditioning’ has been observed in various fields including political science, public opinion and social science. ‘Panel conditioning’ refers to the participants’ experience taking surveys or interviews for a study influencing their responses to subsequent surveys or interviews for the study. The evidence on panel conditioning suggests that participating in surveys or interviews may have an effect not only on respondents’ subsequent responses, but also on their actual behavior.

In order to investigate the effect that surveys might have on participants, we included an optional reflection question at the end of the first wave of a national longitudinal engineering education survey (total respondent n=7197). As the survey was intended to gather information on both participants’ past academic experiences and their intentions for future actions, including their career plans in the near and distant future, it seemed ideally suited to fostering reflection. Here, reflection is defined as intentional thinking whereby an individual recalls and engages in meaning making around past experiences and considers the implications on future actions. Thus the optional final question asked respondents to what extent the survey inspired them to think about their education in new or different ways.

A clear majority out of the 2374 responses to the reflection question affirmed and described how participants were thinking differently after the survey, for example increased awareness of variety in employment opportunities, interest in campus resources, and musings on the engineering education system. Some respondents even reported explicit action goals for the future, ranging from a desire to spend more time thinking about topics the survey brought up, to immediate plans to take advantage of opportunities, to future-oriented goals such as developing skills and preparing for a career. Students with lower class standing and those identified as Under-Represented Minorities were significantly more likely to report new or different thoughts as well as make explicit goals.

Our results also suggest that there may be an association between reflection on the survey and participation in subsequent surveys. Participants who took the time to answer the final question substantively had higher odds of marking that they could be contacted for the second wave of the study, although their actual response rate was not significantly different from that of all other respondents. The current results suggest that including reflective questions on surveys may offer benefits to both survey participants and researchers, although more research is needed. The results also caution educators to be mindful of the impact that the questions in surveys can have on students, highlighting the importance of considering the alignment between educational goals and the operationalization of research goals.

Citation
Format

Levine, A. D., & Björklund, T. A., & Gilmartin, S. K., & Sheppard, S. (2017, June), A Preliminary Exploration of the Role of Surveys In Student Reflection and Behavior Paper presented at 2017 ASEE Annual Conference & Exposition, Columbus, Ohio. 10.18260/1-2--27500

TY  - CPAPER
AB  - This research paper explores the premise that surveys are not neutral data collection instruments. Surveys are often used in educational research to gather information about respondents without considering the effect that answering the questions may have on the survey-takers themselves. However, during longitudinal studies, where the same subjects complete multiple surveys or interviews, a phenomenon called ‘panel conditioning’ has been observed in various fields including political science, public opinion and social science. ‘Panel conditioning’ refers to the participants’ experience taking surveys or interviews for a study influencing their responses to subsequent surveys or interviews for the study. The evidence on panel conditioning suggests that participating in surveys or interviews may have an effect not only on respondents’ subsequent responses, but also on their actual behavior.

In order to investigate the effect that surveys might have on participants, we included an optional reflection question at the end of the first wave of a national longitudinal engineering education survey (total respondent n=7197). As the survey was intended to gather information on both participants’ past academic experiences and their intentions for future actions, including their career plans in the near and distant future, it seemed ideally suited to fostering reflection. Here, reflection is defined as intentional thinking whereby an individual recalls and engages in meaning making around past experiences and considers the implications on future actions. Thus the optional final question asked respondents to what extent the survey inspired them to think about their education in new or different ways.

A clear majority out of the 2374 responses to the reflection question affirmed and described how participants were thinking differently after the survey, for example increased awareness of variety in employment opportunities, interest in campus resources, and musings on the engineering education system. Some respondents even reported explicit action goals for the future, ranging from a desire to spend more time thinking about topics the survey brought up, to immediate plans to take advantage of opportunities, to future-oriented goals such as developing skills and preparing for a career. Students with lower class standing and those identified as Under-Represented Minorities were significantly more likely to report new or different thoughts as well as make explicit goals.

Our results also suggest that there may be an association between reflection on the survey and participation in subsequent surveys. Participants who took the time to answer the final question substantively had higher odds of marking that they could be contacted for the second wave of the study, although their actual response rate was not significantly different from that of all other respondents. The current results suggest that including reflective questions on surveys may offer benefits to both survey participants and researchers, although more research is needed. The results also caution educators to be mindful of the impact that the questions in surveys can have on students, highlighting the importance of considering the alignment between educational goals and the operationalization of research goals.

AU  - Amber Dale Levine
AU  - Tua A. Björklund
AU  - Shannon Katherine Gilmartin
AU  - Sheri Sheppard
CY  - Columbus, Ohio
DA  - 2017/06/24
PB  - ASEE Conferences
TI  - A Preliminary Exploration of the Role of Surveys In Student Reflection and Behavior
UR  - https://peer.asee.org/27500
DO  - 10.18260/1-2--27500
ER  -