Asee peer logo

A Dialectic Data Integration Approach for Mixed Methods Survey Validation

Download Paper |

Conference

2015 ASEE Annual Conference & Exposition

Location

Seattle, Washington

Publication Date

June 14, 2015

Start Date

June 14, 2015

End Date

June 17, 2015

ISBN

978-0-692-50180-1

ISSN

2153-5965

Conference Session

Survey and Assessment Development

Tagged Division

Educational Research and Methods

Page Count

18

Page Numbers

26.35.1 - 26.35.18

DOI

10.18260/p.23376

Permanent URL

https://peer.asee.org/23376

Download Count

63

Request a correction

Paper Authors

biography

Nicholas D. Fila Purdue University

visit author page

Nicholas D. Fila is a Ph.D. candidate in the School of Engineering Education at Purdue University. He earned a B.S. in Electrical Engineering and a M.S. in Electrical and Computer Engineering from the University of Illinois at Urbana-Champaign. His current research interests include innovation, empathy, and engineering design.

visit author page

biography

Justin L Hess Purdue University, West Lafayette

visit author page

Justin Hess is a Ph.D. candidate at Purdue University's School of Engineering Education, Masters student in the School of Civil Engineering and a National Science Foundation Graduate Research Fellow. He received his Bachelor's of Science in Civil Engineering in 2011 with a minor in philosophy and anticipates receiving his MSCE in 2015, both from Purdue University. His research focuses on understanding engineers' core values, dispositions, and worldviews. His dissertation focuses on conceptualizations, the importance of, and methods to teach empathy to engineering students. He is currently the Education Director for Engineers for a Sustainable World and an assistant editor for Engineering Studies.

visit author page

biography

Senay Purzer Purdue University, West Lafayette Orcid 16x16 orcid.org/0000-0003-0784-6079

visit author page

Ṣenay Purzer is an Assistant Professor in the School of Engineering Education. She is the recipient of a 2012 NSF CAREER award, which examines how engineering students approach innovation. She serves on the editorial boards of Science Education and the Journal of Pre-College Engineering Education (JPEER). She received a B.S.E with distinction in Engineering in 2009 and a B.S. degree in Physics Education in 1999. Her M.A. and Ph.D. degrees are in Science Education from Arizona State University earned in 2002 and 2008, respectively.

visit author page

Download Paper |

Abstract

A Framework for Mixed Methods Survey ValidationQuantitative surveys are often used as tools to gauge student attitudes, knowledge, and behaviorsin engineering education contexts. Results from these surveys can be used to guide curricula, asformative assessment, to compare multiple groups, and for many other purposes. In order toensure that students scores, and any action taken as a result of those scores, are accurate andmeaningful in the implementation context, surveys often go through rigorous validationprocedures. Often these involve procedures such as confirmatory factor analysis to verify thefactor structure or comparison to results on previously validated measures. While these methodshave a longstanding track record, incorporation of qualitative data in a holistic mixed methodsmanner can improve understanding of the survey in the desired educational context. This paperwill present a methodological framework for incorporating both traditional quantitative data andqualitative data into the validation of a survey targeted at engineering students.First, we present the philosophical underpinnings of quantitative and qualitative validation anddiscuss connections that allow both traditions to be incorporated into the same survey validation.Briefly, both qualitative and quantitative validation share two common threads. First, bothemphasize thorough understanding of research participants’ context in order to make meaningfulinterpretation of results. Second, both are built upon utilizing a variety of data forms to develop acomprehensive understanding of phenomena in the survey-takers’ contexts.Stemming from these commonalities, we describe a procedure that incorporates both qualitativeand quantitative data into the evaluation of a Likert-type scale survey. We discuss the types ofdata that might be used in a mixed methods survey validation, with examples of this analysisusing authentic data. We then discuss how quantitative and qualitative data can be mixed to forma deeper understanding of the participants, their educational context, and how survey resultsmight be interpreted in that context among those participants. Analysis, in addition to occurringat quantitative and qualitative levels, also occurs at the item and structure levels. Item-levelanalyses are used to understand how item phrasing may affect student responses. Structure-levelanalyses are used to understand how participants connect to the primary constructs in the survey,and how/whether they are relevant in the participant context. Consideration of all data in unisonprovides a holistic understanding of the survey constructs in the participant contexts andimplications of using the survey as an assessment tool. We discuss tangible ways in whichdifferent types of data may be integrated and lead to more nuanced understanding of the surveyand its implications for the research context. We demonstrate this analysis using a previouslysurvey on a sample of 163 engineering students from a variety of disciplines and academiclevels. We expect that this paper will contribute to understanding of mixed methods research inengineering education as well as introducing a novel method of survey validation.

Fila, N. D., & Hess, J. L., & Purzer, S. (2015, June), A Dialectic Data Integration Approach for Mixed Methods Survey Validation Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.23376

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015