June 14, 2015
June 14, 2015
June 17, 2015
Educational Research and Methods
26.35.1 - 26.35.18
A Framework for Mixed Methods Survey ValidationQuantitative surveys are often used as tools to gauge student attitudes, knowledge, and behaviorsin engineering education contexts. Results from these surveys can be used to guide curricula, asformative assessment, to compare multiple groups, and for many other purposes. In order toensure that students scores, and any action taken as a result of those scores, are accurate andmeaningful in the implementation context, surveys often go through rigorous validationprocedures. Often these involve procedures such as confirmatory factor analysis to verify thefactor structure or comparison to results on previously validated measures. While these methodshave a longstanding track record, incorporation of qualitative data in a holistic mixed methodsmanner can improve understanding of the survey in the desired educational context. This paperwill present a methodological framework for incorporating both traditional quantitative data andqualitative data into the validation of a survey targeted at engineering students.First, we present the philosophical underpinnings of quantitative and qualitative validation anddiscuss connections that allow both traditions to be incorporated into the same survey validation.Briefly, both qualitative and quantitative validation share two common threads. First, bothemphasize thorough understanding of research participants’ context in order to make meaningfulinterpretation of results. Second, both are built upon utilizing a variety of data forms to develop acomprehensive understanding of phenomena in the survey-takers’ contexts.Stemming from these commonalities, we describe a procedure that incorporates both qualitativeand quantitative data into the evaluation of a Likert-type scale survey. We discuss the types ofdata that might be used in a mixed methods survey validation, with examples of this analysisusing authentic data. We then discuss how quantitative and qualitative data can be mixed to forma deeper understanding of the participants, their educational context, and how survey resultsmight be interpreted in that context among those participants. Analysis, in addition to occurringat quantitative and qualitative levels, also occurs at the item and structure levels. Item-levelanalyses are used to understand how item phrasing may affect student responses. Structure-levelanalyses are used to understand how participants connect to the primary constructs in the survey,and how/whether they are relevant in the participant context. Consideration of all data in unisonprovides a holistic understanding of the survey constructs in the participant contexts andimplications of using the survey as an assessment tool. We discuss tangible ways in whichdifferent types of data may be integrated and lead to more nuanced understanding of the surveyand its implications for the research context. We demonstrate this analysis using a previouslysurvey on a sample of 163 engineering students from a variety of disciplines and academiclevels. We expect that this paper will contribute to understanding of mixed methods research inengineering education as well as introducing a novel method of survey validation.
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015