Developing a Program to Assist in Qualitative Data Analysis:  How Engineering Students’ Discuss Model Types

Kelsey Rodgers; Angela Thompson; Nicholas Hawkins; Matthew Verleger; Farshid Marbouti

Download Paper | Permalink

Conference: 2022 ASEE Annual Conference & Exposition
Location: Minneapolis, MN
Publication Date: August 23, 2022
Start Date: June 26, 2022
End Date: June 29, 2022
Conference Session: ERM: Exploring Educational Technology in Engineering
Page Count: 17
DOI: 10.18260/1-2--41032
Permanent URL: https://peer.asee.org/41032
Download Count: 291

Paper Authors

biography

Kelsey Rodgers Embry-Riddle Aeronautical University - Daytona Beach

visit author page

Dr. Kelsey Joy Rodgers is an engineer, engineering educator, and a researcher. She previously worked as an Assistant Professor and Visiting Research Scholar for Embry-Riddle Aeronautical University. She received her PhD in Engineering Education from Purdue University and her Bachelor's in Mechanical Engineering from Arizona State University - Polytechnic campus.

Follow her on YouTube: https://www.youtube.com/c/EngineeringwithDrKelseyJoy

visit author page

biography

Matthew Verleger Embry-Riddle Aeronautical University - Daytona Beach

visit author page

Matthew Verleger is a Professor of Engineering Fundamentals at Embry-Riddle Aeronautical University. He received his PhD in Engineering Education from Purdue University in 2010. His research interests include student use of models and modeling, flipped-classroom environments, development of educational software, and gamification of engineering courses.

visit author page

biography

Farshid Marbouti San Jose State University

visit author page

Farshid Marbouti is an Assistant Professor of General (interdisciplinary) Engineering at San Jose State University (SJSU). He is currently the chair of SJSU Senate Student Success Committee. Farshid completed his Ph.D. in Engineering Education at Purdue University. His research interests center on First-Year Engineering student success and engineering design.

visit author page

biography

Angela Thompson University of Louisville

visit author page

Associate Professor of Engineering Fundamentals and Director of Undergraduate Affairs, J. B. Speed School of Engineering, University of Louisville

visit author page

author page

Nicholas Hawkins University of Louisville

Download Paper | Permalink

Abstract

This Research paper discusses the opportunities that utilizing a computer program can present in analyzing large amounts of qualitative data collected through a survey tool. When working with longitudinal qualitative data, there are many challenges that researchers face. The coding scheme may evolve over time requiring re-coding of early data. There may be long periods of time between data analysis. Typically, multiple researchers will participate in the coding, but this may introduce bias or inconsistencies. Ideally the same researchers would be analyzing the data, but often there is some turnover in the team, particularly when students assist with the coding. Computer programs can enable automated or semi-automated coding helping to reduce errors and inconsistencies in the coded data.

In this study, a modeling survey was developed to assess student awareness of model types and administered in four first-year engineering courses across the three universities over the span of three years. The data collected from this survey consists of over 4,000 students’ open-ended responses to three questions about types of models in science, technology, engineering, and mathematics (STEM) fields. A coding scheme was developed to identify and categorize model types in student responses. Over two years, two undergraduate researchers analyzed a total of 1,829 students’ survey responses after ensuring intercoder reliability was greater than 80% for each model category. However, with much data remaining to be coded, the research team developed a MATLAB program to automatically implement the coding scheme and identify the types of models students discussed in their responses.

MATLAB coded results were compared to human-coded results (n = 1,829) to assess reliability; results matched between 81%-99% for the different model categories. Furthermore, the reliability of the MATLAB coded results are within the range of the interrater reliability measured between the 2 undergraduate researchers (86-100% for the five model categories).

With good reliability of the program, all 4,358 survey responses were coded; results showing the number and types of models identified by students are presented in the paper.

Keywords: first-year engineering, modeling, qualitative survey data, data analysis, MATLAB tool

Citation
Format

Rodgers, K., & Verleger, M., & Marbouti, F., & Thompson, A., & Hawkins, N. (2022, August), Developing a Program to Assist in Qualitative Data Analysis: How Engineering Students’ Discuss Model Types Paper presented at 2022 ASEE Annual Conference & Exposition, Minneapolis, MN. 10.18260/1-2--41032

TY - CPAPER
AB - This Research paper discusses the opportunities that utilizing a computer program can present in analyzing large amounts of qualitative data collected through a survey tool. When working with longitudinal qualitative data, there are many challenges that researchers face. The coding scheme may evolve over time requiring re-coding of early data. There may be long periods of time between data analysis. Typically, multiple researchers will participate in the coding, but this may introduce bias or inconsistencies. Ideally the same researchers would be analyzing the data, but often there is some turnover in the team, particularly when students assist with the coding. Computer programs can enable automated or semi-automated coding helping to reduce errors and inconsistencies in the coded data.

In this study, a modeling survey was developed to assess student awareness of model types and administered in four first-year engineering courses across the three universities over the span of three years. The data collected from this survey consists of over 4,000 students’ open-ended responses to three questions about types of models in science, technology, engineering, and mathematics (STEM) fields. A coding scheme was developed to identify and categorize model types in student responses. Over two years, two undergraduate researchers analyzed a total of 1,829 students’ survey responses after ensuring intercoder reliability was greater than 80% for each model category. However, with much data remaining to be coded, the research team developed a MATLAB program to automatically implement the coding scheme and identify the types of models students discussed in their responses.

MATLAB coded results were compared to human-coded results (n = 1,829) to assess reliability; results matched between 81%-99% for the different model categories. Furthermore, the reliability of the MATLAB coded results are within the range of the interrater reliability measured between the 2 undergraduate researchers (86-100% for the five model categories).

With good reliability of the program, all 4,358 survey responses were coded; results showing the number and types of models identified by students are presented in the paper.

Keywords: first-year engineering, modeling, qualitative survey data, data analysis, MATLAB tool
AU - Kelsey Rodgers
AU - Matthew Verleger
AU - Farshid Marbouti
AU - Angela Thompson
AU - Nicholas Hawkins
CY - Minneapolis, MN
DA - 2022/08/23
PB - ASEE Conferences
TI - Developing a Program to Assist in Qualitative Data Analysis: How Engineering Students’ Discuss Model Types
UR - https://peer.asee.org/41032
DO - 10.18260/1-2--41032
ER -