First Approach to Purposeful Sampling for Determining Key Factors on Outcome Bias

Mariana Tafur-Arciniegas; Andres Felipe Lara Contreras

Download Paper | Permalink

Conference: 2018 ASEE Annual Conference & Exposition
Location: Salt Lake City, Utah
Publication Date: June 23, 2018
Start Date: June 23, 2018
End Date: July 27, 2018
Conference Session: Assessment and Research Tools
Tagged Division: Educational Research and Methods
Page Count: 12
DOI: 10.18260/1-2--30518
Permanent URL: https://peer.asee.org/30518
Download Count: 1701

Paper Authors

biography

Mariana Tafur-Arciniegas P.E. Universidad de los Andes

visit author page

Mariana Tafur-Arciniegas is an assistant professor in the School of Education at University of Los Andes, Bogota-Colombia. She is a Ph.D. in Engineering Education from Purdue University, a M.S. in Education and a B.S. in Electrical Engineering from University of Los Andes. She is a 2010 Fulbright Fellow.
Her research interests include engineering skills development, STEM for non-engineers adults, motivation in STEM to close the technology literacy gap, STEM formative assessment, and Mixed-Methods design.

visit author page

biography

Andres Felipe Lara Contreras Universidad de los Andes

visit author page

Andres Lara is an undergraduate student in Chemical Engineering at Universidad de Los Andes, Bogotá, Colombia, Class of 2017-20. His research interests include engineering learning and approach to technological challenges, nanostructured materials, and self-sustainable energy sources.

visit author page

Download Paper | Permalink

Abstract

It is common, in educational research studies, to select purposeful population samples. Independent of the selected method, there is an associated bias to the sampling process because of the subjectivity of the person or group of people who accomplish the task. This paper shows the pilot of a protocol development for observing how people approach purposeful sampling using a real research problem. Engineers and non-engineers were observed completing the task and were monitored for collecting the differences between both points of view, and for identifying possible bias in the purposeful sample. Five stages were conducted for piloting the protocol for gathering information in how purposeful sampling is traditionally performed. Two participants with knowledge in educational research performed the activity for the first iteration of the pilot. In addition, four participants (two engineer students and two non-engineer students) performed a second pilot using an improved protocol. Finally, a panel of experts was asked to review the process for a final protocol. For the activity designed, participants were asked to review 118 profiles of people with diverse academic and social backgrounds. The goal was to choose 3 profiles for each of four categories, aiming to identify those participants who were the best representatives of each of the categories: 1. Engineers with a low level of Lifelong Learning (ELL), 2. Non-Engineer with a low level of Lifelong Learning (nELL); 3. Engineers with a high level of Lifelong Learning (EHL), and 4. Non-Engineer with a high level of Lifelong Learning (nEHL). The time for the assignment was limited to 90 minutes, and a think-aloud protocol was followed for data collection. The iterative process design for the protocol allowed to improve the resources, time management, and activity logistics for the Purposeful Sampling Activity. The time for the whole activity was calibrated and defined as 90 minutes. The time alert was changed from 10 to 15 minutes, and 10 minutes were defined to be given to those individuals who asked for more time. After participants’ feedback, new resources such as pens, sticky notes, eraser, water, and digital database with the information were included. The activity allowed to identify bias in sample selection due to lack of usage of the complete data set. Likewise, each expert defined different criteria for selection, setting diverse start points. This bias was induced by variables such as age, undergraduate background, expertise in a specific field of study and degree of development of specific skills through the professional life. In conclusion, this study showed the design of a protocol for collecting information about how non-intended bias was present in a purposeful sampling. This analysis may guide the process of purposeful selection of samples for qualitative research and provides a tool for measuring the bias reduction between traditional and statistical purposeful selection of information-rich cases.

Citation
Format

Tafur-Arciniegas, M., & Lara Contreras, A. F. (2018, June), First Approach to Purposeful Sampling for Determining Key Factors on Outcome Bias Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--30518

TY - CPAPER
AB - It is common, in educational research studies, to select purposeful population samples. Independent of the selected method, there is an associated bias to the sampling process because of the subjectivity of the person or group of people who accomplish the task. This paper shows the pilot of a protocol development for observing how people approach purposeful sampling using a real research problem. Engineers and non-engineers were observed completing the task and were monitored for collecting the differences between both points of view, and for identifying possible bias in the purposeful sample.
Five stages were conducted for piloting the protocol for gathering information in how purposeful sampling is traditionally performed. Two participants with knowledge in educational research performed the activity for the first iteration of the pilot. In addition, four participants (two engineer students and two non-engineer students) performed a second pilot using an improved protocol. Finally, a panel of experts was asked to review the process for a final protocol. For the activity designed, participants were asked to review 118 profiles of people with diverse academic and social backgrounds. The goal was to choose 3 profiles for each of four categories, aiming to identify those participants who were the best representatives of each of the categories: 1. Engineers with a low level of Lifelong Learning (ELL), 2. Non-Engineer with a low level of Lifelong Learning (nELL); 3. Engineers with a high level of Lifelong Learning (EHL), and 4. Non-Engineer with a high level of Lifelong Learning (nEHL). The time for the assignment was limited to 90 minutes, and a think-aloud protocol was followed for data collection.
The iterative process design for the protocol allowed to improve the resources, time management, and activity logistics for the Purposeful Sampling Activity. The time for the whole activity was calibrated and defined as 90 minutes. The time alert was changed from 10 to 15 minutes, and 10 minutes were defined to be given to those individuals who asked for more time. After participants’ feedback, new resources such as pens, sticky notes, eraser, water, and digital database with the information were included. The activity allowed to identify bias in sample selection due to lack of usage of the complete data set. Likewise, each expert defined different criteria for selection, setting diverse start points. This bias was induced by variables such as age, undergraduate background, expertise in a specific field of study and degree of development of specific skills through the professional life.
In conclusion, this study showed the design of a protocol for collecting information about how non-intended bias was present in a purposeful sampling. This analysis may guide the process of purposeful selection of samples for qualitative research and provides a tool for measuring the bias reduction between traditional and statistical purposeful selection of information-rich cases.
AU - Mariana Tafur-Arciniegas P.E.
AU - Andres Felipe Lara Contreras
CY - Salt Lake City, Utah
DA - 2018/06/23
PB - ASEE Conferences
TI - First Approach to Purposeful Sampling for Determining Key Factors on Outcome Bias
UR - https://peer.asee.org/30518
DO - 10.18260/1-2--30518
ER -