Board 301: Impacts of the ProQual Institute: Summative Evaluation of Participant Skills, Perceptions, Confidence, and Research Products from a Qualitative Research Institute

John Ray Morelock; Aileen Reid; Ayesha Sherita Sherita Boyce; Chaturved Janaki; Nicola W. Sochacka; Joachim Walther

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: June 26, 2024
Conference Session: NSF Grantees Poster Session
Tagged Topics: Diversity and NSF Grantees Poster Session
Page Count: 16
DOI: 10.18260/1-2--46879
Permanent URL: https://peer.asee.org/46879
Download Count: 145

Paper Authors

biography

John Ray Morelock University of Georgia orcid.org/0000-0001-8043-5060

visit author page

Dr. Morelock is an Assistant Professor of Practice with an emphasis on engineering education research, and the Associate Director of Educational Innovation and Impact for UGA's Engineering Education Transformations Institute (EETI). In addition to coordinating EETI’s faculty development programming, Dr. Morelock conducts research on institutional change via faculty development, with an emphasis on innovative ways to cultivate and evaluate supportive teaching and learning networks in engineering departments and colleges. He received his doctoral degree in Engineering Education at Virginia Tech, where he was a recipient of the NSF Graduate Research Fellowship. His dissertation studied the teaching practices of engineering instructors during game-based learning activities, and how these practices affected student motivation.

visit author page

biography

Aileen Reid University of North Carolina, Greensboro orcid.org/0000-0002-8099-1435

visit author page

Dr. Reid’s research applies culturally responsive frameworks to evaluation and measurement to shape educational research and policy and respond to societal challenges. This work is centered on social justice, access, diversity, equity, and inclusion and challenges researchers and practitioners to look at our own cultural location as well as key dimensions of practice to incite change (e.g., context, relationships, validity, methodology, and design). Within this line of inquiry, Dr. Reid investigates inequities in outcomes among underrepresented and minoritized groups, specifically how the intersection of identities and practice impact engagement and responsiveness to varying contexts and cultures. She also investigates values-engaged teaching, learning, and mentorship in research and evaluation. Dr. Reid presents and publishes her work in program evaluation, education, and the STEM disciplines.

visit author page

author page

Dr. Nicola Sochacka is the director of the ProQual Institute for Interpretive Research Methods and a research scientist in the Engineering Education Transformations Institute (EETI) within the College of Engineering at the University of Georgia. She co-developed the ProQual approach, which she uses to build capacity and community around interpretive educational research methods among STEM faculty and students.

visit author page

biography

Joachim Walther University of Georgia

visit author page

Dr. Joachim Walther is a Professor of engineering education research at the University of Georgia and the Founding Director of the Engineering Education Transformations Institute (EETI) in the College of Engineering. He is dedicated to lowering the barriers for STEM faculty and students to conduct high-quality, interpretive, educational research. He co-developed the ProQual approach, a method that is designed to demystify interpretive research for faculty and students from technical backgrounds.

visit author page

Download Paper | Permalink

Abstract

Building on work presented at ASEE 2023, this NSF Grantees Poster Session paper reports on the impacts of the [NSF Project]—a $1M NSF award via the NSF ECR-EHR Core Research program in 2019—as it nears the end of its final year. The [NSF Project] aims to build national capacity for STEM education research by engaging technical STEM from across the U.S. in cohorts that participate in a semester-long course on qualitative and mixed methods educational research techniques. Faculty from underrepresented backgrounds and Minority-Serving Institutions were given priority consideration in terms of recruitment and admission to participate in the project. This project was funded based on impact rather than knowledge generation; thus, this paper will report on the [NSF Projects]’s summative outcomes and impact revealed through external evaluation.

Using the Qualifying Qualitative Research Quality (Q3) framework pioneered by Dr. Joachim Walther and colleagues as a foundation, the project team guided three cohorts of faculty (48 faculty total) in designing qualitative or mixed methods studies to address research questions they wanted to answer about their educational contexts. The project team has also hosted four follow-up research incubators (each one semester long) as spaces to allow graduates of the [program] to continue working together as a community to continue developing educational research projects to completion or apply for extramural funding opportunities (serving 27 faculty total.) Furthermore, the team has funded graduates of the [program] to lead communities of practice focused on areas of shared research interest among graduates. Finally, in the past year, the team has funded professional grant proposal coaching for interested graduates and has worked with graduates to design and execute workshops on the [program’s] approach to qualitative research design.

The evaluation results presented in this paper derive from a primarily quantitative, mixed methods survey conducted by the project’s external evaluators. The survey aimed to gauge participants' engagement with [program] activities; their knowledge, perceptions of, and comfort with qualitative and mixed methods; assessment of training outcomes and curriculum; the influence of [program] training on research dissemination; and self-reported behavioral changes before and after training. Results indicate substantial self-reported improvement in knowledge of qualitative research methods, as well as application of those methods and confidence to do so. Many participants have gone on to pursue funded research projects using qualitative and mixed methods research approaches, and many have reported increased capacity to collaborate on and disseminate research projects successfully. Constructively, the results point to a need to support nascent qualitative researchers in attending educational research conferences and integrating into these conferences’ communities. Participants also indicated a desire for more practical experience with real-world qualitative research scenarios, pointing to a need for further hands-on professional development opportunities. Overall, the survey paints a picture of a successful program that nearly all respondents reported as being worthwhile and a desire from participants to continue seeking opportunities to grow as qualitative and mixed methods researchers.

Citation
Format

Morelock, J. R., & Reid, A., & Boyce, A. S. S., & Janaki, C., & Sochacka, N. W., & Walther, J. (2024, June), Board 301: Impacts of the ProQual Institute: Summative Evaluation of Participant Skills, Perceptions, Confidence, and Research Products from a Qualitative Research Institute Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. 10.18260/1-2--46879

TY  - CPAPER
AB  - Building on work presented at ASEE 2023, this NSF Grantees Poster Session paper reports on the impacts of the [NSF Project]—a $1M NSF award via the NSF ECR-EHR Core Research program in 2019—as it nears the end of its final year.  The [NSF Project] aims to build national capacity for STEM education research by engaging technical STEM from across the U.S. in cohorts that participate in a semester-long course on qualitative and mixed methods educational research techniques.  Faculty from underrepresented backgrounds and Minority-Serving Institutions were given priority consideration in terms of recruitment and admission to participate in the project.  This project was funded based on impact rather than knowledge generation; thus, this paper will report on the [NSF Projects]’s summative outcomes and impact revealed through external evaluation.

Using the Qualifying Qualitative Research Quality (Q3) framework pioneered by Dr. Joachim Walther and colleagues as a foundation, the project team guided three cohorts of faculty (48 faculty total) in designing qualitative or mixed methods studies to address research questions they wanted to answer about their educational contexts.  The project team has also hosted four follow-up research incubators (each one semester long) as spaces to allow graduates of the [program] to continue working together as a community to continue developing educational research projects to completion or apply for extramural funding opportunities (serving 27 faculty total.)  Furthermore, the team has funded graduates of the [program] to lead communities of practice focused on areas of shared research interest among graduates.  Finally, in the past year, the team has funded professional grant proposal coaching for interested graduates and has worked with graduates to design and execute workshops on the [program’s] approach to qualitative research design.

The evaluation results presented in this paper derive from a primarily quantitative, mixed methods survey conducted by the project’s external evaluators.  The survey aimed to gauge participants&#39; engagement with [program] activities; their knowledge, perceptions of, and comfort with qualitative and mixed methods; assessment of training outcomes and curriculum; the influence of [program] training on research dissemination; and self-reported behavioral changes before and after training.  Results indicate substantial self-reported improvement in knowledge of qualitative research methods, as well as application of those methods and confidence to do so.  Many participants have gone on to pursue funded research projects using qualitative and mixed methods research approaches, and many have reported increased capacity to collaborate on and disseminate research projects successfully.  Constructively, the results point to a need to support nascent qualitative researchers in attending educational research conferences and integrating into these conferences’ communities.  Participants also indicated a desire for more practical experience with real-world qualitative research scenarios, pointing to a need for further hands-on professional development opportunities.  Overall, the survey paints a picture of a successful program that nearly all respondents reported as being worthwhile and a desire from participants to continue seeking opportunities to grow as qualitative and mixed methods researchers.
AU  - John Ray Morelock
AU  - Aileen Reid
AU  - Ayesha Sherita Sherita Boyce
AU  - Chaturved Janaki
AU  - Nicola W. Sochacka
AU  - Joachim Walther
CY  - Portland, Oregon
DA  - 2024/06/23
PB  - ASEE Conferences
TI  - Board 301: Impacts of the ProQual Institute: Summative Evaluation of Participant Skills, Perceptions, Confidence, and Research Products from a Qualitative Research Institute
UR  - https://peer.asee.org/46879
DO  - 10.18260/1-2--46879
ER  -