Work in Progress – Development and Validation of the Ambassador Questionnaire

Melissa G. Kuhn; Joanna K. Garner; Shanan Chappell

Download Paper | Permalink

Conference: 2019 ASEE Annual Conference & Exposition
Location: Tampa, Florida
Publication Date: June 15, 2019
Start Date: June 15, 2019
End Date: June 19, 2019
Conference Session: ERM Technical Session 8: Survey and Instrument Development
Tagged Division: Educational Research and Methods
Page Count: 7
DOI: 10.18260/1-2--33571
Permanent URL: https://peer.asee.org/33571
Download Count: 461

Paper Authors

biography

Melissa G. Kuhn Old Dominion University

visit author page

Melissa G. Kuhn is a PhD Student in Educational Psychology and Program Evaluation at Old Dominion University. Additionally, she works at the Batten College of Engineering and Technology in educational projects and program coordination.

visit author page

biography

Joanna K. Garner Old Dominion University

visit author page

Dr. Garner is Executive Director of The Center for Educational Partnerships at Old Dominion University, VA.

visit author page

author page

Shanan Chappell Old Dominion University

Download Paper | Permalink

Abstract

Motivation and Background:

This Work in Progress describes research to develop a measure of the impact of engineering outreach participation on undergraduate students. Studies have suggested that this common co-curricular activity may play a role in students’ professional preparation, but more research, including precise measurement tools, is needed. Our objectives were to (1) examine existing questionnaires measuring constructs of relevance including academic confidence, engineering career motivation, engineering beliefs, professional skills, and interpersonal skills; and (2) create and test a measure that can be used to assess the potential impact of outreach activities on students’ professional perceptions and aspirations.

Methods:

A survey of 30 outreach programs in the U.S. revealed that less than 10% routinely assess the impact of ambassadorship. Follow-up interviews and focus groups indicated that program leaders were interested in a measure.

A review of literature revealed existing sources including the Longitudinal Assessment of Engineering Self-Efficacy (LAESE), the Project to Assess Climate in Engineering (PACE) survey, the Laanan Transfer Students Questionnaire (L-TSQ), the National Academy of Engineering Changing the Conversation survey, and ambassador chapter exit interviews. Items were grouped and rated for relevance and uniqueness by subject matter experts. Highly-rated items were then included in an online Ambassador Questionnaire (AQ), which was administered to 168 outreach ambassadors from 11 colleges/universities. The sample was primarily female (67%) and white (69%), with most respondents in their first year as ambassadors (51%). We conducted two phases of exploratory factor analysis (EFA), and calculated internal consistency reliability via Cronbach’s alpha coefficients.

Results:

Diagnostic procedures indicated the removal of four items with excessive low intercorrelations and six items that failed to load on any factor at or above .40. The Kaiser-Meyer-Olkin value was .83, indicating sufficient sampling adequacy. Bartlett’s test of sphericity (p>.001) revealed that the correlations among the 47 retained items were suitable for EFA.

The EFA initially extracted six factors and explained 54% of the variance. We retained theorized subscales of Academic Confidence, Engineering Beliefs, Professional Skills, and Interpersonal Skills. The Engineering Career Motivation subscale did not hold, with some items loading on Academic Confidence and some items breaking into their own factor, which we labeled Ambassadorship. Two items focusing on equity in engineering careers loaded on a separate factor. Cronbach’s alpha coefficients suggested strong internal consistency, with a whole scale coefficient of .93 and subscale coefficients ranging from .81 to .90.

A follow-up EFA specified five factors to align with theory. This solution explained 50% of the variance, and moved the two equity items to the Ambassadorship subscale. Descriptive analyses revealed sensitivity to group differences. For example, female respondents were less likely to expect fair and equitable treatment in their future workplace than males (p<0.001).

Significance:

Our preliminary findings include a robust structure to the measure and the potential to investigate group- and time-related differences in students’ perceptions and experiences. After further validation with a larger sample, we plan to make the measure available to researchers and practitioners.

Citation
Format

Kuhn, M. G., & Garner, J. K., & Chappell, S. (2019, June), Work in Progress – Development and Validation of the Ambassador Questionnaire Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. 10.18260/1-2--33571

TY - CPAPER
AB - Motivation and Background:

This Work in Progress describes research to develop a measure of the impact of engineering outreach participation on undergraduate students. Studies have suggested that this common co-curricular activity may play a role in students’ professional preparation, but more research, including precise measurement tools, is needed. Our objectives were to (1) examine existing questionnaires measuring constructs of relevance including academic confidence, engineering career motivation, engineering beliefs, professional skills, and interpersonal skills; and (2) create and test a measure that can be used to assess the potential impact of outreach activities on students’ professional perceptions and aspirations.

Methods:

A survey of 30 outreach programs in the U.S. revealed that less than 10% routinely assess the impact of ambassadorship. Follow-up interviews and focus groups indicated that program leaders were interested in a measure.

A review of literature revealed existing sources including the Longitudinal Assessment of Engineering Self-Efficacy (LAESE), the Project to Assess Climate in Engineering (PACE) survey, the Laanan Transfer Students Questionnaire (L-TSQ), the National Academy of Engineering Changing the Conversation survey, and ambassador chapter exit interviews. Items were grouped and rated for relevance and uniqueness by subject matter experts. Highly-rated items were then included in an online Ambassador Questionnaire (AQ), which was administered to 168 outreach ambassadors from 11 colleges/universities. The sample was primarily female (67%) and white (69%), with most respondents in their first year as ambassadors (51%). We conducted two phases of exploratory factor analysis (EFA), and calculated internal consistency reliability via Cronbach’s alpha coefficients.

Results:

Diagnostic procedures indicated the removal of four items with excessive low intercorrelations and six items that failed to load on any factor at or above .40. The Kaiser-Meyer-Olkin value was .83, indicating sufficient sampling adequacy. Bartlett’s test of sphericity (p&gt;.001) revealed that the correlations among the 47 retained items were suitable for EFA.

The EFA initially extracted six factors and explained 54% of the variance. We retained theorized subscales of Academic Confidence, Engineering Beliefs, Professional Skills, and Interpersonal Skills. The Engineering Career Motivation subscale did not hold, with some items loading on Academic Confidence and some items breaking into their own factor, which we labeled Ambassadorship. Two items focusing on equity in engineering careers loaded on a separate factor. Cronbach’s alpha coefficients suggested strong internal consistency, with a whole scale coefficient of .93 and subscale coefficients ranging from .81 to .90.

A follow-up EFA specified five factors to align with theory. This solution explained 50% of the variance, and moved the two equity items to the Ambassadorship subscale. Descriptive analyses revealed sensitivity to group differences. For example, female respondents were less likely to expect fair and equitable treatment in their future workplace than males (p&lt;0.001).

Significance:

Our preliminary findings include a robust structure to the measure and the potential to investigate group- and time-related differences in students’ perceptions and experiences. After further validation with a larger sample, we plan to make the measure available to researchers and practitioners.

AU - Melissa G. Kuhn
AU - Joanna K. Garner
AU - Shanan Chappell
CY - Tampa, Florida
DA - 2019/06/15
PB - ASEE Conferences
TI - Work in Progress – Development and Validation of the Ambassador Questionnaire
UR - https://peer.asee.org/33571
DO - 10.18260/1-2--33571
ER -