Quantifying and Assessing Trends on the National Science Foundation's Broader Impact Criterion

Dina Verdín

Download Paper | Permalink

Conference: 2017 ASEE Annual Conference & Exposition
Location: Columbus, Ohio
Publication Date: June 24, 2017
Start Date: June 24, 2017
End Date: June 28, 2017
Conference Session: The Big Picture in Engineering Education
Tagged Division: Liberal Education/Engineering & Society
Tagged Topic: Diversity
Page Count: 21
DOI: 10.18260/1-2--28778
Permanent URL: https://peer.asee.org/28778
Download Count: 586

Paper Authors

biography

Dina Verdín Purdue University orcid.org/0000-0002-6048-1104

visit author page

Dina Verdín is a Ph.D. student in Engineering Education and M.S. student in Industrial Engineering at Purdue University. She completed her undergraduate degree in Industrial and Systems Engineering at San José State University. Dina is a 2016 recipient of the National Science Foundation’s Graduate Research Fellowship (GRF). Her research interest focuses on first-generation college students, specifically around changing deficit base paradigms by providing asset base perspectives for understanding this community.

visit author page

Download Paper | Permalink

Abstract

American Innovation and Competitiveness Act (S.3084) reapproved the National Science Foundation’s (NSF) merit review criteria i.e. intellectual merit and broader impact, called for an update of the policy guidelines for NSF staff members and merit review process participants, and emphasized the importance of transparency and accountability. Evaluating proposals on the basis of intellectual merit and broader impacts has been the standards of maintaining excellence and accountability since 1997. Intellectual merit consists of proposing activities that advance knowledge, while broader impact statements “describe the potential of the proposed activity to benefit society and contribute to the achievement of specific, desired societal outcomes.” While intellectual merit has been widely understood since its inception, the broader impact criterion has continuously been debated among Principle Investigators (PI), panel reviewers, NSF Committee of Visitors, and the broader scientific community. This small-scale study used a convergent parallel mixed method design, collecting both quantitative and qualitative data, simultaneously, to answer two research questions 1) What trends are Program Officers seeing in the broader impact criterion and 2) Which broader impacts statements are being address in proposals submitted to the National Science Foundation. The quantitative approach consisted of appraising 82 awarded abstracts to obtain a quantifiable assessment of the extent to which PIs addressed the broader impacts suggestions outlined in NSF’s Proposal and Award Policies and Procedures Guide. Broader impact proposed activities outlined in the NSF guide and quantified in this study include a) Increase public scientific literacy, b) Increase public engagement with science and technology, c) Address issue of developing a diverse STEM workforce, d) Develop a diverse STEM workforce, e) Develop a globally competitive STEM workforce, and f) Increase economic competitiveness of the U.S. Inclusion and exclusion factors were created for the six aforementioned activities. The qualitative approach involved interviews of four program officers from the NSF Engineering Directorate regarding the trends in addressing the broader impact criterion. The data were collected in parallel strands, independently from each other, and were brought together for comparison. Results from the appraised abstracts indicate that PIs were more oriented towards stating broader impact statements around increasing public scientific literacy, public engagement in science and engineering, and addressing issues of developing a diverse STEM workforce. While the interview data from Program Officers indicated that PIs were more inclined to write “pie in the sky” statements around broader impacts. Program Officers also indicated that reviewers tend to place more weight on intellectual merit than broader impact even though they are informed to put equal weight in both criteria. There exists no formal metric for measuring or assessing broader impact statements PIs propose in their NSF awarded grants. This small-scale study is not proposing a metric or measure of broader impact activities, rather it is an exploratory attempt to unpack what is currently being funded using awarded abstracts and outline tensions around addressing broader impacts.

Citation
Format

Verdín, D. (2017, June), Quantifying and Assessing Trends on the National Science Foundation's Broader Impact Criterion Paper presented at 2017 ASEE Annual Conference & Exposition, Columbus, Ohio. 10.18260/1-2--28778

TY  - CPAPER
AB  -         American Innovation and Competitiveness Act (S.3084) reapproved the National Science Foundation’s (NSF) merit review criteria i.e. intellectual merit and broader impact, called for an update of the policy guidelines for NSF staff members and merit review process participants, and emphasized the importance of transparency and accountability. Evaluating proposals on the basis of intellectual merit and broader impacts has been the standards of maintaining excellence and accountability since 1997. Intellectual merit consists of proposing activities that advance knowledge, while broader impact statements “describe the potential of the proposed activity to benefit society and contribute to the achievement of specific, desired societal outcomes.” While intellectual merit has been widely understood since its inception, the broader impact criterion has continuously been debated among Principle Investigators (PI), panel reviewers, NSF Committee of Visitors, and the broader scientific community. This small-scale study used a convergent parallel mixed method design, collecting both quantitative and qualitative data, simultaneously, to answer two research questions 1) What trends are Program Officers seeing in the broader impact criterion and 2) Which broader impacts statements are being address in proposals submitted to the National Science Foundation.
        The quantitative approach consisted of appraising 82 awarded abstracts to obtain a quantifiable assessment of the extent to which PIs addressed the broader impacts suggestions outlined in NSF’s Proposal and Award Policies and Procedures Guide. Broader impact proposed activities outlined in the NSF guide and quantified in this study include a) Increase public scientific literacy, b) Increase public engagement with science and technology, c) Address issue of developing a diverse STEM workforce, d) Develop a diverse STEM workforce, e) Develop a globally competitive STEM workforce, and f) Increase economic competitiveness of the U.S. Inclusion and exclusion factors were created for the six aforementioned activities. The qualitative approach involved interviews of four program officers from the NSF Engineering Directorate regarding the trends in addressing the broader impact criterion. The data were collected in parallel strands, independently from each other, and were brought together for comparison.
	Results from the appraised abstracts indicate that PIs were more oriented towards stating broader impact statements around increasing public scientific literacy, public engagement in science and engineering, and addressing issues of developing a diverse STEM workforce. While the interview data from Program Officers indicated that PIs were more inclined to write “pie in the sky” statements around broader impacts. Program Officers also indicated that reviewers tend to place more weight on intellectual merit than broader impact even though they are informed to put equal weight in both criteria. There exists no formal metric for measuring or assessing broader impact statements PIs propose in their NSF awarded grants. This small-scale study is not proposing a metric or measure of broader impact activities, rather it is an exploratory attempt to unpack what is currently being funded using awarded abstracts and outline tensions around addressing broader impacts.
AU  - Dina Verdín
CY  - Columbus, Ohio
DA  - 2017/06/24
PB  - ASEE Conferences
TI  - Quantifying and Assessing Trends on the National Science Foundation's Broader Impact Criterion
UR  - https://peer.asee.org/28778
DO  - 10.18260/1-2--28778
ER  -