June 24, 2017
June 24, 2017
June 28, 2017
Liberal Education/Engineering & Society
American Innovation and Competitiveness Act (S.3084) reapproved the National Science Foundation’s (NSF) merit review criteria i.e. intellectual merit and broader impact, called for an update of the policy guidelines for NSF staff members and merit review process participants, and emphasized the importance of transparency and accountability. Evaluating proposals on the basis of intellectual merit and broader impacts has been the standards of maintaining excellence and accountability since 1997. Intellectual merit consists of proposing activities that advance knowledge, while broader impact statements “describe the potential of the proposed activity to benefit society and contribute to the achievement of specific, desired societal outcomes.” While intellectual merit has been widely understood since its inception, the broader impact criterion has continuously been debated among Principle Investigators (PI), panel reviewers, NSF Committee of Visitors, and the broader scientific community. This small-scale study used a convergent parallel mixed method design, collecting both quantitative and qualitative data, simultaneously, to answer two research questions 1) What trends are Program Officers seeing in the broader impact criterion and 2) Which broader impacts statements are being address in proposals submitted to the National Science Foundation. The quantitative approach consisted of appraising 82 awarded abstracts to obtain a quantifiable assessment of the extent to which PIs addressed the broader impacts suggestions outlined in NSF’s Proposal and Award Policies and Procedures Guide. Broader impact proposed activities outlined in the NSF guide and quantified in this study include a) Increase public scientific literacy, b) Increase public engagement with science and technology, c) Address issue of developing a diverse STEM workforce, d) Develop a diverse STEM workforce, e) Develop a globally competitive STEM workforce, and f) Increase economic competitiveness of the U.S. Inclusion and exclusion factors were created for the six aforementioned activities. The qualitative approach involved interviews of four program officers from the NSF Engineering Directorate regarding the trends in addressing the broader impact criterion. The data were collected in parallel strands, independently from each other, and were brought together for comparison. Results from the appraised abstracts indicate that PIs were more oriented towards stating broader impact statements around increasing public scientific literacy, public engagement in science and engineering, and addressing issues of developing a diverse STEM workforce. While the interview data from Program Officers indicated that PIs were more inclined to write “pie in the sky” statements around broader impacts. Program Officers also indicated that reviewers tend to place more weight on intellectual merit than broader impact even though they are informed to put equal weight in both criteria. There exists no formal metric for measuring or assessing broader impact statements PIs propose in their NSF awarded grants. This small-scale study is not proposing a metric or measure of broader impact activities, rather it is an exploratory attempt to unpack what is currently being funded using awarded abstracts and outline tensions around addressing broader impacts.
Verdín, D. (2017, June), Quantifying and Assessing Trends on the National Science Foundation's Broader Impact Criterion Paper presented at 2017 ASEE Annual Conference & Exposition, Columbus, Ohio. 10.18260/1-2--28778
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2017 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015