Results from a Survey of National Science Foundation Transforming Undergraduate Education in STEM (TUES) Program Reviewers

Jason Feser; Maura J. Borrego; Russ Pimmel; Connie Kubo DUPE Della-Piana

Download Paper | Permalink

Conference: 2012 ASEE Annual Conference & Exposition
Location: San Antonio, Texas
Publication Date: June 10, 2012
Start Date: June 10, 2012
End Date: June 13, 2012
ISSN: 2153-5965
Conference Session: Thinking About the Engineering Curriculum
Tagged Division: Educational Research and Methods
Page Count: 13
Page Numbers: 25.1126.1 - 25.1126.13
DOI: 10.18260/1-2--21883
Permanent URL: https://peer.asee.org/21883
Download Count: 472

Request a correction

Paper Authors

biography

Jason Feser American Association for the Advancement of Science, Science & Technology Policy

visit author page

Jason Feser is a 2011-12 American Association for the Advancement of Science, Science & Technology Policy Fellow currently placed in the Division of Undergraduate Education at the National Science Foundation. He completed his graduate studies in molecular biology in 2010 at the University of Colorado, Denver, Anschutz Medical Campus in Aurora, Colo. His graduate work explored how the aging process affects chromatin structure and how lifespan can be extended by altering histone expression in Saccharomyces cerevisiae. He received his B.S. in biotechnology from North Dakota State University in 2004, while investigating the occurrence of an infant meningitis causing bacterium Enterobacter sakazakii in bird populations in North Dakota. Prior to attending graduate school, Feser worked at the Center for Nanoscale Science and Engineering, developing high-throughput screening methods to test naval coating efficacy in preventing biofouling.

visit author page

biography

Maura J. Borrego National Science Foundation

visit author page

Maura Borrego is an Associate Professor and former Director of the Graduate Program in the Department of Engineering Education at Virginia Tech, currently serving as a Program Director in the Division of Undergraduate Education at the National Science Foundation. She recently held a 2010-2011 AAAS Science & Technology Policy Fellowship at the National Science Foundation. Borrego’s engineering education research awards include PECASE, CAREER, and two outstanding publication awards from the American Educational Research Association for her journal articles. Her research interests include engineering faculty development, specifically how faculty members decide to apply the results of educational research, and interdisciplinary graduate education in STEM. She is an editorial board member for Journal of Engineering Education and chair of the American Society for Engineering Education’s Educational Research and Methods Division. Borrego has developed and taught graduate level courses in engineering education research methods and assessment from 2005-2010. All of Borrego’s degrees are in materials science and engineering. Her M.S. and Ph.D. are from Stanford University, and her B.S. is from University of Wisconsin, Madison.

visit author page

biography

Russ Pimmel University of Alabama

visit author page

Russell Pimmel retired from the NSF after serving as a Program Director in the Division of Undergraduate Education for eight years. Before that, he was a faculty member at the University of Alabama, the University of Missouri, University of North Carolina, and Ohio State University. He also has held industrial positions with Emerson Electric Co., McDonald-Douglas Co., and Battelle Research Labs.

visit author page

author page

Connie Kubo DUPE Della-Piana National Science Foundation

Download Paper | Permalink

Abstract

Results from a Survey of NSF TUES Program Reviewers Russ Pimmel, Connie Della-Piana and Don MillardVirtually all proposals submitted to the NSF Transforming Undergraduate STEM Education(TUES) Program are reviewed by a panel in which three or more reviewers submit individualreviews and then meet to discuss and prepare a summary of their judgments. The event usuallyconcludes with a wrap-up session during which a representative of each panel brieflysummarizes their reviews of their proposals, highlighting common strengths and weaknesses. Inorder to better document this process during the 2011 review of the Type 1 TUES proposals,reviewers completed a survey questioning how they saw the proposals in light of the increasedexternal (e.g., non-local) focus incorporated in moving from CCLI to TUES.The questionnaire contained thirteen forced-choice and two open-ended items. The first twoitems asked reviewers to rate their experience in reviewing proposals and in working oneducational projects. The next eight asked them to indicate what fraction of proposals (almostnone, about ¼, about ½, about ¾, and almost all) addressed specific expected aspects, such asappropriate measureable outcomes and efforts to facilitate adaptation at other sites. The nextthree asked reviewers to gauge what fraction of the proposals (same choices as above) wouldaccomplish specific desirable outcomes, such as producing adaptable materials or contributing tothe understanding of STEM education. The following expression was used to compute aweighted average for each question: 0.1* (percent selecting almost none) + 0.25 * percentselecting about ¼) + 0.5 * (percent selecting about ½) + 0.75 * (percent selecting about ¾) + 0.9((percent selecting almost all). The last two questions were open-ended items asking respondentsto list the most common strength and weakness in the proposals they read.Weighted averages ranged from 36 % to 55 %. In general, the questions that dealt withestablished concepts in the investigator’s own implementation (i. e., local effects) had higherweighted averages; for example, the questions with the two highest weighted averages were“Proposal had a discussion building on existing knowledge about STEM education” (55%) and“Proposal had well conceived activities” (53%). In contrast, questions that dealt with external,more broadly focused issues (i. e., non-local effects) had lower weighted averages; for example,the questions with the two lowest weighted averages were: “Proposal had creative, original, orpotentially transformative concepts” (36%) and “Proposal had a significant effort to facilitateadaptation at other sites” (38%). Questions that dealt with local aspects (relatively new to theSTEM community) such as evaluation and broadening participation produced intermediateweighted averages. A preliminary analysis of the open-ended responses on strengths andweaknesses appeared to agree with these observations.These finding indicate that reviewers were aware of the non-local or external focus that has beenemphasized in moving from the earlier inward-looking CCLI framework to the new, outward-looking expectation in the TUES solicitation. The data also indicates that approximately onethird of the proposals addressed this newer focus to some extent.

Citation
Format

Feser, J., & Borrego, M. J., & Pimmel, R., & DUPE Della-Piana, C. K. (2012, June), Results from a Survey of National Science Foundation Transforming Undergraduate Education in STEM (TUES) Program Reviewers Paper presented at 2012 ASEE Annual Conference & Exposition, San Antonio, Texas. 10.18260/1-2--21883

TY  - CPAPER
AB  -              Results from a Survey of NSF TUES Program Reviewers                       Russ Pimmel, Connie Della-Piana and Don MillardVirtually all proposals submitted to the NSF Transforming Undergraduate STEM Education(TUES) Program are reviewed by a panel in which three or more reviewers submit individualreviews and then meet to discuss and prepare a summary of their judgments. The event usuallyconcludes with a wrap-up session during which a representative of each panel brieflysummarizes their reviews of their proposals, highlighting common strengths and weaknesses. Inorder to better document this process during the 2011 review of the Type 1 TUES proposals,reviewers completed a survey questioning how they saw the proposals in light of the increasedexternal (e.g., non-local) focus incorporated in moving from CCLI to TUES.The questionnaire contained thirteen forced-choice and two open-ended items. The first twoitems asked reviewers to rate their experience in reviewing proposals and in working oneducational projects. The next eight asked them to indicate what fraction of proposals (almostnone, about ¼, about ½, about ¾, and almost all) addressed specific expected aspects, such asappropriate measureable outcomes and efforts to facilitate adaptation at other sites. The nextthree asked reviewers to gauge what fraction of the proposals (same choices as above) wouldaccomplish specific desirable outcomes, such as producing adaptable materials or contributing tothe understanding of STEM education. The following expression was used to compute aweighted average for each question: 0.1* (percent selecting almost none) + 0.25 * percentselecting about ¼) + 0.5 * (percent selecting about ½) + 0.75 * (percent selecting about ¾) + 0.9((percent selecting almost all). The last two questions were open-ended items asking respondentsto list the most common strength and weakness in the proposals they read.Weighted averages ranged from 36 % to 55 %. In general, the questions that dealt withestablished concepts in the investigator’s own implementation (i. e., local effects) had higherweighted averages; for example, the questions with the two highest weighted averages were“Proposal had a discussion building on existing knowledge about STEM education” (55%) and“Proposal had well conceived activities” (53%). In contrast, questions that dealt with external,more broadly focused issues (i. e., non-local effects) had lower weighted averages; for example,the questions with the two lowest weighted averages were: “Proposal had creative, original, orpotentially transformative concepts” (36%) and “Proposal had a significant effort to facilitateadaptation at other sites” (38%). Questions that dealt with local aspects (relatively new to theSTEM community) such as evaluation and broadening participation produced intermediateweighted averages. A preliminary analysis of the open-ended responses on strengths andweaknesses appeared to agree with these observations.These finding indicate that reviewers were aware of the non-local or external focus that has beenemphasized in moving from the earlier inward-looking CCLI framework to the new, outward-looking expectation in the TUES solicitation. The data also indicates that approximately onethird of the proposals addressed this newer focus to some extent.
AU  - Jason Feser
AU  - Maura J. Borrego
AU  - Russ Pimmel
AU  - Connie Kubo DUPE Della-Piana
CY  - San Antonio, Texas
DA  - 2012/06/10
PB  - ASEE Conferences
TI  - Results from a Survey of National Science Foundation Transforming Undergraduate Education in STEM (TUES) Program Reviewers
UR  - https://peer.asee.org/21883
DO  - 10.18260/1-2--21883
ER  -