Kevin Wright AEA Annual Conference Nov 3, 2011

1. Estimating the Impact of Hypothetical Portfolio Reductions on Production of Major Discoveries Funded by the National Institute of Allergy and Infectious Diseases Kevin Wright AEA Annual Conference Nov 3, 2011 Co-Authors Brandie K. Taylor (NIH), Jamie Mihoko Doyle (STPI), Brian Zuckerman (STPI)

2. Overview • Task description and approach • Part I Exploratory Analysis: Data and results • Part II Counterfactual Analysis: Data, methods, results, and assumptions • Conclusions

3. Question • What might the impact be on the research community and public health if NIAID received a budget reduction? • What would have been the impact had NIAID received smaller budgets?

4. Approach • Two sets of analyses were conducted: • An initial exploratory analysis to characterize the discoveries • A comparison of characteristics of R01s associated with discoveries with those that were not

5. Part 1Exploratory Analysis

6. Data • 138 unique discoveries (or “advances”) between 2005 and 2009 were identified by NIAID and a list was provided to STPI • A complex script was written to scan the attributes of each discovery for grant numbers, query them in QVR, and populate a database • A total of 10,323 awards were pulled from QVR • These records contained all award types (ex. 1, 2, 5, etc.)

7. Data • Variables that were downloaded from QVR include: • Priority scores and percentiles (if available) • Award amount and year • Study section • Full grant number • Institution and institution state • Principal investigator (PI) name

8. Data Exclusions • Only the most recent Type 1 or 2 was retained • R37 (i.e. MERIT) and R56 (i.e. High Priority, Short-term Project) awards were excluded, but the associated R01s were kept • Contracts and intramural awards were excluded from the analysis

9. Data • 138 unique discoveries were associated with 305 unique awards (247 of which were NIAID) • 106 out of the 138 of the discoveries were funded exclusively through NIAID awards (77%)

10. Results and Findings

11. Distribution of Discoveries by Year Number of Discoveries (N=138 discoveries) (N=106 discoveries)

12. Non-NIAID Award Funding Sources Number of Awards

13. Number of Awards Cited per Discovery Mean: 2 awards Number of Discoveries (N=138 discoveries)

14. Distribution of Activity Code • All Awards, All ICs • (N=305 awards) • NIAID Only • (N=247 awards)

15. Part 2 Counterfactual Analysis

16. Counterfactual Analysis • A comparison of R01 associated with discoveries with those that were not • An analysis of discovery-associated awards affected by reductions in appropriations

17. R01s Only • Highest percentage of awards funding NIAID discoveries • Primary funding mechanism for basic and applied research • Percentile and payline data easily accessible

18. Reference Sample • All Type 1 and Type 2 R01s were extracted from QVR for FY 2004-2008 • R01s associated with the discoveries were then removed • R01s with percentiles that were missing or outside of the payline for that particular FY were also excluded • A total of 2,320 awards are contained in the reference set

19. Sample • The analysis is restricted to the 40 discoveries associated with R01s • Of the original 138 discoveries, 59 were omitted because they were not supported by at least 1 R01 • 29 were omitted because none of R01s associated with the discovery were within 2004-2008 period • 10 discoveries excluded because the percentile information for the R01s were missing (N=4) or outside of the payline (N=6) for that FY • 38 R01s are considered for the analysis

20. Methods • The amount of appropriated funds were allowed to vary, but several factors within each FY were held constant. These factors include: • Percentage of allocated funds dedicated to extramural awards • The percentage of extramural funds dedicated to Type 1 and Type 2 R01s • Cost per R01 • The cost per R01 estimated by the budget office (based on previous years) is equal to the actual cost per R01 reported for that FY

21. Distribution of R01 Awards Percentage of Awards

22. Distribution of Percentiles Percentage of Awards (N= 38 awards) (N= 2,320 awards)

23. Procedure for Counterfactual • For each FY, calculate the reduction in extramural allocations, total funding for Type 1 and Type 2 R01s as a function for each percent reduction in overall appropriations • Calculate the number of awards lost by holding the cost per R01 constant • Order all R01s by percentile (high to low) and eliminate awards according to number of awards lost at each percent reduction in overall appropriations

24. R01s Affected by Appropriation Reduction % R01s Affected

25. Conclusions • Results from the descriptive analysis demonstrate the complexities of how discoveries are generated and reinforce the challenges of attributing major advances to a single award • Even small reductions in the budget could result in the loss of discoveries • Applications having percentiles outside of the payline, but were funded, were no less likely to yield a discovery than those within a payline

26. Contact Info Kevin D. Wright, MPA Deputy Chief Strategic Planning and Evaluation Branch OSPFM, OSMO, NIAID, NIH, DHHS (p) 301-402-3574 wrightk@mail.nih.gov

27. Thank You

28. Mappingof award percentiles by discovery (N=40)

29. Awards Above the Payline • Whether select pay or other awards outside of the payline would have been lost is uncertain. The assumption is that these awards would not be affected and therefore excluded from the analysis. • An ancillary analysis to determine whether discoveries had higher percentages of awards outside of the payline was conducted