Manuscript

Also available as a PDF on SocArXiv

Research equity: Incentivizing high-risk basic research with market mechanisms

Highlights

This paper proposes that funders pay researchers and institutions for completed research based on its scientific and socioeconomic value.
Valuation of completed research could be informed by a novel “chess rating” peer review method.
Payment based on research value could stimulate more rapid scientific discovery and progress, especially in high-risk research.
The proposed funding model could easily coexist with existing funding models.

Abstract

Innovation in basic research is vital to scientific progress and technological development; however, such research finds insufficient support in the current research environment (Boudreau et al., 2016; Foster et al., 2015; Pagano, 2006; Rzhetsky et al., 2015; Wang et al., 2017). To stimulate high-risk, high-reward basic research, this paper proposes a “research equity” funding model in which funders—such as government agencies and philanthropies—would pay researchers and institutions for completed research: The more valuable the research, the greater the reward. The valuation of completed research could be done with a novel “chess rating” method: A peer reviewer would be presented with a pair of research papers and would decide which of the two is of greater value, and a large number of comparisons would produce a numerical rating to inform payment. Payment based on research value would enable many of the qualities found in healthy markets. Initial capital for basic research would be provided by research institutions, which would be financially incentivized to invest in a diverse body of basic research that includes both low-risk, conservative research and high-risk, innovative research. Institutions would be motivated to demonstrate the value of completed research in their portfolios, which may accelerate recognition of important results. By motivating researchers and institutions to produce and promote valuable research, the research equity model could stimulate more rapid scientific discovery and progress. Notably, the research equity model could coexist with grant funding.

Introduction
The research equity model
Benefits of the research equity model
A path forward
Conclusion
References

Introduction

Today’s conservative funding environment obstructs innovation in basic research.

Basic research advances scientific knowledge without any particular application in view (National Science Foundation, 2018). That knowledge enables diverse technological and societal developments, which typically occur after significant delay—in many cases, 10 to 30 years (Casadevall & Fang, 2016; Contopoulos-Ioannidis et al., 2008; Loellbach, 1968; Salter & Martin, 2001). For example, development of the laser in the 1950’s was based on decades of gradual research on stimulated emission—an obscure phenomenon theorized by Einstein in 1917 (Bortoletto et al., 2011). The laser is now essential to telecommunications, data storage, microsurgery, experimental research equipment, and materials processing. Similarly, even the most enthusiastic practitioners of mathematical number theory were once hard-pressed to think of practical applications for advances in the field—advances which today form the backbone of modern cryptography, essential for digital communication, electronic commerce, and national security.

Long investment horizons and unexpected applications make it unlikely that an investor in basic research can fully capture its value. Accordingly, funding for basic research is predominantly provided by institutions willing to eschew profit to advance the public good: governments, higher education institutions, and philanthropies (National Science Board, 2020). In the US, the government provides the majority of funding for basic research. Much of this funding is distributed as grants wherein research proposals are evaluated by peer review panels and funded based on the estimated cost of the research project.

Basic research varies in its degree of novelty. Conservative research develops established directions and reliably produces timely results, while innovative research explores new conceptual territory and is more likely to have a greater impact in the long run, spurring further scientific discoveries and enabling greater advancements in technology (Casadevall & Fang, 2016; Foster et al., 2015; Ke et al., 2015; Kuhn, 1962; Rzhetsky et al., 2015; Wang et al., 2017). Despite the importance of innovative research, government grant agencies are systematically biased against novel research proposals, instead strongly preferring short-term, conservative projects (Boudreau et al., 2016; Foster et al., 2015; Lee, 2007; Pagano, 2006; Rzhetsky et al., 2015; Wang et al., 2017). Of research proposals rejected for federal funding, about one-third to one-half are abandoned (Carter, 1978; Chubin & Hackett, 1990; Gillespie et al., 1985). It is likely that some of these abandoned projects would have produced valuable—even transformative—results. Among Nobel laureates in medicine, physics, and chemistry from 2000-2008, the research behind 30% of their pivotal papers was done without grant funding (Tatsioni et al., 2010). Many of these laureates stressed that the flexibility and freedom of non-grant funding provided by their institutions was essential to their work, and several deplored that such open-ended funding was becoming increasingly uncommon. Similar concerns are frequently reiterated by other Nobel laureates, who warn that today’s increasingly conservative research environment would have failed to support their groundbreaking research (Clauset et al., 2017; Lee, 2007; Tatsioni et al., 2010). Nobel laureate Roger Kornberg observes, “In the present climate especially, the funding decisions are ultraconservative. If the work that you propose to do isn’t virtually certain of success, then it won’t be funded. And of course, the kind of work that we would most like to see take place, which is groundbreaking and innovative, lies at the other extreme” (Lee, 2007).

Support for the value of risk tolerant funding can be found in philanthropic funding, where greater risk tolerance and longer investment horizons have enabled many notable advancements. Evidence indicates that researchers produce more innovative and impactful results when they receive long-term, risk-tolerant, philanthropic funding than when they receive short-term, conservative, government funding (Azoulay et al., 2011). Furthermore, philanthropies have sometimes funded risky, early-stage research which then yielded the promising preliminary results requisite for subsequent government funding—strong evidence that conservative government grants fail to fund important research and that greater risk tolerance is crucial to enable valuable results (Grant, 2017).

While awards and accolades for innovative results offset the pressure for conservative research, the average payoff insufficiently compensates the risk of failure (Foster et al., 2015). The probability of success of innovative research is sufficiently small that a researcher’s career is better served by adopting conservative strategies that reliably generate citations and other markers of consistent productivity upon which rest university hiring and tenure decisions (Foster et al., 2015; Laudel, 2006; Rzhetsky et al., 2015).

To motivate innovation in basic research, this paper proposes a “research equity” funding model that manages risk with market mechanisms. A reallocation of risk would allow the government to support basic research with minimal risk while incentivizing researchers and institutions to gamble on bold ideas. In the following sections we describe the proposed system and elaborate on its features.

The research equity model

In the research equity model, funders—such as government agencies and philanthropies—would pay researchers and institutions for completed research: The more valuable the research, the greater the reward.

In the proposed “research equity” funding model, government agencies and philanthropies would represent the public interest by paying for completed basic research when its value is evident: the more valuable the research, the greater the reward to the researchers and institutions that produced it.

The basic mechanics of such a model are outlined as follows. Upon completion of a research paper, researchers may submit the paper to an online repository. The repository generates a number of shares in that paper—for instance, 100 shares. Those shares are then allocated to the researchers as well as the investors—for example, the research institutions—that supported the research. The allocation of shares is determined by consensus among those parties. For example, researcher A may receive 25 shares, researcher B 25 shares, and their research institution 50 shares.

Share price would begin at zero. As the value of the research becomes evident, buyers—such as government agencies and philanthropies—would publicly indicate on the online repository their valuation of the research—for example, $2000 per share. At any date, shareholders may sell their shares to the buyer at the specified price. Buyers may modify their valuation, establishing a new purchase price for the shares. For instance, it may become clear that the research is more valuable than the previous price reflected, and buyers may increase the price to $5000 per share. Shareholders would hold their shares until they choose to sell—for instance, when they believe the shares to be adequately valued or when they require cash flow. Buyers would be motivated to purchase shares for a similar reason as in the grant model: not for intellectual property rights to basic research, but rather to stimulate scientific progress. Purchase of shares can be likened to the distribution of prize money which stimulates participation and rewards excellence.

Payment based on value is an extremely desirable property in a market, and many benefits may naturally emerge, which we discuss in the section “Benefits of the Research Equity Model”.

Valuation of completed research

A novel “chess rating” peer review system could inform valuation of completed research.

In the research equity model, shares in completed research would be purchased at a price that is based on the value of that research. This paper proposes a particular method of research valuation that resembles the algorithm by which chess players are assigned numerical rating based on the outcome of chess games (Glickman, 1995).

In the proposed “chess rating” system, every research paper would have a numerical rating. Ratings for research papers would be updated using peer review. Each reviewer would be presented with a pair of research papers. The reviewer would then decide which of those two papers is of greater value, taking into account both its scientific and socioeconomic impact. The outcome of the peer review comparison would update the ratings associated with the two papers in a way similar to the method by which chess ratings are updated after a match (Glickman, 1995). The chess rating system could choose which papers to show to which reviewers in a way designed to optimally resolve the ratings. After a large enough number of comparisons by a large number of reviewers, the rating of the papers will either converge or the algorithm will reach a point of diminishing returns in its process of seeking convergence. The ratings associated with papers can then be used as the input to a valuation function that assigns a monetary value to the shares of each paper. For example, a paper with a rating of 1000 may be assigned a total valuation of $100,000 while a paper with a higher rating of 2000 may be assigned a total valuation of $1 million.

While valuation of research may include some ambiguity, government grant agencies today operate with far inferior information. Research proposals are evaluated by a relatively small number of peer reviewers who operate in a bottleneck of confounding factors, including limited information about the future value of research and systemic grant agency bias against novelty (Boudreau et al., 2016; Kaplan et al., 2008; Pagano, 2006). By contrast, the chess rating system would leverage the expertise of a large, diverse group of peer reviewers in the valuation of completed research, and buyers would make purchase decisions once the value of completed research is evident.

In the chess rating system, it is likely that some reviewers will favor research that resembles their own as a way of indirectly increasing the value of their own shares. While this bias creates an obstruction to fair valuation in the short term, researchers are also motivated to pursue research that they believe will be valuable in the long term—even research that presently competes with their own. After some lag, promising new directions will become areas of active research, attracting higher valuation from a larger community of reviewers.

There are also potential financial conflicts of interest that may arise if the chess rating system requests a reviewer to perform a comparison between papers in which the reviewer or the reviewer’s employer owns shares. To prevent such a conflict of interest, the chess rating system would avoid requesting these comparisons.

A fair and robust method of valuing completed research is critical to the success of the research equity model. Notably, the valuation method is separable from other mechanics of the research equity model. Aside from the chess rating system, other valuation models for informing ex post funding have also been suggested (Squires, 1986; Tsao, 1989). The valuation method warrants further consideration.

Secondary markets

Trading of shares in basic research could provide liquidity in financing research as well as draw attention to valuable results.

An extension of the research equity model would allow for shares in basic research to be traded in a secondary market. This would provide several advantages. Purchase or exchange of shares between researchers or research institutions could increase motivation for collaboration, while purchase of shares by external investors would motivate those investors to serve as advocates who promote and explain the value of the research. Speculative investment could also offer more immediate liquidity to researchers and research institutions. High prices for shares would be a signal that players in the research equity market expect that the research will be highly valued in the future, which may bring attention to important results.

A secondary market would require careful oversight and regulation. Further consideration is necessary and falls outside the scope of this paper.

Benefits of the research equity model

It is difficult to overstate the significance of funding decisions in shaping the landscape of basic research. The research equity model may influence the speed of scientific progress and the quality of research, among other things.

Incentivizing innovation

Payment for valuable research would directly incentivize researchers and institutions to pursue high-risk, high-reward research.

The research equity model motivates risk-taking in basic research. In the grant funding model, research is funded based on cost. For this centrally planned system to maximize research value, government grant agencies must accurately fund research proposals that have the greatest future value; however, grant agencies systematically undervalue novel research proposals, resulting in chronic underinvestment in high-risk research (Boudreau et al., 2016; Foster et al., 2015; Pagano, 2006; Rzhetsky et al., 2015; Wang et al., 2017). The research equity model would decentralize the pursuit of basic research by offering researchers and institutions direct financial incentive to attempt risky results. Payment based on the value of completed research would incentivize researchers and institutions to pursue new lines of inquiry and gamble on novel approaches (Foster et al., 2015). Such risk tolerant investment in basic research would likely enable more innovative and valuable results (Azoulay et al., 2011; Grant, 2017; Tatsioni et al., 2010). Because innovative research is more likely to be highly valuable, there would be additional incentive for exactly the kind of research that is under-funded currently (Foster et al., 2015; Wang et al., 2017).

Payment based on the value of research—rather than its cost—is necessary in this decentralized system because innovative basic research has a high variance: While some projects will produce high-value results, many others will not. Payment based on research value incentivizes institutions to invest in basic research while managing this risk. The expected payment for completed research would determine the amount of risk that institutions are willing to accept.

Illuminating valuable research

Research institutions would be motivated to demonstrate the value of their completed research, which may accelerate recognition of important results.

In the research equity model, promoting the value of completed research would be an important part of realizing financial gain for that research. A research institution would support a collection of researchers and receive shares in research produced, accumulating a portfolio of research. The institution could increase the value of its portfolio by demonstrating the value of research in it. This may involve promotion in which the institution explains the relevance of complex or underappreciated research. The institution may also support follow-on research that expands on the significance of work in its portfolio. Such behaviors may accelerate the recognition of truly transformative research—a process which has often required years or decades (Casadevall & Fang, 2016; Ke et al., 2015).

Promotional work for past results could be done by universities or other stakeholders, which would enable researchers to focus on research rather than engaging in the bureaucratic processes of financing that research. Survey data indicate that faculty at top US universities presently spend on average 8% of their time preparing grant proposals, which represents 20% of their time available for research activities (Link et al., 2008). For some faculty, seeking funding comprises more than 50% of their working hours (Testimony before the Senate Committee on Appropriations: Subcommittee on Labor, Health and Human Services, Education, and Related Agencies, 2007). By some models, the value of the science forgone while seeking funding may approach or exceed the value of the science funded, and this waste increases as grant funding paylines continue to decrease (Gross & Bergstrom, 2019). By contrast, the research equity model would motivate institutions to employ researchers primarily in the production of research.

Amplifying productive research institutions

Productive research institutions would receive capital that could be reinvested in future research.

Because share price is based on research value, institutions that produce excellent results would accumulate resources, enabling them to invest in additional research and creating a virtuous cycle for scientific progress. When investing in future research, some institutions may elect to employ a process similar to the current government grant peer review process, but presumably other institutions would explore other methods, such as funding people instead of projects (Azoulay et al., 2011). Institutions most successful at producing impactful research would be rewarded and thereby be able to invest in more research in the future.

Enabling flexible exploration and discovery

Researchers may freely explore research directions.

The research equity model provides researchers with flexibility in choosing their research directions and allows agile adoption of new ideas. Currently, government grants require that researchers commit to a particular research direction and method of investigation for several years, inhibiting innovation and ready application of novel insights. Likewise, because government grant agencies prefer to fund researchers who have a consistent track record of work on a specific topic, researchers are often confined to established lines of research (Laudel, 2006). Grant agency bias against novel research proposals compounds the issue (Boudreau et al., 2016; Foster et al., 2015; Pagano, 2006; Rzhetsky et al., 2015; Wang et al., 2017). In the research equity model, researchers need not predefine their lines of inquiry or methods of discovery. Instead they may employ novel approaches in the pursuit of whatever discovery they believe to be valuable and within reach. Payment based on research value would reward researchers who expediently pursue productive new lines of research, allowing researchers to change direction mid-project or mid-career.

Attracting talent to high-risk research

Institutions would be motivated to hire researchers who do innovative work and to gamble on promising but unproven researchers.

The research equity model would create researcher career paths both for conservative work and also for exploratory and innovative work. Currently, institutions are financially incentivized to promote researchers who successfully obtain external funding, and many university departments use external funding as a significant part of their professional evaluation for promotion and tenure. This magnifies grant agencies’ conservative funding decisions, strongly pushing researchers to pursue conservative research that produces reliable results (Foster et al., 2015; Laudel, 2006; Rzhetsky et al., 2015). In the research equity model, institutions would manage financial risk by accumulating a well-rounded portfolio of both low- and high-risk research. They would thus be motivated to hire not only researchers who excel in producing reliable, conservative research but also researchers who adventurously explore novel ideas. If their research proves valuable, both the researchers and their institution would benefit. This would engender meritocratic career paths ascended through the pursuit of valuable research, enabling researchers to find jobs both for doing conservative work as well as innovative work.

The research equity model would also aid in the discovery of researchers who might otherwise go unnoticed. Currently, government grant funding decisions are informed by measures of researchers’ past performance—such as citation performance, journal impact factor, and history of funding (Wang et al., 2017). For example, funded researchers are more likely to receive future funding than unfunded researchers, despite similar qualifications and grant peer review scores (Heggeness et al., 2018; Mason et al., 2013; Tesauro et al., 2014). Reliance on historical measures creates a rich-gets-richer dynamic and may overlook undiscovered researchers. The research equity model would reward contributions from researchers at any stage in their career and at any institution, inviting diverse participation and motivating institutions to gamble on promising but unproven researchers.

Encouraging high-quality research

Payment based on research value would motivate research excellence.

The research equity model would encourage high-quality research. Current pressures in academia reward frequent publication, especially in journals with high impact factors (Casadevall & Fang, 2014; McKiernan et al., 2019; Nosek et al., 2012). However, these measures are not necessarily indicative of underlying research quality and may actually encourage low quality science. Many journals—including journals with high impact factors—preferentially publish novel and positive results, a practice which favors false positives, underpowered studies (since they are more likely to produce unexpected results), and even poor research practices (McKiernan et al., 2019). The pressure to publish thereby inflates the rate of false effects in published science (Casadevall & Fang, 2014; Ioannidis, 2005; Nosek et al., 2012). In the research equity model, the total value of a large volume of low-impact publications would not inherently outweigh the value of a few high-impact ones. Institutions and researchers would both benefit from valuable and high-quality publications, rather than high-volume publications. Furthermore, because buyers would make purchase decisions after the value of the work becomes clear—for instance, after other researchers have validated the work—shares in unreplicable studies would be unlikely to be purchased. Similarly, the statistical power of completed research would be evident, making it easier for buyers to avoid rewarding underpowered studies. This could help ameliorate the replication crisis facing many fields (Baker, 2016; Lilienfeld, 2017).

A path forward

The research equity model is a decentralized system with a natural path to implementation. Researchers would submit their work to the research equity repository in a relatively easy process that would be similar to submitting their work for consideration for prize money, and philanthropies could be the first buyers, purchasing shares of research with demonstrated value. If the research equity model seems to be an effective way of financing valuable research, then a larger proportion of researchers and philanthropies would be incentivized to participate. When the model has demonstrated reliability, government funding agencies could begin allocating some of their budget for purchase of research equity shares. The allocation of funds for research equity relative to other funding models would be a matter of choosing which funding model is most appropriate for a particular form of research.

Conclusion

Basic research has enabled multitudinous technologies and societal developments previously unimaginable; however, such diverse and far-reaching benefits imply that investors in basic research—such as funders, research institutions, and researchers—cannot fully benefit from the value they help to create (Bortoletto et al., 2011; Casadevall & Fang, 2016; Contopoulos-Ioannidis et al., 2008; Loellbach, 1968; Salter & Martin, 2001). The proposed research equity funding model would help to bridge this gap. In the research equity model, organizations—like government agencies and philanthropies—would represent the public interest by paying researchers and institutions for completed research. The benefit of the research to the public would be reflected in the purchase price. Payment based on the value of completed research would enable a healthy, competitive market for the creation of basic research, and many benefits would naturally emerge. Researchers and institutions would be directly incentivized to invest in bold new ideas and creative approaches in order to produce unexpected discoveries and groundbreaking results. Efforts by institutions to demonstrate the value of research in their portfolio may accelerate recognition of important research results, a process which has often taken costly years and even decades. Productive institutions would accumulate capital which they could reinvest in future research—a virtuous cycle for advancing the frontiers of science. Institutions would also be motivated to hire researchers with adventurous research strategies and to gamble on promising but unproven researchers, attracting talent to high-risk research. These factors would contribute to a thriving research environment, rich in talent and abundant with incentives for extraordinary discoveries.

Scientific discovery is inherently unpredictable, which makes it important to plan for the unexpected. Given the tremendous significance of basic research to the development of society and technology, it is imperative to find ways of funding research that open the door to the creativity and innovation of the research community. By representing the public interest in a market for basic research, the research equity model would reliably reward valuable results, enabling the scientific community to produce discoveries beyond our current understanding.

References

Azoulay, P., Graff Zivin, J. S., & Manso, G. (2011). Incentives and creativity: Evidence from the academic life sciences. The RAND Journal of Economics, 42(3), 527–554. https://doi.org/10.1111/j.1756-2171.2011.00140.x

Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. https://doi.org/10.1038/533452a

Bortoletto, D., Cornell, E., Corlette, K., DeSimone, J., Fonseca, I., Hawley, S., Reichmanis, E., & Aizenman, M. (2011). At the Foundation of the Foundation: Basic Research at the NSF.

Boudreau, K. J., Guinan, E. C., Lakhani, K. R., & Riedl, C. (2016). Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science. Management Science, 62(10), 2765–2783. https://doi.org/10.1287/mnsc.2015.2285

Carter, G. M. (1978). The consequences of unfunded NIH applications for the investigator and his research. Rand Corporation.

Casadevall, A., & Fang, F. C. (2014). Causes for the Persistence of Impact Factor Mania. MBio, 5(2), e00064-14. https://doi.org/10.1128/mBio.00064-14

Casadevall, A., & Fang, F. C. (2016). Revolutionary Science. MBio, 7(2), e00158-16. https://doi.org/10.1128/mBio.00158-16

Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review and US science policy. Suny Press.

Clauset, A., Larremore, D. B., & Sinatra, R. (2017). Data-driven predictions in the science of science. Science, 355(6324), 477–480. https://doi.org/10.1126/science.aal4217

Contopoulos-Ioannidis, D. G., Alexiou, G. A., Gouvias, T. C., & Ioannidis, J. P. A. (2008). Life Cycle of Translational Research for Medical Interventions. Science, 321(5894), 1298–1299. https://doi.org/10.1126/science.1160622

Foster, J. G., Rzhetsky, A., & Evans, J. A. (2015). Tradition and Innovation in Scientists’ Research Strategies. American Sociological Review, 80(5), 875–908. https://doi.org/10.1177/0003122415601618

Gillespie, G. W., Chubin, D. E., & Kurzon, G. M. (1985). Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Science, Technology, & Human Values, 10(3), 44–54. https://doi.org/10.1177/016224398501000306

Glickman, M. E. (1995). A comprehensive guide to chess ratings. American Chess Journal, 3(1), 59–102.

Grant, B. (2017, December). Philanthropic Funding Makes Waves in Basic Science. The Scientist Magazine. https://www.the-scientist.com/careers/philanthropic-funding-makes-waves-in-basic-science-30184

Gross, K., & Bergstrom, C. T. (2019). Contest models highlight inherent inefficiencies of scientific funding competitions. PLOS Biology, 17(1), e3000065. https://doi.org/10.1371/journal.pbio.3000065

Heggeness, M. L., Ginther, D. K., Larenas, M. I., & Carter-Johnson, F. D. (2018). The Impact of Postdoctoral Fellowships on a Future Independent Career in Federally Funded Biomedical Research (Working Paper No. 24508; Working Paper Series). National Bureau of Economic Research. https://doi.org/10.3386/w24508

Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLOS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

Kaplan, D., Lacetera, N., & Kaplan, C. (2008). Sample Size and Precision in NIH Peer Review. PLOS ONE, 3(7), e2761. https://doi.org/10.1371/journal.pone.0002761

Ke, Q., Ferrara, E., Radicchi, F., & Flammini, A. (2015). Defining and identifying Sleeping Beauties in science. Proceedings of the National Academy of Sciences, 112(24), 7426–7431. https://doi.org/10.1073/pnas.1424329112

Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago.

Laudel, G. (2006). The art of getting funded: How scientists adapt to their funding conditions. Science and Public Policy, 33(7), 489–504. https://doi.org/10.3152/147154306781778777

Lee, C. (2007). Slump in NIH funding is taking toll on research. Washington Post, A06.

Lilienfeld, S. O. (2017). Psychology’s Replication Crisis and the Grant Culture: Righting the Ship. Perspectives on Psychological Science, 12(4), 660–664. https://doi.org/10.1177/1745691616687745

Link, A. N., Swann, C. A., & Bozeman, B. (2008). A time allocation study of university faculty. Economics of Education Review, 27(4), 363–374. https://doi.org/10.1016/j.econedurev.2007.04.002

Loellbach, H. (1968). Technology in retrospect and critical events in science (TRACES). Chicago: Illinois Institute of Technology.

Mason, J. L., Lei, M., Faupel-Badger, J. M., Ginsburg, E. P., Seger, Y. R., DiJoseph, L., Schnell, J. D., & Wiest, J. S. (2013). Outcome Evaluation of the National Cancer Institute Career Development Awards Program. Journal of Cancer Education, 28(1), 9–17. https://doi.org/10.1007/s13187-012-0444-y

McKiernan, E. C., Schimanski, L. A., Muñoz Nieves, C., Matthias, L., Niles, M. T., & Alperin, J. P. (2019). Meta-Research: Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations. ELife, 8, e47338. https://doi.org/10.7554/eLife.47338

National Science Board. (2020). Research and Development: US Trends and International Comparisons. National Science Foundation, Science and Engineering Indicators 2020. https://ncses.nsf.gov/pubs/nsb20203/

National Science Foundation. (2018). Definitions of Research and Development: An Annotated Compilation of Official Sources. https://www.nsf.gov/statistics/randdef/rd-definitions.pdf

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7(6), 615–631. https://doi.org/10.1177/1745691612459058

Pagano, M. (2006). American Idol and NIH Grant Review. Cell, 126(4), 637–638. https://doi.org/10.1016/j.cell.2006.08.004

Rzhetsky, A., Foster, J. G., Foster, I. T., & Evans, J. A. (2015). Choosing experiments to accelerate collective discovery. Proceedings of the National Academy of Sciences, 112(47), 14569–14574. https://doi.org/10.1073/pnas.1509757112

Salter, A. J., & Martin, B. R. (2001). The economic benefits of publicly funded basic research: A critical review. Research Policy, 30(3), 509–532. https://doi.org/10.1016/S0048-7333(00)00091-3

Testimony before the Senate Committee on Appropriations: Subcommittee on Labor, Health and Human Services, Education, and Related Agencies, (2007) (testimony of R Siliciano).

Squires, A. (1986). The Tender Ship: Governmental Management of Technological Change. Birkhäuser Basel. https://doi.org/10.1007/978-1-4757-1926-0

Tatsioni, A., Vavva, E., & Ioannidis, J. P. A. (2010). Sources of funding for Nobel Prize-winning work: Public or private? The FASEB Journal, 24(5), 1335–1339. https://doi.org/10.1096/fj.09-148239

Tesauro, G. M., Seger, Y. R., DiJoseph, L., Schnell, J. D., & Klein, W. M. P. (2014). Assessing the value of a Small Grants Program for behavioral research in cancer control. Translational Behavioral Medicine, 4(1), 79–85. https://doi.org/10.1007/s13142-013-0236-x

Tsao, J. Y. (1989). Consumer preferences and funding priorities in scientific research. Science and Public Policy, 16(5), 294–298. https://doi.org/10.1093/spp/16.5.294

Wang, J., Veugelers, R., & Stephan, P. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46(8), 1416–1436. https://doi.org/10.1016/j.respol.2017.06.006