Innovation Inducement Prizes: Connecting Research to Policy

Innovation inducement prizes have been used for centuries. In the U.S., a recent federal policy change—the America COMPETES Reauthorization Act of 2010—clarified and simplified a path by which all federal agencies can offer innovation inducement prizes, thus intensifying interest in how government agencies can most effectively design and apply such prizes. This paper aims to review and synthesize the academic literature on innovation inducement prizes, to clarify what has been learned that is relevant to current policy discussions, and to highlight unresolved questions that would be fruitful areas for future academic research and policy experimentation. Relative to the existing literature, this article aims to bridge two gaps. First, I synthesize the academic literature in this area with an eye towards drawing lessons for the types of innovation inducement prizes under consideration by federal agencies under the America COMPETES Reauthorization Act. Second, I discuss the problem of how to evaluate the success or failure of innovation inducement prizes, arguing that careful empirical evaluations of innovation inducement prizes are needed in order to provide guidance to federal agencies (and others) on how to most effectively apply and design innovation inducement prizes. Innovation inducement prizes 3 Innovation Inducement Prizes: Connecting Research to Policy INTRODUCTION The idea of offering innovation inducement prizes as rewards for new inventions has a long history. Famous examples of innovation inducement prizes date back at least to the 18th century, such as the frequently discussed Longitude Prize that was posted by the British government in 1714 for a method of measuring longitude. In more recent years, innovation inducement prizes have been offered by governments, private firms, and a variety of philanthropic institutions— ranging from NASA's Centennial Challenges and TopCoder's programming competitions to prizes offered by the X Prize Foundation. In the U.S., a recent federal policy change has intensified interest in how government agencies can most effectively design and apply innovation inducement prizes. Specifically, the America COMPETES Reauthorization Act of 2010, signed into law in January 2011, provides all federal agencies with broad authority to offer innovation inducement prizes. While some agencies—like NASA—had previously offered innovation inducement prizes, this policy change aimed to clarify and simplify a path by which all federal agencies could offer innovation inducement prizes (Kalil & Sturm, 2010). This change was preceded by the September 2010 launch of Challenge.gov—an online platform that allows federal agencies to post problems and open calls for solution submissions from all interested parties. More generally, President Obama's Strategy for American Innovation has encouraged the use of innovation inducement prizes as part of a broader strategy to promote open government. Innovation inducement prizes 4 With this policy backdrop in mind, my aims in this article are to review and synthesize the academic literature on innovation prizes, to clarify what has been learned that is most relevant to current policy discussions, and to highlight unresolved questions that would be fruitful areas for future academic research and policy experimentation. A long academic literature has analyzed various aspects of innovation prizes; for recent reviews, see Gallini and Scotchmer (2002), Kremer and Glennerster (2004), Maurer and Scotchmer (2003), Scotchmer (2004), Kalil (2006), National Academy of Sciences (2007), and McKinsey (2009). Relative to this existing literature, this article aims to bridge two gaps. First, much of the past literature has focused on analyzing aspects of innovation prizes that are not directly relevant to the types of prizes under consideration by federal agencies under the America COMPETES Reauthorization Act. For example, many papers have analyzed the relative merits of incentivizing innovation through intellectual property rights versus innovation prizes; in contrast, current policy discussions are essentially exclusive in their focus on using innovation prizes as a complement to the patent system, not as a replacement for it. While this broader literature provides many relevant lessons, in this article I focus attention on what has been learned that is most relevant to the types of innovation inducement prizes that are the focus of current policy discussions. Second, the previous literature has placed little emphasis on the problem of how to evaluate the success or failure of innovation inducement prizes. While economic theory can offer guidance on how to design innovation inducement prizes, empirical evaluations are critical given the strong policy interest in understanding when innovation inducement prizes would be effective tools for Innovation inducement prizes 5 spurring innovation, and what forms of innovation inducement prizes are most likely to be successful in any given context. The fundamental evaluation problem is to credibly estimate how innovation would have evolved in a counterfactual world where a given innovation prize had not been offered. I discuss a few recent studies that have made progress in addressing this evaluation problem, but the harsh reality is that we currently have a very small evidence base from which to draw policy conclusions. In light of a clear need for further evidence in this area, I highlight the potential for field experiments (potentially conducted by federal agencies) that could shed light on these issues. Reflecting the orientation of this discussion towards prizes of the type that federal agencies may implement under the America COMPETES Reauthorization Act, I limit the discussion in several ways. First, in the language of Kremer and Williams (2010), I focus on innovation inducement prizes that are “voluntary” in that participants can continue to rely on existing institutions such as the patent system, in contrast to “mandatory” prizes that would replace existing institutions; Gallini and Scotchmer (2002) discuss the strengths and weaknesses of mandatory innovation prizes as an alternative to intellectual property rights systems. Second, given current policy interest in open innovation inducement prizes that encourage participation from a wide variety of entrants (e.g., via crowdsourcing efforts like Challenge.gov), I do not discuss procurement mechanisms such as auctions for the right to be paid when a given invention is developed; Laffont and Tirole (1993) discuss procurement design. Third, in the language of Maurer and Scotchmer (2003) and Scotchmer (2004), I focus on “targeted” prizes that request solutions to pre-specified problems, in contrast to “blue-sky prizes” (such as Nobel Prizes) granted to inventors for innovations that were not specified in advance. Innovation inducement prizes 6 Although the specific focus of this paper is on innovation inducement prizes of the type federal agencies may implement under the America COMPETES Reauthorization Act, McKinsey (2009) argues that the distinctions between government-sponsored, philanthropic-sponsored, and corporate-sponsored prizes are modest in terms of design and administration—implying that much of the discussion in this paper may also be relevant for non-government sponsored innovation inducement prizes. Although the size of the prize sector is difficult to estimate due to a lack of centralized data on offered prizes, McKinsey (2009) estimates the prize sector is currently on the order of $1 to 2 billion dollars. This figure can be compared to the $55 billion U.S. universities spent on research in 2009, about 60 percent of which was federally funded (see Stephan, 2011). However, the McKinsey (2009) data suggest the prize sector has seen tremendous growth in recent decades, with more than a 15-fold increase in value since 1970. They estimate that much of the growth to date has come from the philanthropic sector, but the America COMPETES Reauthorization Act may lead to accelerated growth in prizes offered by the government sector. The second section discusses the rationale for public policies in incentivizing innovation, and outlines the key aspects of the innovation policy levers that are the focus of this paper. The next section reviews the academic literature on innovation inducement prizes, with a focus on synthesizing what has been learned that is most relevant to current policy discussions. The following section lays out the problem of how to evaluate the success or failure of a given innovation prize, describes how recent studies have addressed this evaluation problem, and Innovation inducement prizes 7 discusses potential ways forward. The final section concludes with a summary that looks ahead to future use and evaluation of innovation inducement prizes. REWARDING INNOVATION What is the Rationale for Innovation Policies? There has long been concern that competitive markets may not provide adequate incentives for innovation. In formally articulating explanations behind this concern, economists have focused on the fact that ideas are non-rival and can also be non-excludable (Nelson, 1959; Arrow, 1962). Non-rivalry implies that the use of an idea by one individual does not limit its simultaneous use by other individuals. Units of labor are rival, in the sense that one unit of labor cannot be used simultaneously by more than one firm, but ideas are non-rival in the sense that the use of an idea by one firm does not preclude its simultaneous use by other firms. Ideas can also be nonexcludable in the sense that it may be difficult to block individuals from using ideas once they exist. This would be the case if, for example, imitators could easily copy or reverse engineer a new technology once it is developed and marketed. As discussed below, policy mechanisms such as patents can be used to make ideas excludable, or in some cases inventors may have natural mechanisms for excludability even in the absence of patents (for example, because of first mover advantages). But in cases where ideas are both non-rival and non-excludable, ideas then fall into the category of public goods, which may be under-provided by a competitive market. Innovation inducement prizes 8 Given this concern, combined with the presumed role of new ideas in promoting economic growth (Romer, 1990; Aghion & Howitt, 1992), there has been tremendous interest in the question of how to design public policy institutions to provide incentives for the creation of new ideas. As discussed by Gallini and Scotchmer (2002), the general framework outlined above clarifies why some policy mechanism is needed to incentivize innovation, but it does not shed light on which form of public policy intervention is most appropriate. A variety of policy levers exist that can be used to incentivize innovation, and the current innovation policy system in the U.S. and other countries utilize several of these levers simultaneously. Intellectual property rights, including patents and copyrights, aim to incentivize innovation by granting inventors exclusive rights to market their inventions for a fixed period of time. Innovation inducement prizes of the type I focus on in this paper are (when sponsored by governments) financed via general revenue and paid to inventors conditional on developing an invention that satisfies the pre-specified needs laid out by the government. Both intellectual property rights and innovation prizes are, in the language of Kremer and Glennerster (2004) and others, so-called “pull mechanisms” that reward the successful development of specific products, as opposed to “push mechanisms” that instead subsidize research inputs. Research grants are one example of a push mechanism. Designing Institutions to Promote Innovation Intellectual property rights systems are perhaps the most common form of policy aiming to increase incentives for innovation. Broadly defined, intellectual property rights create incentives for innovation by granting inventors exclusive rights to market their invention for a fixed period Innovation inducement prizes 9 of time. During this fixed patent term, inventors may charge prices that are higher than the cost of production without fear of being undercut by competitors. This opportunity to earn additional profits has the goal of providing dynamic incentives for the development of new technologies. However, the benefit of these dynamic incentives comes at the cost of making goods more expensive to consumers during the life of the patent. These higher prices generate deadweight loss, because some consumers will not purchase the good even if their willingness to pay exceeds the marginal cost of production. Thus, the key trade-off that arises under the patent system is that society chooses to provide incentives for research at the cost of deadweight loss from monopoly pricing during the (fixed) life of the patent. Stated in this way, it is not surprising that the patent system has long been controversial (see the discussion in Machlup & Penrose, 1950). In addition to concerns about the deadweight loss from monopoly pricing, patents have also been criticized for at least two other reasons. First, some have criticized whether patents are necessary for stimulating innovation (for a recent critique, see Boldrin & Levine, 2008). Second, as discussed in more detail below, some have argued that in contexts where innovation is cumulative—in the sense that new innovations build on past innovations—transaction costs in the market for technologies may cause patents on existing technologies to hinder cumulative innovation (see Heller & Eisenberg, 1998). Motivated by these concerns, a number of academic papers have analyzed the conditions under which intellectual property rights will be the best incentive system for incentivizing innovation. A key insight in this line of academic literature is well-summarized by Levine (2009): If the government knew how much it would cost to develop technologies, and what the value of the Innovation inducement prizes 10 technologies were, then it would be relatively easy to design an appropriate compensation scheme for innovators; however, in practice there is asymmetric information about both the costs and values of new technologies, and different individuals may substantially disagree about one or both. In an early contribution, Polanvyi (1944) proposed prizes as a means of patent reform. Demsetz (1969) offered an influential critique of the arguments in Arrow (1962), highlighting that there is likely to be asymmetric information between inventors and the government, and that in such cases patents will likely guide innovative effort better than will other mechanisms such as government-sponsored research. Wright (1983) gives perhaps the first formal discussion of how ex ante information asymmetries between inventors and the government influence the choice of mechanisms for spurring innovation, and characterizes several important results. First, if the benefits and costs of research projects are known to both researchers and the government, and if patents impose a higher deadweight loss than do prizes or contracts, then prizes or contracts are preferable to patents. Note that the deadweight loss here takes into account both the deadweight loss from monopoly pricing in the case of intellectual property rights, and the deadweight loss of taxation necessary to fund an innovation prize or government contract in those cases. Intuitively, conceptualizing the monopoly mark-up as a tax, the deadweight loss of funding prizes or contracts will in expectation be less than the deadweight loss imposed by the monopoly pricing that arises with intellectual property, because the latter imposes a higher tax on a narrower base of consumers. Second, if there is asymmetric information between researchers and the government about the costs and benefits of new inventions, and the terms of the award must be fixed before this informational asymmetry is resolved, then prizes and contracts may not be Innovation inducement prizes 11 superior to patents. The key idea is that with intellectual property, the government does not need to know in advance which projects are socially beneficial—rather, firms have incentives to use their superior knowledge to screen out low-value research projects. Gallini and Scotchmer (2002) review the academic literature in this area, synthesizing Wright's work as well as later papers, and conclude that intellectual property is probably the best mechanism for screening projects when value and cost are not observable to the government. More recent analyses of related issues include Chari, Golosov, and Tsyvinski (in press); Hopenhayn, Llobet, and Mitchell (2006); and Weyl and Tirole (2010). Thus, a number of academic papers have provided frameworks for analyzing the conditions under which a system of intellectual property rights will be preferable to a system of innovation prizes as a means of incentivizing innovation. However, that is not the central question in current policy discussions. Rather than being focused on the idea of replacing the patent system with a system of innovation prizes, current policy discussions are essentially exclusive in their focus on the question of how to most effectively use innovation inducement prizes as a supplement to existing institutions like the patent system. In the language of Kremer and Williams (2010), policy discussions are focused on voluntary innovation prizes for which participants can continue to rely on existing institutions such as the patent system, in contrast to mandatory prizes that would replace existing institutions. In the remainder of this paper I focus on discussing voluntary innovation inducement prizes. Innovation inducement prizes 12 INNOVATION INDUCEMENT PRIZES: CONNECTING RESEARCH TO POLICY In many discussions, innovation inducement prizes are assumed to have a single goal: to incentivize the creation of a desired technology. Looking beyond this goal, innovation inducement prizes can also be designed with at least two other additional objectives in mind. First, prize sponsors may be interested in orienting research effort towards designing a product that is capable of being used at scale by consumers, in contrast to what can be thought of as demonstration projects. A well-known example of a demonstration project is the Ansari X Prize, which was awarded in 2004. The Ansari X Prize was awarded to aerospace designer Burt Rutan and financier Paul Allen for being the first private team to “build and launch a spacecraft capable of carrying three people to 100 kilometers above the earth's surface, twice within two weeks” The focus of this prize was on showing that it was possible to accomplish the stated goal—a demonstration of feasibility—rather than on the development of a technology that was ready to be broadly adopted by consumers. In contrast, two examples of innovation inducement prizes that have aimed to spur the development of technologies capable of being used at scale are the Super Efficient Refrigerator Program (SERP) and the Advance Market Commitment (AMC): • Super Efficient Refrigerator Program (SERP). In 1992 a group of electric utility companies offered a $30 million prize aiming to spur the development of an energyefficient refrigerator meeting certain specifications for energy efficiency and environmental standards. Whirlpool won the competition in 1993 on the basis of technical specifications, but the bulk of the prize award payments were based on a market test that required Whirlpool to sell at least 250,000 of the winning unit by 1997. Sales of Innovation inducement prizes 13 Whirlpool's winning model were lower than expected (many cite a high price tag of the awarded unit as a barrier to adoption), and as a result Whirlpool was never awarded the prize. While this Super Efficient Refrigerator Program (SERP) prize “failed” in the sense of not spurring the development of a new technology that was widely adopted, the presence of the market test facilitated a backstop in the sense of not awarding prize money to a technology that was not desirable to consumers. • Advance Market Commitment (AMC). Discussions of Advance Market Commitments (AMCs) have largely focused on encouraging research on vaccines for diseases concentrated in poor countries (see Kremer & Glennerster, 2004). An AMC is a legal contract detailing a guaranteed top-up price that will be paid by sponsors (say, $15 per treatment) for a pre-defined maximum number of vaccine purchases, conditional on (1) the vaccine fulfilling a given set of technical specifications, and (2) poor countries expressing demand for the product (paying, or other purchasers paying on their behalf, a more affordable price of, say, $1 per treatment). The key idea is that the subsidized price provides a financial return for the vaccine developer, but in exchange the developers must agree to a cap on the long-run price charged for the product (or agree to license the technology to other manufacturers). Hence, price subsidies layered on top of the existing patent system achieve close-to-marginal-cost pricing in the short-run, and the agreement for developers to provide the technology at an affordable price (or license the technology to other manufacturers) achieves close-to-marginal-cost pricing in the longer term. In June 2009, a group of country governments together with the Bill & Melinda Gates Foundation launched a pilot AMC for a vaccine against pneumococcal diseases prevalent in developing countries with a $1.5 billion commitment. Four suppliers subsequently Innovation inducement prizes 14 made offers to supply pneumococcal vaccine under the AMC, and vaccines have been rolled out in several countries. Second, in some cases prize sponsors may be interested in placing the technology in the public domain with a goal of incentivizing subsequent research. An example of an innovation inducement prize with this goal is the Daguerreotype photography patent buyout: • Patent buyout for Daguerreotype photography. Kremer (1998) discusses how the government of France, in 1839, bought out the patent for Daguerreotype photography and placed the technique in the public domain. He notes that after the patent was bought out, Daguerrotype photography was subject to myriad technical improvements—likely because the patent buyout eliminated the costly reverse engineering that had previously been necessary in order to do subsequent research on improvements to the photography method. I mention these two additional objectives because most policy discussions of innovation inducement prizes are focused solely on spurring the development of a new technology, with relatively less attention being paid to what will happen to the desired technology after it has been developed—in terms of both pricing and cumulative innovation. As discussed more in the following section, each of these three issues is central in evaluating the social value generated by different forms of prizes. Hence, each of these three issues deserves careful attention from agencies interested in designing innovation inducement prizes. Innovation inducement prizes 15 The remainder of this section discusses each of these three objectives in more detail. Initially I discuss some of the conceptual issues that must be navigated in order for innovation inducement prizes to successfully spur research investments into socially valuable technologies. Then I discuss how innovation inducement prizes can be designed to encourage access to technologies conditional on their development—for example, rewarding innovators through payment mechanisms that result in marginal cost pricing such as in the AMC, thus avoiding the deadweight loss that arises under monopoly pricing. Finally I discuss how innovation inducement prizes can be designed, as in the patent buyout, so as to reward the technology in a way that places it in the public domain. In contexts where innovation is cumulative—in the sense that an innovation may exist in a technology market where it is useful as an input to future discoveries—this may be beneficial in encouraging subsequent innovation, as Kremer (1998) argues was the case with the Daguerreotype photography technique. For more detailed discussions of the many of the topics covered in this section, see Abramowicz (2003), Kremer and Glennerster (2004), Maurer and Scotchmer (2003), and Kremer and Williams (2010). Fisher and Sayd (in press) present a detailed discussion of prizes as applied to health problems in developing countries, discussing AMCs as well as mandatory prize systems such as the Health Impact Fund proposal, the Medical Innovation Prize Fund proposal, and their own proposal. Incentivizing Innovation A major challenge in designing prizes is how to design the prize so as to spur innovation into desired technologies. Below I discuss issues relevant to setting the size of a prize, and then Innovation inducement prizes 16 discuss ways of structuring the prize payments so as to attempt to reward innovators in a way that roughly coincides with the social value of the technology. I next discuss how sponsors might choose and refine prize targets. Importantly, most innovation inducement prizes are predicated on sponsors being able to define in advance the characteristics of the technologies they wished to award (as in the SERP and AMC prizes); this will not be possible for all desired innovations. Setting the Size of a Prize How should sponsors set the size of a prize? If the size of the prize is set too low, it may fail to spur research. If the size of the prize is set too high, sponsors may overpay relative to what was needed in order to spur the development of the technology. It is worth noting that an analogous question arises in patent design, with policymakers needing to choose the breadth and length of the patent. Identifying an example of a prize that failed to spur innovation because of being too small in size is difficult: Many prizes have not yet been claimed, but it is difficult to know whether the desired technology would have been developed had the size of the prize been larger. That said, one frequently cited example is a $1 million prize established by the Rockefeller Foundation in 1994 for a low-cost, rapid, easy-to-perform diagnostic test for gonorrhea and chlamydia that is appropriate for use in resource-poor developing countries. The prize was never claimed, which Kalil (2006) argues was because it had too many specific requirements—including that the test would be 99 percent accurate, cost less than 25 cents per usage, use noninvasive samples, and provide immediate and reliable results that could be interpreted by health workers with a primary education and no more than 2 hours of training. Innovation inducement prizes 17 Of course, on the other end of the spectrum are concerns that if the size of a prize is set too high, sponsors will overpay for a technology that they could have obtained at a lower cost. As with the case where the prize is set too small, identifying examples of such contexts is complicated by the lack of a clear counterfactual of whether the technology would have been developed had the prize been smaller. A related issue, as I will discuss in the next section, is that larger prizes may spur the technology to be developed more quickly, or spur the development of a better technology, either of which could generate social value. Many observers focus on the concern that setting the size of the prize to be too large may induce too much research, which could result in inefficient races that duplicate innovative effort. Importantly, the possibility of such inefficient races is not unique to prizes, and can arise in the absence of prizes as well (see Loury, 1979). On the other hand, Kremer and Glennerster (2004) note that in many cases it may be helpful to pursue many different leads simultaneously when searching for solutions for important problems, or it may be useful to have multiple competing teams with their own ideas of how to execute the project. Along the same lines, Berndt et al. (2007) argue that sponsors contemplating an increase in prize size must trade off the risk of inefficient duplication of effort against the benefit of a shortened expected wait for the technology and the possibility of obtaining a better technology due to more entrants. As is hopefully clear from the above discussion, there is not one magic correct size of any given innovation prize, below which no research will take place and above which further resources will be wasted. Berndt et al. (2007) provide a detailed analysis of the question of setting the size of a prize for the example of an Advance Market Commitment (AMC) for a malaria vaccine. Their Innovation inducement prizes 18 strategy is to estimate how large an Advance Market Commitment for a malaria vaccine would need to be in order to give inventors incentives that are roughly on the same order of magnitude as the incentives embodied by product markets for diseases prevalent in rich countries. The basic idea is to estimate the net present value of revenues earned by a sample of recently launched commercial pharmaceutical products, and to then adjust this figure on a number of margins. This type of analysis provides one example of how to calibrate the size of an innovation inducement prize. Designing Prizes to Reward Socially Desirable Technologies Despite its shortcomings, one benefit of the patent system is that it creates at least a rough link between the reward an inventor receives and the value consumers place on the invention. Under the patent system, if a firm develops a better product that is more desirable to consumers, the firm will be able to charge a higher price for that product—thus giving firms incentives to invest in product quality improvements that might be difficult for a social planner offering a prize to anticipate at the time the prize is offered. In the case of prizes, an important challenge is to design a prize mechanism that can at least partially replicate this (desirable) feature of the patent system. That is, poorly designed prizes may reward innovators who create technologies that meet an ex ante set of technical specifications, but for some reason are not valuable to consumers. Going back to the Super Efficient Refrigerator Program (SERP) example, if prize payments had been paid solely on the basis of technical specifications, Whirlpool would have collected the prize funds even though ex post it turned out that the characteristics of the winning technology were not very desirable to Innovation inducement prizes 19 consumers. The fact that the prize payments were designed so as to be linked to a market test avoided that outcome. The SERP example relates to a broader question of what event should trigger the payment of rewards. Kremer and Williams (2010) discuss various reward triggers in detail. First, prizes can be awarded solely on the basis of ex ante fixed technical specifications. In some cases, such as the Wolfskehl prize rewarding the first person to prove Fermat's Last Theorem, there may be little room for problems to arise under this approach. However, in other cases, such as the SERP example if there had not been a market test, this approach may be problematic. Second, prizes can be awarded based on metrics of ex post use, such as the allocation of subsidies per person immunized under the Advance Market Commitment (AMC) or the market test under the SERP prize. Basing prizes of metrics of ex post use is one example of how the size of an innovation prize can be linked to ex post information, which may often be desirable as a way of orienting research towards the most socially useful technologies (within the set of possible technologies that would qualify under the technical specifications of the innovation inducement prize). Kremer and Williams (2010) argue that essentially any mechanism for rewarding innovation involves some degree of ex post discretion, but that mechanisms vary in how much ex post discretion is allowed and to whom ex post discretion is allocated. Both the patent system and prizes rewarded on the basis of ex post use leave ex post discretion relatively more in the hands of consumers—which may be desirable as a way of avoiding potential problems such as inventors viewing the preferences of a prize committee as time-inconsistent. Innovation inducement prizes 20 Patent buyouts are an alternative mechanism through which ex post information can be used to set the size of an innovation prize. Kremer (1998) proposes a mechanism for implementing patent buyouts in which the private value of a patent would be determined using an auction. In Kremer's mechanism, in most cases the government would purchase the patent at a (mark-up of the) price determined by the auction he proposes, and place the technology in the public domain; however, in order to provide auction participants with an incentive to truthfully reveal their valuations, a small share of patents would be randomly selected to be sold to the highest bidder. The auction mechanism Kremer proposes, and the idea of basing prize payments on ex post use, both aim to overcome the types of informational constraints highlighted by Demsetz (1969) and Wright (1983), and award prizes in a way that roughly corresponds to the value of the innovation. Of course, clear concerns arise if firms can collude or otherwise manipulate these market signals (as discussed in detail in Chari, Golosov, & Tsyvinski, in press). Kremer (1998) and Abramowicz (2003) discuss possible forms of collusion in patent buyout mechanisms, and possible ways to address them. Choosing and Refining Prize Targets A natural question that arises is what types of technologies may be a good fit with innovation prizes. Here, it is helpful to first lay out some concrete examples. Kalil (2006) discusses how prizes might be applied in five areas. First, he suggests that NASA pursue prizes for Earth-Moon solar sailcraft and a lunar lander-rover. Second, he proposes that the U.S. and other developed countries offer prizes for technologies relevant to African agriculture, listing the specific goals Innovation inducement prizes 21 suggested in Kremer and Zwane (2005) such as disease-resistant bananas, cassavas, and millet. Third, he highlights the then-in-progress Advance Market Commitment (AMC) for vaccines needed in low-income countries. Fourth, he proposes that the Department of Energy investigate prizes for zero-energy buildings, methods of reducing urban greenhouse gas emissions, and fuelefficient cars. Finally, Kalil proposes that the Department of Education and other agencies investigate prizes for technologies such as software for early reading, math, and science software for middle schools, and software for introductory college courses in math and science. More broadly, National Academy of Sciences (2007) discusses the issue of choosing prize targets in detail for the case of the National Science Foundation. Based on a review of prizes offered in the past by the Defense Advanced Research Projects Agency (DARPA), the National Aeronautics and Space Administration (NASA), and the X-Prize Foundation, the National Academy of Sciences (2007) report concluded that choosing prize topics requires extensive consultation with experts, affected parties, and categories of potential participants. For example, a recent round of NASA's prizes were chosen following an internal process of idea generation as well as a two-day workshop attended by over 200 representatives from industry, universities, and government agencies. Although the National Academy of Sciences (2007) report examined a few potential technologies in detail—such as nano self-assembly and green chemistry—the report stresses that these possible prize topics were discussed not with an eye toward recommending them, but rather as a means of exploring and better understanding how prize contests could be structured around them. Innovation inducement prizes 22 One possible model for the type of consultation process recommended by the National Academy of Sciences (2007) report is the working group that was set up to assess the Advance Market Commitment (AMC) proposal before it was implemented. As described on the website for the working group, members included individuals (participating in their individual capacity) from the World Bank, the Brookings Institution, the biotech industry, life sciences venture capital, academia and the Bill & Melinda Gates Foundation, and the group's work was complemented by the pro bono assistance of experts in contract and life sciences law from Covington & Burling. This working group's conclusions were set out in a report that laid out a specific proposal for the design of a prize (Barder et al., 2005), together with draft contract term sheets to illustrate how the AMC might be translated into legally binding contracts. For agencies thinking of implementing prizes under the America COMPETES Act, the recently streamlined legal process (as outlined in the March 2010 memorandum from the Office of Management and Budget) likely means that the contractual side of prize design will require less attention than it did in the case of this AMC Working Group. However, the process of bringing together this type of consultation group—or similar consultation processes such as those undertaken by DARPA and NASA, as described in National Academy of Sciences (2007)—seems likely to be a useful model for agencies looking for a process through which to choose and refine prize targets. Defining the Desired Product in Advance: Prizes and Research Grants In each of the Super Efficient Refrigerator Program (SERP) and Advance Market Commitment (AMC) examples, prize sponsors laid out a detailed set of technical specifications in advance, clarifying the technological and market requirements that a given innovation must meet in order to be eligible to receive prize payments. However, in many cases inventors will have ideas for Innovation inducement prizes 23 new technologies that no prize sponsor will have thought of in advance. Such serendipitous scientific discoveries may often have high social value, but are difficult to stimulate using innovation inducement prizes because the desired technology would not be known at the time the prize is announced. Building on this type of argument, many authors (including National Academy of Sciences, 2007) have argued that this feature of innovation inducement prizes—being unable to stimulate research on serendipitous scientific discoveries—complicates the use of prizes to stimulate investments in basic, fundamental scientific research. Kalil (2006) takes the position that innovation inducement prizes will not work for long-term, fundamental research, because it is often impossible to specify the goal of fundamental research in advance. He argues that many of the most interesting discoveries in science are serendipitous, implying that even if the goals of a prize would be generally understood, it may be difficult to develop appropriately specified proxies for those goals. In general, there seems to be widespread agreement that basic research typically cannot be supported via pull incentives like prizes given that it is very difficult to define the desired output in advance. Direct funding of research through institutions such as the U.S. National Institutes of Health (NIH) and the U.S. National Science Foundation (NSF) has been very successful in generating innovation. For at least two reasons, the grant financing provided by these institutions may be more appropriate than prize mechanisms. First, Stephan (2011) argues that because prizes award funding only after project completion, as opposed to grants that cover research costs in advance, academics may only be able to complete prize-incentivized research if it complements research Innovation inducement prizes 24 supported by other means or if partnerships can be built with industry. Second, prizes and research grants may have different implications for the disclosure of scientific research results. Maurer and Scotchmer (2003), National Academy of Sciences (2007), and Boudreau and Lakhani (2009) discuss how research grants may encourage more sharing and publication of scientific results than will prizes, which may stifle sharing and publication if researchers revert to secrecy in an effort to win the prize. To the extent that sharing of early-stage scientific research results generates spillovers that encourage future technological progress, this would provide an additional justification why research grants may have advantages relative to prizes for funding basic research. Avoiding Monopoly Pricing Distortions In some cases—such as a prize seeking to encourage a demonstration project—the only goal of the innovation inducement prize will be to encourage innovation on the desired technology. However, in other cases the prize will aim to spur the development of a technology that will actually be used directly by consumers. In such cases, evaluating the social value of the prize requires taking into account how the rewarded technology will be priced. As previously discussed, patents provide dynamic incentives for innovation at the cost of making goods more expensive to consumers during the life of the patent, which generates deadweight loss. Prizes can be designed to minimize deadweight loss from monopoly pricing in at least two ways. Innovation inducement prizes 25 First, prizes can be designed so as to reward innovators in a way that allows marginal cost pricing, avoiding the static efficiently losses from monopoly pricing. The Advance Market Commitment (AMC) structure gives an example of one mechanism for doing this: The AMC distributes rewards through a price subsidy system that explicitly aims to provide close-tomarginal-cost access to the technology for consumers in both the shortand long-run. Second, prizes can be designed so as to use pricing conditions as part of the technical specifications. The Archon X Prize for Genomics provides one example. The Archon X Prize is a $10 million prize, sponsored by Medco (a U.S. pharmacy benefits management company), calling for a method of sequencing 100 human genomes in 10 days according to a set of guidelines for accuracy, completeness, and cost (less than $10,000 per genome). If the prize conditions were to stipulate that the technology needed to be sold to consumers at $10,000 or less, this would be a way of minimizing monopoly pricing distortions. To summarize, while prize mechanisms frequently focus almost exclusively on providing incentives for innovation, in cases where prizes aim to spur the development of products useable by consumers, it is important to focus attention on pricing—since pricing will be a key determinant of consumers' access to technologies conditional on their development. Encouraging Cumulative Innovation Kremer (1998) argues that when the government of France bought out the patent for Daguerreotype photography and placed the technique in the public domain, it was subsequently subject to myriad technical improvements—likely because the patent buyout eliminated the Innovation inducement prizes 26 costly reverse engineering that had previously been necessary to do subsequent research on improvements to the photography method. This example relates to what the economics literature calls cumulative innovation—the idea that a given technology may exist in a technology market, where it is an input into subsequent technological innovations. The idea that the patent on Daguerreotype photography may have discouraged innovation, and that placing it in the public domain may have encouraged cumulative innovation, relates to the more general question of whether property rights impose transaction costs in the market for technology. As discussed below, whether (or to what extent) property rights impose transaction costs in the market for technology is ambiguous from a theoretical perspective, and there currently exist a relatively small set of empirical studies providing evidence on this question. The tentative conclusion that can be drawn from this small available empirical literature is, at least in some cases, it appears that property rights such as patents can hinder cumulative innovation. This suggests there may be social value generated from awarding innovation inducement prizes in a way that places the rewarded innovation in the public domain, avoiding the potential discouragement of cumulative innovation. While more empirical evidence is needed to support this tentative conclusion, it raises the possibility that one useful context for innovation inducement prizes may be to reward innovations where patents may be expected to substantively distort cumulative innovation. Next I discuss the need for prize administrators to clearly define intellectual property rights, and subsequently discuss the theoretical and empirical literature exploring whether property rights hinder cumulative innovation. Innovation inducement prizes 27 Defining Intellectual Property Rights Innovation inducement prizes, if effective, will result in the development of new technologies. A question naturally arises of who has ownership over these technologies—namely, innovators, sponsors, or some other arrangement. McKinsey (2009) discusses a variety of arrangements that have been implemented in the past: Some prize sponsors (particularly businesses) insist on controlling intellectual property rights on all resulting technologies; others—such as Red Hat— ask participants for exclusive licenses; yet others leave intellectual property rights completely in the hands of the inventors. In the case of patent buyouts, prize sponsors could chose to place the technology in the public domain. The key point here is simply that clear definition of intellectual property rights in the initial prize offering is essential. Do Property Rights Hinder Cumulative Innovation? When research is cumulative in the sense that a given product is the result of several steps of invention and research, intellectual property rights on an existing technology can affect the incentives for subsequent innovation building on that technology. To give a concrete example, consider a patent on a human gene. Sequenced genetic data is a research input into subsequent, cumulative innovation: Scientists can study the sequenced genetic data in order to learn about the links between genetic variations and diseases, and can then apply that knowledge to develop medical technologies such as pharmaceutical drugs and gene-based diagnostic tests. Innovation inducement prizes 28 Why might gene patents hinder cumulative innovation? In 2001 pharmaceutical firm Bristol Myers reported to The New York Times that it had abandoned research on more than 50 cancerrelated proteins due to conflicts with gene patent holders (Pollack, 2001). This example illustrates that if innovation takes place in multiple firms—if one firm patents a human gene and a second firm like Bristol-Myers discovers a gene-based pharmaceutical treatment that requires licensing that gene—then the effect of patents on cumulative innovation depends on the ease with which an agreement can be reached for the first firm to license its gene to Bristol-Myers (see Green & Scotchmer, 1995; Scotchmer, 1991; Scotchmer, 1996; Scotchmer, 2004). In a classic Coasian framework (Coase, 1960), the two firms would always negotiate licensing agreements such that cumulative research is not hindered. Many have argued this is the policyrelevant view of the world. For example, George Church—head of the U.S. NIH-funded Personal Genome Project at Harvard—has said regarding gene patents: “Pull out your cell phone. There are hundreds of patents in that phone. Each one gets pennies from that phone sale. The marketplace will sort this stuff out” (qtd. in Goozner, 2010, p. 757). Church's view echoes conclusions from pioneering survey research by Cohen, Walsh, and colleagues that has consistently suggested researchers adopt “working solutions” to patents such that patents do not interfere with cumulative innovation (Walsh, Arora, & Cohen 2003a, 2003b; Cohen & Walsh, 2008). The Coasian framework assumes away transaction costs. However, if—for example—firms’ research costs are private information, negotiations may break down, and some socially desirable research may be deterred, as is claimed in the Bristol Myers example. Academics have long Innovation inducement prizes 29 expressed concerns about this potential problem (Machlup & Penrose, 1950; Merges & Nelson, 1990; and Heller & Eisenberg, 1998, among others) have argued problems are particularly likely to arise when researchers must negotiate multiple licenses. The concern that gene patents may be hindering research was one of the key controversies in the recent set of court cases investigating the validity of gene patents. Thus, the question of whether patents discourage cumulative innovation settles on the question of whether licensing markets for patents operate efficiently, or whether patents induce transaction costs that hinder cumulative innovation. Empirically, it is difficult to estimate the effects of intellectual property on subsequent innovation, largely because technologies that are held with patents in most cases will be inherently different than technologies that are in the public domain. For example, Moser (2007) finds evidence that higher quality innovations are more likely to be patented. Such patterns of selection into patenting complicate simple comparisons of the levels of cumulative research on— say—patented and non-patented technologies, since the levels of cumulative research may reflect both any effects of patents on subsequent innovation, and the selection effect of which technologies are patented in the first place. A number of recent papers have attempted to circumvent such selection issues in order to isolate the causal effect of intellectual property rights on subsequent innovation. Murray and Stern (2007) use data on life sciences technologies, and find that patent grants decrease citations to scientific papers on the patented technology, relative to scientific papers on similar non-patented technologies. Murray et al. (2008) find that the removal of patents on certain types of genetically engineered mice increased citations to scientific papers on affected mice relative to scientific Innovation inducement prizes 30 papers on unaffected mice. Murray et al. (2008) also find evidence, consistent with the model of Aghion, Dewatripont, and Stein (2008), that patents reduce the diversity of scientific experimentation. Finally, Williams (2010) finds evidence that non-patent contract-law-based intellectual property on sequenced human genes appears to have discouraged both subsequent scientific research (in terms of gains in knowledge about the functions genes have within the human body) and subsequent product development (in terms of the use of sequenced genes in genetic diagnostic tests that are available to consumers). To summarize, although only tentative conclusions can be drawn given the small number of empirical studies, the body of available empirical evidence suggests that patents may substantively hinder both subsequent scientific research and subsequent product development. Across a relatively heterogeneous set of technologies within the life sciences, and examining various forms of intellectual property rights, the available empirical evidence suggests that property rights hinder cumulative innovation—with declines on the order of 30 percent. Clearly much more work is needed in order to examine the extent to which these patterns generalize to other technologies and other forms of intellectual property, but the best available evidence suggests that mechanisms that reward innovation in a way that places the technologies in the public domain—such as patent buyouts—may have substantial benefits in terms of encouraging cumulative innovation, at least in some contexts. EVALUATING INNOVATION INDUCEMENT PRIZES: LEARNING WHAT WORKS Innovation inducement prizes 31 Federal agencies considering offering innovation inducement prizes likely have a number of questions about prize design issues. In what contexts can prizes successfully incentivize innovation? How large should the prize be? What is the appropriate time frame for prizes? While economic theory can offer some guidance on how to design innovation inducement prizes, empirical evaluations are needed in order to understand the specific contexts in which innovation inducement prizes can be effective, and what forms of innovation inducement prizes are most likely to be successful in any given context. One might think of evaluating the effectiveness of innovation inducement prizes by asking whether the desired technology was developed (that is, whether the prize was awarded). Looking at the list of approximately 200 innovation inducement prizes compiled by McKinsey (2009), all but a handful of these prizes have been awarded. However, for several reasons this type of analysis does not clarify answers to many types of questions. Would the technology have been developed even in the absence of the prize? Would the desired technology have been developed more quickly if the structure of the prize had been different? Would a more effective version of the technology have been developed if the structure of the prize had been different? Answering these questions requires careful construction of a clear counterfactual that posits how innovation would have evolved in the absence of the innovation inducement prize. Unfortunately, only a small number of studies exist that have conducted this type of counterfactual analysis, and as a result there is currently a very small base of empirical evidence from which to draw conclusions. Innovation inducement prizes 32 Maurer and Scotchmer (2003) argue that when looking across incentive schemes—for example, prizes relative to intellectual property—no single incentive scheme dominates, and different models of knowledge creation call for different incentive schemes. An analogous lesson almost surely applies to designing prize features in different contexts. The general approach of small-scale field experimentation paired with careful evaluation seems very promising as a way to shed light on many of the specific questions about how best to design prizes, and in which contexts prizes can be most effective. The sub-section below lays out the fundamental evaluation problem that complicates assessments of the success of innovation prizes. In light of a clear need for further evidence in this area, I highlight three types of analyses that have provided useful evidence on these questions: case studies, historical data, and field experiments. The Evaluation Problem Consider the Archon X Prize for Genomics discussed above, which is a $10 million prize for a method of sequencing human genomes that meets a set of criteria. If the prize were to be awarded, would that mean the prize was successful? Answering this question requires the construction of a counterfactual analysis in order to estimate how innovation in genetic sequencing would have evolved in the absence of the Archon X Prize. The difficulty in answering this question is immediate, since the path of genetic sequencing innovation can be observed either in the presence of the Archon X Prize or without it—but not both. Comparing innovation before and after the prize announcement is complicated Innovation inducement prizes 33 by concerns that other factors are also changing at the same time the prize is announced. To estimate a counterfactual, one needs to construct a comparison group that could reasonably be expected to have had similar outcomes to the treatment group in the absence of the prize. In this case, the comparison group might be a set of technologies that had similar trends in research investments to genetic sequencing technologies prior to the Archon X Prize announcement. Why is this type of counterfactual analysis important? One reason is that a number of firms were making investments in improved methods for genome sequencing prior to the announcement of the Archon X Prize. Moreover, the sponsors of the Archon X Prize may well have chosen to establish the prize in order to reward the type of technology that they expected to be most useful—but other investors and firms may have had similar expectations about which technologies would be most useful (or profitable), and could already have been orienting their research effort towards this type of sequencing method. Thus, it is not straightforward to answer the question of whether the Archon X Prize spurred innovation, even if the prize is eventually awarded. A closely related issue is that the margin on which many prizes may operate is to move forward in time the date at which a given technology is developed. Consider the case of the AMC for a pneumococcal vaccine. The AMC was formally launched in June 2009, and in October 2009 four vaccine suppliers made offers to supply vaccines under the AMC. In March 2010 two companies made long-term commitments to supply vaccines under the AMC, and the roll-out of the pneumococcal vaccine began in Kenya in February 2011 (and subsequently expanded to a series of other developing countries). The natural question is the following: Can these events be Innovation inducement prizes 34 attributed to the introduction of the AMC? Answering this question requires undertaking a counterfactual analysis to estimate what the roll-out of the pneumococcal vaccine would have looked like in the absence of the AMC. An early policy report on AMCs (Barder et al., 2005) argued that AMCs could likely be applied cost-effectively to both technologically closer products, such as a pneumococcal vaccine, and to technologically more distant products, such as a malaria vaccine. In the case of technologically closer products, the goal of the AMC would less be focused on spurring vaccine research, and more focused on speeding the adoption and diffusion of new vaccines. The recent historical record has suggested that adoption of new vaccines in poor countries is often delayed by 10 to 15 years after their introduction in developed countries; for example, even 14 years after the introduction of the Haemophilus influenzae B (Hib) vaccine in the U.S. and Europe, less than 10 percent of infants in the world’s poorest 75 countries were routinely receiving this the vaccine (Levine et al., 2004). Such delays in the roll-out of existing vaccines to poor countries contribute to the approximately 2.5 million deaths that occur annually in poor countries as a result of vaccine-preventable diseases. In the case of the pneumococcal AMC, it is likely that the vaccine would have eventually been rolled out in poor countries. However, given the substantial morbidity and mortality burden that accrues annually as a result of vaccine-preventable diseases, the pneumococcal AMC likely generated substantial value even if its primary purpose was to move forward the point in time when the pneumococcal vaccine was distributed in poor countries. Berndt et al. (2007) estimate that if a malaria vaccine AMC advanced vaccine development by 10 years and accelerated access Innovation inducement prizes 35 in poor countries by 10 years, it would cost about $23 per additional disability-adjusted life year (DALY) saved; even in the extreme case where the malaria vaccine AMC accelerated vaccine development by only 1 year and adoption in poor countries by only 2 years, the program would still cost less than the oft-cited $100 per DALY cost-effectiveness threshold for the poorest countries Thus, a counterfactual analysis is needed in order to answer several questions. First, would the technology have been developed even in the absence of the prize? Second, did the prize result in the desired technology being developed more quickly? Third, did the prize result in a more effective version of the technology being developed? Of course, there is no single right counterfactual analysis, and any counterfactual analysis will rely on some assumptions. A general goal is to ask whether, under a range of reasonable counterfactuals, the prize would be considered successful on cost-effectiveness grounds. The Berndt et al. (2007) analysis provides one example of how such an analysis can be conducted in advance of a prize being announced. Case Studies One approach for learning about the effectiveness of past prizes is to conduct case study analyses. McKinsey (2009) presents (in Appendix 1) a number of case studies giving detailed discussions of various past prizes, but not in a way that is focused on counterfactual analyses. The only case study of which I am aware that attempts to conduct a counterfactual analysis is the in-progress counterfactual analysis of the pneumococcal vaccine AMC. A 2008 monitoring and evaluability study report tasked a study team with selecting comparator vaccines, and making the Innovation inducement prizes 36 adjustments necessary to allow for a reasonable counterfactual or range of counterfactuals to be estimated. The report clarifies that, ideally, the comparison vaccines would control for the influence of factors such as the availability of funding for vaccination in low-income countries, technological characteristics that may affect the price or production of vaccines, and market size. Note that to the extent the introduction of the pneumococcal AMC had spillover effects on other vaccines (for example, shifting available non-AMC donor funding away from pneumococcal and towards rotavirus or other potential comparator vaccines), the use of historical vaccines as controls may be desirable. Although still in progress, this report provides one example of the type of analysis that can be undertaken to evaluate the effectiveness of innovation inducement prizes. Historical Data Two recent papers have looked deeper into the historical record to construct counterfactual analyses in order to assess the effectiveness of prizes in spurring innovation: Brunt, Lerner, and Nicholas (in press) and Nicholas (2010). Brunt, Lerner, and Nicholas (in press) collect a novel dataset in order to analyze innovation inducement prizes awarded by the Royal Agricultural Society of England (RASE) between 1839 and 1939. The goal of RASE was to encourage scientists to apply their skills to improving agricultural technologies. Starting in 1839, RASE held annual prize competitions. One year in advance of the competitions, RASE announced which technological areas would be targeted as well as the number and value of prizes to be awarded in each area; judges authorized payment of awards, or withheld them if the criteria for winning were not met, and were also given discretion Innovation inducement prizes 37 to award additional ex post prizes. These competitions awarded substantial monetary prizes (in excess of 1 million pounds in current prices) as well as prestigious but non-pecuniary medals. Between 1839 and 1939, 15,032 inventions competed for these prizes and a total of 1,986 awards were made. In order to examine the question of whether these prizes encouraged innovation, the authors assemble data on all applications for (and grants of) British patents from 1839 to 1939, matched to information on competition entrants, prize winners, and prize schedules (that is, the preannounced targeted technological areas as well as the number and value of prizes). Following previous work, they also collect information on whether renewal fees were paid for granted patents as a proxy for the quality of patents (since inventors should be more willing to pay renewal fees for more valuable patents; see Schankerman & Pakes, 1986). Using this data, the authors present a number of empirical results. First, they find that the RASE contests attracted large numbers of entrants. This is true for both pecuniary and non-pecuniary prizes, with the largest entry effects arising from the non-pecuniary RASE gold medal. Second, they find that prizes are associated with “real” changes in contemporaneous patenting activity in the technological areas targeted by the RASE contests. This result suggests that RASE prizes were spurring not only the entry of technologies into RASE contests, but actually spurring the development of new technologies (as measured by patents) that would not otherwise have been developed. Importantly, the induced innovation seems to be composed of “high quality” inventions as measured by the renewal fee metric described above. Within the sample of high quality patents as defined by this measure, the authors find that a doubling in monetary prize Innovation inducement prizes 38 value is associated with a 4 percent increase in contemporaneous patents, and that an additional medal is associated with a 20 to 21 percent increase in contemporaneous patents. In a second recent paper, Nicholas (2010) examines a similar research question in the context of Japan's Meiji era—during which patents were introduced in Japan (in 1885) and a large number of mostly non-pecuniary prizes were awarded (by 1911, 1.2 million prizes were awarded at 8,503 competitions). Using a methodology similar in spirit to that in Brunt, Lerner, and Nicholas (in press), he finds evidence that prizes increased patent outcomes on the order of 30 percent. To summarize, both studies suggest prize awards—including non-pecuniary prize awards—can encourage not only entry into prize contests, but also real innovation, as proxied by patenting activity. While the results of these studies clearly do not imply that prizes will successfully spur innovation in all cases, they are suggestive that the types of prizes that have been implemented in the past can be successful on this metric. Field experiments A promising area for future research investigating the effectiveness of innovation inducement prizes is field experiments. By construction, randomized field experiments provide a clear counterfactual analysis: Because treatment and control groups should have been similar in the absence of the treatment, any differences in outcomes between the treatment and control groups can be attributed to the effect of the treatment. Innovation inducement prizes 39 A recent example is the work of Boudreau and Lakhani (2011), which involved a field experiment conducted in collaboration with TopCoder—a private firm that administers computer programming contests. Boudreau and Lakhani report the results of a 10-day randomized field experiment during which over 500 elite software programmers developed solutions to a particular algorithmic engineering problem from NASA's Space Life Sciences Directorate. Working with TopCoder allowed them to utilize detailed data on workers skill (as measured ex ante, prior to the start of the experiment) as well as effort and objective measures of problemsolving performance. The focus of this experiment was on the effectiveness of allowing participants to self-select into competitive versus team-based regimes. Specifically, the authors were interested in evaluating whether allowing workers to sort into different institutional settings, based on their intrinsic preferences, would increase performance. An additional, cross-cutting randomized treatment involved some groups competing for a cash prize of $1,000 while others competed for a noncash prize. Their results suggest that both self-selection and the cash prize of $1,000 had significant economic effects, with each treatment nearly doubling problem-solving performance. The authors found that the effect of cash incentives was significantly greater for higher skilled participants (as measured by ex ante skill, prior to the start of the experiment) relative to lower skilled participants, consistent with higher skilled workers having more of a chance of winning