NBER Entrepreneurship and Innovation Policy and the Economy Conference, 2026 edited

Added: 2026-05-12

[00:00:00]actually the 26th edition of the broader series that we that Ben and I are organizing now and in earlier generations uh Scott Stern and Adam Jaffy were involved with as well. And we have today what's a super exciting program dealing with some of the uh biggest issues on innovation and mostly innovation policy though we get a little into entrepreneurship as well. Um, and I think it's going to be a great day. Um, a couple housekeeping things beforehand.

[00:00:34]One of which is the NBR is strict on code of conduct. So, please behave yourselves. And again, it's nine o'clock in the morning, so probably the odds of misbehavior are pretty low, but we just make sure. Um, the other thing is that we do to try to broaden the reach of this, we do a live stream. So somewhere out there in the ether is people watching it, which is great. But this has two uh preconditions. One of which is of course if you don't want somebody in Beijing or Moscow to hear what you're saying, probably don't say it. Uh but the other thing is uh uh we are going to have plenty of time for questions and so forth but please do use the uh please do use the mics because if you don't use the mic the people on the live stream can't see it and then they start sending me grumpy emails which I have enough of as it is. All right with with that without further ado I'm going to turn it over to Ben um and we will get into the first of the three sessions we have today.

[00:01:40]Thanks Josh. Uh good morning everyone.

[00:01:42]Uh it's it's great to see everyone here.

[00:01:44]Uh and our first uh two papers are one from CBO and one from JCT. And I know Josh and I are just delighted to have uh CBO and JCT participating here uh and bringing their insights and questions uh into this conversation. You know, this conference is very much about a sort of two-way street between sort of academic research, about entrepreneurship and innovation and science policy and the practice of that policy. And so bringing a community together uh to sort of make sure we're asking the right questions to sort of synthesize recent economic literature to kind of inspire academics to sort of choose questions of of of great practical importance and that's what this is all about. So, you know, it's really great to get this community together. All right. So, let without further ado, um we're going to first have the paper uh from uh CBO uh estimating the economic effects of federal investment in research and development. Uh we're delighted to have Caleb presenting. Uh and so we're going to plan about 22 minutes for the presentation. Then we'll have some Q&A time.

[00:02:39]>> I think you have a mic there.

[00:02:42]>> Okay.

[00:02:58]And then where's the Oh, can you hear me?

[00:03:00]>> Okay. Okay. Thanks. Um, so thanks so much for the to the organizers for um for giving us an opportunity to present this and to write this paper. Uh there's a few of us from CBO who are on this paper, but the work here reflects the work of lots of different people at the agency. um former employees uh some of whom are here um who've been contributing to this work over the last few years. So in this paper we're going to ask how changes in federal investment and research and development affect the macroeconomy.

[00:03:29]So in recent years Congress has shown interest in making policy changes that affect research and development and incentives to uh perform research and development. A couple of examples. So a few years ago, the chips and science acts modified uh federal funding directly modified federal funding for R&D investment. More recently, as part of the 2025 reconciliation act, there were changes in corporate tax policy that affected uh tax provisions that um affect the after tax price of R&D. And so that's the the the folks at JCT, their paper is about, you know, the R&D tax credit and and how that affects incentives to do R&D in the private sector.

[00:04:06]So in this presentation, what we're going to do is we're going to talk about um how uh the current basis of assessment that we have at CBO for modeling the economic effects of changes in federal funding for non-defense R&D.

[00:04:17]Uh the goal for this is sort of twofold.

[00:04:19]So on the one hand, there's a bunch of people in the room who have thought really hard about this and so we're hoping to get feedback on on how we're modeling it um so that we can provide better answers to Congress the next time that there's um because there's continued policy interest in this. and then also sort of point out places where we think there are um where we've had trouble applying the academic literature in ways that um that we would like to and so places where we think there's opportunities for more research and where we'd like to see more work. Uh the way that we're going to sort of quantify this is we're going to model the effects of a a number a couple of different policy experiments where we study what we're where we estimate what would happen if there were either increases or decreases in federal funding for non-defense R&D of about $30 billion per year over 10 years. So starting in 2027 and this is similar to some even more preliminary analysis that we did about a year ago again in um response to interest from Congress um where we uh presented the results of a similar set of policy experiments.

[00:05:16]Okay. So the sort of road map for um what we're going to present here. So we have two analytic frameworks that we use for modeling how R&D affects the economy. Uh we think these are complimentary approaches. So they connect in different ways with how researchers, academics, and policy makers think about how R&D spending affects the economy. Um so the first is what we're going to refer to as the R&D capital stock approach. Here we're going to model R&D investment like we do other types of investment that's done by the private sector or by the federal government. So, uh, when there's investment in R&D from either the private sector or the government, that's going to contribute to an R&D to a capital stock, an R&D capital stock, which, you know, you can sort of analogize to like a a stock of knowledge. Um, that stock of knowledge is then over time going to sort of depreciate away and that's going to reflect the obsolescence of knowledge over time. And then that capital stock is going to sort of directly affect macro variables that we care about, things like total factor, productivity, and output.

[00:06:16]What's nice about this is that it's directly connected with the standard way that R&D appears in the national income and product accounts as produced by the BEA. It's also closely connected to a bunch of recent academic work. Um, and so it's very easy to sort of take the results from those academic papers and apply it in our setting. It's also very closely connected to prior work that we've done at CBO when thinking about the economic effects of other types of uh investment, in particular federal investment and physical infrastructure, things like highways. And so uh there's sort of some consistency between the way that we model these different types of investment.

[00:06:49]The second approach is what we're going to call the human capital approach. Um here what we take really seriously is that lots of um federal spending on R&D goes to higher education institutions and in particular uh we think is going to train new researchers.

[00:07:04]Uh so the way that we operationalize this is we sort of separately allocate federal funding for R&D to labor and capital in a way that's consistent with historical spending patterns. And then the subset of that funding that goes to labor is going to support both the work of additional researchers. So if you receive uh more grant funding, you can hire an RA for a couple of years. That person is going to sort of directly contribute to the innovation process.

[00:07:27]But it's also going to contribute to the training of new researchers. And this is going to have potentially um significant effects on things like innovation and um and R&D because if this person has more education, they're able to contribute more to innovative activities over their sort of entire career. So that's going to have potentially much longer lasting effects um and it's going to have different effects than sort of directly hiring someone to work for for uh two years in a lab.

[00:07:51]This analysis is is also consistent with some prior work that we've done. in this case sort of estimating the e economic effects of changes in immigration policy and in particular immigration um for foreign nationals who work in uh highly skilled foreign nationals who work in STEM fields. Um it's also consistent with the sort of traditional way that we model capital stock because we're going to allocate some of this money to labor, some of it to capital, and then capital is going to affect economic variables in the way that we sort of the standard way we think about investment affecting the economy.

[00:08:21]Okay, so I'm going to start off by presenting some background, some stylized facts that are going to kind of frame the analysis and and motivate these two methodologies.

[00:08:30]So this is just showing federal outlays for R&D. This is for 2023, which is the most recent year for which data are available. What we want you to take away from this slide is that you can sort of broadly characterize R&D that the federal government does into non-defense and defense R&D. Um, you know, defense R&D was about $80 billion in 2023. Most of that went to the Department of Defense. non-defense R&D goes to a wider set of um departments and agencies although most of it is going to the department of health and human services and is spent out through organizations like the NIH. So you know when I'm thinking of this work like the sort of modal grant that I have in the back of my head is is a grant that's being um spent by the NIH to fund health research or science research.

[00:09:11]Okay. So then once that you know those are where the outlays are coming from who is receiving this money the bulk of the funding is going to for non-defense R&D is going to institutions of higher education. This is going to sort of motivate this human capital approach because again we think a lot of this money is going to universities and then is being spent on the training of researchers. So about 40% is going to uh these institutions of higher education.

[00:09:34]um smaller amounts are going to about a quarter is going is being spent within the federal government and there's also significant scums going to the private sector and to the FFRDC's. Uh we'll see papers later on the FFRDC's. And one thing I think to note is that we think there's um that potentially some of the money that's even some of the money that's not going directly to the institutions of higher education is sort of indirectly um supporting uh the training of uh graduate students and postocs and researchers. So I was at Berkeley before I was at CBO and there were lots of graduate students and postocs there who were being funded in part through work that they were doing at the Lawrence Berkeley lab or were getting post-docal training at LBL and so we think even some of the non uh the money that's not being directly spent by the higher education institutions is still potentially uh helping to train researchers.

[00:10:20]Okay. So now I'm going to talk uh uh lay out our two frameworks. I'm going to start with the R&D capital stock framework. So here there's sort of three sets of parameters that are going to be really important for determining the sort of timing and magnitude of the effects of spending on the economy.

[00:10:37]We're going to need to think about um how much money is spent by the federal government and over what time period uh that money is spent out. Then once you uh have accounted for the outlays, you need to think about the sort of lag structure between when money is actually spent out on additional innovative activity and when that affects sort of macro variables that we care about. And then of course the magnitude of those effects and then finally uh you know you need to uh estimate how persistent those effects are and that's going to be parameterized by the depreciation rate for R&D capital.

[00:11:10]Okay. So what this graph is going to show is this is going to this these are our estimates of the lags between when money is appropriated by Congress and when it's actually spent out by the agencies. So we've shown here is on the horizontal axis you have uh years since the funding um has been appropriated and then the y- axis is the share of the money that's being spent out. So these are for the five largest players of non-defense R&D. They represent about 75% of non-defense R&D. Um what you want to I think take away from this is so first of all there's quite a bit of heterogeneity in how different types of actors uh within the federal uh R&D infrastructure spend out money. So the NSF basic research accounts, those are spent out over longer periods over say seven years. In contrast, uh when you're looking at say NASA deep space exploration, most of that money is going to be spent out within a year of Congress appropriating that money. Um what we're going to do here, uh we also think it sort of matches uh at least anecdotally some of the uh sort of institutional um features of of grant funding. And so uh like for example the NIH basic and applied research grants are spent out over five years which approximately matches the average length of those NIH grants or the usual length of those NIH grants. Um so the way that we're going to operationalize this in the modeling is we're just going to take a simple weighted average where the weights are going to reflect the sort of share of non-defense R&D spendings R&D spending that's done by each of these different accounts.

[00:12:34]So then once you know sort of the time path of spending, you also need to think about the magnitude of spending. A key parameter for estimating this is going to be how spending by actors other than the federal government endogenous indogenously responds uh to changes uh in federal funding. So this is again this uh is a really important parameter when we think about other types of federal spending. So infrastructure spending um the federal government spends a lot of money on infrastructure but the state and local governments also spend lots of money on infrastructure.

[00:13:04]And so if you want to think about the aggregate effects of that you need to estimate when the federal government spends an additional dollar how much more or less is being sort of crowded out or crowded in by the state and local governments. In the context of innovation spending the major player uh the major other player besides the federal government is not the state and local governments. It's mostly the private sector. And so here what we need to estimate is when the federal government spends an additional dollar on innovation, how much more or less money is spent by the private sector?

[00:13:30]The way that we're going to do this is we're going to draw on the academic literature. So here in this graph, what we've shown is each dot represents an estimate from the academic literature on the response of either spending or innovative activity in the form of patenting in response to additional spending by the federal government.

[00:13:46]Um the thing to note here, so we're going to end up taking just a median of this. The thing to note here is that nearly all of the estimates are positive, which implies that when the federal government spends more money, it's not crowding out spending from the private sector. It's actually crowding in spending. So, uh, when the federal government spends a dollar, we're going to estimate that that leads to an additional 27 cents of spending from the private sector. And that's going to be close. So, the most, uh, the sort of the most recent uh, estimate of this that we're drawing on is from recent work by Andrew Fieldhouse and Carol Merrens where they estimate something like 20 cents on the dollar of crowd. So that's going to be pretty close to what we're going to what we're going to get just looking at the sort of wide swath of of estimates and papers here.

[00:14:24]So the key point is that the private sector response is actually going to amplify what the federal government does. It's not going to it's not going to be crowded out in this in this case unlike with the infrastructure case.

[00:14:35]Okay. So that determines the magnitude and the time path of spending. Then what happens uh to macro variables? So here now uh what we're uh again we're going to draw on the academic literature to look at the sort of response of productivity in this case total factor productivity to changes in federal spending on non-defense R&D. These are coming from a set of macro papers uh that are using time series style methods to estimate the productivity effects of changes in innovative activity or non-defense R&D spending.

[00:15:05]Um here what we've done is we've plotted sort of the the effect and we've normalized it. So here you're just seeing uh how the time paths compare across these various papers.

[00:15:14]It's difficult to make apples to apples comparisons. These are looking at different types of shocks. The shocks have different time profiles. Um and so uh you know you have to do some assumptions to compare these. But what we're um going to sort of show is that they have somewhat different spending paths and yet there are also some similarities. So for most of these papers you're seeing that there's some delay between when the outlays start and when there's actually uh productivity effects. So if you look 5 years out, the productivity effects are smaller than if you look out 15 years. And so that's how we're going to parameterize that there's going to be some delay between when these innovative activity is actually being taken place and when we're going to see macro effects on variables like total factor productivity output. In practice, what we're going to do in this paper is we're going to sort of lean very heavily again on the work of Fieldhouse and Merrens. Um you know I this here it looks like something that's sort of in the middle or it's it's not too different from these other papers.

[00:16:04]Um but in practice that's how we're going to we're going to look at this for for the analysis that we've done today.

[00:16:11]This is the same graph only we've allowed the magnitudes to differ now across these different papers. So this is the total response of TFP um across these different horizons. Again we're going to lean pretty heavily on the fieldhouse and mers. you know somewhere in the middle of these but um I think this is also a place where we we would love to see more research and um this type of time series um these types of time series estimates have been very very helpful to us and so this is a place where we think uh the research literature has uh has offered a lot and where we'd uh like to see more work as well.

[00:16:40]Okay, so that is going to give us the sort of initial time path of the of the spending and the time path of the sort of productivity effects. The final piece is you know how persistent do we expect this to be? The sort of parameter that's going to determine that is going to be the depreciation rate of R&D capital.

[00:16:57]Traditionally, what the research literature does is it applies BEA standard R&D depreciation rate which is about 16%.

[00:17:05]Um, and alternatively, what one could do is you could take work by Crawford and co-authors where they tabulate, you know, different depreciation rates for different categories of R&D. If you take a weighted average of that where again the weights are going to be the share of federal spending in R&D you're going to get a depreciation rate close to something like 10%.

[00:17:23]We think an important conceptual issue here is that the depreciation rates as measured by BEA are arguably intended to capture the private depreciation rate that's relevant to the owner of an R&D asset. So if I'm a private firm, I conduct R&D because it's going to have some effect on my profitability.

[00:17:40]Um, and then the depreciation rates that BEA uh calculates are sort of meant to capture the sort of declining way that that R&D that the ownership of that R&D asset continues to affect my productivity. But for economists, a potentially important issue of uh with sort of innovation and R&D is that there are these large spillover or that there are these spillover effects um where you know if I invent calculus then many other people can use calculus.

[00:18:04]Um that's going to affect the productivity not necessarily of my own firm but potentially many other firms.

[00:18:09]uh other people are then going to draw on those follow-on innovations. And so the relevant depreciation rate for us is not the private depreciation rate, but the economywide depreciation rate that accounts for all of these spillover effects. So the way that we're going to try and estimate that is we're going to sort of uh step away from the BA methodology a little bit and we're going to do an analysis based on patent citation rates. So what we're going to do is we're going to look at how citations to patents decline as a function of the time since the patent has been introduced. Based on that, we're going to estimate depreciation rates between four and 6%. And so we're going to use a 5% depreciation rate in our analysis.

[00:18:43]Okay. Now, I'm going to talk about the human capital framework. So, um, in this framework, again, we're focusing on the extent to which federal funding for R&D goes to train new researchers. The way that we're going to do that is we're going to estimate what share of R&D spending goes to labor. We're going to do this by drawing on data from the NSF's National Center for Science and Engineering Statistics and the BEA's R&D satellite account. We're going to estimate that on average 54% of federal funding for R&D is going directly to labor. In our estimates, we're trying to account for indirects as well, although the data for that is less good. So, we've done our best, but um this is a place where we think more data would be helpful. For higher ed institutions, we're going to draw on analysis of the um metrics data in a variety of papers to allocate this labor spending across different types of occupations and different types of workers. So people with different levels of education.

[00:19:29]We're again going to account for crowdin of spending from the private sector. And then for the labor component, we're going to estimate the effects through two channels. One is what we're going to call the reallocation channel. So here is where federal funding for R&D is going to allow additional people to work in the STEM sector contributing to innovative activities. Um and that additional work measured in researcher years is going to directly contribute to TFP. Then there's going to be the training component where the federal support for higher ed institutions is going to partially support the training of additional researchers because of the the time associated with that training and because of the fact that once you've trained a new researcher they can potentially contribute to innovative activity over their whole career.

[00:20:06]There's going to be uh those are going to contribute to TFP but with much longer lags. And so for capital we're going to apply then for the capital share of the spending we're going to apply our sort of standard capital investment framework. So how are we going to estimate this? For the reallocation part, we're again going to draw on Fieldhouse and Merton. We're going to start with their estimate of the economywide change in the number of researchers. We're going to apply estimates again from this Tari uh at all paper to allocate that change across people with different education groups.

[00:20:34]And then we're going to estimate education specific effects on TFP. We're going to start with uh Marta Praau's job market paper where she estimates the TFP effects per researcher involved in innovative activity. And then we're going to allow for those productivity effects to differ across workers with different education levels. We're going to parameterize this using differences in patenting rates across people with across different levels of education as well as differences in wage rates across uh different education levels.

[00:21:00]For the training part, we're going to start with uh an estimate of the economywide change in the number of workers with PhDs. So the number of people who receive PhDs in STEM fields after a change in R&D spending. This is again coming from the work of Fieldhouse and Merens. That's what we're going to sort of show here. We're in the process and in the final draft of the paper we'll we'll provide some estimates of the economywide change in the number of workers with M's degrees as well. Um again based on the work of Satari and co-authors right now we currently lack a basis for estimating a training component for posttos relative to PhDs for undergraduate researchers. Um we think these are potentially quite important. We think this is a margin where potentially more funding is allowing people to you know hire additional undergrads in their labs to hire postocs for longer. We think that those training components are very important but right now we just don't have a way of connecting the existing research literature to the sort of um uh what we're doing and so that's a place where we'd appreciate more work. Okay.

[00:21:54]So now I'm going to show you some results. These are going to be um great.

[00:21:58]Yeah. Yeah. Yeah. So now we're going to show some results. So, these are going to be here we're showing the um response of variables to a $30 billion increase um in federal funding that occurs over 10 years. They're very preliminary.

[00:22:11]They're going to look different in the paper, but we're just going to give you some some sense of what we have so far.

[00:22:15]So, this is just showing the effect on um the number of researchers who engaged in innovative activity in response to this $30 billion increase in appropriations. So, on the horizontal axis, you'll see the years since the change in appropriations. our estimate is going to be based on this Fieldhouse and Merton's paper. Reassuringly, it's very similar to the Satari at all paper.

[00:22:34]You know, if you benchmark to the Satari at all estimates, um it's much less than if you just divided the change in spending by the number by the cost of a full-time equivalent researcher. We think the reason this is going on is because um these sort of causal papers are sort of better counting um accounting for a counterfactual. What would happen that many of the researchers who are being hired in these labs as a result of federal funding would have worked in the STEM STEM sector anyway. And so it's not actually adding to the number of people engaged in innovative activity. It's just sort of poaching from different labs.

[00:23:05]Okay. This is how we're going to sort of build out the macro effects. We're going to start with these models that we've developed for this paper. What these are going to do is they're going to give us a sort of partial equilibrium change in total factor productivity and private investment. We're then going to feed that into our sort of standard set of macro models where there's going to be sort of aggregate demand and aggregate supply effects. there's going to be general equilibrium where prices are going to clear the markets. Those economic effects are then going to affect the federal budget which is going to feed back in because there's going to be um you know if the federal if the debt to GDP ratio goes down there's going to be sort of crowd in of spending and vice versa if there's the debt to GDP ratio goes up. These are baseline results. So this is the change in total factor productivity using these two frameworks. Um you see that these results have different magnitudes and different time paths. Uh for the R&D capital stock approach again we've benchmarked very heavily to this Fieldhouse and Merton's paper. The TFP effect peaks about 20 years after the change in policy and then starts to decline um back to zero. The human capital approach again because these sort of new researchers are having effects over a much longer time horizon.

[00:24:10]Um it takes much longer to peak. It doesn't it's actually still increasing within this 30-year window. Um so these are sort of baseline effects on TFP.

[00:24:20]Then what happens to GDP? Here I'm showing you um four lines. So the black lines are the human capital model. The blue lines are the R&D capital model.

[00:24:30]And then the sort of um the solid lines represent a deficit financed increase in R&D spending. The uh dotted lines are a deficit neutral increase in spending. So here you should imagine we're just taking money from other parts of the government consumption spending and allocating it to R&D.

[00:24:47]What do we see here? So first thing to note is that initially early on in the window the the deficit or the um human capital model has larger um output effects. This is even though there aren't TFP effects yet. Why is that?

[00:25:00]It's because some of this money is being spent initially on capital and that's um leading to changes in output in the sort of way that we always m model changes in investment leading to changes in output.

[00:25:10]The other thing to note is that initially the deficit finance spending um has larger effects. This is because again there's an increase in G. So there's sort of standard Keynesian multiplier effects um that are initially uh leading to a boost in aggregate output over time. The deficit financed effects though have smaller impacts on output than the deficit neutral effects. This is because they're uh the fact that we're sort of deficit financing them is increasing the debt to GDP ratio that's leading to higher interest rates and that's crowding out private spending. So as those sort of initial Keynesian um demand effects fade those uh those deficit finance increases have smaller effects.

[00:25:51]Okay, there's lots of places of uncertainty. I'll just highlight a couple. So first for the human capital approach as we mentioned we need estimates of the economywide changes in the number of researchers posttos and workers with PhDs. We think this is a place where there's a slight disconnect between the way that we usually do things in the research literature and the way that sort of we operationalize things at CBO. Lots of papers, you know, because you want to find a nice control group, we're estimating relative effects and say more exposed areas or more exposed industries. We're capturing local spillover effects. Um, but those are sort of different from the sort of economywide aggregate effects that that we sort of use for these models. And in macro, we call this like the missing intercept problem. And so that's a place where there's sort of a disconnect between the research literature or much of the research literature and what we And that's why we drawn on the time series analysis. Another place where um we think there's some uncertainty is thinking about interactions of federal funding and R&D and other types of policies including immigration policy.

[00:26:41]Um and that's something that we haven't built into the models yet but we think is potentially important. Thanks.

[00:27:00]Okay.

[00:27:02]Thank you very much, Caleb. Uh great presentation. Um if you have questions, please come to the microphones. We'll have a few minutes uh for open questions. Maybe I'll just start us off by asking um so you know, you get quite a different result with the human capital versus the R&D approach. So So as you think conceptually about that and the way they're modeling it, do you have a sense of of why you would find a much larger effect on the R&D side than on the human capital side? Yeah, I think it's um it's just sort of differences in the sort of um the existing empirical literature. So the the Fieldhouse and Merkens paper is again the one that we're benchmarking to for the R&D capital or most strongly for this R&D capital approach. Um so that's one where uh you know we're sort of just directly taking the estimates that they that they get from uh in their paper for the human capital approach. It's sort of we're building up again from this from these estimates of the sort of per person TFP effect that come from uh work like for example Mara Praau's work on on immigrant innovators. Um this is also a place where again the sort of human capital modeling especially is not quite done yet. There's more channels that we're hoping to build in and so um you know it could it could go either way but but uh it might be the case that those end up being closer in the end than they are now.

[00:28:10]>> Great. Thank you.

[00:28:11]>> Yes. Have you considered Jim Besson's work on the depreciation of R&D capital stock in the capital stock approach in which he finds substantially greater rates of depreciation uh because because uh R&D uh by one firm often blocks out R&D by other firms.

[00:28:30]>> Uhhuh. Yeah. So um >> he has that in a couple of center of economic studies working papers and I a lot of people find that rather convincing.

[00:28:40]>> Uhhuh. Uhhuh. Yeah. So um we we have thought a lot about that work and just in general uh there there there are very wide opinions about what the depreciation rate of R&D should be. Some people say it should be much higher than we were doing. Some people say it should be zero or or even negative. Um I think you know one thing is again we're so we're initially what we've estimated and thought a lot about is this crowd out or crowd in from public sector spending to the aggregate private sector spending.

[00:29:04]So there we find crowd in not crowd out.

[00:29:06]we haven't done as much thinking about um uh the sort of competitive uh uh the competitive response between different private actors of R&D. Um but yes yeah yeah we have we have looked at that work pretty carefully. Yeah. Rich.

[00:29:23]>> Hi. Um, I was also surprised at the depreciation rate. Um, for some reason I I thought the depreciation rate for these like intangibles was at the private level was higher than that. But >> but anyway, the my question is can you compare the the effects the productivity effects of say these investments to investments in physical capital or investments in other intangible capital? I mean I know CBO oftentimes models the effects of physical capital on on these outcomes right so >> is there a difference quantitatively or qualitatively between these investments and physical capital investments?

[00:30:04]>> Yeah I don't know if uh someone else uh maybe Sheila wants to answer that who who did the the who's done a lot of the physical infrastructure modeling Sorry. So, uh, in we did work on infrastructure five years ago, I think, and I think that the return numbers were smaller than what we're seeing here in the R&D approach. Um, but I'm sorry to say I don't remember the exact number.

[00:30:35]>> Thank you. Let me um just to keep us the on track with our timing today, let's take two more questions and then maybe respond to both of them to conclude.

[00:30:42]>> Yeah. Yeah.

[00:30:43]>> This is an amazing paper. It's like 10 different papers in one. But I'm most excited by the economywide RD depreciation rate. Just a quick clarifying, is this USwide or worldwide?

[00:30:53]Because it matters to me last time.

[00:30:55]>> We're using the US uh patent data um for this. Yeah. Yeah. Yeah. But okay, I think there's sort of more we were talking yesterday about there's I think more we can do thinking about uh differences how this rate might be different across different patent classes um different types of university you know university uh patents versus commercial patents and things like that.

[00:31:14]So yeah.

[00:31:16]>> Yeah. Fantastic paper. My question is really about um to what extent do we expect federal R&D to show up as GDP improvement? I mean we have environment health research basic versus applied.

[00:31:31]There's a huge mix of stuff that's going on there.

[00:31:34]>> Yeah. Um so we're here we're sort of taking kind of an aggregate approach.

[00:31:39]We're thinking about sort of average R&D spending across um average average R&D agg uh average non-defense R&D spending across the sort of wide set of um uh you know different types of R&D spending that's funded by the federal government. Um again so the majority of it in our estimation um is looks like it's sort of HH HHS NIH spending. Again, we're sort of benchmarking to this fieldhouse and burns paper where again there's a mix between development, basic, and applied. We don't have sort of right now like estimates of what would happen if we change the composition of that spending to do more basic or less basic or more development or um or less. Um but that's I think uh potentially quite an important thing and and we'll think more about that. Yeah.

[00:32:24]Thanks.

[00:32:25]>> Okay. Thank you very much Caleb and thanks everyone at CBO for this really excellent paper.

[00:32:35]Just segueing on that last question, I do think that you know with a lot of the research money is on health and the target of that is health as opposed to productivity. Uh you know it's is or the productivity would be fairly indirect.

[00:32:46]One might think that the returns might even be higher on the sort of non-health side and we might we could think about the welfare consequences also more more broadly. Um, okay. So, for our next paper, uh, we're delighted to have, uh, JCT, uh, and, uh, Brandon, uh, presenting on estimating the present value of R&D tax benefits in the United States. Thank you.

[00:33:08]>> Yeah, someone's going to need to tell me if I am uh, actually sharing it.

[00:33:13]>> That's you, That's urban college being like, "Oh, the person at the podium doesn't know how to use any of the technology." Now, now that's me.

[00:33:39]>> Yeah.

[00:33:46]>> Great. So, thank you everyone for coming. Thank you to the organizers for inviting us. Um this is work that um uh me and several of my colleagues at the joint committee on taxation have uh spent considerable time over the last year on. Um so most people have heard or familiar with the congressional budget office of the CBO less familiar with joint committee on taxation JCT with broad brush strokes in our in our role as analysts for Congress. CBO does spending JCT does tax. So, we're going to be talking about tax here. Um, so, uh, as we are all, uh, employees for the joint committee on taxation, the usual disclaimer applies. Um, but we're going to be talking about, uh, the present value of R&D tax benefits in the United States. uh in as this audience very well knows the US federal government provides substantial R&D tax incentives through uh a deduction from income for general R&D expenses and a non-refundable tax credit for certain R&D activity and we and like other uh advanced countries that do this we think it because it has effects on productivity that will uh help economic growth. Now these are sub these are substantial in size. So in 2016 a decade ago the statutory tax value of these tax benefits jointly the deduction of the credit was more than 110 billion in that single year which is about four times larger than the direct federal funding for business R&D and that was like 15% I think I if I remember correctly on Caleb's slide of total federal spending and R&D.

[00:35:33]there's a but despite the size firms cannot always use these immediately or even quickly. Um the the easiest way to think about this and I'll get into some more detail is that loss status so an accounting loss can delay utilization of deductions credits. Why? If you earn a loss you have no taxable income. You if you have no tax if you you're limited from using additional deductions. If you have no taxable income, you have no tax liability. There's no tax liability to use credits against. So eventually, as for reasons I'll discuss as proceeds, eventually some of these benefits can be used in the future, but this delay in usage will lower the present value of this tax subsidy to R&D. And you wonder, well, why do we care about this? is is ultimately statutory to the extent that statutory tax incentives are a poor measure of effective R&D tax incentives then any policy analysis that is based upon statutory tax incentives is going to have mismeasurement and that if you think about it at the macro level if you're trying to estimate a user cost and user cost elasticity for investment to using statutory tax incentives you have the wrong user cost object and you have the on elasticity. Uh at the micro level, if you're trying to think about the tax subity, how it's distributed across firms, if you're not accounting for utilization in terms of how it's actually offsetting tax, you are not you are not correctly getting at that distribution to which you want to say something about. So that's where we try to come in with this paper and we develop and apply a methodology that's that tracks at the annual level firm level cohorts of R&D deductions and credits that jointly generate this tax benefit from the year that it is the investment is made in the tax benefits generated all the way until when they're either used or expire. And to do this, we use confidential IRS corporate tax return data over the period 2012 to 2016. And we measure this wedge between the statutory tax benefits and the realized tax benefits. So whenever I say a realized tax benefit, I'm talking about the amount that is actually used to ultimately reduce tax liability and present value. The statutory is what is claimed on tax returns. And what is the difference between that? And our contribution is well, no one's done this before. So these are the first firm level estimates of the intertemporal utilization pattern and the ultimately the realized R&D tax benefits. We find that on average uh uh the uh there's 41 cents per dollar of statutory tax benefits that is per dollar of R&D spending but only 36 cents realized in present value which implies a 12% decrease in the R&D tax shield. And that's on average across firms not income weighted. Income weighted it's bigger decrease. Uh the realized tax benefits though have substantial heterogeneity across loss status, age and size. So at the other end is young small firms in loss status realize an average of 23 cents per dollar in R&D benefits in present value which is a 44% decrease in R&D tax shield. And among that group there's also substantial heterogeneity. So the implication is that these tax asymmetries which is ultimately the unequal treatment of profit and loss status could s significantly reduce the R&D tax shield and this is especially protracted for young and small firms in law status.

[00:39:14]This is not a tax crowd um other than my JCT colleagues and CBO colleagues. So I do a little bit of tax background so to contextualize what we're ultimately doing here. So there's for R&D activity at the federal level there's two primary provisions a deduction section 174 R&D deduction deduction it is for general R&D expenses if deductions reduce taxable income whether it's an R&D deduction or a deduction for depreciation or deduction for wages they reduce taxable income there's also an R&D credits reduce tax uh net tax liability They while the same R&D outlay can qualify for both a credit and a deduction which is unusual but not unique to this what qualifies for the R&D credit is a subset of what qualifies for the R&D deduction. What we are going to be looking at in this paper is we're identifying outlays of R&D based upon the credit because that's something that we can observe and we are saying for those AR we're looking at tax benefits generated and ultimately realized for the subset that jointly satisfy for credit and deduction and how this all flows through and then sort of a high level whether we're talking about R&D deductions and credits or other types of deduction and credits started income gross receipts minus deductions. That's your taxable income. The tax rate is applied to your taxable income to get your tax liability before credits. So deductions reduce taxable income and the benefit is at the rate of the rate of tax.

[00:40:54]It after you have income tax liability before credits, credits are are subsequently subtracted from that tax liability. They come second after deductions. um and you ultimately get at at a net tax liability. In a frictionless world or world without tax asymmetries, oh taxable income is negative, then that would imply you get a tax refund back in the same year or if net income tax liability is negative, uh you get a refunded via the credit. That is not the case in the United States. In reality, the tax asymmetries is that those are limited um not only to be non- negative but sometimes to be strictly positive which I'll get to in turn.

[00:41:41]Starting with deductions, deduct deduction usage operates in the context of the net operating loss system or NOLs and loss status specifically is when deductions exceed income. So taxable income is negative. The immediate benefit of the current year deduction is then limited to that amount of income.

[00:42:00]The unused portion of current deductions that is the the portion that does not reduce taxable income becomes an NOL carryover which then can be used to reduce taxable income in other years.

[00:42:12]During our sample period it can be carried back two years or carried forward 20 years. So that means from the point of view of I a year where I'm a corporation making an expenditures generating a deduction I can't use it all. I'm in loss status. If I elect to if I elect to carry it back I look back two years is there positive taxable income? If so it's applied uh oldest years first. If there's a residual left over then that is carried forward up to 20 years. So it eventually could get used if you get out of loss status and you start burning through your stock of net operating loss carry forwards. For purposes of of our work, we define a contemporaneous benefit. Meaning a benefit you get that will be undiscounted is whether you use the deduction today against current taxable income or it gets carried back because even if it's applied against past year's income, you still get it today. So there's no discount factor applied. The implication is loss status can delay deduction utilization and lower the present discounted value of realized tax benefits. There's a not quite parallel but very similar set of rules for the credits. The R&D credit operates in what's in what's called the general business credit system. A lot a lot of other credits. There's credit constraint status. Credits that are general business credits are limited from offsetting tax liability before credits of approximately 75%.

[00:43:37]Um, so the immediate benefit of the current year credits is limited to that allowable tax offset. So if you're in a loss status and you have no taxable income, you have no tax liability. 70% of zero is zero. You get to use no credits in the current year against current tax liability. So it's that credits go second after deductions is pretty important. Um the as with NOLs the unused portion of current year credits become a GBC carryover which can use to reduce tax liability in other years. Should sound familiar and the difference in our sample period is while NOLs can be carried back two years GBC's can be carried back one year and carried forward for up to 20 years. Similarly, we define a contemporaneous benefit as the portion that of the credit that is used in the current year or carried back. Implication of this is credit constrained status whether you are constrained by the 75% limitation itself or already constrained via loss status can lower the present value of R&D benefits.

[00:44:40]The last piece of tax context is uh a first in first out rule in the tax code.

[00:44:46]Um so think about uh a firm that's starting up goes through a history of loss years. It is it is accumulating net operating loss and general business credit carry forwards and accumulating these stocks that they can only use once they have tax liability. Again the first in first out rule is that the carry forwards must be used in the order that they were generated. So there's a difference between old vintage. So from the perspective to today when I'm making an investment stuff that I've already accumulated and new vintage things that might be created from the investment that I make today the old vintage has to be used first which means that for firms that are have a history of losses and have this larger stocks of carry forwards that further pushes new tax benefits into the future lowering their present value. So where we're going with this is we're going to account for all of these things um using tax data. So we use confidential statistics of income or SOI corporate tax returns um to do this analysis. Our unit of observation is a firm expenditure pair meaning uh we're we're looking at each year 2012 to 2016 there's a firm they make an R&D investment that is a unit of observation um we focus on Ccorporations two reasons they account for most over 90% of the R&D credit generation also they are a consistent uh entity that makes this exercise um valid. If we start mixing different types of entities like escorps and partnerships, it becomes a little more complicated and the picture isn't as clear. To give you an idea of how large uh these are over over those same years 2012 to 2016, on average each year corporate R&D credit eligibles are about 235 billion for a tax value on average of 79 billion for deductions and 13 billion for credits.

[00:46:47]So we construct a balance panel u from looking at at these the c the repeated cross-section of the soi corporate tax returns every year from 2010 to 2017 and collect the firms that observed in every year. Why 2010 to 2017 for every year that we want to look at an annual cohort of R&D we need two prey years to account for carrybacks and one postear to identify carrybacks from carry forward.

[00:47:15]So that's why we're truncated a little bit. We stop at 2017. There's a major tax change going to be confounding factors. We're the data are right censored and we have an extrapolation routine to um estimate usage outside of that. In a typical panel year, there are 22,000 firms unweighted by weight.

[00:47:34]That's 650,000 filers. what we're calling R&D firms or firms in a given year that make an R&D investment uh that is jointly eligible for the credit and uh and the deduction a little under 3,000 by weight a little over 5,000. The panel accounts for about 70% of the R&D credit eligible corporate expenditures.

[00:47:54]Um in the paper we talk about um the corporations we don't account for but in interest of time uh I will not talk about that. Um so the methodology so we have our balance panel start in in each year 12 through 16 we observe take as given the generation of these tax benefits the deduction and the credit uh we determine looking at taxable income tax liabilities and other characteristics of each firm whether usage is contemporaneous meaning used against current year income or tax liability or carried back or if it becomes carried forward to be used in the future. For those that are used in the future, we track carry forward utilization. Uh the appendix of the paper is a very detailed procedure that we've developed which is part of the contribution of how to actually do this.

[00:48:44]Um along with since the data is right censored, how we extrapolate. In short, the extrapolation um I is a bend procedure based upon uh credit constraint and loss status that applies uh um rates of usage on these carry forwards. Uh we've done a number of sensitivity checks to see how much um the assumptions we end up matter.

[00:49:08]Quantitatively it I was shocked at how much it does not matter and I'll explain why. Finally, we have this utilization pattern for over 20 years before these before these tax attributes expire and we discount that to present value and our main object is this present discounted value tax shield. So results um so am I in time?

[00:49:29]>> Good. Okay, great. So first I'm going to talk about exposure to delay. So on the x- axis we have uh number of loss years and the vertical axis we have median age of firm in 2012.

[00:49:43]Um so we separate firms by uh small and large. The cutoff is median assets in 2012. The main point here is that small firms uh tend to be in losses more frequently.

[00:49:57]Small uh uh and those firms are also younger.

[00:50:02]So there's an if we if the argument is that loss status implies you cannot utilize deductions today also implies directly credit constraint right from the bat. So to the extent that there's a group that's more exposed to this they are more exposed to delayed utilization.

[00:50:19]So we're throughout the rest of the results we're going to talk a lot about the small younger firms because of this evidence that they are highly exposed to delay. So what sort of portion of activity uh is rep is does R&D R&D um make uh for these types for the firms that we're looking at? We have density functions at the top. It's for all firms as a share of total deductible expenses. The R&D deduction we call that R&D intensity.

[00:50:53]The median firm in our panel is about 11%. Young small firms substantially higher 17% and young small firms in a loss even higher at 19%. The point of this is that these are firms the firms that are exposed to delayed usage are more exposed to the benefits of tax benefits as a proportion of their own size. Um so keep that in mind as we go through these results and talk about the actual utilization patterns which is is uh on this slide. Start off uh on the left panel for deductions in black. All firms on average deduction utilization is pretty high. Um for young small firms since they're more likely to be in law status, it's substantially lower at about 70% in year one.

[00:51:45]Um what we have u as the x-axis is years since the original investment. When we get to year six, you see this the the gray area all that's all of the utilization is extrapolated. Um and the part that's white that before that is that's that's mostly uh uh observed um uh utilization. Credits are a different story. credits are uh are are more exposed to further delayed usage than deductions. For all firms, first year credit usage is 56%. By the end of 20 years, when they expire, it slightly exceeds 70%. For young and small firms, it starts at 14% and barely exceeds 30% at the end of uh 20 years. These are also dollar weighted. So if there's a given amount of dollars of credits and deductions generated, you can apply these proportions to see how much is used at a given year. Now the main payoff is here's this utilization that we've estimated over 20 years. After this time the tax attributes expire discount this.

[00:52:57]So we have at the top distribution for loss and non-loss firms in the year of investment for the full sample and then the subsample of small uh firms small young firms. For non-loss firms these densities are quite similar. The solid line is the median. The dot the dash line is the mean. For the green density for non-loss firms it's kind of tight. Um I said the average across all firms was 41% in in uh statutory benefits. Uh so on average they're pretty the realized value is pretty close to that but the reason why there's a distribution is because you could still be in a non-loss status and still be credit constrained which is reflective of the previous slide.

[00:53:46]Um now loss firms that ma laws status matters on average for all firms. It also matters especially for younger and smaller firms that distribution is wider. There is a mass that's larger at zero present value per dollars. You think of how is that possible?

[00:54:07]Part of it is your if young small firms tend to be in loss status more frequently. they have a large st stock of carry forwards that they have to burn through before they can utilize the benefits that are generated in the current year's new vintage uh um uh deduction uh and credits. Um so I I mentioned I mentioned uh that our extrapolation routine is is uh if we perturb the growth rates or the utilization rates quite a bit even as far as doubling it the quantitative results don't change much. It is precisely because there the old vintage stock of carry forwards for these loss firms is just enormous. Um and we go into more detail about that in the paper. Um and we in the paper we have some applications for well how does JCT think about this when this um uh when it comes to our official job for Congress of providing revenue estimates of changes to tax law. Um and we also have how do you think about this if you really if you want to account for this in a user cost object. Um so to conclude statutory and realized tax benefits whether they're for R&D or anything else they're not the same thing for R&D in particular realized tax shield is 12% lower the it is this delayed utilization of deductions and especially credits that creates these systematic differences in the effective subsidy across uh to R&D across firms and like young small firms that it's precisely because of this more frequent loss status their R&D tax shield is substantially lower 44 44%. So take away if you remember anything from this is that statutory rules alone are not the complete story that if you are trying to do a policy analysis of of of R&D tax credits you should attempt to account for the timing the value the distribution of these realized benefits because they're going to matter for both micro questions and macro questions and right on time.

[00:56:21]>> Thank you very much, Brandon.

[00:56:27]>> Again, uh please line up with any questions. Let me just ask I mean you're inspiring me to think about the following. You've got two instruments, a deduction and a credit, and there are different design criteria around them, when you can and cannot use them. Like you could imagine you should you could allow people to have somewhat of a loss and still take the credit. Or you could imagine reallocating credit statutory dollars to a larger deduction, $1.2 $2 per dollar of R&D or something. Have you thought about uh analyzing uh the present value uh implications of these kinds of reallocations in policy?

[00:56:56]>> Well, in the in the paper we look at um a highly stylized refundability which means that the tax credit under present law is not refundable. Um so if you would make it fully refundable meaning if you are in a law status a credit uh credit constraint you get a refund today. Um, of course that lowers the uh I'm sorry, that increases the present value and it makes it align the statutory and realized values. Um, however, um there's lots of evidence that there's um non-compliance with refundable tax credits more generally, and that's something that um also needs to be paid attention to. Uh I'm going to invoke my deputy chief of staff, Chris.

[00:57:43]Has there ever been a deduction that is worth more than 100% of the value?

[00:57:51]>> So, I don't think that's a novel idea.

[00:57:54]And um it's I I don't Yeah, I don't know of that ever existing, but uh the sky's the limit with ideas, >> right?

[00:58:04]>> Good or bad creativity, I don't know, but it's okay. Uh Josh.

[00:58:08]>> Um so this has been an issue that a number of uh law professors writing about venture capital have sort of claimed venture capitalists are really stupid.

[00:58:17]They should really set up one company and then they could take advantage of the uh tax losses. And presumably what the law professors miss is that you know having the incentive effects of having people have equity in their own company and not in 18 other companies is why you know essentially Sequoia has been so much more successful than IBM or whatnot. So the question is, is there are there ways to sort of potentially do some of the things that the law professors suggest without destroying the incentive structures? For instance, having something where you could have some pooling across a portfolio and be able to redeploy the tax losses from the, you know, the R&D credits from the tax losers to the relatively small number of winning companies there and would that create a whole bundle of other problems or is it actually a brilliant idea?

[00:59:10]>> You think like a a market for tradable?

[00:59:12]>> No, not even that. is just say, you know, if you're Sequoia and you've invested in 50 companies and 45 of them have t R&D tax credits and losses >> that already exist >> that already exist consolidated entities on tax five winners and just do that on the tax side of things without doing it on the business side. So a number of the returns we look at I don't I don't know the portion offh hand but they're consolidated tax returns which means they're multiple entities >> right >> and but it is a single accounting identity >> sorry accounting entity >> right >> so that one subsidiary can generate R&D credits and that can be allocated to another entity that does no R&D >> and then they can also sell subsidiaries there are limits though to how when you sell subsidiaries that carry forwards um to how much uh of the carry forward can be used and how quickly it can be used um that uh are pretty stringent.

[01:00:10]>> Um so there is some of that under present law. There's no there's no market for trading >> um at least a formal market for trading credits. Um but we yeah >> R&D credits but that you that there's precedent for that. He said it in in what credits?

[01:00:29]>> Energy.

[01:00:29]>> Energy credits. Okay. So, there's precedent for that.

[01:00:34]>> And what?

[01:00:34]>> Partnerships pass through.

[01:00:36]>> Yes. And you can allocate that to partners differently. And that's why it's much more complicated to do the same exercise with part with not just escorps but partnerships.

[01:00:47]>> But going back to your questions, I mean, yes, you can do that with partnership. So, so you can answer the local questions, take care of the partnerships.

[01:00:55]>> Yeah. because some of Yes. A lot of that is kind of in present law.

[01:01:02]>> Jeff, >> you mentioned you were surprised by the lack of sensitivity to the extrapolation and it looked like it's because it's really the credits that matter and in the extrapolation the credits are basically flat in your whole gray window around 70%.

[01:01:21]Can you just say more about what makes that happen as it those firms are just they don't last more than five years so it doesn't really matter that they can carry forward because the firm isn't there or what's happening?

[01:01:34]>> Yeah. So I think I think a couple things are going on there. Um I think first you know the in in terms of the usage being flat as Brandon mentioned these firms have a sort of a large stock of these old vintage carry forwards that they have uh that it takes time to burn off.

[01:01:54]Uh the other thing is that you know when we extrapolate forward we're modeling these transitions in and out of loss status and credit constrained status.

[01:02:04]And so the other thing that's happening in the background there is that you know firms are firms are sort of doing this in our projections right they are uh they are sort of ending up with years down the road where they're either in a loss status or or in a credit constraint status and therefore not able to to sort of burn down those uh those stocks as fast. So I think th those are the things that contribute to that kind of being flat uh when we project it out forward.

[01:02:33]>> Yeah. And and we have this in I know this is not a great for slides. We have this in the paper. It separates out the old vintage and the new vintage >> and the deductions take precedent.

[01:02:42]>> Mhm.

[01:02:42]>> In the stacking order. Even if you start becoming uh profitable, you got a large stock of NOLs.

[01:02:49]>> You got to burn through those. You're generating more and like you you you you have to get through those before you can start using your GBC's and you end up with very low utilization.

[01:03:02]Um I was I was shocked. I was just toying with like the assumed uh or we start with an estimated usage rate. I'm like what happens if I double it?

[01:03:11]>> Stop changing or triple it. What's going on? Let me look at these firms and I'm like whoa those >> carry forward stocks are just enormous.

[01:03:22]>> Thank you. So with apologies we're actually past time. So let me just take one more quick question and and then if we can just ask maybe in the break uh other questions and apologies.

[01:03:30]>> Great. Ezra Ker, Federal Reserve Bank of Chicago. So, I I'm wondering about two mechanisms. One is inflation. So, you estimate these effects during a time when inflation was lower. Uh it seems like if utilization is low and it takes a while for these companies to actually use this when you depreciate it, I assume you don't get to inflation adjust your credits. Uh and so that seems like it would matter a lot for these effects.

[01:03:50]And the second is mergers and acquisitions. So, if you can uh you mentioned balancing your panel and I've worked with some of this firm level data before. I worried a little bit about what that would do if you have a few winners who are taking advantage of these credits by acquiring smaller firms that didn't win. And I'm wondering if that has an effect on measurement here.

[01:04:08]>> Yeah. Well, uh it's related to like two questions ago is if the bigger firms are in our panel, which they likely are, and they are the acquirers, uh then that shows up in their carry forwards that we're tracking. Um, and it's they are li they're limited to using that at a at a low rate. Like they can't use it all the way to uh before 2017 eliminated all their taxable income. Uh they there's a a statutory rate that they have to use it. I don't quote me on it. Somewhere between I think three and 4% of the value that they that they got it at.

[01:04:47]They can only use that much each year.

[01:04:49]So it's not like there's not a case where a firm goes belly up that has a lot of these carry forwards, a big firm buys them and then in one year uses all of it. That's that's not allowed under uh tax law.

[01:05:02]>> Yeah. I also think on your inflation adjustment point, I think that that would, you know, sort of even further push us in the direction of what we'd concluded, right? Because if you're a firm who's facing delays in using these, not only is the delay going to lower the present value, but inflation is going to sort of erode the the present dollar value of those.

[01:05:21]>> Yeah, it's the second second time in a week that I've heard about the idea of inflation indexing carry forwards.

[01:05:29]>> Very good. Thank you very much the audience and uh Brandon and team for this great presentation. Okay. Um great opening session. We're now taking a 10-minute break, slightly shorter than originally planned. and then we're going to come back and do two papers around the economics of innovation in space.

[01:05:46]>> We have two more great opportunities for networking. Don't fear. So uh so first up is so this is space as Ben said and first up is Benois with uh defining innovate innovatization a new word for me. All right.

[01:06:05]>> Hi it's all yours. You've got 22 minutes starting now.

[01:06:09]>> Thank you.

[01:06:10]>> All right. So, good morning everyone. Uh it's a real pleasure to be here and thank you uh to the organizers for allowing us to present this work today.

[01:06:19]Uh so this work that has been conducted with Dr. Shandre Marandre Shi McDonald from the University of San and Professor Dominic from EF. So our work deals with a new concept so innovatization um and the space sector and we we try to show how this new concept can explain some of the uh many many changes that we see happening in the sector.

[01:06:43]So okay um okay so to give you a bit of context about uh the space sector. So this uh the space sector is undergoing through uh dramatic transformations uh in the past 20 25 years especially in the US uh where you see new roles emerging for um government agencies, new technologies uh new entrance not only startups but also uh nonspace incumbents like can think about Amazon and its constellation but as well as new economics with venture capital investing in the sector but also new business model appearing like space tourism for example.

[01:07:22]And so what seems to be appearing is the largest boost um in in the sector since the 60s and 70s basically since Apollo uh with not only innovation arriving in the sector but also uh a change in the nature of the demand signal that is coming not only from the government but also more and more from the private sector. And so what we are trying to do in this paper is to try to explain u to try to find you know some querance and to provide an explanation about what is happening.

[01:07:52]So proposition is that uh these changes that are happening in the sector uh result from the arrival of innovation uh and this phenomenon. So the arrival of innovation in the sector we call it innovatization.

[01:08:06]uh and so what I will try to argue in this presentation and what we do in the paper is to to to show that actually innovation was not really the rule of the game in the space sector for a very long time. Of course there were great achievements missions were achieved. Uh but eventually it led to uh low value for for the masses of people and by this we mean we we don't mean that it didn't have an economic effect. Of course it had huge economic effect. Uh but in terms of allowing us for example to go to the moon like I think in this room not many people were able to go to the moon. Um so this is what we do. Uh because to become an innovation a technology must not only work technologically but it also needs to work economically meaning that there is an economic discovery process that is realized. Um whether trying to understand whether the product or the technology that is developed is it attractive for consumers. Is it produced at an acceptable cost? Does it allow innovators to generate some profits?

[01:09:04]Uh so we argue that this was not really the operating mode in the sector for a very long time, but it's becoming more and more uh the the the way that the sector operates. Uh and so innovation is seems to be becoming the norm in the sector. And so what we what we try to do in this paper so to to to discuss about this innovatization uh concept. So where how a sector can go from a mode of producing technological achievements to producing innovation and so what we do is we develop we start by developing a framework uh and then we try to see how it can explain um yeah what is happening in the space sector.

[01:09:42]Uh just as a side note here we are not talking about causality etc. What we are trying to do is just to see how this theory can explain some of what we see in some of the things that we see in the space sector.

[01:09:55]All right. But so at the start to make sure that we are all um in uh in sync with uh with what we mean by uh innovation and the concept that we're trying to develop just go quickly uh through a few definitions. Uh so first what is innovation? So our definition of innovation is based on um so shed's definition and and pel and most of innovation uh literature which is that it u it consists in doing things differently but it also has to be in the realm of the economics of the economy and it also needs to be adopted by the market. This is also the definition that is used in the empirical literature by the OECD Aostat etc. So again um doing things differently but things that also needs to be introduced on the market or that needs to be brought into use by the firm.

[01:10:46]So innovation is a discovery of what works economically and it's something that generates economic value in the form of economic surplus or either consumer surplus or profits.

[01:10:56]So by contrast what would be technological achievements? So techn technological achievements uh would be uh technological advances uh where so technological goals are achieved but without necessarily passing market validation. These would still be in use in the economy but uh because for example they would be pushed by the government for prestige or national security purposes. Um but so how do we differentiate technological achievement from innovation and invention? So invention would be confined within the world of the lab. Uh but by contrast innovation and technological achievement would be in using the economy. But the difference is that technological achievement do not pass the test of the market. And so with business model for example subsidized by by the government.

[01:11:45]um technological achievement also not failed innovation again because they don't uh actually pass the test of of the market while failed innovation would uh be uh put into use would try to pass the test of the market but they just don't succeed.

[01:12:03]So what we try to do in this paper is to provide a simple framework for uh this innovatization concept and try to understand how uh a stector can go from uh producing technological achievements to producing innovation. And to do that we uh actually build a framework uh using uh Ben's paper. So uh which is a paper which tries to understand why innovation happens and why it does not.

[01:12:27]So and try to understand how different sectors can have so why different sectors have different propensity to innovate. Uh and so here though argue that there are some determinants and characteristics uh that determine the returns on innovation according to three categories. So demand supply and institutions and based on how this criteria based on how these characteristics are developed in the sector then it will incentivize companies to invest in innovation or not. So demand factors are for example associated to market size etc. So here's the intuition is that the bigger the potential market the more companies will be incentivized to invest in innovation.

[01:13:09]uh supply criteria are more about the cost and yeah how um how how costly it is for companies to to innovate and here the intrusion is that the lower the cost for example the lower the cost of experimentation the more incentives there will be for them to invest in innovation and then there is also um the institutions so how easy it is for companies to appropriate the the returns generating by their investments in innovation And so uh the innovatization process would involve shifting from a situation where these uh features uh are not very uh conducive to innovation where a situation where they become conducive to innovation. And this eventually would lead to a change.

[01:13:56]And so when we look at the space sector we can um what we will argue that what we would see that uh through the history we see this transformation and the transformation of these characteristics that determine the state uh the the return on innovation investments. And so if we look at Apollo for example um and the market structure of the structure of the market during Apollo uh we can really see that. So if you consider a competitive market, so in the competitive market you have many actors on the supply side and many actors on the demand side and uh market transaction and the price mechanism will reveal consumer preferences and will reveal the technology of complex. Uh and so here the economic discovery process is um realized through this market transa continuous realization of market transaction and the price mechanism.

[01:14:46]By contrast during Apollo what we had was a monopony oligop market. So on the demand side we have only one barrier which is uh the government and so the mark we don't have here this continuous realization of market transaction. The market is discrete which mean that transaction only occur when the state wants to buy. Uh but it's also specific meaning that it has the state has very specific requirements. uh and these uh are revealed very rarely only when it wants to buy and so these um consumer preferences need to be revealed explicitly through uh through contracts and very detailed contracts and requirements etc. And on the supply side what we have is an oligopoly so few suppliers so high concentration and high barrier to entry uh sector that is highly capitalistic um and high product differentiation. So here also on the supply side there is a lack of competition and it's very difficult for the buyer to know whether it's procuring at this cost and so here the mask structure is um doesn't really allow this process of economic discovery.

[01:15:54]We also see it at the level of contracts. Uh so what was used to develop the technologies at the time was cost plus 50 con cost plus fixed fee contract. So called CPF meaning that so it's highly regulated and the idea was that uh NAZA would refund all the cost incurred by the suppliers uh but also be uh provide a fixed fee which with percentage of the cost as profits. And so here we can see that the the contracts are not really uh providing the right incentives of so by designs they generate in attention to cost uh making profits and realize to performance because maximizing proofy would uh just consist in maximizing costs and so there was a symmetric information here and leading to high monitoring and so it was also costly to administer because uh just a lot of monitoring from NA was required.

[01:16:46]Um so this um this construct insulated the industry from the economy and generated inefficiencies.

[01:16:53]But of course um this was uh quite well known and very early on uh in the in the program uh there was this report from David B the director of the bureau of the budget who tried to propose to implement some new contract that would better align contractor incentive with another objective in term of time schedule etc. And so he suggested the implementation of cost plus toward fee contracts and cost plus incentive fee construct where here we are still in a cost plus logic but the idea was to have portion of like the proof would be linked to the performance of the company. Uh there was also a fixed price contract but this at the time of Apollo were not implemented. So fixed price contracts is just a contract where the the price would be set in advance. uh but this was difficult to do at a time where it was uh there was a lot of R&D and we didn't know exactly how how to develop the technology and so this contract so CPAF and CPIF were uh very quickly implemented and represented approximately 20 or more than 20% actually of Apollo of the Apollo program but so did they work uh so actually the results were mixed so one thing that the mission was accomplished so the the the US was able to to land on the moon before the Soviets. Uh but there was a bit of a debate about whether this contract were able to actually yeah move the project forward and were actually fulfilling the mission that they they they had to to fulfill. U internal task force said that it actually improved deliveries contain cost growth and um led to less uh contractor surveillance.

[01:18:34]Um but more independent assessments said that actually it was increasing uh procurement cycles because of course things needed to be much better defined in advance. Um also because now the industry was bearing uh much more of the risk um then they were asking for higher fees putting into question whether this actually uh led to containing the costs.

[01:18:59]uh and some also argued that it generated wrong incentives pushing companies to actually take shortcuts in order to be able to get the award and and the incentives.

[01:19:10]Um so what does it say about today? So actually these contracts have been u extensively used uh for many uh programs. So for the space shuttle, international space station and the SLS uh and many reports have shown that actually they were not very good at um containing cost etc. And so despite efforts to simulate market conditions um these contract didn't seem to have succeeded. Um and apart from a few exceptions until the 2000 the space sector was still mostly driven by uh by the public sector and producing technological achievement rather than innovations but so what we are so today we see that there's many changes in the sector that there is the arrival of innovation and so um how do we explain this these changes the fact that the sector for example and later on produce technological achievement to producing today uh more innovation.

[01:20:12]So this is the innovatization of the spectro which we argue happened in two uh in two steps. Um the first one was um coming from the outside. So what we call external originators um that there were favorable conditions uh in the demand and supply side as well as new entrance.

[01:20:32]So in this case disruptive innovators.

[01:20:34]So building on the disruptive innovation theory um and so here like changes came from the outside. So so we saw that there has been attempts to to change with you know these new contracts attempts to change the system from the inside but this was difficult but then um here we argue that change came from from the from the outside. Uh and so this better conditions and new entrance eventually led to increase in uh investment in innovation. uh and these first successes um started to attract investors and other entrepreneurs uh which also improved the demand side because there was this very first successes on the demon side they could see that actually um yeah products were working so um lowering uncertainty increasing the willingness to pay um and eventually this led to increases in production generating learning effects uh etc and then there is feedback effects that are happening and reinforcing the the system And then at the same time institutions evolve to accompanize this phenomena.

[01:21:36]Uh and so possible measures to try to to to evaluate this would be uh R&D expenses or human capital for the input in innovation and also otherwise patent for for the output. So what I will show now is to try to just show uh some okay just to try to show some data very quickly uh to to see how we can see you know what is happening in the model how we can see that actually these factors have actually changed in the space.

[01:22:04]So in terms of external originators so this these new favorable conditions are also on the demand side. So increase in connectivity demand uh due to it revolution also the development of GPS applications and then on the supply side there was like more for example and so this phenomenon who decreased potentially experimentation cost here in this figure. Uh so what we are trying to see how actually experimentation costs decreased in the sector. Um on the x- axis you have time and on the y- axis you have the cost of building and launching satellites. U in gray what you see is the cost of um so building and launching prototypes satellite prototypes. Um and uh here the the dots. So the the red dots are about um telecom satellites and the green dots are about uh earth observation satellites. And we can see that the the cost of launching of building and launching um one satellite also decreased through time and the cost and so the the bubbles represent constellations and the cost of building constellations also seem to have decreased.

[01:23:14]Here we also saw how uh so this pioneering entrepreneurs. So these entrepreneurs that arrived especially at the late in the late so on the y- axis we have the number of entrepreneurs and again through time um and yeah through the mid and 90s into the 2000 that we have many more entrepreneurs arriving in the sector and entrepreneurs able to uh to raise more money. In terms of feedback system we see significant decrease in uh in development production and operation cost. So here there is a video the slides will be online so you can see it. It's another scientist who actually was trying to explain why we saw such decrease in cost in spaces versus NASA. Um and uh yeah so I explained that through bringing bringing uh yeah new uh new designs so focusing on standardization and larger series etc. And here we try to show how so we again plot um uh cost to orbit uh through time uh and we see uh how by increasing batch number we also seem to have u decreasing cost to to orbit and this is for spaces. So here we try to uh to shed light on some learning effects in the industry.

[01:24:30]Finally and then I will I will conclude something that is important is also the changing institutions and there is a lot of change that was happening at NAZA especially the beginning since the beginning of the 2000 with a new philosophy so supporting innovation now rather than the development of technologies uh and the implementation of new types of contracts so and especially a private partnership uh where here we the idea is not to refund all the cost but more to be in a so partnership logic uh having like co-inancing So forcing company to also have skin in the game. Uh but also enabled them to appropriate so the return on innovation by uh giving them the yeah the the possibility to retain intellectual property which then make them able to commercialize the technologies. NAZA also acts as an customer uh starting so buying services so transportation services rather than rockets just here. So here we are putting uh procurement contracts um of NAZA. So in blue you have cost contract and in orange you have fixed price contracts and we can really see that in 1993 uh cost contract represented 10 uh 90%. So fixed price contract 10% of of NAZA procurements uh but fixed price and in 2019 fixed price contract represented approximately 30% of NASA contract. So we can really see a change um in in the way NAZA interacts with the industry uh going from this cost price contract to to this more fixed price contract forcing companies to to co-inance with sort of the project and then enabling them to commercialize the project here I will pass so just to to here we try to to see okay so if we have this innovatization of the space sector and this increase of innovation in the sector we should see it also in the data and one way to measure this increased propension to innovation is uh through patents and here we see uh those first bump in the 90s potentially coming from this pioneering entrepreneur and then in the 2010 from this feedback system.

[01:26:33]So to conclude what we propose here is a novel concept to try to understand the the significant transformations that are currently happening in the sector uh and to try to explain why the sector has become so innovative and very close to some of the most innovative sectors like bio biomedical pharma etc. And we suggest that this could potentially come from these two steps. So uh yeah uh external originators and and feedback.

[01:27:03]However, the space sector still has very specific features. So fixed cost especially on some segments will never really be zero which leaves a critical call for a critical ro sorry for for NAZA and also for patient capital to help develop this uh these technologies.

[01:27:18]Um, also a key ingredient for innovation is freedom to experiment. And it will be interesting to see how this evolves in a sector that has always been very u subject to to to the the public sector.

[01:27:33]Uh, and so values are persistent. But also something that is very important especially for policy policy that as market takes more important role in the sector also arise market failure and so it will be important. So this leaves a role for policy. Um and so this concept can has been used for the US space sector but can also be used in other either in other sectors or in other geography.

[01:27:58]So >> great well thank you very much.

[01:28:06]We have a few minutes for um a few minutes for questions. Um so maybe while people are getting up I'll ask the first one. Do you think I mean if you think about the recent Entare mission right that was orbital sciences and that's a little different from SpaceX right in the sense that SpaceX is sort of doing a bunch of stuff really geared towards commercial >> things as well as government contracts but as I understand it >> the entire program and orbital science more generally has been really fully focused on meeting NASA's needs.

[01:28:36]>> Yes. Exactly. And how do you think about the you know the incentive issues and contracting issues of that multiclient as opposed to single client kind of kind of setting?

[01:28:46]>> Yeah. So that's actually a very interesting question. So um yeah so orital actually apart from NAZA indeed has not been able really to develop other contracts or at least to go like commercial and and and get a private client.

[01:29:01]It's interesting to so I don't really know if it's a strategy of the company to focus only on the need of NAZA or if it's the inability of the company to decrease cost so significantly that then it would have been able to compete with companies like NAVA and so I think one of the difference is that uh so orital has been created in the 80s uh and so it's still working on a maybe older logic compared to SpaceX that has been created you know uh much later on and with also like different values a different philosophy and bringing Yeah, new way of being organized and developing space products. Uh, and so companies that is also more competitive and maybe that's why we we see this difference. So maybe it's just an issue of competitiveness from the company.

[01:29:45]>> Please.

[01:29:45]>> Hi. Um, Jacob Martin, International Technology and Trade Associates. Uh I was just wondering uh how uh whether you've thought at all about uh the fragility of some of the feedback systems that you talked about that reinforce and perpetuate the cycle of innovatization.

[01:30:01]>> Um for example, the there seems to be a strong confidence among big aspects of the space sector that we're going to see continued decrease in launch costs, >> but we haven't actually seen that the last few years, particularly on SpaceX ride share. It's up like 20% since 2021.

[01:30:17]Um people seem to approach calculating this based on the um just math of like engine cap engine capabilities but without confidence but but still just maintaining confidence that uh cost savings will be passed on to the consumer there and then for the the other feedback systems you have venture capital and influenced by stuff like interest rates and so on. um have you thought about how if v variability in those feedback systems would change your um like the overall trajectories that we're seeing here?

[01:30:49]>> So this is actually a very good point.

[01:30:51]Um so we have not really um thought about that in the paper but it's definitely something we should think about especially when it comes to decreasing cost especially like there is actually two things that actually would contradict each other in a way. So first is like the dominant position that basics currently has and indeed the the passing through of the the decreasing cost to the customer like they did it for a while but now that such a dominant position they tend to do it a bit less.

[01:31:18]So indeed like this the decrease in cost will not go so much. Maybe they will also waiting to see what um Starship will do >> and this could and according to space it's it would very significant decreasing cost. So potentially this decreasing cost will happen still thanks to >> thanks >> please.

[01:31:41]>> Hi Richard Davo joint economic committee. I've been interested what lessons we could draw about the trade-offs between the cost sharing arrangements versus a fixed price and its less popular cousin I think the X-P prize uh possibility for government policy from this context.

[01:31:59]>> Sorry, can you repeat the question?

[01:32:00]>> Yeah. So, you you show you know that there's been a movement between cost plus and fixed price. There's also some X-prise >> situ. I don't think the government does that very often, right?

[01:32:10]um like what lessons can we draw from the government's choice between these different policy options from this history of space innovation?

[01:32:21]>> Um yeah, so so that's a good point. So yeah, the potentially it has to deal with how advanced the technologies are. So potentially when there is still a lot of R&D to to be made um cost could still be useful because when it's very not very clear how yeah the amount of which needs to be done you still need to induce companies to invest in this program. So potentially there is still like room for cost contracts in some areas uh the very early stage of development of the technologies. Uh but then the later on fixed price contract would put when it's closer to commercialization fixed price contract would be more adaptive because they would force company to yeah again have skinnings again and then uh find ways to decrease cost and to yeah make the the the technology more affordable.

[01:33:20]This is, I think, kind of related to Josh's question, which is sort of more of a history question, which is how do you think about the transition from NASA really being integrated into military and military technology and their developments being of interest to defense into this sort of civilian world? And does that relate to the kinds of contracts that they choose or the ways that they choose to get products delivered or who they buy from?

[01:33:50]So, so from my understanding um development made at NAZA was really mostly civilian. So there was still like a lot of indeed space is a lot of interest for for military actions. But then so the spa the department of defense is also a very huge investor and a very huge actor in in space. Um so yeah I think when military issues are um are raised etc then it could go to to the do so yeah so so NA would start to develop some technologies maybe but then when there is a different the would come >> yeah the um space toilet was actually done on Artemis was actually done through a contest, right? That at least the key elements of a design and it lasted like six hours before it failed. So I'm not sure that that's a ideal alternative in terms of procurement. Heidi ask you >> um the benchmark with biomedical is kind of interesting. So if you read like Dan Carpenters's book on the history of the FDA, like one of the things that he thinks is really important is consistency and predictability of kind of demand on the public sector side. And so that's part of something that people often emphasize when they're talking about basically the growth of the space sector in recent years is that across administrations there's been kind of a very predictable focus on certain technological goals that allowed kind of the private investment to um develop out in a way that had you know a perception of certainty over which which industrial policy goals essent essentially within the space sector were going to be have demand in the future. And I was just curious if you had anything that you had run across on kind of how important you think that is and also whether the international kind of context gives reasons to think that was more or less important.

[01:35:43]>> Um so so sorry can you repeat your question?

[01:35:46]>> We have one minute.

[01:35:47]>> Okay. Yeah.

[01:35:48]>> No, it's fine. I'll just ask you offline.

[01:35:52]>> Without further ado, let's conclude by >> Thank you.

[01:36:03]We've got the dynamic duo here, Alex and Ruben. Not sure who's going first, but we will find out. Sounds like Alex is my slides.

[01:36:36]Maybe we'll have a 15 minute intermission while we >> should be over here on the right.

[01:36:55]>> Sure.

[01:37:00]Yeah.

[01:37:01]>> Yep.

[01:37:02]>> Cool.

[01:37:03]>> Yep. It's looking promising.

[01:37:08]>> Full screen.

[01:37:10]>> Wow.

[01:37:12]>> Rob, you're a hero. Try Acrobat Pro with a 7-day trial.

[01:37:18]>> Maybe another time.

[01:37:21]>> Okay. Well, thanks thanks so much. Uh great to be here. Thanks to to Ben and Josh for having us as part of the conference. Uh this is I'm Alexander Wallally from University of Calgary. We have Ruben Gatani who's gonna going to take over as we go here. Um and this is a a paper we've been thinking about for a little while and we really want to think about these different eras that we just we just heard something about old space and new space and we want to ask is this really a commercial uh revolution in innovation or is it more of an evolution? That's really what we're interested in. Um, and just trying to understand that that time series and what does it tell us about policy levers that can can really move uh innovation.

[01:37:59]>> I have to click on it once.

[01:38:01]>> Yeah, you got to kill him.

[01:38:04]>> Now try.

[01:38:05]>> Okay, now >> click on the screen once.

[01:38:11]>> Okay, here we go. All right, so we're at second slide. Um so the thing that really motivates the study is if you look at the space launch launch boom it's just it's just tremendous. So if you're maybe outside of this sector you haven't spent a lot of time paying attention to it uh when you when you look at the graph it really jumps out at you and this is really the the number of objects launched into space. This is like number of satellites being launched and what you see is it's really flat for a long time period. If you go back to uh early 50s, the launch of Spudnick is the first satellite and the number of satellites is pretty constant um until about 2010. Then it takes off uh like a hockey stick graph. Um and it's just amazing how many how how different it is now. We've about 2,700 satellites launched in 2024. Um you know before 2015 there's never more than 500. Um in operation today we have about 11,000 satellites. It's a sector that's growing really fast. Uh so the world economic forum is projecting a 1 point tra 1 trillion uh dollar sector by 2035. Uh the rate of growth is about twice uh global GDP.

[01:39:12]And so what we're interested in today is trying to understand this this transition moving from kind of old space uh to new space. So we think about the old space era. you know the first launch of of people to the moon and returning them uh to earth is really government-ledd NASA very riskaverse kind of culture um maybe stagnant and bureaucratic uh we have like cost plus type contracts uh it's heavily involving legacy aerospace giants and then we have the era today which is which is quite different so we have the new space era we sort of dated around 2005 and onward these are entrepreneurial uh ventureback companies uh famous companies like SpaceX Blue Origin really disrupting uh the sector and we have, you know, quite different technologies. Reusable rockets are pretty pretty amazing. Um lower launch costs, which is really what's driving that graph, right? It's the the price is going down. A lot of launching objects into space. The weight of the actual satellites are getting smaller.

[01:40:08]So, it's just much cheaper to launch things into space than it used to be.

[01:40:11]And so, we're going to ask is do we see a sharp break between these eras and kind of innovation data? That's really what we're going to we're going to ask today. Um and if you just take a quick look at the you know your ocular econometrics of not even econometrics not even not even data just looking at images you think wow you know space technology has changed a lot. So we have kind of old space on the left here the launching the astronauts to the moon the first time 1969 uh and then new space kind of this timelapse uh photography of the the uh reusable rockets coming back to their station. And you would think people in 1969 would think wow that's incredible. I don't know uh how how you did that. It's quite a quite a big technological leap.

[01:40:49]At the same time, a lot of things look fairly similar, right? If you go to old space and we think about what does a rocket look there like it's kind of, you know, these different stages that get lost in the launch process. You can't reuse different items. And you look at today Artemis 2, you just kind of stack them side by side. There's some similarities there. You have, you know, a similar multi-stage rocket. There's some boosters on the side, but it looks looks fairly similar. So, what we're going to try to think about um through the lens of patent data is how different are these different eras. There's been some quite big changes uh over time. Um we think about three different eras here today. The first is the shuttle era. Uh you know that's the first aspect of reusable kind of space launch vehicles.

[01:41:28]And then we think about postcold war about 1990 to let's say 2005. And the third era is going to be uh new space.

[01:41:36]This is our timeline today. We're looking at 1975 uh to 2025. Um there's a few really important elements that happened during this time period. uh one is the commercial space launch act. So really uh setting up the legatory and regulatory environment uh behind the commercial space sector. Another um is the NGSO spectrum allocation. So when satellite communications uh are actually allocated communications frequencies that's really important uh in the sector. Um and then we sort of see the changes in new space kind of happened in 2005. Uh the commercial procurement model is a big change where the CS program for example and then from there we sort of see the new space era uh come online. So we try to classify these in two broad broad buckets. One is kind of market creation you know setting the property rights setting the rules of the game spectrum licensing procurement creating is appropable markets and that's kind of 1980s 1990s and then market entry policies so allowing the private sector to enter much easier after 2000 and that's more like fixed price contracts reusable launch entry barriers. So our empirical evidence we're going to look at today is sort of the timing of space innovation. How does it line up with these different eras? Do we see a big surge in one era versus another? And again, there's there's sort of two uh main economic forces at play.

[01:42:50]One is sort of market driven innovation.

[01:42:52]So we we think a lot about uh private sector innovators following profitable opportunities, right? When there's a a market created or market sizes that are big, uh you know, innovators are looking to serve serve those markets. And so when government uh through these policies sets the rules of the game, makes the market size bigger, we get the private sector to respond. And so that might be an important role for government here, setting that that appropriable market, not necessarily uh directly funding R&D.

[01:43:18]Another one is thinking about kind of entry-driven innovation, right? So there's a lot of discussion about creative destruction, how incumbents in in uh uh new new entrance might be different in their innovation intensity.

[01:43:30]Incumbents might be really concerned about cannibalizing their own products.

[01:43:33]So they might be concerned about doing too much innovation. Um entrance might not have that concern. they may be really important. On the other hand, incumbents may have complimentary assets. They may have things that give them um some kind of advantage in this new sector. So, we're going to look at is when does the innovation surge occur and who drives it. And so, we'll look at the timing of that in the data.

[01:43:52]Now, we put in a 2 x two box. We're really thinking about kind of market creation as being low or high or entry barriers being high or low. If we're in this far left lower quadrant, that's what we think of as kind of old space.

[01:44:02]Uh market creation is not really there.

[01:44:04]Entry barriers are quite high. We're in sort of a stagnant area from an innovation perspective. Um and market creation kind of begins in the 80s and 90s. So we kind of move to the upper left quadrant where you get incumbents could lead innovation surge. Uh and then new space is really moving us we think to the upper right cell where entry barriers are going down.

[01:44:23]And we're going to do with this analysis is look at patents as a measure of innovation. So a very commonly used uh data source um from the US PTO United States patent trademark office. The key the key challenge here the hardest part of this exercise is what is space related technologies. So uh a lot of space technologies are dual use. You can use them in space you can use them in something else. So how do we figure out what's going with space innovation?

[01:44:45]That's going to be a key challenge.

[01:44:48]We're going to track changes in the scale composition and geography of space patenting. So what we're going to see what Ruben will show us um is the largest surge in space patenting occurs in the 1990s and it's really driven by kind of satellite and communication sector. Um the composition is going to be quite interesting is that incumbents are really going to lead the way. It's going to be these already established firms that are going to really dominate space patenting. Uh we're going to see entrance grow but still remain a minority. So sort of near the end of our sample, they're start to come in a little more strongly but they're still going to be a minority. And there's also a lot of persistence in geography. So the areas of of the United States that were dominant space innovation producers in the cold war still remain still remain dominant today.

[01:45:33]Okay. So what we try to learn from the p patent data is the 1990s we move from this lower quadrant of kind of stagnation to this incumbent innovation surge. Um that's sort of happening earlier in the sample that's happening earlier before the new space era. And then maybe we're just starting to see um some movement from this incumbent innovation surge to kind of this creative destruction just now and post 2010. There's some early early signs of that. So the big transition is kind of market creation that drives the incumbent led patent surge um even though everyone often talks about barrier reduction. So we're sort of advocating for another another key element there. And so what I want to do now is turn turn the keys over to Ruben who will go ahead and take us away through through the patent data. All right. Thank you, Alex. And yes, as as Alex said, we're going to be using primarily uh patent data, which is a great starting point as a measure of innovation.

[01:46:34]Yeah. Oh. Uh is is a great starting point as a measure of innovation. It's widely available, comprehensive, a reach of information. Uh at the same time we do know it has some major limitations.

[01:46:45]Uh different sectors different their propensity to patent. Some technologies are very easy to protect through patents. Others are much less easy to protect. Um and uh uh these challenges which are very common whenever we are using patents data are really compounded or really amplified when trying to use patent data to study a sector such as space uh which is extremely heterogeneous. uh most technologies have this dual use nature. They have applications in space as well as as well as on earth and the boundaries of the sector itself are not very well defined.

[01:47:19]Um it is there is no really a agreement on what constitutes a space innovation and what doesn't. Uh and just to illustrate the type of challenge that we encountered as we started to approach patent data uh to study space uh here are two patents. Both of them come from SpaceX which is you know quintessentially space related organization but one of them the one on the right is very obviously space related. The one on the left is a satellite constellation. The one on the left is what you might call a hidden space patent. Something that would be very difficult to detect as a space patent if you were just to naively rely on the information in the title or the text. So how are we going to try to address this challenge? Well, first of all, by recognizing there is not going to be one unique classification that will work for everything we're we are trying to capture. Instead, we're going to try to address this by we're to approach this really by looking at multiple classifications spanning from very narrow classifications to very broad. So very narrow uh um kind of meaning of what we mean by space patterns and very uh broad uh one that will load differently on different areas of the space economy. We can implement these classifications at scale and the data is publicly available. The detail everyone is really welcome to use them.

[01:48:35]The details on where to find them are in the paper.

[01:48:38]So these classifications will span from very narrow to very broad. Let me tell you a little bit about each one of them.

[01:48:44]The first one is the narrowest of the three, but it's one in which we can very uh confidently identify patents as being related to primarily space transportation and space launch. So these are all the patents that have at least once in their CTC uh classes B64G which is cosmonautics. So what kind of patents are these? Um well for example, this is a this is kind of a nice example. This is a patent filed by uh by uh Blue Origin which is the company that Jeff Bezos founded in 2000. The the first inventory is Jeff Bezos himself.

[01:49:19]And if you look at the uh picture in the patent which kind of illustrates this uh these dynamics of launch and the C platform in which the uh launch vehicle is finally uh collected is exactly the same picture with slightly better graphics slightly better aesthetic but exactly the same picture which is used in on Blue Origin's website uh as a way to kind of describe the technology behind their flagship spong launch vehicle which is blue um new glam.

[01:49:50]Um so this is uh these are patents uh that are very confidently identified as space related but also have this kind of narrow coverage. They only cover space launch and and space transportation. So the next step is to try to extend this uh narrowly identified set by including patterns that do not belong to that technology class necessarily but they are semantically close to it. Uh we're going to use a predictive approach. Um first of all we're going to build text embeddings so vector representations of each patent in the sample um using a large language model and then we're going to train a logistic classifier that uh uh uh using this set of narrow space transportation patterns as the training set and try to kind of predict the probability or the proximity of each patent to that narrow set. That's going to give us a predicted score, predicted probability of being uh space. You may think of this really as a space score.

[01:50:46]Uh the blue distribution here is the distribution of space scores for the uh uh for the for the narrow space patterns. The orange one is for everything else. And you see really the degree of separation between the two. So we set the threshold. Everything to the right of the threshold will be classified as a space patent under these classification. And what we get is a whole set of uh uh uh of patents that are very related to space. They're just not captured by the B64G kind of narrow class. Um this is a patent filed by NASA. It's about the Odium program which is the NASA's program for outer space exploration. It doesn't have B64G. This is really kind of reminds us that uh CPC classes are there mostly to make the job of patent examiners and patent practitioners easier not the job of us as innovation scholars uh uh uh easier.

[01:51:37]So using this technique we can kind of expand the set to uh include more patents but at the same time we're kind of linked or tied that original core uh which is used as training data. So the third classification we try to step out of that uh of that classification entirely and take an industrydriven or technologydriven approach by uh using a taxonomy of space related technologies that was developed by deal room and the European uh space agency uh which creates created this taxonomy which essentially goes into a fair amount of detail of everything that is happening in space technology nowadays uh spanning from satellite design and construction ractions, space exploration in space operations. So for each of these categories, the taxonomy provides a concise but quite detailed description.

[01:52:26]And since we have both the text of the description and the text of the patents, we can look for co- occurrences of words and expressions. We can try to identify which of these five categories uh each patent may belong to. Of course, the majority of patents will not be space related, will not fall under any of these categories. But for a subset of the patterns we can actually place them under one of these five categories.

[01:52:49]Um so these classifications again we implement them at scale. They are they go from very narrow to very broad. There is a lot of overlap but there is also a lot of patterns that do not overlap across these classifications. One thing that I want to mention this methodology is general purpose. It's very common for you know when we study innovation using patents to look at a sector have no idea how to define it what the boundaries are. So using this kind of methodology we can we can kind of define these boundaries to some extent and we can use this also for other sectors. So in the remaining uh >> five five minutes I will uh go over uh the results and we'll show you primarily what you know time series what happened using this data broken down in different ways what happened uh to uh space innovation since the end of the Apollo era so around mid of the 1970s until uh very recently. uh as Alex said, the first 15 years uh is this uh we're still in the cold war and the the the the kind of the second block of 15 years is where these uh policies that created a propriable market for innovation started to emerge and that predates what happened later which is uh this new era this new space era in which which was spearheaded by the emergence of high-profile startups such as SpaceX and Blue Origin. And the first thing that we notice is that there is this big spike in space related patenting around the 1990s which predates the later increase in space patenting that comes with the new space.

[01:54:22]This is for narrow space transportation.

[01:54:24]These are the blue uh the blue line here. When we look at the uh more extended classifications, we see that the spike in patenting in the 1990s is visible in both in both, but it's extremely pronounced in the broader space uh uh category. Um and thankfully for that classification, we do have this breakdown in all the different categories. And when we look at that, we see that really the category that is driving this big surge is satellites. satellite manufacturing, satellite um uh design and communication.

[01:54:59]And the spike really starts around 1992, which is exactly when the set of policies that created these probable markets for private actors to join the space economy and start kind of increasing their involvement in the space economy. That's when this uh uh this surge in patenting starts to uh become evident.

[01:55:18]In fact, where is this surge coming from? Is it coming from the private sector or the public sector? Primarily from the private in fact entirely from the private sector.

[01:55:29]>> Um the red line which is public sector space patenting stays entirely flat throughout this period. Private sector patenting uh increases quite dramatically again starting around the 1990s.

[01:55:43]Is this surge coming primarily from incumbents or new entrance? Primarily from incumbents. So incumbents are the ones that are responsible for the great majority of this large increase in patenting that starts in the 1990s.

[01:55:58]uh new entrance grow in relative terms but remain a minority in absolute terms which is really consistent with what Alex was suggesting which is that you know when these new markets are created and can be appropriated incumbents can leverage their existing assets and their existing capabilities to appropriate these markets but at the same time this creates opportunities for new entrance to join the race and contribute perhaps new technological paradigms Um something that really tells us that something big is happening in the 1990s in the in the space sector as far as patents are concerned is this big spike in uh uh breakthrough patents. So in the share and the frequency of breakthrough patents. These are patents that have exceptionally high um exceptionally high novelty and exceptionally high uh impact. There is this spike which is visible also in the non-state sector but it's much more pronounced in the space sector. And when we look at it a little bit closer in the paper see that really what can explain this is that the space sector was one of the early adopters of uh of uh the digital revolution one of the the the first adopter of digital revolution uh and the co occurrence between ICT and space happens much earlier than in other sectors such as automobiles.

[01:57:16]And as a last step, we're going to look at the geography of space patenting. Um, and uh, what I'm plotting here is, uh, total space patents by commuting zone in the cold war era. They're on the left and the new space year here on the right. The map for the intermediate period uh, looks very similar. And we see that there is a lot of overlap between them. Um this is something that in economic geography we might call persistence which is areas that are uh dominant in space innovation in this context uh the beginning of the period so in the cold war era in this in this case remain dominant for the same areas that are dominant in space innovation uh nowadays and actually this is something that we can so you know to us this really tells us that uh this continuity between old space and new space is something that is visible both in the identity of who patent but also on where this innovation takes place. And this is something that we can test formally uh by running a simple regression of current space patterns by commuting zone on cold war era space patterns on the right hand side and the coefficient of beta which is around one tell us that there is persistence. So there is not too much for shuffling which is what we find for space which contrast with what we find for land transportation. So automobiles where actually find some evidence of convergence which is new locations rising to prominence at the expense of older ones. Amazing. And uh so let me just try to wrap up. What is um what can we learn from this? So is there a big uh um you know a sharp break a sharp transition between old space and new space? And uh the evidence here really suggests that uh the transition is more nuanced than what uh current uh prevailing narratives seem to imply.

[01:59:01]There is this really this continuity and this intermediate period which was characterized by the emergence of these policies that created appropriable markets where uh they were kind of fertile ground for incumbents to use their capabilities and at the same time slowly progressively allowing for new entrance. They kind of created a continuity between the two. Of course, this analysis is descriptive. Next step of this agenda is to try to think about the causal impact of specific policies.

[01:59:29]And the reason why we think this is important to understand exactly the role the policy has is because there is still a lot of value that can be created and captured in space. Uh energy generation, data centers and you know the exploration of space for the benefit of science and the progress of humanity.

[01:59:45]These are all important things that we have to do in space or know it would be nice to do in space. It's a there's a lot of value that can be captured and understanding the role that policy can have in this process is extremely important and this is true for innovation for space as well as innovation for her. U thank you very much and we're really looking forward to your comments.

[02:00:07]>> Do you want to kick things off Bitsy?

[02:00:09]>> Um I'm really curious to know more about the assignees of these patents. I know you split them into incumbents and entrance, but I'm interested in what other industries they're in, how related the space patents are to the other patents that the uh assignees are working on, um sort of how the geography of the space patents compares to, I guess, related fields like ITC. Um and and just thinking about like, you know, sort of are they building off of of new things or are they they sort of getting into a new area just like what's going on with the sneees there >> right I mean just to push on that a little bit more right I mean in financial innovation which we spend a lot of time looking at there you see the sort of big techization of financial innovation right that basically so much of the stuff is being driven by the Apples and Googles of the world and so forth rather than the traditional banks and so on do to what extent do you see crossover of tech companies into this space or is it still just basically Loheed Martin suddenly they woke up and said let's be innovative instead of being the sleepy people we've been for the last uh century right please >> okay um actually mine's related to that as well because one thing I'm very interested in um just I have a lot of experience in the telecom sector which had another kind of kind of discontinuous breakup from policy and so I'm kind of curious about the supply chain in particular so you should see greater diversity because you'll have a move from vertical integration to more you know suppliers becoming more diverse that kind of a thing. Um the other thing that I was curious about your view and this goes to the previous one I'm kind of curious about the absence of discuss about the change in the risk kind of profile for space. So the fact that in the Apollo the government-driven era right you talk tax dollars failures were very high-profile there's a lot of reluctance to take big risks whereas in the commercial sector we're using private capital there's a lot more appetite for risk but also therefore you know that kind of changes the the the nature of what innovation is able to do so I'm kind of curious you speak about those two things other the um the deconstruction of the monopolies and then also about kind of where you how you might integrate this question of risk into your um analysis Do >> you want to take a take a few of these?

[02:02:29]Yeah.

[02:02:30]>> Yeah. Amazing. So, thank you. So, thank you for this question. So, about the nature of uh uh the uh firm. So, in the paper, we look at different ways when we break them down in different ways and uh um we find the same patterns when we define incumbents in different ways and we see that for example this big surge is driven by non venture capital uh uh backed uh uh firms. So one thing that we do and this is um uh this is this is more kind of part of a validation of whether we are identifying patterns in the correct way is how predictive it is. We we assemble a kind of a fairly big list of space related organization and this includes both large space companies and space startups and we look at how much overlap there is between them and our classification. We find there is a lot of overlap. Now big space firms are involved in space are also involved in other things. For example, Boeing you know does other stuff. So it's difficult then to to separate the two. Space startups are very involved in space and uh you know they have a very high uh uh their patenting is very concentrated in these kind of space related patenting. So there is not too much kind of uh uh uh diversification there. Um and uh so that's as far as we have gone kind of in looking deep deeply into the identity of uh of asenees and these are all kind of things that we can look into and so that's that's great. Yes. And regarding the risk, so there is a literature on risk mitigating innovation and uh that's actually something that we could look at, you know, if uh risk mitigating innovation is something that is more uh uh prevailing under some context. For example, when uh uh when projects or um uh or programs are government related versus private related, I think these are measurable things. So it'll be interesting to look at.

[02:04:19]>> I think we've got time for at least one more. Yes, >> please. Great.

[02:04:22]>> Yeah. Uh great talk. Thanks. Uh I I loved the map uh which uh you had some points that where uh there was persistence uh and we we recognized those points. There was Johnson Space Center, Huntsville, Alabama, Cape Canaveral, JPL. So uh yeah, does proximity to NASA explain a lot of that.

[02:04:42]And then that that got me thinking part B. uh yeah contractors and such even though yeah few of the patents come from NASA directly some contractors are closer to NASA than others I I think the the big difference SpaceX is just another NASA contractor but also they have a lot of other outside contracts >> as well uh and so is there a way to describe sort of proximity to to NASA in a more general space beyond geography so that's part B but >> yeah uh amazing So uh with the geography you know we're Yes absolutely. So one of the biggest predictors of where space innovation happens nowadays as well as back in the cold war era is proximity to NAZA research centers and that's what the red dots that you immediately recognize where uh we can define exactly the same thing uh using other measures.

[02:05:33]I think we've looked at specific contracts that's you know that's part of the data collection that is ongoing for future projects. uh of course uh that is different because uh the uh geography is fixed uh the identity of potential contractors and the contracts and the programs change over time. So yes, it is absolutely possible to kind of try to think about how connected to NASA private firms are and how exposed they are for example to specific policy changes. uh you know the the the match is not going to be as clean as thinking about geography where we know the NASA centers are there and the cities are there and they're not going to move but yes it's absolutely doable and uh that's that's a great suggestion >> great well thank you so much guys for the fascinating panel 12 minutes and despite what the program looks like which looks a little blank we have three brilliant papers showing up after the break thank you That was good.

[02:06:57]>> What do you close up like scratch?

[02:07:15]Hi.

[02:07:24]>> You're a joint tax now.

[02:07:26]>> Okay, cool.

[02:07:29]I was at census. I left a year ago. I'm at the economic innovation group.

[02:08:02]later this afternoon.

[02:08:04]>> I hope everything turns out Thanks for all your input.

[02:09:15]Brother.

[02:09:34]>> Oh, thank you.

[02:09:58]I'll introduce you to >> Okay, let's go over there.

[02:10:36]six It's already a mix.

[02:11:26]Well, I think You Maybe.

[02:12:32]There's alcohol.

[02:13:02]Yeah, I know.

[02:13:36]Are you antirm?

[02:14:03]Yeah.

[02:14:19]What's the What do you think?

[02:14:54]You're already ready to start asking questions before they start presenting or >> what?

[02:14:59]>> Oh, that would be smart. Get ahead. You read the paper. You're way ahead.

[02:15:08]Yeah. Yeah.

[02:15:25]>> All right.

[02:15:34]Yeah.

[02:16:03]Hey, >> hey, it's not Sorry about that.

[02:16:58]Okay, welcome back. We're now going to continue into our our final uh three papers. Um the first of which um Sue Halper will present uh local returns to federal innovation investment evidence from the national laboratories.

[02:17:15]>> Great. Um I'm really pleased to be here and thank the organizers for the chance to talk about this. Um and uh so we think that this question of whether federal R&D can spark local innovation and prosperity is an important question. We have uh both the magnitude and the structure of our innovation uh investments in the federal government is being discussed and challenged in a way that hasn't been in many decades. uh and meanwhile we have growing regional divergence in both incomes and innovation and we have some significant new institutions being created around this stuff. Uh but yet whether these things can be effective or not is a really hard question to answer.

[02:17:56]Uh typically R&D funding flows to already innovative places and so it's really hard to know what the incremental impact of that funding was. Uh, and we're going to argue the national labs are a great case because they sidestep this issue. Um, and as I'll I'll repeat, we are not saying that the national labs are the optimal way to do regional economic development. We're just saying that this gets us some uh uh identification uh help here. Um, so we're going to look at three questions.

[02:18:27]First is do we see spillovers beyond the lab? So are we actually creating some kind of innovation ecosystem with these investments? Um and this is going to be a summary of some work uh that's available in an NBER working paper. Um and then some new work uh about do local gains reflect increased productivity or is it just reallocation from other places and then second what features predict the largest returns.

[02:18:52]Uh so just to kind of fix ideas I want to talk about a particular national lab um in DuPage County Illinois found in 1946. it. Uh they moved some nuclear bomb research from underneath the University of Chicago football stadium.

[02:19:07]Perhaps not the safest place to do explosion research. They moved it to this rural forest. Um a second feature that's also important is like almost all the national labs, argon is governmentowned, contractor operated.

[02:19:23]The uh contractor in this case is the University of Chicago. And I'll spend a lot of time talking about why that matters. Uh and particularly because of the kind of open norms, graduate pipelines, access of to expensive equipment that is uh made available to both companies and other universities.

[02:19:40]Um what do they do? So they got their start doing nuclear weapons, nuclear reactors. Um and then they moved into uh advanced computing, clean energy and materials uh based on their initial expertise. And then that expertise in materials had led to led to things like uh improved lithium batteries and did DVDs. Um so these are pretty large institutions uh 2700 uh employees in 1957 uh and about 3,700 today. Um the annual budgets are around a billion dollars. So it's about the size of a medium-sized research university.

[02:20:21]Uh why are national labs a good setting?

[02:20:24]Um so they are large scale and long lasting. Um as I've said they are kind of we we argue that they are exogenously cited. We did not go there because they had really good innovation capabilities.

[02:20:38]Uh in fact almost the opposite seeking isolated places uh where things going boom would not be so dangerous. Um, and we're we're comparing we're in every all of our results we compare to control counties and we have two ways of creating control counties and the results are pretty much the same. So I'm going to flip back and forth between them. Uh, one is in some cases we actually know who the runner-up counties were. Uh, and so you can argue these are pretty close substitutes. Um, and uh, and and so we have that in some cases.

[02:21:13]In other cases, uh, we have synthetic controls. So we construct a control group using data from neighboring counties. Um, and so we argue then that the lab impact may actually be a lower bound. As we'll discuss, uh, you may think that you the optimal way of doing such investment as you go to a place that has at least a little bit of existing pre-existing research infrastructure.

[02:21:39]Um so just to preview some results um we're focusing here on um innovation as measured by patents. Um and particularly what the graph shows is um local patents by inventors not affiliated with the lab. Um and so what we're looking at is comparing in each case um the uh counterfactual counties um and that so that you can see that before the lab is founded before that dotted vertical line uh they look pretty similar and then you get this increase uh over time uh in some cases quite dramatically um uh and these gains you know it's not just we we threw a bunch of money into a lab and the innovation stayed in within the walls of the lab building, we actually see some evidence that it spills over to other inventors. So we have this evidence on non-lab patents.

[02:22:37]We can see that inventors in these counties move toward the areas where the lab is inventing and we see uh that the citations uh to the lab nearby inventors are disproportionately citing lab research. Um this graph is unweighted.

[02:22:56]Uh but if we wait by citations, we get even stronger results. Um in the the companion paper, we also talk about gains in household income, retail sales, and high school graduation. And I'll talk about those as as I go, but I want to move on to the the kind of the second question, which is do these local gains come at the expense of other regions?

[02:23:17]Um, and so you can imagine that there's these local effects are a mix of reallocation. You know, we moved Enrico Farmy uh from I think Princeton wherever to to Illinois, middle of Illinois. You know, what happened to the place he left? Um, and you might think and but then the another possibility is we actually get some net additions to productivity and wages and innovation. Um, and so we're going to look at this in three ways. So, we're going to look at wage gains, co-author effects, and then some big push dynamics.

[02:23:52]Um, so on incumbent wage gains, uh, we see big increases in economic activity and household incomes in lab counties, so like 25%.

[02:24:04]Uh, after 20 years. Um, but this could just be we, you know, we've moved in a bunch of highwage scientists. Um and so what we want to do is look at what happened to the people who were in these regions in 1940. Most of the labs were founded in the late 40s. Um and so we can actually link we can find uh people in 1940 and find them again in 1950.

[02:24:25]Um and we look at men um and we see overall uh on average these pre-existing residents saw a significant wage increase of of almost 5%.

[02:24:38]uh the college educated workers did really well 25% but even the non-oled educated saw something 1.4%.

[02:24:46]Um and so this suggests that there is actually some kind of economic development going on uh in in these counties.

[02:24:55]A second thing you might worry about is that co-author networks get disrupted that uh you know you lose your co-author to the middle of nowhere in Idaho or Tennessee and you can't publish together anymore and so overall p uh co-op publishing falls. So we looked at this and didn't find much evidence and so so the way this graph works um is we have we look at the set of people who publish in a set of prestigious physics and chemistry journals and um the control group are people who didn't ever publish with anybody who went to a lab. That's the red line. The treatment group are people who uh lost co-authors to a national lab. You might think if if it was just reallocation that that blue line would sort of start trending downward after the lab is founded. You don't see that. And if you squint, maybe you can convince yourself it goes up.

[02:25:55]But but there's some idea that we're actually creating new knowledge here.

[02:26:00]And this is not inconsistent with some other research um around uh you know formation of new groups, strength of weak ties etc. Um the third piece is a kind of uh you're sort of thinking about a big push what's going on here that we're getting thinking about kind of the development economics literature where we thinking about you know can we get to critical mass in these regions um and um and so what we see is we have a a way of kind of constructing a multiplier. uh we found uh some data on retail sales uh from the 1930s. This is sort of before GDP exists and stuff um and uh the Cleveland Public Library turns out to be one of the places where this complete set of this stuff existed. So we digitized it um and then compared it to lab budgets. And so uh two things to see from this graph. Uh one is that particularly university operated labs really saw an increase over time. So that these multipliers start out um and and so what we're looking at is excess retail spending. So how much did retail spending go up in this uh county compared to the control county? And so in 1956, uh, a dollar of lab budget, uh, is correlated with about $2 of, uh, excess retail spending. It rises over time to be about $5, uh, over, uh, on average over time. Um, and you can see that the effect is really driven by the university operated labs. And I'll talk about why we think so in a sec. Um so all this is sort of leading us to tell a story that we think that we actually are with these labs creating new innovation ecosystems with kind of their own dynamics. Um and it's not just reallocation.

[02:28:03]Uh and so then we want to know which labs do particularly well and we're taking advantage of the fact that this is a chapter to kind of suggest and push. We only have 16 data points never going to divide them. Uh so um the first thing that you might look at you you might think that really matters here is what is the baseline um economic you know innovation potential. And so since this is 1940 we think about high school grad as being a kind of useful cutoff.

[02:28:35]Um, and you can sort of see that the the biggest returns in terms of excess patents per dollar of lab budget, um, you see it in this kind of mid-range.

[02:28:46]Uh, so Ames, Iowa, Oakidge, uh, Pacific National Labs, and the ones that really are in the middle of nowhere like Los Alamos, you know, have a little trouble getting started. um whereas the ones kind of at the far end so even uh in uh the 1940s Slack and Berkeley you know were highly educated places and it's a little harder to find the impact of of the national lab spending there but uh what really matters in terms of uh impact is the university affiliation of the contractor and I want to thank uh Josh for uh suggesting that we look at this uh because it it really does drive and you saw in the uh multiplier uh graph just how different they are. Um you also see this difference um when you look at patenting uh where the university affiliated uh uh labs have um what two and a half times the excess patent per dollar of lab spending that that the non non-lab uh non uh university contractors do. Why do we think this is the case? There's open publication norms. Uh there's grad student pipelines. Faculty join appointments. They really see as a mission when they have expensive equipment. So there is this giant computer at Argon. Um and part of their mission is to make that available broadly. Um and uh whereas the uh industry operated ones tend to see a lot of what they do as trade secrets much more insular and so you see much less of these economic development spillovers.

[02:30:31]Uh again this is not to say that this is a perfect way to do it. Um so uh Tatari and Stern have a really interesting paper where they look uh they compare labs to universities in terms of the quality of their entrepreneurship spillovers and uh the labs are way worse. Um and you know when you go to these labs they are not designed for transparency you once you get to Argon you are not done driving. You want to see somebody in another building you get in your car and you drive. Lawrence Liverour is, you know, in Berkeley, but it's on a very steep hill. So, this is Anyway, um the point is that even under these kind of difficult conditions, we're seeing spillovers.

[02:31:15]Um so, uh this is really I'm going to rename this. This is policy implications. We do not make policy uh recommendations here. Um but just sort of thinking about what what does this what do we learn from this? Um and so the first thing is just that governance really matters. Uh and it matters because it in it gives it affects the incentives to embed in local uh econ institutional ecosystems.

[02:31:41]Um a third thing is innovation spending has a greater multiplier than traditional stimulus programs. So we're seeing multipliers you know four to$5 per dollar of lamp spending. Um the typical studies like Choro Reich get like uh less than $2. You know, we think about just stimulus and kind of transfers. The innovation really matters. Um but these these returns only emerge after uh you know many years. Um and so uh you know some of these newer programs for example the Trump administration think about something called tech labs that would be five to seven year projects. Similarly, the focused research organizations, you know, they have many goals, but to the extent the goal is regional economic development, that's way too fast. Um, and and then then the last implication that we'd like to urge people consider is thinking about what the outcomes are.

[02:32:37]Often we look just at jobs. Um, and we want to argue instead you should look at non-affiliated patenting. um I also say co-authoring uh education research or mobility and wages are other things you might look at to judge the success of of uh some of these uh uh investments.

[02:32:58]um more broadly just sort of thinking about researchers um this is a case study um and um you know we just have one type of institution but I think what they do is illustrate how you might think about applying organizational economics to innovation studies and we have a few papers that do this start to do this we have uh Heidi's paper on like innovation ecosystems we have the re some research on universities Um but you know we sort of think here's this is an organizational innovation of the 1940s.

[02:33:33]Uh this go operated government-owned contractor operated model was meant to combine a public mission with long-term funding and managerial flexibility. Did it actually do that? I mean if we compare NIH where actually the researchers that work directly at NIH are in fact government employees. What difference does it make? Um and I think a key thing that I also want to think is not just kind of static incentives at a point in time but the trajectory and what do we do how do we build that trajectory and these capabilities over time. So what one of the purposes of this paper I think today is maybe to um set the stage for the really great paper by Clancy and Snyder and at all about you know what choices should a um policy maker make about how to fund investment and what this paper does I think at least point how do we think about what choices are even available to that policy maker what kind of expertise exists what kind of incentives are the um the researcher is going to leap to uh respond to an instruction from a policy maker etc. Um and so anyway there's a whole tons of questions that keep us busy for a long time. Key dimensions of institutional design. Um you know we really just pointed to one which is the the university uh versus private industry contractor thing but there's a whole lot of other kinds of things we could look at. Um uh and then we also need to think about not just the labs uh within themselves but their relationship to other uh institutions. And so there's a a bider program called Tech Hubs that tries to think about existing institutions and knit them together more effectively. Um and I guess make from my experience as a policy maker these information barriers are really large. Um, so I, for example, you know, you you often find or one of the examples of some of the Obama era um, uh, ecosystem development programs was that the workforce director in a relatively small city met the economic development director in that city.

[02:35:51]Usually, it's seen as completely separate. So promoting this kind of interaction can can, I think, be very important.

[02:35:59]Um uh so I'm already talking about future research. Um I think uh thinking about the functional form of these spillovers.

[02:36:08]So Johnson and Gruber have this great book about startup uh or jumpstarting America with this essentially an assertion that there are a bunch of um midlevel um regions that have a little bit or a moderate amount of R&D and research capability. and we could jumpstart them and that would benefit those regions and it would not hurt the regions on the coast that currently do the vast majority of R&D in the US. And so I think you know we have some kind of suggested results that maybe that works that there's a lot of congestion in these coastal areas both of ideas and and also traffic um but that this kind of happy uh uh reallocation could happen and and get us net new uh uh ecosystem development. Um there's a really cool uh forthcoming paper by a guy named Stuart Buck of thinking about the innovation ecosystem as a garden and it's a really different kind of metaphor than a machine. You know a garden you select, you read but you also get some kind of endogenous growth that you don't control. So I think it's a really suggestive metaphor. Uh and then just to point out that we're not only running this experiment as increases. We are currently seeing the potential of some very large decreases and we should think about potentially over time some really large impacts on regions and not just nationally.

[02:37:35]Um so to uh conclude um three three points to leave you with. Um one is that um um these federal research infrastructure generates large and persistent local spillovers even when we start from near zero innovative capability. Um and that we think that this is actually new productivity uh creation and not just spatial reallocation. And then finally, that institutional connectivity is really important for making the most of our research dollars. So, it's not just spending more uh but thinking about how it's spent, how people talk to each other, etc. So, thank you.

[02:38:25]>> One quick question as people are lining up. Um, with regard to the the non-lab local patenting, um, do you have any evidence that the human capital there is was formerly at the lab and then kind of left the lab and started something or are they being drawn in from elsewhere?

[02:38:41]>> We we have looked at this um and and some of it is measuring what does it mean to be at the lab. Um and so um we've been if someone was ever affiliated with a lab, we tend to call that as a lab patent. Um and um but often what you see Yeah, I maybe I'll just stop there. Yeah.

[02:39:08]>> Yeah. Hi, great paper. Uh really interesting. Um although you can imagine I could have a bunch of kind of caveats I'd like you to consider. So obviously one is that the the diversity of the national labs themselves is actually quite broad. Um so you have national nuclear security administration labs like Los Alamo Sandia for Liverour versus Office of Science Labs like you know Argon Oakidge the office of science ones are by nature more open and also there's the issue of like some labs are really focused around user facilities like Slack or um Brook Haven right versus something like Oakidge is more multi it has user facilities but it's more multi-dimensional. So one thing I might think is looking at the areas in which those labs are publishing which is something that I've looked at with some colleagues and there are certain areas of science that are more open. So if you look at nuclear physics that tends to be pretty closed field because it's related to nuclear technology but then if you look at the ones that are more looking at like environment which argon went to more environmental science um or ENL right you're going to see a very different kind of dynamic. So maybe one idea to think about is the openness of the institution is going to be a big issue and especially the new programs like I do work with engines I do work with tech hubs those aren't even facility based they're actually built as a consorcia that are themselves not even institutions so I I just maybe think about kind of how can you maybe create a variable to measure the openness of the institution itself and how that might have an influence on the um spillover effect.

[02:40:32]Yeah, I mean I think this is uh gets at kind of you know causality and um it's not you know some of these lab uh contractors are assigned for somewhat exogenous and historical reasons but sometimes they're also assigned because of the initial mission of the lab but then there's a third possibility which is the you know the mission evolves based on the contractor and so we you know we I think Uh yeah, we should be more humble. We'll try in the paper to about this. This is just correlation and just something to think about. Um and and I guess maybe my main point is that in innovation studies, we might want to think not just about institutions and incentives, but also the trajectory and what kinds of uh things, you know, unfold over time based on some initial conditions. But but look forward to talking to you more and getting that paper. real quick thing they'd look at actually is a number of postocs at the lab and maybe has a a proxy for how open they are. So that data is hard to get and I know from a source who may not want to be identified that the private contractor labs actually see that as a trade secret and so you cannot um if if the source wishes to say she say more but anyway it it is hard to get >> uh yes thank you for that really interesting paper. What what do we know about the scale required do you think to get get this kind of effect in terms of federal support? I mean these are big labs billion dollar budgets. What do we can we expect the same sort of dollar for-dollar impact from smaller programs?

[02:42:16]>> Yeah. Because I mean if we think about these current programs I mean like the entire tech hubs program is a billion dollars which you know we thought of as like giant but it's a billion dollars spread across like 20 different cities.

[02:42:29]Um and so on the one hand you know we're going to establish innovation ecosystem so the ground may be more fertile on the other hand the amount of money that we have is 120th uh so you and and the question is I mean there's a whole literature in economic development about takeoff and and is 50 million enough to get you takeoff and I think we can't actually answer that with this work but it's clearly a really important question.

[02:42:57]Hey, uh I was was wondering instead of a comparison uh with um stimulus spending if you have or could look at like uh universities that start in a rural area and me and measure those spill up over effects to then try to infer is it well if stimulus is different than a community being developed but now you can control for the community being developed around like say a liberal arts school in a rural area and the difference would be then the scientists there that are doing this lab research if that's even a possible comparison >> there's really amazing work by Andrews who actually uh argues >> here >> oh great great great uh but you know you find like counterfactual locations for uh uh many many universities uh and get my or my understanding is kind of order of magnitude similar results Uh we also have you know the NASA work from Alex that kind of that actually we we kind of looked at the kind of dollar implied dollars per patent there and it's a lot higher. So our the labs are better than that. I mean you know these things are are uh SBIR looks really great but this is some something we hope to do in follow-on work.

[02:44:20]>> Two more uh here. Oh. Um, this is fascinating and I really like how much you got from 16 op observations. But as usual for the audience, I have another suggestion on how to split them. Some of these were nuclear labs and some of them were accumulating huge amounts of really dangerous waste that the federal government is still paying large amounts of money to store. And maybe if you could somehow subtract that storage cost from like county output to get uh you know real county output.

[02:44:52]>> Yeah, as I'm saying I'm not sort of saying causally that this is the way we should allocate all our research dollars but yes and this is another cost of these things. Thank you.

[02:45:01]>> In NEPA I don't know how storing nuclear waste goes into retail sales but it might but but you know I don't buy that anyway but um >> um thank you s for an interesting paper.

[02:45:11]I believe the Stevenson Widler Act requires each lab to have an office of technology transfer to actually push the innovations and and research findings out of the lab and commercialize it with with a focus on the regional area. So do you have a sense of the the efficacy and the impacts of the offices of tech transfer in these labs?

[02:45:34]>> Yeah. So, so we mostly stop in 1975 because that's it gets harder to identify to attribute patents to the national labs after that. Um, but I think just qualitatively it really matters which lab. So I think Sandia and Los Alamos, you know, they had these um entrepreneur and residence programs um that that were actually fairly successful. others are just you know that is they even despite the existence of this office they don't they still don't feel like it's their job u so I think very uneven effects and and then the Scott Stern and Tartari work is actually done post Stevenson Wler and finds kind of not a huge amount of you know not not effective compared to the best universities okay thank you so much Sue Uh our next paper will come from uh Dewey Murdik and Richard Clavvens. It's called Keeping Pace with the Frontier National uh portfolio dynamics.

[02:46:46]We have here a collaborator with someone who That was my business partner who did that.

[02:47:07]>> Yeah, I'm just trying to find myself here.

[02:47:16]Oops.

[02:47:28]I'm not seeing the share button. Sorry.

[02:47:31]>> I thought I would say something clever while I did this, but I don't have enough cognitive energy left. Um.

[02:47:39]>> Oh, there.

[02:47:42]>> Yep, that's us.

[02:47:58]>> Great.

[02:47:59]>> Well, um, really excited to be part of this discussion. Really interesting conversations. Um, thank you Ben and Josh for the invitation. Um, so Dick and I are going to talk a little bit about the community level. This is more of a microlevel view of how uh the innovation ecosystem works and we're going to kind of look at a couple different community level measures that will help us understand a little bit about the scorecard if you may about uh national R&D agility and we'll get into some examples of some interesting details that will hopefully um light up some future discussions. So, one of my best and most uh favorite jobs was being a program manager at the intelligence advanced research project activity uh DARPA's cousin uh our sister agency.

[02:48:41]Um and I got to ride the wave of innovation really try to leverage that innovation wave to uh achieve you know national security relevant uh capability developments um and really fill critical gaps. The problem is when you're at that level or even if you're at the office level or even at the agency level, you've got a lot of competing decisions that have to be made and they're really quite challenging. Um, you have to figure out, you know, where the state-of-the-art is, a bet wave you're likely to catch, um, and which one you need to wait for the next cycle of innovation before you leverage that wave for whatever capability you're trying to build. Um, you have to find the right partners. You have to figure out who will actually respond to your solicitations. A lot of people don't. uh who will actually do the work and who will walk away with IP often very different entities. Um and then you need to be able to figure out what your open science approach is. So open science demonstrabably at least in my experience if that's where you get better innovation but in the process you expose a lot of capabilities that uh adversarial competitive uh entities might walk away with and you have to figure out what that calculus is. Um and then you know in in the process of that which is very hard being able to figure out what the capability versus intent of every adversarial entity uh to be able to do this calculus appropriately. So um and uh I did my best as a program manager as an office director as a R&D portfolio manager. We were trying to figure out where to hedge the bets and appropriately. Um and you use your network and you use and you read what you can and you just hope it works out okay. But there's a lot of need and opportunity for tools and anal an analytic frameworks economic wise or R&D wise and that's what we're really uh talking about today. In fact in 2011 of the programs I started running was a program a research program to try to figure out what the infrastructure is necessary for this type of work. It was called the fuse program foresight and understanding from scientific exposition. Um and uh Dick was one of the performers and we've been kind of like that was 15 years ago. We've we did okay. uh we kind of met the the goals um and uh but we've we we're uh unwilling to give up the fight and continue to make this innovation and we've hit some really interesting thresholds that I think are worth your time today.

[02:50:55]So big high level two findings uh and we'll get into the details kind of a bottom line upfront thing is that um for researchers um I mean for monitoring research doing this at a micro level at the 100,000 research community level gives you new insights and allows you to do um community level uh analysis at a way that allows you to break down aggregates and we'll talk a little bit about why this is so important for actually making effective policy decisions. Um and then we have to be able to think about um we we've hit a threshold in terms of the predictive accuracy that is actually finally useful and we'll talk Dick will talk about that a little bit. We're not breaking forecasting any breakthroughs in individual breakthroughs. This is community level. Um and then we've starting to see how the agility can be measured um at the national level and give kind of a scorecard of how nations and regions are able to respond to the the pressures and reallocate funds. Um we're using we we've over 15 years we've tried a bunch of different measures uh everything from growth rates to term usage novel term usage and we're using a pretty useful one about uh citations u as a proxy for attention from a scientific community um anyway so what this enables is enables to um you to start managing re R&D portfolios in some interesting new ways because you have some new insights that are much more timely and more proactive and that's really to the point of what we're talking about. So Dick, would you like to get into the details?

[02:52:31]>> Well, the joy this is okay. So the joy of working with Dwey is that actually I had a a totally different trajectory before I met Dwey. I came out of strategy at PhD at Penn at Wharton and I asked a very uncomfortable question in 1990 when I was there as a student and I said look I'm looking at patents and all this other stuff but I'm noticing that science is becoming more and more important as a competitive advantage and a locus of innovation and if that's the case what do we know about science and how it's structured and how it evolves.

[02:53:04]Okay. So you read Cune which was a major part of the um of that uh oh you're on the the wrong slide. There should be a slide before this. Here we go. You got it. And uh so one of the foundations was basically Cun's idea of a research community. Very very simple. But I was more at the micro level. So as a doctoral student brought it out to a couple people that I knew in industry.

[02:53:26]Got a contract from Smith Klein to do portfolio analysis. Got one from Warner Lambert and I said why go to the academic track? I'll set up a little research labs with a bunch of people that I know and we'll do that to see if we can improve this technique that was being developed at the Institute for Scientific Information. The father of this technique is Henry Small who was director of research at that at that time. That's pretty much what happened.

[02:53:50]And then it was I think 2011 when we worked together um and I turned 65 in 2013. So, uh, I was about to retire and so I handed the company over to Kevin Boyak who was at the Sandia Labs and luckily had the, uh, uh, entrepreneurial leave and, uh, uh, ended up being on Dwiey's team, I could say, of how do we crack this problem.

[02:54:15]So, let's get to the next slide.

[02:54:20]So, Kun's idea is very simple and how to do this is extremely simple. You take a huge database. In this case, we do open Alex. You kind of cluster the papers based on who sites whom. That's much more important than the textbased one.

[02:54:33]And there's a reason for that because these are dependency relationships. When you site someone, it's a communication.

[02:54:39]You are depending on the prior person.

[02:54:41]So, we're trying to model the communication problems. Cun said it's about 100 people in a community. There's 10 million researchers. So set the dial for 10 for a 100,000 research communities. Each dot in those things is a research community. Basically that's it. Um and there's about I don't know 40 art articles I've been co-author about that describes these details a lot more difficult. Just do Google Scholar and you'll find all of them. Let's go to the next one.

[02:55:09]But the best way to really show that is to show Josh.

[02:55:14]So, here's this picture with all these dots and and in all those dots, let me first give you the color things. We had a lot of fun with that. Purple is physics. That's the royal royal discipline. Next to that is dark blue.

[02:55:27]That's chemistry.

[02:55:29]Pink is light. You physics light. It's called computer science. And the other ones light blue are the engineering fields because they're not the, you know, they're not the basic areas. Earth science is brown. Duh. Biology is green.

[02:55:45]Duh. For medical, you go from yellow to red depending if you're likely to die.

[02:55:49]So if you have, you know, problems remembering things, that's a yellow area of research. If you're it's infectious disease and you're dead in the year, that's the dark dark red. Luckily, social science is that peachy area in the middle. And you can see that his we just took his publications and said, what community does he belong to? Now, there's a hundred,000 dots here. And the reason for showing that is look, he has three major areas all in venture capital and they're really different. And when you actually would do that and you show it to them, yeah, they are really different communities. This happens so and so and this happens so and so. And it's almost a visceral reaction of this community is my life. He's a member.

[02:56:33]It's like he travels from from town to town and these are the companies he spends a lot more time in. That's not an unreasonable way of thinking about it.

[02:56:41]That this is basically what we're trying to get at. What communities do people belong to that really work about a particular problem that they care about?

[02:56:50]Oh yeah. One other thing about that actually we can go back on that is is very important.

[02:56:56]You all talked about spillovers.

[02:56:58]What you look at in research is a community is linked in terms of a discovery affects another community.

[02:57:06]There's more importance in the spillover in the links between community than what's in the community. If you really talk about its impact, its impact is in what's spilled off the cup, not the coffee in the cup. So, we much more focus on spillover effects. Now, let's get the next one.

[02:57:26]So, there are a couple things that are here that we measure. The first two, first one is about uh predictive um uh things. And this is what Dwey said. We have to figure out whether or not we're sufficiently predictive that first of all he had two criteria. One is it's not crappy. Okay, it's not bad. It's at least okay. And then the other one is it's good enough that you can make decisions on. So started in 2011 when he said we did okay on that. I kind of went like that because nobody hit the not crappy threshold by the time the program was over. Okay. That's what research is.

[02:58:03]I admire somebody who does things that fail um very much so. Uh we didn't hit that just that first threshold until about uh was about 2020 and then it's a breakthrough that happened about a year ago that said suddenly a lot more predictive in that level of 60 to 70% which even I didn't believe so it took a year to just see if I could break it. Um that has a major major effect. This is the first time we're talking about it and doing that. It still has its warts.

[02:58:33]Just realize that it's because it's, you know, proof of principle, but it's it's it's an expected one. Expected papers is really what a nation's expected to published. That's normalization. You all do that. The portfolio agility is based on a very very simple idea. You would expect if there's a 100,000 research communities, there'll probably be 10,000 that is unlikely to come up with any discoveries in major methods. There's probably 10,000 that is highly likely to. Wouldn't it make sense that in anticipation of the communities that you're in, if you're in the communities that you know are going to really take off, you put more money in that? In this case, you publish more in that area. And if you're in the 10,000 that aren't going to publish anything, yeah, take some money away. That's basically all we did. That's what the agility measure is.

[02:59:21]So you take China and India and United States. The first one is what their growth weight overall in publications for all 100,000. Then you take the 10,000 of the ones where you don't expect a huge amount. You subtract the the expected values and sure enough they're all positive. They're growing a lot faster in anticipation of impact, not after impact, in anticipation of impact. The same with the bottom 10,000.

[02:59:51]You do the same thing and they're all negative.

[02:59:57]Then you do it by nations because we did this for 20, 2021 and 2024.

[03:00:03]Now the interesting thing is the 2018 and 2023, 2021 has actual future information that says whether or not the community had high impact. So you can use that future information, but you can't do that for 2024 because that's future information. That's the only place that predictive model was for was to see okay how can we do a profile for 2024. So 2018 and 2021 are based on actual data and the 2024 is based on predicted impact of a community and as I said the predicted impact of 100 communities on R squared of about 67 68.70 that's pretty darn good uh when you're dealing with that large scale and cross-sectional and what you do also see is tremendous stability in these profiles.

[03:00:55]So common sense would uh tend to indicate that a R&D organization would want to put resources where there's likely to be future innovation um and remove resources where there's probably less interest. You know things we've tried maybe it's move more to a more applied state um and um so that moving faster than baseline versus um as an incentive. Um this so this is kind of the common sense assumption. So basically India and China in this analysis are moving more agile. they're move they're moving resources more quickly into the areas that have by this proxy measure uh more more discovery rich potential the EU and uh the US are less so but the pattern is pretty stable was just mentioning important caveats that we just need to make sure and caution um you know where this measures pace not intent you know once a a funer actually puts money down it takes a little while to see it so there there's some lags in there also u it's measuring activity we're not trying to say citations are quality. I mean, this is a it's it's different. Um, and so this is a proxy of influence, not a verdict on the quality of the science. And then, you know, there are other things that are important to rank. Um, just um, you know, resilience, public health, application, industrial competitiveness.

[03:02:09]These are all things that you would want to roll in here. So, just saying kind of obvious things, but I just want to recognize that that's a caution. Um, so if we just step back for a second, the institutional infrastructure for monitoring science, at least in the US, was built when we were number one in many situations. I'm not trying to say that we've lost that entirely, but it's definitely a much more competitive landscape. We're seeing evidence of the portfolio being updated much more agile, much more quickly, sorry. Um, and so, uh, these are just really important, uh, characteristics that are different. And I think the really important thing for us is looking at that now we need to start tearing apart and understanding where the innovation is going on and so we can start understanding what a particular agency or directorate or program officer would actually do with this information and this really appreciated Ben encouraging us to dig a little deeper into this uh vignette approach. So, imagine the tip, not imagine, you've we've seen the tip, uh, road map. Um, and it's got these 10 priorities. Um, and it's got a nice little report. And so, we basically pulled this out and started saying, okay, let's find a couple areas or we actually did all of them. And I'm just going to show you a couple quick uh explorations into uh a few of the areas.

[03:03:23]So, the first one is uh lithium ion batteries. Um, so this is this first panel is observed. So, this is 2018. So imagine being in 2018 and you start saying so the x- axis is that observed versus expected ratio. So if you're far on this side, you're publishing in this area much more than you would u on an average research community. Um and then on if you're on the other side of the dotted line, you're publishing much less. Um and then the y-axis gives you some strata basically u and this is observed. We're using real citation information with with basically low citations, middle citations. You're highly cited, but most of it your citations come within your community um within your research clusters or communities. Um and then the top one is one that you're getting a lot of citations, but most of them are coming from other research communities. You're transferring that knowledge elsewhere.

[03:04:17]And so now you can start to see what's happening in the lithium ion battery space. And in 2018 you saw anode and cathode materials work largely being done u by China uh in this comparison.

[03:04:30]Um and then US in this case for example three here is leading in u molecular sim in simulation of uh the battery characteristics and the d the interfaces between the chemistry um electrolytes and the anode cathode elements. So we're starting to see a really important dynamic as you you have different characteristics depending on um so you can't just look at the aggregate view of what's happening looking I am and you can see where the investments are happening and then you can start to say oh well do we want to change our investment strategy or is this really where we want to double down and these are the kind of questions that this is enables within that agility context now we're moving forward into the next um and now this is predicted so this is still observed the publications versus um you know what's actually happening versus what was expected. Um and you can see where you know once again this is a very China dominant topic area as you might have guessed a lot of red uh with um and we're still seeing the materials work being led by um the um by the Chinese uh but the now we're starting to see some really interesting work in the US where we're focusing additionally on calendar aging trying to figure out the lifetimes of batteries figuring out why they die and how to simulate the process the simulate process to try to improve the lifime time and age of batteries performance. But once again, you're seeing a very different picture when you start to break down the story. Next story is uh large language models. So 2018 large language models were definitely happening. Um but it was a very small community. These um pre-trained large language models was where or pre-trained language models is what it was typically called. And you can see where it was going on and you can see some of the context in which this was happening. Um and you can now say okay what would your investment strategy be in 2018 where do you want to do now this was obviously a very industrydriven one so this got some extra dynamics in it but then you can say okay this is what now 20 24 looks like um very different dynamics you see China working very hard to catch up US is still engaging but the dynamics of what areas are being focused on is is very different and um you can start to when you disagregate you get a lot value and that's really the main point is that a program manager needs to see or a program officer directorate needs to see these things at a much higher detail. Um so the big big takeaway here is you know aggregate volume is not the same thing as the frontier presence and you need to be able to have the ability to look at the frontier presence and countries with different or even same total works have very different profiles about where they're putting those bets. Um and it's really helpful to see. Um so a PM can use this in a variety of different ways.

[03:07:19]They can stress test their current portfolio analysis. They can understand uh where um because now they've got a predictive element. They can start to see that view here which is now looking into the future about what tier these things will be. And once again because of Dick was just talking about we have about a you know 7ish 67 uh so this is not total rubbish. it gives you enough information to start providing value at in how you operate. Um so just to to bring things to a close here. Um so you really do have to watch uh where you're putting your resources and you know looking at where communities are for forming where uh adversarial or competitors are concentrating their patterns and adjust capabilities appropriately based on this type of analysis. Um I think this brings a lot of a lot of very interesting question a lot of follow-on work. So once again this is enabling and it could connect with a lot of economic uh analysis um and um use some additional data sources but the main point is that we have a proof of principle that allows us to look at community level shifts and are visible over time. You can see the predictive accuracy is enough to start to make be a proactive analytic capability and allows you to we need to obviously do a lot more robust testing robustness testing and a lot of data integrations really start to rich make uh this able to tell us much more rich stories. Um but that's where we're at.

[03:08:48]Thank I'll add one.

[03:08:58]Ask for questions. But in the meanwhile, while people, >> do you have a did you have a last time?

[03:09:02]>> I just want to to mention one thing. A lot of a lot of what I've heard is about venture capital and the importance of venture capital when the innovations are based in technology. Well, if think about the role of the program officers like Dwey are essentially venture capitalists when it's science, they're each spending maybe a hundred million dollars and they have to decide where to place those bets and they have to understand the competitive environment for each one of those bets.

[03:09:30]When I've talked to program officers such as ones at NSF in the chemistry division, none of them monitored what was going on in China in chemistry. I always thought that was amazing. Now, this was about 10 years ago. I'm sure they all are now. But regardless of that, it's the principle. If the equivalent of a venture capitalist, you still have to look at the competitive environment to discern where you place your bets. That's not, I think, being done as well as it could be done in this country.

[03:10:02]>> Okay. Um Jeff, >> thanks. Uh great talk. As as you guys know, I love this stuff. Um full disclosure, do we actually sponsor some of my work early on from Fuse? So, thanks for that. Um but my question is um just from the funders I work with, right? There's a a fundamental question between do you fund projects or basically ideas or do you fund people?

[03:10:22]Like can we spot transformative ideas or can we only spot transformative people?

[03:10:26]And I'm kind of curious going back to the first talk about how a lot of R&D spending actually goes to training the research workforce. Do you integrate that into your model about it's actually the movement of people among the communities and the the agility of the performers or is it really the agility of the funders that's determining the the dynamic?

[03:10:44]>> Yeah. So I mean to me it's at least in my experience it's an integration of those two. You pick your top you pick your priority areas and then you put really exciting people within those spaces. Um you know um so there is sorry you do it with an awareness of the technical areas that they're going to bring and whether you think it's actually the right bet to make. So to me it's an integrated at least in the ARPA models uh you really you know it really is a program manager centric uh type approach and so you really do have to find the right people um and find them at the right time of the the wave of innovation um but generally you let them run with a whole bunch of different experience and let them learn iteratively from what they're getting from the proposals and what they're doing. So that that's my I don't know if that's overly simplistic but uh that's >> and I'll give a different answer as we always do. Okay. So um it's a great question and and in disclosure we've known each other for I don't know decades. Um I tend to lean on General Dorio's old belief that it's about the person not about the idea. Uh, one of the things that Dewey did when he put together Fuse is he made sure all of the people that he funded were in a collaboration where they could talk to each other. The ones that you can bet on are the ones that talked, shared, and learned.

[03:12:04]It's you get into the door with having a good idea, but sometimes those don't play off. And what you learned helps you succeed. And so to me it's the people yes you get them in the door based on their idea but it's still the person and risktaking. This is why I'm also concerned about the prior talk saying that we should be giving the money to the universities. My impression is that most university top people are less are more risk averse as opposed to risktaking. So you have to look at whether or not in in the faculty that you're providing money are they risktakers? Do they support risk-taking amongst their students? Do they support those kind of entrepreneurial behaviors?

[03:12:49]>> Real quick about this, does your model though take into account personal characteristics? When you're looking at them, >> we have included an indicator about the personal characteristics. We've tested a number of them, but the only one that worked was Tedlo's old idea of foxes and hedgehogs. And you know, Josh is is a fox and I'm a hedgehog. So that's >> I don't know how useful that is but anyway >> the other dynamic that's relevant here is the the science is done as communities and so this research community based centric view you're whether you might not you might not be able to articulate exactly what every belief of that community is but they do have a shared uh environment in which they're trying to solve problems together and have a de facto or a very real community. So to me it is very communitycentric the entire model that we're rolling out >> and the gatekeeper is the key person usually in that community is not the entrepreneur >> coming back to agility. So you had kind of a rank ordering where India and China look extremely agile US EU less so is that partly because do you think India and China are kind of growing their R&D footprint so they can enter green field areas where and they just choose the ones that are hot as as they're recalibrating or is there something else going on? It would be true if it was only on the hot area, but on the cold area they actually cut faster.

[03:14:01]>> Yeah.

[03:14:02]>> So I I I might maybe we differ slightly.

[03:14:06]Um but uh >> we will always differ.

[03:14:08]>> But uh the when you have a lot of investment, it does allow you to refresh. Yeah.

[03:14:12]>> More easily because you you you can put money and and allocate resources more uh without killing all your darlings at the same time because you have new resources to allocate. So it does provide a little bit more help. Um, I think the real challenge is figuring out from a practical sense how to dep prioritize attention uh with because that's where you get your biggest complaints of like how in the world could you not fund this important research? Look at all it's brought us. Um, and that's a really hard decision for a constrained economy.

[03:14:43]>> Absolutely.

[03:14:44]>> Yeah, this is really interesting to me too and actually I don't know if this is thing is it working?

[03:14:50]>> It is working great. Um, so to tie into the agility question, you know, something we hear a lot too is you have to be patient. So agility is great, but how do we make trade-offs against the patience in order to see returns on, you know, moving into spaces that we actually want to be in?

[03:15:05]>> You want to answer that one?

[03:15:06]>> No.

[03:15:07]>> Yeah. Yeah.

[03:15:08]>> That's a great question.

[03:15:09]>> So it's a great question. So we have 100,000 research communities. Uh, we're saying the bottom 10% are often dying. they're off sorry dying is they don't actually die but they're they're not as relevant uh to today's innovation space um and they're often are more applied and it's there's a larger percentage of that work that's going on outside of the scientific community. Um not saying that's bad work that's another phase of the innovation system that are really spending more and more effort in that or adjusting. Um there's a large section in the middle that 80% where there's a lot of incubation going on. So, we're not saying that you don't. This is why to to Jeff's uh question. There is, you know, bringing people in and hedging your best bets across a whole R&D portfolio is really helpful. When I managed the R&D portfolio for DHS, we had a large hedge situation where we spent so much of our money on the big bets of what we think would break through. we have another set of our money which we're trying to like you know continue to make innovation and then other ones we're like well let's just try something crazy and and and try it out and so I think you need to have a R&D portfolio managed uh at that at that cradle front in a way that allows you to think through this >> so so actually another answer to that question now you gave me a chance to think about it >> the program officer is take is needs to be like a venture capitalist a risk taker and make that decision and be and be encouraged encouraged to fail.

[03:16:39]>> That's really important. If you are really not encouraged to fail and you always are measured by okay, how many papers were published by the people you funded, you are going to be risk averse and you will be funding an area that had earlier hype and because of that earlier hype, even though the indicators say this is now not going to be working out, it is not working out. It is it should be a personal judgment and to me that's one of the things that it's personal judgment to venture capitalist they ain't stupid >> they do a lot of work to do that but I much more trust that kind of providing the risk takingaking in the hands of people like that than a bureaucratic system >> okay we should uh conclude but actually let me give you one very last very quick question for looking for very quick answer which you look at your top 10k growth communities are predicting the best growth and what share of those is the US uh dominating versus say China or others.

[03:17:36]>> So I was just looking at this and I just completely blanked on the numbers when you asked that. Um but um I can look up the number but basically the US is present but um not uh it's like at a 10 or 12% or something like that. Yeah. Um where you see China more like a 35ish percent. I need to check those numbers because I might be hallucinating slightly but it is a >> that direction. Okay. Thank you. But I think we should that's okay just because we're we're at we're at time. Thank you guys. Thank you both.

[03:18:10]>> Okay. And so for the final paper of our program, we are delighted to have uh Chris Snder and and others uh uh uh on a policymaker centric guide for selecting innovation funding mechanisms.

[03:18:25]>> Good. Excellent. Please do.

[03:18:41]successful.

[03:19:08]Thanks. Thanks very much for uh to the organizers for inviting this paper. Um it's a team effort as you can see from the list of co-authors. It's a joint undertaking of the market shaving accelerator which is a policy shop that I'm one of the faculty co-directors of and the institute for progress. So you can see and also Matt Clancy from coefficient giving is also an author but we have some of the uh rest of the team up here. Matt Ashe from uh Institute for Progress, Sid Haria from Market Chipping Accelerator. Claire McMahon was an RP at the market shipping accelerator. She's now a grad student at Harvard. Um there's Claire and I'm Chris Snder from Dartmouth College and uh Caleb Watney from the Institute of Progress is also another co-author.

[03:19:51]So um you can picture yourself as the program manager or policy maker. This could be a government official or uh somebody who works at an NGO or maybe a philanthropy. And you think of all the set of possible funding mechanisms for innovation uh R&D contracts inter mural science you can run a in in-house lab loan subsidies grants for researchers or or projects and add to that some of these you know newer funding mechanisms pull funding mechanisms like advanced market commitments prizes relative versus absolute prizes how in the world you have your uh research budget how in the world should you allocate that so that's what we're about in this paper. Um, so the natural response of a policy maker is you just default to precedent. Um, or you might say, "Hey, this is a shiny new idea. I'm just going to push that forward. Is it really relevant?" Maybe you flip a coin. Um, so this paper is sets out to be a guide to the policy maker um to solve this difficult question. Um, one thing we're going to do is catalog the uh funding mechanisms that are out there. And we have this web tool we're putting together um called the Atlas of Innovation. the TM is is kind of a a joke. Um, so we're certainly going to include Oops.

[03:21:09]See, how do we go back?

[03:21:17]Inadvertently, >> is this the one?

[03:21:24]Um, we need tech support here.

[03:21:30]There it is.

[03:21:31]>> Um, okay. Um, so we're going to certainly integrate all of the traditional innovation policy concerns, asymmetric information, moral hazard, adverse election, coordination among parties, spillovers and things.

[03:21:44]But we're we're going to in a sense um integrate this uh in a sense maybe even more fundamental set of questions on what's contractable on the set of of feasible contracts. Um both because we think they're primary but also because they're in a sense simple for a policy maker to get a handle on and answer. Um you can't sign a contract on information the funer never obtains. Um and so um we're going to think about questions you can ask to determine the the nature of contracts in this setting. And and the reason contracts are important is if you can sign a contract, you can commit u to either to not reigging on the basis of that and that gives a target for the innovators to to try to achieve. So can can you identify the team? Is there somebody you can on the other side of the contract to to sign a contract with?

[03:22:33]Can you specify a problem and at least sign a contract on specifying what makes a valid attempt at a solution? Um are things even more contractable and say you can specify a valid solution and val verify success that you can make a payment on the basis of. So those are the the uh what we're bringing to bear here. Um so one example of why it matters not just the level of funding but how it's organized u what sort of funding mechanism and we've seen a bunch of papers on space and so this is a an example I mentioned the international space station uh resupply um initially it was a sole source contract with Kler aerospace um it turned out they never ended up flying their uh K1 rocket they went into financial distress there was some issue as a non-competitive tender and and issues are raised so NASA went who said, "Well, let's go back to in-house provision. We'll use the space shuttle." The GAO said, "Hold on a minute, you know, let's let's re-evaluate." And so that led to this uh CS program. Uh part of the issue was that maybe we should try to stimulate the commercial space industry. Anyway, um so they moved to a more competitive set of tenders to multiple firms. They integrated milestones. Uh KSL actually won one of the contracts. Uh turned out they failed to meet the uh target financing. couldn't raise venture capital and they didn't meet the technical milestones, so they sort of dropped away. SpaceX won a contract and and the rest is history. Uh they're private, but the estimates are that they're valued at um upwards of 1.25 trillion dollars. Um it kind of also suggests both the the form of the contract and in this case it's like how do you identify the sort of winners? It looked like Kesler was was a good firm.

[03:24:12]In fact, SpaceX turned out to be a better firm. Um this is a a a paper uh hopefully in in the volume um related to other literature. There's a a huge literature on the advantage of of various public funding mechanisms.

[03:24:27]Closer are those papers that try to horseer race um different mechanisms. Uh a bunch of different practitioners guide and ours is one of those. Um I think um perhaps the the I'd like to identify our relationship to literature on and contractual verifiability. uh you know incomplete contracts that um you know some so there's known unknowns and unknown unknowns that you can uh put into contracts and that actually leads to sort of organizations that try to um navigate that the problems of the absence of contracts. Um I' I'd say the the most seminal paper for us is this paper by Josh uh and Yure that in the AR that that talks about um you know integrating this contractual uh contractability into innovation. So I'll I don't have a picture of Josh but um so uh just quickly with the conceptual model um we're taking the perspective of a principle this will be uh the policy maker is seeking to achieve an innovation goal we'll say fairly vaguely and they're going to allocate a fixed budget to uh some kind of chosen funding mechanism. qu the the question is what sort of funding mechanism to choose the uh agents that are going to undertake the innovation um you know they have some commercial incentives we're going to just presuppose that those are not strong enough to drive innovation and so there's some public funding that's needed here so it's an optimization problem um the the principal wants to maximize expected program benefits minus expected costs subject to a fixed budget constraint so what does that you know suggest should be in the objective function well You want to sort of maximize the probability of program success which is um advanced by having more innovators and having more skilled innovators enter. Of course it depends on the intensity and the direction of their effort. Um and uh subject to that you'd rather not have wasteful expenditures that don't advance the goal. Um there's information frictions.

[03:26:30]We think a lot about asymmetric information. So adverse selection and moral hazard. But really there's another issue. There might be just symmetric uncertainty. some things might not be subject to contracts. We have known unknowns and and unknown unknowns.

[03:26:46]So, let me be a little modest in in what we do and don't do in this paper. Um, we're going to not answer the question of is public funding needed. We're just going to assume that. Um, and is the particular innovation goal that the policy maker has in mind, is that the right goal? We're going to not try to address that. Um, we're also going to take as given the the fixed budget and not talk about trying to size that. And this is hardly the final answer and and many different caveats. Um, hopefully we raise policy makers awareness of the issues. Say here's the potential catalog of of funding mechanisms. You might not have considered all of them. We view this as a first pass guide. Hopefully, we'll revise the atlas in light of uh comments and suggestions uh from this audience and others. Maybe this is just uh beginning to frame the debate, take uh further input from experts and u maybe raise some questions for further uh research.

[03:27:41]So here's the framework.

[03:27:43]Um so the it's kind of like a threepart approach. The the first part asks us to kind of look at these contractual issues and u framed around three core contractual questions and it's going to determine which of this you know welter of different possible programs is even viable to begin with. Um and those three questions are um can you articulate the the problem um sufficiently so that you can specify and verify that this is an valid attempt at this uh innovation goal. Um even better further along the contractability can you characterize the solution? Um so is this a a success um at uh you know attaining this in innovation goal and can you identify a team Xanti who would be the the best performer um either because you know of their existence or you can run a competition uh or an auction Xanti to determine that.

[03:28:43]Um so that's the the first part. the the second part um then kind of brings in more of these innovation policy considerations that are familiar to this audience things like financial constraints and basically this is we're going to get a box from the the the first part and now within that box we're going to say okay which of those are now efficient and and all of these considerations interdependence and approaches finance constraints spillovers um moral hazard and adverse selection um and then finally there might be some practical details to sort out we talked about you know is security an issue. So that's going to you know mitigate militate towards certain um certain policies and and and against others. Um you know who are the performers here? Are there any kind of legal and political constraints? Are there sort of time horizon constraints?

[03:29:29]And what is the supplier and market buyer market structure look like?

[03:29:35]Um so let's u go back to the this this this first part of you know this we're going to look at these contractability issues and that actually will allow us to take this cloud of different programs and kind of organize them in this in this sort of matrix that is you know the boxes that are darker have more uh contractability there. And as you get uh more and more uh ability to contract on things, you get a larger and larger set of potential um viable programs. Um and in a sense, as you move from the u upper left of this matrix to the lower right, you're actually just u all of those um programs in the boxes are then available to the program manager. Um so let's start at the um the upper left here.

[03:30:24]like in a sense nothing's contractable there. So you can identify the team that's in the rows and across the the columns can you identify the the nature of the problem or can you specify a solution? Um when you're all the way over to the left you can't really sign a contract on anything. So you're in this world like what's open to you? You can uh do field building. um you can provide a generalized R&D tax credit. Uh but it's hard to have sort of a dedicated contract um and any other type of of program.

[03:30:56]Um if you can start identifying the teams, then you can you can do more. So this is moving down um you're you're staying in that that first column, but you're moving down the row. Um you can if you can identify team maybe there's a a star researcher that you could try to um um award a researcher grant to or you can actually build an in-house lab through and let your scientists kind of give them free uh reign to do their research in inter mural science. Um moving back up to the nothing is contractable. If we move across that row, um, if you start to then at least to be able to say this is a valid attempt, you can start to sign contracts that allow for things like relative prizes. Even if you can't validate whether this is definitely a solution that you're looking for, the relative prizes, you're sort of contracting to within the set of things that are at least a valid attempt on the problem. You can say what's the best one. Even though we can't necessarily specify here's a threshold beyond which um we call success, you can still say relative among those that are um performing the in the problem area.

[03:32:02]This is this is potentially the best. Um if you're able to identify a team, you can start to do things like have project grants, R&D contracts, a government uh research lab, joint ventures, or maybe a coordinated like ARPA like programs.

[03:32:18]Um and as you can start to validate a solution like this might be say a a product we're looking for say a um new vaccine against uh some disease that we don't have a vaccine. Um then you can start to write contracts um that are you know conditional on that. So you can start to say you know here's here's a target and we're going to offer you an absolute prize. You can break down that absolute prize into you can sequence it into milestone payments. You can have an advanced market commitment. That's a um a policy where you set a a technical product profile. Maybe this is as I was saying for a a vaccine for a new disease that has a certain efficacy and you put aside a fund, a committed fund that will pay for say a perunit subsidy um until that that fund is fund is exhausted. Of course, for that to work, you need to be able to specify what it is that product that you're seeking. Um also a per unit subsidy like a an EV say tax credit um you know you have to specify what a an electric vehicle is and finally if if um you can start to identify the but what one thing about the advanced market commitment it's um firm agnostic and so in a sense you can't identify the team you're not sure if the Kistler or the SpaceX is going to be the best performer you have this sort of firm agnostic contract that allows some competition there if you can identify the best team then you can move to something like a procurement contract. Um or if you see the the most promising team, you can offer loan subsidies.

[03:33:50]Um so that's kind of the the first stage. And notice that um as you move down to the those boxes where there's um more contractability there. Um in a sense, those plus everything else before it are programs that are available. And then you can in a sense choose the one that then gets the innovation policy right. and maybe meet certain um you know local conditions like if um defense secrecy is is important some of these are going to work better than others.

[03:34:20]You might want to have an in-house team rather than contract out.

[03:34:25]Um the basic trade-off that you can see here is that um as you move uh from one corner of the box to the other, it just requires more and more contractability.

[03:34:34]It just might not be there in that particular economic environment. Um but as you move along you can have sort of uh tighter contracts. Um you can get stronger incentives potentially. Um so one principle would be seek the strongest incentives um subject to um you know the contract being actually allowed for in the environment. Um and in a sense the larger the set of policies it's going to allow you to accommodate idiosyncratic features of your environment more readily.

[03:35:05]Um, so, uh, we also don't just have a paper, we have a web tool and, uh, we call it this atlas of of innovation. Um, and I'll give you a URL. You can actually play around with it and and and see. Um, so here is, um, navigating through this. This should be a little video. We can see if we can, uh, get this go. So, we start off um at, you know, the our little boat here and the question that's asked. And so we're trying to phrase these in a way that will be um you know legible to to policy makers. So um how precisely defined is your goal narrowly or or broadly? Let's see if we can get this. So um we're thinking about we're going to try to get to an advanced market commitment. So we're saying it's it's narrowly defined.

[03:35:49]We know that we want this um this vaccine. Um what form of solution uh prototype? We want a product. We want a vaccine. So we hit that and we're navigating to the next point. Can you identify the team or person? Well, maybe not. We're not sure who the best performer is. So, we hit no and we drive our a little jeep over here. And then does the solution already exist and we just need adoption or is it innovation?

[03:36:13]It's a vaccine that doesn't exist. We need innovation. And that brings us to advanced market amendments. So can read more and we have um you know five or 10 pages of material with pros, cons, literature, uh cross references to other policies that might be that might be useful in a in a nice graphic there that somehow represents uh advanced marketing.

[03:36:36]Um so uh you saw that we got to this page by answering some questions. It's a a narrowly defined goal. um and we're trying to develop a product or or service that we can attach a subsidy to, it's going to be hard to identify the winning team. So we'd rather have a firm agnostic uh program rather than say a procurement contract and the product doesn't yet exist. So it's not an adoption question, it's an innovation question. Now one issue is like what were the alternatives that this ruled out? Um procurement contracts and loan subsidies, those are ruled out. Those might be nice uh programs, but you need to identify the teams to get those to work. And we might be worried that we're uh unable to do that and we'd rather those teams emerge. Um it might be where that we're well in advance of this technology actually happening and and we need that to emerge over time. Um another possibility we'd be an absolute pride that you invent this vaccine, we give you a large sum of money rather than this per unit subsidy. trouble is that that doesn't reward scale up and maybe it will um result in a product that meets the technical product profile but for reasons that were difficult to specify in the contract. Um they don't meet consumer needs and they act it doesn't get adopted. So you end up paying out for something that doesn't get used. Um R&D subsidies is another wonderful policy but it's such a diffuse reward that it doesn't hit your narrow goal. Um how are we on time?

[03:38:01]>> Three minutes.

[03:38:01]>> Three minutes. Okay. So, I'm not going to go through, you know, other things like researcher based grants, what would get you there and why that's better than other things or or loan subsidies. Um, I'll just uh go to the concluding page and and here's the um the uh what do you call that thing? The the URL um Q QR code. Thank you. Um so, this is uh atlasof of.org and the password is shaping progress.

[03:38:28]So, um, since we're this econ, it won't be launched till next month, but please, uh, go ahead and and, uh, play with it and and see what you think. Um, but what are we doing in this in this paper? Back to the paper away from the the tool. Um, one thing here is where we're we've got this really hard question trying to make some sense of this uh 15 or or 20 different policies and we we thought that if you lead with contractability issues that really simplifies the problem down to a much smaller set of uh feasible viable programs. Uh so easy to understand goes a long way to narrowing narrowing down mechanisms and turns out they're of actually first order importance. Uh the three core contractual questions are can you articulate a problem? Can you characterize a solution and can you identify the team? Um and then that can be part of this uh three-part framework for selecting the innovation funding mechanism. You start with these contractability questions. You move to sort of standard innovation policy considerations. Then integrate any practical considerations that are relevant to your particular area. And again, we're I'm shilling for this atlas of innovation um this free web tool u that will be launched next month. Um it applies this framework and sort of easy to understand understand questions uh to navigate the policies. The landing pages are pretty extensive actually put a lot of time in those vetted by experts, definitions, examples, pros and cons. um it actually has the users check their answers and to cross reference other mechanisms and it's going to be a living document that I would love your input on. So thank you.

[03:40:12]>> Okay, again thank thank you all for this great uh integration. I guess one quick question um would be about actually this is not a quick question. Uh so I'm going to pause on that because maybe I'll let Chaian ask a question. I'll come back to mine.

[03:40:26]>> Hi. Um I'm Chaden Bu from the uh TIP directorate technology innovation and partnerships at NSF. So thanks everyone for all the advice today. Um the one suggestion I have and this is something we are facing right now as we try to design these programs is there are costs involved in every one of these options.

[03:40:47]uh in terms of the operational costs of uh the amount of work that the program director or whoever has to do. There are contracts to be signed. There are legal people involved. There are uh transaction authorities that and our folks are very good at underestimating all of that. So the roads that your vehicle is taking, you know, maybe those roads have to be rough and have diversions and so on. So >> thank you. Um We'll add that in as consideration.

[03:41:18]Thank you very much.

[03:41:18]>> Okay. Well, let me let me let me Oh, you want to respond as well?

[03:41:21]>> I was just going to say I think that's one thing that we're hoping also to get out of having this tool out in the world is hearing from policy makers. Oh, like I actually had real difficulties in executing a prize or doing a grand challenge and then being able to adjust or go back and talk to other um folks that can give authorities or smooth things over.

[03:41:37]>> We don't have infinite operating >> that too.

[03:41:41]>> Oh, can I just add on the budget point?

[03:41:44]uh we we we have another tool that cla colleagues have developed which doesn't deal with operational costs but deals with the how much how big these mechanisms have to be. So here we've taken the budget as fixed. In that exercise we don't we figure out how big the budget should be and that's something you should also check out on our website.

[03:42:02]>> Sue.

[03:42:04]>> Yeah. This is so fascinating and so great. Um and maybe maybe it's your next challenge. I mean this is kind of one project at a time and sort of thinking what if you want to build an overall capability and is there a role for like the relational contracts literature? we think about building and maybe that's what the national labs are is we think about we know we're going to have problems of a certain class um and we're just gonna you know in a cosian way we're going to make our solution rather than buy them. I I don't know if any of those thoughts how do you think about those things?

[03:42:36]>> So are you saying um if you say you wanted to combine these say over the life cycle of an innovation goal you know what would that life cycle look like and what combinations of these should be used.

[03:42:48]>> Yeah. So, let's say you want to build a nuclear bomb, right? Or you want to send somebody to the moon. Um, you know, it's a series of projects and steps and grants or maybe it's not even a clear mission. It's we want to have a green transition.

[03:43:07]Um, I don't know it but but just sort of thinking about these more than one at a time.

[03:43:16]>> Yeah. So first of all, this is a great uh teaching tool. So I hope you give us license to use it in the classroom to illustrate because this is actually something that I teach how to use prizes to incentivize innovation when you think as an organization actually you know Google had the lunar prize $30 million if you could get to the moon and uh uh and Netflix had this Netflix prize that also was a was a big hit. So this is used by governments as well as by private firms. So one of the dimensions that I always emphasize in class we were discussing this yesterday at dinner is the rapidness of the problem. So in the in terms of the ENK model the Kuffman uh the Kafman model. So is the problem single single peaked or is it kind of very difficult to identify what direction of improvement you have to push where to invest in what direction to invest because really the the landscape is very rocked and so it's it's difficult to really so in that case so where does that dimension enter?

[03:44:10]feels like it should enter in the contractability. If it's single peaked, it's easy to contract. You know what steps you should take to try to improve.

[03:44:18]Uh whereas it's very if it's very rugged, then it's very difficult to contract. I was wondering if that's kind of how that element matches.

[03:44:25]>> Would you would you say that's the middle uh section, you know, can you that that middle section can you specify the the nature of the the problem?

[03:44:34]Actually, in a sense, I think if you can, that's kind of your single peak. If if you can't that's almost like you called it the rocky >> the ruggedness like ruggedness. Yes.

[03:44:44]Exactly. So that would be if you can't that's more of a rugged solution but >> I'll talk maybe talk offline about that's a different way to conceive of it.

[03:44:55]>> Hi Sean Klein National Institutes of Health. This is really great. I mean I think a lot of our program managers are deepmemes but not in this stuff. So having a resource that they can tap to that has all the information in one place that they can parse through quickly is really valuable as they're thinking about the best way to fund science. One thing to think about is some of the things you're talking about are split across sometimes two, three, four different teams. Acquisitions is a completely different workflow than grants in our organization. So if you're going to be talking to our program managers about cutting a contract, some of them don't even haven't cut a contract, can't manage a contract, and don't even know who to start with. So thinking about some of those aspects I think would be really important. Another aspect is like NIH is not generally in the business of cutting tax subsidies. I think we have maybe one. Um so like some of these are going to be lower barriers than others, which I think is to the the earlier point. I mean, this might be a guide more to the upper level managers saying, "How much should we allocate to the grant makers versus, you know, maybe we'll integrate some pull funding into NIH?" Uh, >> yeah. No, and I totally get that, too.

[03:46:03]They're not going to read five to 10 pages of information for the most part.

[03:46:07]So, I think getting in the hands of like the lower level people could be really important for them to bring those ideas to the leadership level people.

[03:46:17]>> Okay. So, I thank you for all those questions. I was wondering about appropri more of a property rights kind of orientation. So this is a contracts sort of structure which is very very good. Um but like if I were to sort of explain basic research to sort of applied research to somebody and say like the grant system versus like the patent system I would be thinking more about you know grants in science because we believe that the values and the spillovers and there's really nothing to appropriate. It's not ready for a market. going back to innovation while in the you know private sector I could use a decentralized mechanism with its sort of costs of giving monopoly power uh through patents and so I'm wondering how that kind of perspect classic perspective sort of intersects with this or or fits into your categories how easily it fits in maybe it's already subsumed in your categories or maybe this is a somewhat different perspective that would suggest a richer richer dimension but uh what what's your what's your instinct on that >> um I guess my instinct is to in a sense punt um you you know, we're we're sort of saying this is um where, you know, we need public funding here, so the the commercial market's going to be pretty weak. But I guess you would say, well, if you could have add that in as a dimension, you know, now um you know, how would that um change our our scheme?

[03:47:30]So, we're not really patent is not one of the the policies. We're just thinking about spending money and uh how should you uh organize that spending. So um yeah I think uh you know we'll have to think about if there's more of a private market along with the the uh this public market. Can I add? So we have a section towards the end of the paper that talks about other considerations and I think appropri is there and in so far as you're worried about there being larger probabilities I think somebody in an earlier session mentioned the invention of calculus which is going to have very large spillovers which you're not going to capture that's going to push you towards the grants section and the early stage research grants versus sort of procurement contracts and AMC's and partly that's because where there's lots of large spillovers the firm that delivers the breakthrough through may well not be the firm that commercializes and benefits for the technology. So it's a problem for the later stage mechanisms.

[03:48:27]>> Yeah. And I but I think this also comes back to Sue's very interesting observation, right? Which is that there's a there's a system and there's a bunch of interacting agents and actors who you know which one way of saying that is spillovers are really important between players if you went for a green transition or a nuclear bomb or whatever you're doing. And so thinking about the policy design in light of that is also important at an architectural level.

[03:48:50]And I guess I'll add one more thing here that it can be much cheaper um generally for a policy maker with a zero budget to allocate property rights say like um priority review vouchers for example um over allocating a budget necessarily.

[03:49:03]And I think there's also um another thing that we don't touch which is just like regulatory reform areas where there's potential room to um for a policy maker to be thinking about actually is this market failure it does exist but maybe it exists for a different policy reason.

[03:49:18]>> Okay. Okay. Well, I said that wasn't going to be a short question. That's why I didn't go first. Um um well, more more food for thought. Um this was the uh I guess we're going to move on to the real food now uh uh with lunch. So, I I will stop talking. Um let me just thank everyone, all of our presenters for these great papers and contributions, the audience for the great questions. Um >> yeah, and I hope I hope your conversations can continue uh and enjoy the enjoy uh some further networking as well. Okay, thanks so much.

[03:49:48]We got