Recursive Self-Improvement and the Future of AI

[00:00:00] The fundamental idea here is to build AI that improves itself. And I think we're getting there. >> For the first time, we could swap the optimizer as the optimiz. And this leads to a reinvestment of cognitive improvement into itself. >> You can think of RSI at many different levels of the stack, prompts, harness, weights, and you can go all the way to basically chips. Our goal here is to optimize all the levels of the stack. Our approach to RSI is that we have a multi- aent system working together that's been recursively improved. >> I'm so impressed by your belief that like this idea of a model improving itself, >> but it's not a belief. >> It is a dramatic statement, which is I think it's the first time I've heard you say it. >> Because we have been so thorough, we have not released things until we're super sure it's correct. >> Do you envision a system where the amount of energy that goes into it somehow decreases over time relative to the improvement? We can't share too much, but one of the very important things that we've tried to do. >> Welcome both of you uh for joining. [00:01:00] We're going to be talking about everybody's favorite topic, which is RSI, recursive self-improvement. Maybe to start, it'd be great to just introduce yourselves and your backgrounds. Then we'll talk a little bit about what you're working on now, and then we'll dig into the the the content. >> Yeah, I I I think of [clears throat] myself as like um a physicist of some kind. I I studied that. Then I did the PhD in reinforcement learning at the MIT Media Lab. I really like building things and I'm very excited to be building um a cool startup make a maker with Owen and Grace. My name is Owen. Um I was uh first studying math and then one day I realized that all mathematicians are proving theorems and I was taking a class called formal language and logic and they were showing how you can build a theorem prover and I was like oh that would be cool I can just build a machine that prove theorems for me that's how I got into AI >> amazing >> and then for my PhD I worked on continue learning and metal learning then worked [00:02:00] for a few years at deep mind at a research scientist now I'm working on RSI I see a lot of pitches come in and a lot of them use the term RSI and I'll be like, "Oh, so what do you do?" And they'll say, "We use AI to create AI. That sounds very interesting. So what do you do?" We use AI to create GPU kernels. >> Which that doesn't to me kind of match the definition of what I would consider to be RSI. So maybe can we start by you defining what you think or what you would include in RSI >> definitionally? >> Yeah, I think RSI is definitely an overloaded term today. Um many people when they say RSI they mean different things. Um so I think the important thing when you are differentiating them is to ask the question what is the optimizer and what is the optimiz like when you're self-improving who is improving right so um >> there there are people who saying okay we're doing RSI we have human using AI to develop better AI >> but that to us is just um autocatalysis >> in a sense that the improver is still [00:03:00] human with um tools even though these tools are getting better and better but the idea and the improvement come from human brains, >> right? And and just I don't mean to interrupt, but to be clear, most technical epochs have had autoc catalytic effects, right? Like there's no way you can create a computer chip without a computer, right? You could argue that the internet development was sped up by the fact that people that weren't geographically located could coordinate, right? I mean, the steam engine, you could build better steam engines because you could get coal faster, etc. So, this notion of like using the thing to build the thing faster is not new. And so you're saying something different. >> Yes. I think for us um in in the history right over civilization we have this autoc catalysis you know computer using computers to build better computers but the the idea for this change for this improvement come from human brain but that human brain was never the improve never the optimiz right whereas when now for the first time we could swap the [00:04:01] optimizer as the optimiz so the the brain of the agents which is LM the model can be the thing that you're optimizing and this leads to a compound this reinvestment of cognitive improvement into itself. >> But there has to be a reason that you decided to take on this hard problem as opposed to what anybody else would do which is like chew off pieces of the problem. >> Yeah. I mean I guess to answer before I answer your more important question here. The reason I did this is I think it's a hard problem and it's fun working on the hardest problem and I think >> but do you have an intuition that this is is the actual right answer as opposed to like something that's more pragmatic? >> Yeah, I think a pragmatic thing would be just from a control theory perspective. If you have a gap in the system, you can't have a closed loop and autocatalytic has no closed loop, right? So whatever you improve, there's still like a gap. humans have to come in. You have to pour in other resources to make it work. But if you have a closed loop, [00:05:01] it you still have to pour in resources, but the the marginal gains ideally get either constant or larger over time. And that's that's the fundamental difference. What's really different here can be explained pretty easily. >> So if you think of capability, >> yeah, >> what we're trying to do here is we're trying to improve the rate of improvement of capability, right? So like the first derivative of capability with respect to time is the thing that everybody is trying to improve. Yeah. Right. And now you can put that to be equal driven by two things. The nominator and the denominator. The nom nominator would be performance. Yeah. >> Which is what everybody is trying to improve. And then the denominator is some kind of cost. Let's say capital, right? Or difficulty. And this one is pretty is pretty set because the higher the capability, the higher the difficulty of the next step. Right now the numerator is the interesting part for things like steam engines and [00:06:01] whatnot that was fixed. >> Yeah. >> Right. So then you end up with a very simple differential equation where you basically plateau. >> Yeah. >> Right. Now if you have a system where the improve the things being improved is also the thing that's doing the improvement >> you how you now have a new way to think about it and the the simplest case would be that the numerator is also correlated with capability which makes sense right because the better you are the better you are at also improving >> and if you do that the math cancels out and you basically get constant improvement over time that's that could one level of of RSI, right? And then there's one more case which is >> now that the the numerator could be correlated with a higher power of capability. Yeah. >> And then you get exponential takeoff. >> Yeah. Can we um dig in a little bit more between RSIs having closed the loop and actual convergence properties? Like for some reason when people see a a graph [00:07:01] where there's like a a circle with an arrow to another circle then and it goes back and they look at that closed control system then they say take off [laughter] >> right which >> I mean if this is a nonlinear control system >> it's very difficult to know what the hell happens by the way most people don't seem to understand this distinction >> because we very quickly go to like take off and fume and all that stuff when we talk about these things. Well, as a network guy, you know that it depends on the exponent of, you know, your your function. That's that's self-referencing. >> And it's what we found is that it's almost not useful to speculate on what we think could happen because this is really an empirical problem. And the real way to know what's happening is to test things. and testing things is really really hard like reward hacking is one of you know many problems that we have to deal with but the the question you're asking is [00:08:01] very empirical and for some >> interesting so but so I'm a networker that's how I would have answered >> I would have thought math people would have been like we've got this we've got this differential equation and I took out my pencil and I wrote down the answer >> I mean it is possible to write down the differential equation but you still have to empirically diff measure the the variable the exponent. >> So there's no analytical solution. >> I mean honestly we don't even have a learning theory of deep learning. We don't know like fundamentally why it works, >> right? And and then now you're like you know five six layers on top of deep learning and so we don't have a good theory. I think the closest things we have are scaling laws and these are empirical laws. I think one misunderstanding people have about like our approach to RSI is that they think the entire system has to be fully automatic and digital. But actually if you look at our mathematical foundation we have a multi- aent system not just one digital agent. And in in this multi- [00:09:02] aent system human researchers and engineers are actually special case of these agents and it's this hybrid system of multiple agents working together that's been recursively improved. >> I see. And we don't want to leave out any component to not optimize them because like if it's possible to optimize in the model why wouldn't we do that because you will have more gains if you do include those. Anything that you don't optimize will later become your bot. >> So one observation that um uh uh we have we have made here is um there are claims of uh speed is improving which is actually the case right? Right? So if you look at like the model release time, it is reducing. And so on one hand you can be like okay so clearly we're in some sort of a you know rapidly improving phase. But if you divide by the actual amount of capital raised by these companies that's no longer the case, right? And so you're like well maybe they just raise a whole bunch of money and then like AI allows you to apply that money. So, how would you, this is [00:10:04] not for yourself, but how would you encourage people that are either evaluating claims about this or even working on them to think about not fooling yourself by throwing more money at the problem like you're actually showing that you're making an improvement here that's meaningful with the denominator. [snorts] I think for us it's it's pretty straightforward. It's because if you get improvement and they just come at more and more resources being poured in, it's not RSI. You have to be able to back your claim of how much improvement you're getting. But do include like all the manh hours and experimentation that went into that because I I feel like at some level like that like it's like like they'll talk about like the actual flops that go in and money that go in but the reality is this is like the 23rd experiment and it's actually a human burning a lot of mental capital to get to that final configuration like do you actually >> include that in the denominator? >> I we do and I think it's it's good company discipline to do that. [00:11:05] >> Yeah. And we've tried again to very intentionally from the beginning have a culture of tracking where things go cost time all that kind of stuff. I share a funny thing internally we have this thing people talk about leaderboards you know which is like score >> you know what I'm about to say we have something called a bleederboard >> where we keep track of the cost of of experiments of everybody who's running experiments. >> That's awesome. you know and and everybody it's the comp everybody any day can look at like who's who's high up on the on the bleederboard. I can also give an example from AI. Um I don't know if you know this nano GPT benchmark they are running. It's basically asking you to retrain GPD2 which when it just came out is a very expensive model to train and takes weeks I think on multiple GPUs >> and right now people can train on single GPU within hours and that number has [00:12:06] been shrinking constantly. >> Yeah. >> Yeah. But that's also totally separate, right? Like that's just like whatever Moore's law like >> No, no, it's the same GPU. It's the same hardware >> using the same hardware and then you improve the techniques of pre-training. >> Yeah. >> Oh, interesting. >> Yeah. What has improved is like algorithmic >> techniques, data techniques, kernels, all of that kind of stuff. Do >> you know how much it cost to train GPT2? >> I think it's a few hundred. >> $100. I just Oh wow. So the same same GPU >> same GPU same data >> and this is just algorithmic and do those come from humans or do those come from >> agents a lot of them are auto research results >> but that but that's definitely the denominator that's that's not the numerator like what you want to see is like you can do that with GPD5 which we're not seeing [laughter] right I mean like the cost to these things is going up dramatically >> I think one of the most closely guarded secrets in the industry is exactly this thing which is how much compute you can get out of this. >> Yeah. >> Because in some sense, even if all you [00:13:07] were improving was your denominator, you could get faster training because now you can get more rollouts. >> Right. Yeah. Of course. >> Right. And that means now you can sample that what what you were describing like the manifold of the space of of language more extensively and then you can get better trajectories which means you can get smarter models when you use that to train. >> For sure improving the denominator is good. >> Yeah. >> For sure. Improving the numerator is good. Yeah, >> I've yet to see which is great that you have, you know, glimmers, uh, where you can remove both the numerator and the denominator. I mean, that's a pretty meaningful statement, which is awesome. What we found is that a lot of the papers that claim to do RSI, when we test it in our internal infra, we find that they don't work. And a lot of it is because they're cool ideas that humans try to introspect and say, "Oh, you know, this is what I think should happen." >> They're complicated. this sound cool, they get you a good paper, but then when we test it at scale, you know, on benchmarks, on real economic problems, [00:14:08] they don't scale. And what we found is that this the simpler thing that closes the loop is the thing that scales. I would say one thing that we do that's probably unique is that we always start with a very grounded mathematical framework that that basically lays out the map of the whole space. >> Yeah. And the reason is because this space is so large that if you don't do that and you don't have this very principle way of doing it, you get lost very quickly in this complexity. >> And the other reason is that often when you have this really complex system, there's two extremes of how you can build it. You can take an approach where you basically build like 200 different systems >> and you hope that like this this this you know what is it the Rube Goldberg machine will kind of work out. >> Yes. >> Or you take the other way and you you kind of basically spend some time thinking about these 200 things and maybe you realize you know they're actually just 12 um abstractions that [00:15:09] generalize to these 200 things. And so what we have tried um to to create a culture of let's really think from first principles. Let's you know look at the math and then we realize what are the assumptions we make to be practical cuz you have to do both and then we see what works and and when you do that you then can see that oh you know there's like multiple papers or ideas are actually you know subclasses of the same general framework. >> Yeah. And if you do that well, you can scale. And I think um that's that's one thing that I see that's missing and that's hard to do and and takes a very unusual group of people to to really like make that work. >> I think Richard Feman said once that there are two kind of sciences there. There's STEM collections and then there are physics. >> So without a unified math framework, all these research are sort of just stem collections. You could like a lot of this paper we're reading they're just [00:16:10] ideas inspired by some human behavior and then they try to mimic that and say oh look we have RSI but then if you don't put into a unified system you don't know if the overall system is actually improving or you're just like locally applying an idea that that's getting better. So what we have is that we're putting all these components that you could possibly recursively improve into a unified framework and look at it from a holistic view so that improving one part um does not negatively impact the other part. >> Right. >> I see. I see. >> So so for example there are people who are doing recursive improvement just on harness and then there are people who are only doing on wids but weights and harnesses are not isolated component. They have impact on each other. You can think of RSI at many different levels of the stack, right? You can think of um prompts, um harness, weights, and you can go all the way to basically chips, right? >> And our our goal here is to optimize all [00:17:11] the levels of the stack, right? And we started with harness. We we've tried to be always very capital efficient because we care so much about the denominator. And then we started doing weights, weight updates. And what we found is that you get more gains, but you still have to do hardness stuff, right? And you can keep going down the stack. And and my guess is that you probably will need to do a lot of the kind of co-optimization of many different parts of the system to make it work. Can you talk a little bit about the problem domain that you're going after just to prove that this works? Um, we are trying very hard to pick areas of the AI space that are as close as possible to AI itself. Again, we try to be recursive. >> And if you think about it, there's there's not that many problems that are super close to AI and gives you improvement on both the numerator and denominator. [00:18:11] >> And one of those is the inference problem, >> right? because the more you can spend on um sorry the more you can get improvements and and efficiency out of inference the the more everything kind of unlocks right >> and it's not just um for technical reasons but also the most promise of both the numerator and the denominator being improved right >> it kind of breaks the mind right so like just just for everybody that's listening to this there's two improvements going on here one of them is the ability of the model to make inference faster. And then the second one is now that the inference is faster, it can do that faster, right? >> And again, that does not necessarily mean takeoff that can still, you know, as at some point. >> Yeah. >> But it's just wild to think of this kind of compounding impact. >> Yeah. Yeah. We're a startup. We're not a research lab. And one of the reasons we chose it is because everybody in the world right now wants to serve their own [00:19:12] model. >> Yeah. and and everybody wants to use the latest for example open weight model Kim 3 or whatever just coming out and they want to be able to serve it as efficiently as possible right so it's it's an economically extremely important problem right now >> just for the sake of discussion if if I if I wanted to make the inference for Kimmy K3 faster would I use Kimmy K3 in a recursive way or would I use a separate model and system to do that for Kim K3 >> to improve the serving stack of Kim 3. You mean yeah in our >> well our system um is not just using one LLM you have many different LLMs that are doing many different things. >> Oh so you're different the inference side for all of them. Um I mean we again it's it's we pick whichever is is the most effective and because it it's being used the most time but the interesting [00:20:12] thing is the choice of the of which LLM to use for which thing is also not always up to us. We let the model choose to be honest today it's not really us who are deciding what to optimize. It's our agent because it's recursively self-improvement and what we're trying to do is not to limit itself to only one component. So it chooses whether to >> it. Yeah, we try to be like as hands off as possible. Let it do its thing. >> Do you have like a single master prompt? Is it like improve yourself? >> Uh but even that master prompt that has been improving. [laughter] >> Really? It improves a master prompt. >> Yeah. And I mean and it's not one RSI system because you can think of it as there's the thing that that does the improvement but then you also have to have the thing that knows what to improve and that's both a dynamic problems and they are very co right. How general do you think this ends up being? Like right now [00:21:13] you're focused on speeding up inference in a specific domain. Do you think that whatever approaches you're coming up with here generalize or do you think there'll be a different one per target solution space? >> Mathematically we have defined it so that it's one system that generalizes to everything. I think the the one part of the system >> and by everything mean any problem domain that you can point AI to. >> Correct. But then there are physical limits for example and and one big limits of this is is the human element. I think the way that a lot of AI companies are thinking is you know let let us use AI to to replace humans. But what we found is that there's there's so much that if you have good researchers and good engineers that they can bring to the table that's going to be very hard to to replace by agents. And so that's why from the very beginning we've tried to think of our basically entire [00:22:14] company as a multi- aent system where humans are a kind of agent that's cohabitating with other AI agents. >> Can we at least reduce bounds? So for example, do you envision do you envision a system where the amount of energy that goes into it somehow decreases over time relative to the improvement? We can't share too much but one of the very important things that we've tried to do in our company is not just work on the numerator which is what I was describing earlier which is your performance but to work on the denominator which is how much you can get out of the same amount of physics money whatever you want to call it >> right but do you think it's possible to have a system where the numerator continues to increase at the same way with the denominator being fixed or decreasing I I it's not just possible. I think it's happening >> really. >> Yeah. >> So it takes less energy to get better results. >> Yeah. Yeah. Over time. [00:23:14] >> Do you think there's a there's a universe in which like that continues without asmmptoing? Cuz if that was the case then you actually have takeoff. >> I mean >> but it just seems to disobey information theory and the laws of physics. >> Yeah. Yeah. I mean I I think for at some point you're going to reach plank constant you know plank length plank energy plank time right >> so it it will asmtote at some point >> but I think >> oh I see and so you think that asmtote is basically like the computation of the universe >> at least yeah I mean you know how many how many particles are there 10 to the 80 like we know speed of light you know we know we know what it is you multiply those two and you basically get an upper bound of computation and and so given this fixed amount of computation you can do every We know we have a limit. >> I'm so impressed by your your belief that like this idea of a model improving itself, >> but it's not a belief. >> No, you got to tackle that. That's amazing. I mean, >> also it's a question of what metric you use to to view the progress, right? So, if you use um I don't know benchmark [00:24:15] accuracy, it's upward bounded by 100%. You're never going to see exponential improvement on that because eventually bottleneck 100%. So, so for a long time we had this question, how do we even know if something has ticked off right if we don't have the right metric? >> It's not a belief that this is happening. I think we're already seeing glimmers of that happening. >> But it is a dramatic statement which is I think it's the first time I've heard you say it. Wait, is this like a is this been kind of, you know, a continuous result or is it relatively recent that you're starting to see glimmers of this? No, we we've been seeing like in in these results kind of grow and grow and grow over the last few months. >> That's amazing. >> How do you have confidence in the results given you know like it's not working yet and you don't know? >> No, it's not belief. It's it's empirical measurement. But but also as a as a company there's a there's a very funny thing that that is happening which is because we have been so thorough, we have not released things until we're super sure it's correct, right? And [00:25:15] every few months, you know, there's like an announcement of something RSI related, we look at it, we're like, we should have released our results or something. And then we look at it, we try to replicate it, and we realize, oh, it's noise, you know, it's cherry picking. And so, it's very very hard to to know in this in this sea of, you know, AI, I don't know what you want to call it, like everybody's trying to do AI, RSI stuff to know what's legit or not. The the technical answer is you have to be really really good at measuring things. So for example, we have one of our engineers that we we we named him the viseroy of variance. That's that's a real thing. Like his job his job is to go and like destroy the results we have and find what's wrong. And we even have little like slack emojis that we have. There's one where like there's like red laser stuff coming out of his eyes when like he finds [laughter] that there's bad variance in stuff. One of the things that we found [00:26:15] is that you know when when you're dealing with a new benchmark or a new problem of economic value that you want to apply this to. So far we have not found a replacement for what we call quality time. literally running hundreds of experiments looking at it, developing an intuition, trying to figure out what is wrong, what is not wrong, what what is what is just variance, what is not, what is this weird thing that actually shows that there's, you know, some kind of reward hacking like two levels down the agent that built the agent that build the agent >> did hack something and now you have this weird results, >> right? >> It's not easy. So when I have dealt with you know nonlinear control systems there's a bunch of in networking there's a bunch of degenerate behavior you have to deal with it's not it's not like just convergence and divergence you have things like oscillations >> and this is a very multi-dimensional problem so do you have to deal with all of that stuff like potential oscillations and >> absolutely >> meaning like it'll it'll get good and then it'll get bad it'll get good >> absolutely and then and then on top of [00:27:16] that you have noise are fundamentally super stochastic >> so what we find is that specific agent on a specific benchmark. You know, there's levels of noise that's sometimes unpredictable and some of them are extremely like weird like you even have some weird noise that comes from like this GPU and this other GPU. They're both, you know, B200, but they have slightly different behaviors like on the physics and you can see some of that being replicated at the high level. So that's why it has to be an empirical thing. >> So from that you could already see that intelligence is not a single scalar. It's more like a multi-dimension vector and there will be domains and field where this RSI will give you take off and there will be dimensions where you might plateau or you will have relatively slow improvement and I think the the the defining factor is what is the slowest component in it. So you're [00:28:18] an optimization um person. You know that there's this amdos law where you know like the component the the speed up you can get by optimizing a component is >> you just take it you just take it to zero. Yeah. Nothing else dominates. >> Exactly. So so the digital part we could optimize and accelerate. You could get like a thousand times faster or a million times faster. But if it's only 20% of your orbital time you're only going to get 20% speed up. Right? So, so for for domains where you still need to move atoms that interact with physical world that's really slow um you're still going to be bottlenecked by that. So let's say every evol every evaluation you need to do let's say drug discovery you still need to do wet lab experiments and then clinical trials and that take months then every new idea have to go through a months or multiple months uh evaluation. So those field probably won't be accelerated as fast as you know coding or mathematics where you have very fast and cheap evaluations. >> What I've I've yet to see is anybody put [00:29:19] a balance on that >> like to say in order for me to you know improve this by X to solve this problem. You know I need this much net new energy to do it right and if you had that you could actually predict like if I'm like listen like how big of a warehouse do I need to solve like a drug discovery problem or how many different robots I would need to to solve. So, do you think that we're getting any closer to actually having bounds? Because I think this is such a critical question for all of us to answer. >> One of one of the internal jokes we have in the company is that the the secret to RSI is somewhere in a garden hose company in Wisconsin. And and the joke here is that uh if Einstein hadn't discovered relativity. >> Yeah. >> He um would not have these these laws that tell you how speed and you know contraction of time works. >> Yeah. >> And if some unfortunate engineer had to design the GPS system, they would have had to find up find like a correction of [00:30:21] time and they probably would have said like no no it's clearly not the fabric of spaceime that's contracting, [laughter] right? But I don't know why but this is the formula I've devised empirically >> some [clears throat] you know like some fudge >> exactly right and then and then people later would have been like holy this is this is like space time contracting right and and we have this theory that you know like RSI right now is is being thought of let's let's solve all these like super hard kind of um fashionable problem but what we think is that the complexity of the world is such that the the real challenges probably exist in spaces that are not being um applied to right and so that's the joke that there is some workflow somewhere in a garden host company in Wisconsin that >> that if we solve it we get we get you know >> I love that >> I just want to quickly summarize so you're going after RSI in in your case this is not auto catalytic effects this [00:31:22] is actual RSI where the system improves the full system and for you it's an actual system which includes the harness and the weights. Um, if you think about the problem is you've got a numerator and a denominator. The numerator is like how good the system gets and the denominator is how much um energy goes into it. You believe you're seeing glimmers of the sign where both improve at the same time using this type of approach. Is that correct? >> Y >> when can we uh uh expect to hear more from you? >> Um very soon. >> Amazing. Yeah. Thanks so much both of you for joining. >> Yeah. Thank you.