Robots That Predict the Future: Dyna-2 & Physical AI — Jason Ma, Dyna Robotics

[00:00:00] Yeah. So if we want to learn a new task uh you know we basically do a little bit of robot data collection then the model these >> what is a little bit like >> Yeah. Like couple hours. >> Couple hours. >> Yeah. Where in some of the task [music] like just tens of minutes, right? >> Yeah. And then >> the chopping one was 45 minutes. Yeah. >> The open the fridge was on and that's an interview. That's like an onboarding [music] task for one of your engineers. Our robots today fold thousands of napkins for many many [music] different locations, many different gent locations in the United States. So that's actually our biggest scaled customer, right? So I want to make a distinction here. Uh today there's a lot of robotics companies talking about deployments, talking about pilots, but what's really hard is not putting your robot in one location or doing a pilot. What's really hard is that the customers believe or validate that your robot creates enough value that they want to deploy your robots across many of their sites. So that's [music] what's happened for us with Dintyong which to the best of my [00:01:00] knowledge hasn't really happened for other [music] companies doing similar like general purpose robots. So at Dentyon our robots fold thousands of napkins per day. >> Before we get into today's episode I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do it, I promise you we'll never stop working to make the show even better. Now, let's get into it. [00:02:00] All right, we're here on site with Jason [music] Ma, uh, co-founder, chief scientist of Dino Robotics. Thanks for having us in your >> Of course. Thank you for coming. >> Yeah, >> it's been a blast. So, we got to see, you'll see a lot of B-roll, but we got to see uh drinks being pulled, a lot of evils, robots with knives for the first time. Uh, so under the hood of all this, you have a world action model. What is that? >> Yeah, so last month we released our world action model called Dina 2. So, it's the first world action model trained on more than 1 million hours of data. So going back to your question first, world action model is a model that's able to predict the future video, the future state of the world. So hence the name world model. But in addition, you could also use that future generation to also produce actions robots can execute in the physical world to control the robot. So why do we do that? It's a very powerful architecture because if you think about how humans act in the physical world in our brain, we are implicitly thinking about how the world will evolve based on our actions. [00:03:01] So now we have a model architecture that's able to do that to guide robot actions. So what we find is that by doing so the model is much more dextrous, much more robust and much more capable of generalizing to environments to scenarios is never seen before. >> Yeah. >> Uh I want to prompt this question a bit. I think a lot of people hear world model and they think you're outputting video or world or 3D state. Um you're also doing action. I think most people don't understand what models are under the hood of robotics. So what is input output? What's the training data? What is the architecture? Give us all the details. >> Yeah. Yeah. So uh for inputs the most important input is video, right? You know, you can think about videos that the robot is capturing with its cameras, right? So video or images of the, you know, past many seconds. That's the input. But for robots, what's really interesting is that robot also has a lot of actuator, a lot of joint encoders on itself. So the robot also knows, you [00:04:01] know, the angles of its joints. So those can also be inputs to these world model. That's something that's very different from just a pure video model that's only looking at videos, right? So that's the input. And additionally for robots you want to be able to talk to it to instruct it to do different tasks right. So you can also feed it like language instruction. So those are what I consider the basic set of inputs that go into any robotics models including world action models. And once we have that the output at the uh basic level it needs to output actions the robots can execute on the hardware. But in the case of world action model, what's really interesting is that in addition to the actions, you can also output future video, right? So basically you can almost output the future video or some representation of it first and then use that to guide robot actions. So the way I think about is like this is kind of analogous to you know chain of thought you know basically the robots the models are reasoning in the visual the physics space and then outputting actions. So that's the key [00:05:02] difference between world action models outputs versus other kind of models, robot models outputs. >> Um you put out a good blog post around your model. I think a lot of people understand language models or transformers next token predictors. In this case you shared quite a bit. There's like diffusion flow matching. You're predicting next world state plus action. Um one thing you say is a million hours. How do you determine the architecture? Like what are the choices there? What is the architecture? I feel like in language models there's a lot of take something off the shelf, take trillions of tokens, do prediction SFT, RL and you have a model. What about this? How do you go from nothing to your model? Is there >> vanilla architecture off the shelf? Do you determine all this and what's what is the current state? >> Yeah, I think yeah, so the mental framework to think about how do we build models for robotics is that there are essentially a couple things to look at. one's data and the second is like how do we make a model learn as much as [00:06:02] possible from that data. So let's first talk about the model actually right. So the blog post we released is our D2 word action model. So under the hood it's a diffusion transformer. So transformer is essentially like the language model you know uh uh sequence sequence match decoder only. Yeah. But what's really interesting is that in visual domains right if you think about generating images generating videos and in this case robots looking at images to generate actions people have found the diffusion kind of architecture to work really well. So what is a diffusion model? A diffusion model is a model that takes some random noise and through iterative computation can go from noise to something that's really highdimensional. something really highly like defined and have very fine grain feature. If you think about generating images, right, you want sharp edges, sharp corners, color contrast. And for robotics, even though you're not generating image, you're generating actions that needs to be very precise, [00:07:02] very like smooth, right? So it has kind of the same kind of uh consideration of the sidora as generating image. So that's why from first principle you know the whole field not just us have been using this kind of diffusion transformer architecture to train robotics foundation models you need the transformer aspect because transformer is the thing that can just ingest the most amount of data can learn then diffusion helps you to refine the outcome to get to the smooth precise fine grain motion you need to actually execute on the robot. So that's the architecture that goes behind a lot of our recent breakthroughs. >> And we and we've covered flow matching before on the podcast. It is mostly an efficiency speed up. Like let's say you normally would take 40 to 100 diffusion steps, you can reduce it to four. >> Yeah, that's right. So with a uh diffusion model, one thing that's really hard in practice is that you need to tune the scheduleuler like like by how much you're tuning or converting the initial noise input into the final output. So flow matching is kind of like [00:08:03] a special case of diffusion that makes the transformation much simpler. In practice, you don't have to tune as much and it works just as well. That is why I think about two to three years ago most generative modeling for image and video has shifted from a diffusion model to a full matching model. But the concept is the same and full matching is just a special simpler to tune variant of diffusion. >> Yeah, it actually makes me think that so flow matching obviously has trade-offs in quality. >> Yeah. But actually, it's very suited for action diffusion and and and robotics because you get to re-evaluate in real time. >> Whereas in in images, you're it's basically one shot. >> That's right. >> Which is pretty fun. We got to mess with the laundry folding robot like you know throw a random towel in. Um the models adapt, right? >> Yeah. >> I think one key thing about your models is that they're >> they're made to be generalist. So one foundation model and you adapt them a lot. Uh how does that work? Basically in 45 minutes of mapped data you can [00:09:03] generalize towards we saw an example of um pulling drinks out of a fridge. >> What's the what's the mindset there of one general model and then how do you adapt to different tasks? >> Yeah. So the way we think about it is that uh for robotics we need to build a very general strong base model right and then because if you think about how a human interacts and behaves in the physical world right we have very strong base knowledge we can just readily do a lot of tasks but if you want to become an expert at any any given task there's some amount of training that needs to happen and I think to train a general purpose robots like the one we're building here the same principle has to apply. So our approach is to first train a general model that's training on as much data as we can find. So that is when our dino report our model is trained over a million hours of human data as a starting point and then given that strong foundation what we found is we can quickly make the robot or make [00:10:04] the model do many many different tasks with a very minimal amount of fine-tuning to just tell the model here's the task I wanted to do and here's very few amount of demonstrations and the model can readily do bunch of the tasks that you have seen office yeah >> um we actually have quite a to people who deal in data, who sell data. >> Yeah. >> Um >> what is good data? What is bad data? What do you want to buy? >> Uh yeah. >> Uh I think that these are all open questions. Obviously, I think the the top um important thing is egocentric data of humans doing tasks. Uh what else would you modify on that basic statement? >> Yeah. Yeah. So I think data is the most important thing in robotics today because unlike language model, there is no internet, right? So every company that's building model, building hardware has to think about data. So there's uh roughly couple different ways of operating here. One is that you can collect a lot of data yourself which is what we are doing as well. >> You have employees here doing that. [00:11:05] >> I mean that robots everywhere in the office, right? Uh but uh if you think about how much data you need for robotic foundation model, I think the amount of data any individual company can collect by themselves is just nowhere near sufficient, right? It's kind of like, you know, you're trying to build an internet by just having Reddit or just having Facebook. There are great websites, a lot of users, but it's not enough, right? So, when we think about data, there has to be some sort of data acquisition happening, right? You buy data, you look at data that's already out there. And then there's many different tiers, right? Um, you mentioned data quality. I think data quality in robotics is a non-stationary problem in the sense that what is high quality data for your model changes over time like when your model doesn't know anything like when it's zero any data is useful right you can use a lot of video data uh you know in the case of egocentric data I think it's actually a very good way of pre-training robots because human data is so diverse [00:12:08] like it covers every single physical activity you can possibly imagine and that gives robot the model such a strong prior on how things how the world can evolve around a embodied actor human. But the issue with human data is that before your robot looks exactly like a human there's going to be some gap between human actions and the robot actions. Right? So just by training on human data uh today it's not very likely you can just execute robot actions directly. So you still need some amount of adaptation using robots's own data, right? But the robot's own data collected on a robot is going to be much more expensive and much harder to collect. So the kind of recipe we're thinking about is like how do we take the largest source of data we have today like egocentric data to give the model a strong player over everything physical manipulation can cover and then combine that with a small amount of robotics data to make the robot you know master [00:13:09] the whatever task we're interested in. So data quality and getting data in robotics is really I think it's a fascinating multiaceted problem but the reality is that there's not a single winning strategy like oh here's one kind of data I'm only going to collect on that which will give you a strong foundation model you really need to think about uh this human data covers this part of robot cognition I need robot data covers low-level reflex dexterity the robot also needs and put them into one model and And I think that's what gives a really adaptable really strong model at the end. >> Yeah. I I'll I'll prompt you uh just for again for the people listening who want to work with you. >> Yeah. >> Um what uh what data vendors do you want? Uh this is your call for data. What >> Yeah. Yeah. No, that's a great question and uh yeah. So in our Dino 2 blog at the end we actually have an email address called data partnerships at dina.co. Okay. Okay, so if you're a data vendor that has a lot of high quality, let it be egocentric data, uh, robot [00:14:09] data, video data, or anything you think could help robots, you know, send it to the email. Our team will take a look and we'll get back to you. >> Yeah. But like at your stage, are you you're you're probably not buying any more egocentric data off the shelf, right? You probably want to be like, here's what I need. You >> is there a niche that's lacking? >> You need a scale AI, you need a core for like doing this. I see >> the the custom that don't work in robotics are probably thinking like okay >> some niche factory like you know you showed putting tubes in there's only so much data you'll find about that but like >> would a lot of that help you >> yeah no I mean if you really want to nail a task having data from that task will certainly help you a lot right so for us it's like we still think you know just a million hours of data is is not enough it's just the beginning like we show the first model on a million hour scale but we need more than that right. So we're still looking at a lot of just raw diverse data uh that we can get but we're also thinking about how do we get [00:15:09] more custom data either collected inhouse or by our partner for like really you know specific niche tasks because I think you still need that to get to you know excellence that you know any model needs to actually be useful in the physical world right it's kind of like humans you know uh you know if uh you know Sean or me you know we go work at Rosewood Hotel tomorrow, we're not going to be able to make the bed the way they want, right? We need a lot of training. So that kind of data like special data, it's kind of like what Mark C is doing today for language model, I think will be very very valuable for robotics as well. >> Yeah. So hotels, you know, strap a GoPro on your employees heads and >> and there's probably more themselves in their in their wig. Uh so so one thing that we discussed is like the the closing the sim to real uh and uh what's very interesting is like now you can basically fabricate environments on the fly. >> Yes. >> Uh have you tried that with the the frontier models um what's your experience? Are they good enough? >> Are you using any world model robotic [00:16:09] like world model companies to generate these and train on them? >> Yeah. So uh I think generating synthetic environments, synthetic data, simulation data for robotics is going to be one of the next frontiers right because today people have looked at egocentric data, robot data, you know those are scaling but I think synthetic data will be a new dimension of scaling. So going back to your question like yeah we have been exploring like using frontier models to generate a synthetic uh environments for us on our data on our environments that we care about and surprisingly I think now it's at a point where this technology is actually very much worth like going really really deep into so for some context so uh in my PhD I primarily focused on simulation based training for robotics back then simulation wasn't ready but we had this paper called Eureka uh which was the first paper taking like a large language model at the time is GBD4 which just had to like selfgenerate [00:17:10] environments generate reward functions or reinforcement learning and just like do things in simulation and at the end I think uh we had this really nice demo of the robot spinning pencils right that was really cool at the time but that kind of thing was still very limited at the time because back then the model didn't really have good vision capability didn't have a innate 3D understanding, right? So, it was basically just generating code and like using very limited feedback from text to try to come up with new simulation environments. But today, like the vision capability of these frontier models are crazy. They can do like real to sim cadding, there's 3D reconstruction, all of that. I think now it's at a tipping point where this technology could really accelerate robotics. And for us, right, we think about that as like a way to accelerate our internal progress, right? because we're building models, we're building hardware. So if a frontier model can give us more data that way, like generate really tailored data for ourselves, that's extremely useful for [00:18:11] us. And another thing we've been thinking about is instead of using these models to generate like 3D environments, we have also used our own world model, right? The world action model I was telling you about because the model not only generates action, can also generate videos, the model can also generate its own training data. So that becomes a you know self-fulfilling like flywheel that I think will also be powerful. So overall I'm just really excited about this direction because I think even though we have scaled to a million hours of data we are deploying bunch of robots. The robots are actually you know already starting to do very cool things. We're just at the beginning just the tip of iceberg for like what is there to come. Yeah. >> For those on the research side we recently had runway on the podcast. One of their things is robotic role models. um shifting a little towards there. So, you know, next people to look at it. Um one of the questions I have about you making your own models is what is the constraints you operate in? So, if you draw a parallel to language models, you [00:19:12] see like GPOSS 20B is very fit for your average MacBook. Yeah. Then there's a larger one that's fit for a single GPU. Um do you work in the constraints of robot of model on on device in cloud a hybrid and then how does that affect your scaling law of data? >> Yeah. >> Um you know a million hours is great but like do we just scale up parameters data? >> Yeah. No that's a great question. So I think what's really different about robots versus language model and other applications is that for uh coding agents language model it's okay if it's it has latency right >> well like you know today whenever I ask chat GPD it takes like couple seconds before it spits out anything but for robots like the inference latency has to be like very short like the robot needs to act like it can't just hold a knife and not do anything for three second that's going to freak people out right [laughter] >> yeah we saw someone put their real hand And >> our model's pretty smart. So that's [00:20:12] okay. But uh in general like if you think about very dynamic task like for example you want your robot to like you know hold a I don't know a tennis racket and you know juggle the ball right? This is very dynamic, very reactive. You can't just like the phys physics gravity doesn't wait for you, right? So for that reason, robotics model needs to be able to inference very fast. And the fastest way to do that is by serving on prem like on device, right? Because if you call a model >> network, >> there's no guarantee drive a car. >> There's no guarantee, right? Like so for that reason, the model that gets deployed eventually has to be of reasonable size that can be served locally. But that doesn't mean that the initial model you train also has to be small right like in language model >> can distill >> you have a large >> people do distillation right like the best way to make a small model very smart >> is by doing distillation right you train the biggest model then you distill right so I think in robotics we're getting there but I think that's what people are [00:21:13] going to converge on yeah >> how about like uh inference you have your own inference stack or a lot of this is architecture you've pulled diffusion and whatnot. I'm guessing you can pull off of something. Anything you want to share? >> Yeah, so we build our own inference stack for whatever model we deploy because I like I was saying like latency is so important. So only the whole stack is extremely important for robotics, right? You want to optimize like the inference speed for the kind of model you're running and there's a lot of work in there. So that's what we do. Yeah, you're really full stack like everything from like uh you know the actuators all the way to in the industrial design ID which I didn't really know as a acronym uh obviously to the models and data as well. >> Uh I'm curious like are these your own actuators or they how much of this is your own like of course the design you know it's not a humanoid but >> actuators like how how far down the robotic stack? >> Yeah. So the way we think about uh what we need to own versus what we can buy is [00:22:14] like if the uh input output and uh of a component is very very clear and like there's already extremely good mature solution then there's no point to building a house like for example on this robot uh you see the lightars right so those black circle those are lighters to help the robot detect obstacle um navigate surroundings so that's a very mature technology ology that people have been using for now decades is on self-driving car. So we don't need to build lighter from scratch, right? Same goes with some cameras. But for anything that we think is extremely crucial for the whole systems performance at the end, our vision is to be able to own that and build that ourselves. So one thing is uh you mentioned actuator. So that's something we're working on and we're going to own ourselves. The reason is that like the quality and how you design the actuator really matters for how well you can model the robot, right? Actuator is what drives the robot. To be [00:23:14] be able to model your robot very well in simulation or to do any kind of modeling, you need to select actuator is very good that you know can actually be easily modelable. So that's what we care about eventually. So we're going to own that. Same thing with you know the robots proportion. you know this robot's proportion is like we measured against real people. Why do we do that? Because we're pre-training on human data. >> Oh yeah. >> Best transfer, right? So there's a lot of these kind of decision that goes into hardware that's very much tied to like because eventually you know it's hardware that carries intelligence. Intelligence is the extremely important thing. We want to make sure the hardware serves for that. >> Yeah. Yeah. Uh, I'll just highlight I I don't know if this is relevant to you, but uh, one of the most interesting robotics discussions I've had is someone who was very stressed out about actuators and specifically magnets because apparently we don't make magnets in the US. >> No, we don't. >> And there's only two countries that make magnets. >> Uh, rare earth material. >> Yeah. Right. I don't know what the what's the chem what's the material? I [00:24:14] don't know what the thing is, but yeah. >> Uh, so yeah, if anyone's interested in that, that's apparently a bottleneck that we have. >> No. Yes. >> So, what do we do? Yeah, I mean I mean it's a good question, right? I think there's a lot of uh exciting discussions on bringing manufacturing back to the US, but you know for robots especially that's actually much harder than uh than what it seems. And I would say even now there isn't a company in the US that's truly building everything from scratch in the US, right? like even when they say they assemble robot, they build whatever actually in the US, they're still sourcing the raw materials from other places in the world. And I think that's something to really be aware of as a robot builder, right? Like I think uh like as a full stack robotics company, we realize just how long the supply chain is and how long the whole problem is is actually extremely hard. And I think these days a lot of the discussion [00:25:16] centers around a robot foundation model, model data. But I think hardware and just how hard it is to build good hardware is very undermphasized. But I think >> that's going to be extremely important. >> This is a platform, right? >> Actually platform. >> Yeah. There there's some companies that choose to be let's call it platformless. They only do the model and you'll say we deploy on any platform. We deploy on Optimus >> offtheshelf robotics. >> Uh and so it's a choice, right? like obviously you you you own everything so you can do that. Uh do you see yourself building more in the US or do you see building your own global supply chain? >> What what's the direction that is likely? >> Yeah. So we are currently exploring many different options right because with global supply chain then there is some you know geopolitical things >> shipping >> shipping also shipping delays and all that stuff >> literal ships for robots. >> Yeah. and uh in the US you know it's just that going from zero to one will take some time right so we're trying to think about a [00:26:16] solution that can both allow us to build more in the US while leveraging global supply chain right but I do think uh going back to your previous question on like some companies choose to just build model we build everything I think I think it's very important to get this right I feel that for robotics the hard thing is uh like building the whole system the necessary things to build a whole system like previously I was at deep mind I was at Nvidia primarily just building models and I saw the upper bound of the system when you don't control the hardware when you don't get to optimize that right because if you think about the value creation for robots eventually robots are here to deliver service to solve some physical tasks so if you want to solve physical task you want to be able to optimize everything and that's important >> I mean we have the robot you got to tell us about the Fine. Um your own custom platform. >> What do you call this? >> What do we call it? >> Yeah. So it's not humanoid. Um it's very smooth, but what's the what's the [00:27:16] rotating arm? >> Yeah. So we think of this is generally like a semihumanoid. So basically the upper body is you know like a humanoid. It's got you know of course shoulder got arms. It also got like four degrees of freedom on the waist. So you can basically do all the upper body motion that a human can do. But it's a semihumanoid because the lower body is wheels, right? So why do we choose wheels? Because I think there's a difference. There's a you know optimal like practicality versus performance trade-off. I think with legs the energy the energy consumption will be a lot higher. You'll be a lot more unstable. You know safety will be extremely extremely hard to guarantee. So right now for legged robots, there's actually no guarantee that the robot will stay upright, balanced the entire time. I'm sure you guys have seen a lot of those robot dancing videos like like the robot dancing. Yeah, they are they fall over very easily. >> No, but you also see the demos that you can tell are not production where you can box the robot and you can push it [00:28:16] over and it's very good, but that robot is just good at balance, right? Like >> there's a >> I think I think the more interesting thing that you guys have innovated so like we have one of the prototypes uh which is the Mewtwo. >> Yeah. >> Back here. >> This is this is just like an up down uh axis and you've chosen to then move towards something like this which um has the the ability to lean and we're going to show the laundry demo where you have to lean into machines. Right. That's another degree of freedom that I don't even have the words for it. It's not pitch. It's not yaw. >> Yeah. Yeah. So yeah, how did we think about that, right? like uh so actually it's very useful exercise like you know I think sometimes a couple years ago when I was exercising I accidentally like you know like pull a muscle >> right I'm sure you guys have that experience too right then the next few days like you just couldn't move or like you pull where you slept in the wrong position so your neck is very stiff we have all experienced that and then whenever you experience that you kind of realize wow so many muscles so many [00:29:17] degrees of freedom that we use in our daily life that we take for granted But when it's disabled like it's very awkward for you to do some task that you feel are very natural. Right? So that's also how we think about like when we design robots when you have like a this kind of design which is this purely elevator could only go up and down. We realize a lot of the like human torso movement like we use our like you know waist back a lot for like day-to-day physical activity right even like if I want to get this bottle like I'm not just moving my arms. It's actually very strange for me if I I have to intentionally think about fixing my upper body. I would always like do this, right? >> So there's just a lot of like subconscious way that humans move their body that I think will be equally useful if robot has a degree of freedom. And if you further think about okay, we want to learn from human data. The humans are moving their torso all the time. You want to be able to like ingest that into the model to control the robot. But if you have a form factor that doesn't allow for that, then the transfer [00:30:18] becomes less and the robot becomes less stressous and it's just more awkward. >> What's the data labeling side of this? Like I, you know, you've seen people um is this the last like post-training fine-tuning side of actually the hour of using the roboty operated or remote controlled mapping motion or like does some of it generalize over? Uh when you say labeling you mean >> like taking the model you have that predict state Yeah. >> to mapping it to your specific hardware. >> Oh yeah. Yeah. It generates this very well. So for all the demos that I have shown you on the new robot cuz the new robot we you know it got ready not so long ago. So we quite literally do not have a lot of data compared to the stationary arms we showed you which had much more data. Right. But a new robot, the model has not seen this in pre-training in some of our pre-trained model. It's able to learn the task that you saw like very quickly. And I credit [00:31:18] a lot of that to like you know like even before the hardware is ready, we already committed to pre-training on humans because we knew at some point our robots will be like human. You can get ahead on the training side while hardware is getting ready. And I think now we're seeing that like the transfer is actually there and is actually very helpful. >> Talk about the eval side of this. So you can have predictions, you can have it do stuff and you know every now and then we see it miss stuff. You test on very niche things as well like filling a pill bottle. Um >> how long does that process take? How do you think about the aspect of actually deploying on a new task evaling iterating what does that look like? >> Yeah. So if we want to learn a new task uh you know we basically do a little bit of robot data collection then the model these >> what is a little bit like >> yeah like couple hours couple hours. Yeah, we're in some of the tasks like just tens of minutes, right? Yeah. And then >> the chopping one was 45 minutes. Yeah. >> The open the fridge was on >> and that's an interview. That's like an onboarding task for one of your [00:32:18] engineers. >> Yeah. Yeah. Yeah. So, uh >> fun fun thing if you want to work here, you know. >> Find the task, have the robot do it. >> Yeah. Yeah. Pitch me a new task and I'll come to the office >> uh you know, collect some data, try a model and see what happens. >> Tie your shoes. >> Julie is tiring. Yeah, that'll be cool. But uh yeah so you know a little small amount of data collected train a model then we just deploy the model in the real world and then we look at the performance then >> to to fill in collected means like you are you have a person actually operating remote control >> doing the task for a bit >> and then you train >> and it does okay you improve how do you measure >> yeah yeah so when you're training the model there's uh you know certain metrics you can look at Right? Just like actual model it converges nicely. Now after that for robotics today we just run the model in the real world right on the you know same setup that the data is collected to see how the model does. Sometimes the model just works great out of the box, right? You don't really need [00:33:19] to do much more. But when the model's, you know, like maybe like 70% 80% there, then you see how the model behaves. You figure out what it's not good at. Then you can like collect a bit more data, then iterate, right? I think of this is like the robotics version of like reinforcement learning from human feedback, right? the robot rolls out, you can score it good, bad, uh somewhat good and then that data, the robot's only experience can be used to train the model. And you can also just collect more data yourself to train a model, right? Eventually, you know, our goal is like, you know, this robot is going to be our platform. Anyone should be able to uh train tasks on it. >> I want to get a bit of a prediction and like timeline to where that goes. Right now, the tests are um somewhat simple, like take laundry out. It's a towel. You can't crush it. Um >> chop some food. >> Yeah. >> There's also the side of like, okay, set up this filming studio, right? Move stuff around. Um bring a table, set up [00:34:20] lights, I think. >> Manage my data center. >> Yeah. Yeah. Plug in stuff. Um >> and and also uh re like recharge another robot. >> Yeah. Yeah. But when do we see the factory or like what do we need to scale up? Um is it model? Is it everything? It seems like your hardware is set for that, but where do we see it go to a point where I can go from instead of just cut one thing to you know, okay, here's a recipe like full send or >> Yeah. >> Yeah. I would say to get to truly general purpose robots. uh like like basically a robot or a model that does anything like >> not not not that like okay so to keep it somewhat grounded on a technical level like >> when do we start to see us pushing that feedback right so instead of very simple chopping uh maybe a 5 to 10 minute sequence right where now the eval step it can break at many more points instead of just fill one pill bottle uh 10 different pills all in a sequence right what's limiting us what are we pushing [00:35:20] on >> yeah I think yeah I think a couple of And one is that like uh the laundry demo that that you have seen yeah it's going to be like 1 hour to finish a whole loop right so the longest duration that a robot can do like a full job is already in the hour range >> nice >> right but you know to be able to do that for any task you know I think the bottleneck is coming back to data and uh hardware right so for some of the tasks you know for example setting up the space you know you might need more than like a you know the parallel jaw gripper right you might need something more dextrous like a hand, like a human hand, right? But then if you want to train a model on the hand, then you're bottlenecked by the hardware stability. Like hand today, the best hands, dextrous hands are still not uh durable enough. They overheat pretty quickly, right? So that becomes a bottleneck, right? Like you we you know, you quite literally have to put a you know ice bag >> to we're not just auto researching make better model. We have we have hardware [00:36:20] limit. >> Yeah. So hardware limitation is a big thing but then if hardware is limited then that will slow down everything right because to know that you're making progress on any task you eventually have to eval on the hardware right so then without the eval being very like spontaneous very fast that also limits your research progress right because then how do we collect the right data right I think um how do we collect the right data to do extremely complex like setting up this uh studio right because you can imagine like today most of The data is like vision based, right? It's like video streams and like, you know, some uh uh encoding of your hand motion where the robots have joints. But humans have a lot more sensors. You know, we hear things, we feel things, >> smell. >> Yeah. You smell things, right? So, we subconsciously use all of those to help us be so good at manipulating the physical world. But today like if you're not capturing that at all then even when we scale to millions of hours you might [00:37:20] just be missing critical dimensions of the task. So you'll never be able to solve the task right. So I think those are like what I consider like you know what's dragging out the timeline. But for tasks or for workflows that you can collect data today to high fidelity then I don't I don't necessarily think there's a fundamental bottleneck right just like what we are demonstrating uh to the public this week you know we have a physical agent that can do the entire laundry workflow right as a human would right because for that task we don't really have like oh you actually need more than a parallel job but you actually need some sensing modality that's just not available today, right? So, I do think the biggest bottleneck to have a much more general thing is still hardware and data. >> Yeah. I want to take this chance to talk let you brag about your customers and partners. Yeah. >> Uh you have some announce uh relationships. Um we can start with Best Western and then maybe some of uh some of the others that you're that you're proud of, >> cool stories, anything like that. [00:38:20] >> Yeah. So, we have a deployment blog post uh on our website that goes into much more detail. I really recommend watching the video because I think the video is >> Do you want the best Western one? >> Uh, no. We have like a one video for all the customers we're working with. It's a really really nice video. Shows what we're doing and uh yeah, I think it's great. But yeah, we have customers uh you know, Best Western that's in the hotel space. We fold a lot of uh laundry like uh you know, towels for them. Every hotel needs it. Every gym needs it. Every fitness center needs it. What >> What's the deployment so far? Uh how like where can I go to? Which best person do I go to to see it? >> Yeah. So if you want to go somewhere, I actually recommend going to our other deployment DYon. Have you guys been to Yes. >> Yes. >> Yeah. You know the best >> very thin rapper. >> Yeah. Very >> We should have filmed there. >> Yeah. We should have [laughter] just done there. >> Yeah. But uh in Dinton like if you go to Dinty phone or other like uh you know somewhat upscale restaurant, there's napkins on your table, right? So our [00:39:20] robots, our robots today fold thousands of napkins for uh many many different locations, many different gentle locations in the United States. So that's actually our biggest scaled customer, right? So I want to make a distinction here. Uh today there's a lot of robotics companies talking about deployments, talking about pilots, but what's really hard is not putting your robot in one location or doing a pilot. What's really hard is that the customers believe or validate that your robot creates enough value that they want to deploy your robots across many of their sites. So that's what's happened for us with Dinty phone which to the best of my knowledge hasn't really happened for other companies doing similar like general purpose robots. So at Denton our robots fold thousands of napkins per day. It's the quality standard is extremely high, you know, cuz they want like okay uh the second forward seam of the napkin is only like uh couple centimeters below the uh I don't even know how you call it the the pocket so [00:40:21] you can stick the chopsticks in. So it's actually very fine grain high precision has I can show you >> at some level that's what robotics are good for. Robots are good for too, right? Um humans are not the most consistent. >> Yes, >> especially on small details whereas a robot can somewhat do it perfectly. It's interesting that's the example you bring for general purpose robotics versus >> uh at some level I think people would assume you can have a specialized robot that will fold napkins perfectly to the exact you know millimeter as opposed to something like this but >> yeah I'm sure uh maybe you can right but the hard thing about manipulating deformable object right like napkin towels everything is deformable there just infinite infinitely many possible configuration so that's really hard for a specialized robot, right? And what we find is that by developing a general purpose model in a robot and adapting it to these use cases, you can actually get to a production quality faster. And what's really nice is that now that Dinton has scaled, has signed a much [00:41:22] bigger contract with us, they not only want our robots to fold napkins, now they want our robots, you know, like cut, you know, that's why we had the example to cut food to. >> Is that actually the second use case? >> That's one of the use case they're thinking. >> Oh, wow. I think that point needs to cut so many vegetables per day. >> They need to do the dumplings. >> Exactly. >> Do the full. That's the final boss. That's that's the final. [laughter] Uh >> I think you can. I think you have the dexterity and precision to do this. >> Yeah, >> you have little poll. >> But like I know don't break the dumplings. >> Yeah. Yeah. I mean that's that's the obvious benefit is you can adapt them anymore. I think the other thing you really touch on is the speed, right? To make a custom napkin folding with the exact ply, whatever. >> You have to make the machine. You can't just Amazon this. Yeah. But this can do it pretty fast, right? >> Yeah. This can do it very fast. And now it's actually it's at human parody. >> Yeah. >> You know, and this robot doesn't get tired, doesn't need to uh, you know, you can and I think I really like the fact that you mentioned consistency, right? Like one issue with Dent Tyon is like [00:42:23] you know their LA location with New York location Seattle location they the full napkin study differently but from their p perspective they want consistency but you know that's exactly what these models are good at because models are you know just it's a set of weights you can just like put the weights in the different robots across the can ota update them as well. Yeah, you can just update all of them at once, you know. It just once it works, I think it's truly like incredible. >> Final boss is Cheesecake Factory. If you know the story there, they're there from scratch. Everything is very consistent down to the measured gram. Every sauce is made like anywhere you go in the world, you'll have the exact same thing. If you don't know, you don't know. But, you know, that's the that's the fun. >> Anyway, the term that comes to mind is like sort of like bitter lesson for robotics, right? uh instead of the specialized robots which is a lot of uh industrial engineering you build a generalized platform that can uh do a last malineet tuning and and do all those things. >> Give the expert add-on outside of your what is what is more bitter bitter less [00:43:24] in robotics? >> Yeah, I think there's several bitter lessons for robotics. One is that robotics is going to scale just like any other domains that machine learning has solved. Right? So what does that mean? That means have as much data as possible. But in robotics, the one thing that's a lot harder is like what is the right data is much harder to figure out than other domains. Right? In language is text. Uh in video games, just video game data, but for robotics like what sensors you choose determine what kind of data you have, right? But if you have the right set of sensors and just scale that data, I think you'll get predictable return on model performance. So in that way, robotics isn't so different. So that's the bitter lesson as a corey of that is that like there's a lot of research in robotics that's coming up with like very specific priors like for example we need to make the robot see objects as they are draw some boxes that will give robot better [00:44:24] perception right but I think the bitter lesson here is that just keep the algorithm simpler and scale the right kind of data then generalization better behavior will emerge so that's not so different And the second I think that's the main bitter lesson which is like at the end of the day scaling will work but in robotics to scale and how to scale just much harder because there is no internet you have to come up with your hardware come up with the sensors get all of those right and then scale then I think uh >> I mean there's the argument that you know the world is very made for humans it should be a human right as hard of a hardware problem that is think 10 20, 100, 200 years down the line, can we make a really efficient human? But you have to be practical with what we have now, right? >> Yeah. Yeah. So the way I think about it is like yes, eventually the robot should if you want to build a general purpose robot, right? Like a machine that can be adapted to as many use cases as possible. I think human form factor will [00:45:26] be the best one just because that's what we have been doing uh for ages, right? But I think the best way to actually build a human robot is to take an iterative approach because robotics is so hard the you know the whole problem the whole stack is so long it's really hard to know apuri what's the right decision to make right and because hardware decision is much more expensive much more costly just because hardware iteration is a lot slower you actually want to take a incremental iterative approach not going all the way to the final form factor because the chances that the thing you build actually doesn't last. It's not good enough. So, you just end up scratching it anyway. You know, like it's public news like you know Optimus you know already gone through many many iterations and they had to scratch their designs right and I think by iterating with a semihuman form factor and actually putting them out to real customers or and getting models really deployed you get that feedback loop that you wouldn't get if you're just trying to build like you know warehouse humanoid and you spend like [00:46:26] five 10 years building it. I think the iterative approach will actually get us closer to the final robot faster. >> And I will say, you know, for those listening and not watching, it's probably the smoothest robot we've seen. Like every humanoid you've seen is >> jerky and even humans don't walk as smooth as >> because Yeah. The model is can can move every single motor at once. >> Yeah. More than two cohort. Yeah. >> Yeah. Yeah. And that's actually a very hard problem because if you just think about like uh the model you know it's outputting action there's an action you know dimension right the bigger the dimension is the harder to fit uh the model to the data but here the action space is quite large but uh uh we did very good job to make sure the model fits well and gives you a smooth >> can you bound the order of magnitude of the action space or is that >> yeah so it's in the orders of like uh you know about a 100 action dimensions >> yeah and these are all you continuous, right? >> Continuous, right? So, the model is just basically predicting at very high frequency, you know, like tens like at [00:47:28] least 50 dimensional vector like 10 20 times a second to get that continuous motion >> and potentially predicting one to two seconds out. So, that is another 100 frames or whatever. >> There's a bit of a history um similar to context of previous state I guess. >> So, context is a very important thing in robotics, right? Because otherwise if you're just making decision based on the current observation sometimes it's really hard to tell like what to do next right for example like if you think about opening a microwave like in the middle part it's unclear whether or not you're composer or trying to open it right so like context is also really important >> yeah and I I have a screenshot of the the fridge demo where is there's aentic planning and two hauling and all those things so we can show that for listeners >> I think that's an interesting side after the model >> how do to wrap it around into aentic harness like what is >> kernel deployment clock UI you have a lot of cool UIs u but you know what what goes into [00:48:29] wrapping model output towards >> final thing is this custom per use case is it you have your own SDK >> yeah yeah so I mean eventually we will have our own SDK but uh the way the way we think about is we want a general like harness layer for robotics foundation model today that is our own model right the way I think about is like for any like real task long task in the physical world there's a lot of like highle decision- making you have to do right it's not just like robot you know only reasoning about the physical world in the action continuous trajectories right because that's like very like low-level the thing you're actually executing the physical world but that's not the intent that you know like when we are doing decision making we think about steps of a recipe we think about oh like I finished this I need to do that while I'm waiting for this part to finish I need to go somewhere else to do something else there's a lot of high level cognition decision making that's happening and I think for robots to [00:49:29] handle even you know the laundry task we are showing you it has to have that capability because you know it needs to reason that okay the washer is running so I should go do something else while the washer is finishing but now the washer is done I should take I should take the towels from the washer and put it in dryer immediately Right? Because if not, then that becomes a bottleneck in the whole workflow. Right? So, there's a lot of reasoning that needs to happen for robotics. And I think in theory, you could just, you know, try to learn that with a, you know, low-level model that's outputting actions, but it would be extremely inefficient. So that is why having a hierarchical system like putting a harness like a agentic system around our world action model is what allow the model to compose different capabilities different skills to solve a much longer horizon task. It's very much like in language model, right? Like two years ago, three years ago, the model was only good at autocomplete. Like humans have to like tap tap tap to get the model to keep spitting out code. But today, just like agentic system just [00:50:30] carries the end to end task and that's and I think that's important because I think today in robotics people are still thinking about autonomy at the very task level like a one minute 30 second task. But really the value is about tasks or like workflows that go on like unsupervised for hours and that's what we're building towards and that's why I thinking about the robotics problem not just as a modeling problem but rather as a system problem you need good agent good harness engineer good model good hardware everything it's like how we think general purpose robots ought to be solved >> do you guys see benefits from language model coding tools getting better So new Opus 55, new Astra. Um, do you have language models in your agent harness? How does AGI being developed in language model space affect you guys? >> No. No. Yeah. No, absolutely. Like it affects us very deeply in many ways. One [00:51:30] is that like we're extremely AI native company, right? Like coding agents, all of that is a very fundamental part of our work and then we're very welcoming of that. Like whenever there's a new model, we're the first one to test it, right? Like we didn't like when GBD6 came out, we had like a companywide hackathon the same week and the thing that our engineers built was incredible and many of them are many of the features we built is already in main in production. Right. So we really like using coding agents to accelerate our work. >> Uh just for people interested, uh anything surprising in GPC6 that you found? Yeah, I think uh it's about the ability for it to like construct virtual environments, right? Turns into simulation. We talked that uh 20 minutes ago. I think that's really exciting. >> It's interesting. There were so many like CAD Zillow to 3Dish. >> He lost the hackathon to a CAD. Uh >> I won I did the CAD >> other way. Other way. No. Oh, >> that was I won a GPT hackathon just for [00:52:31] fun. I was like I know nothing CAD. um might as well play around with it cuz every demo is doing it. I'm like it's going to be garbage and I'm like it's kind of grounded. It's kind of good but it's just hackathon stuff. >> Yeah. >> Um cool. I I I was going to round out uh some of the use cases and then go into a little bit of company culture and your personal background. >> Yeah. I have two uh two technical questions. Uh, one is are there any like goals you want to set for yourself like 6 months, 1 year, 2 year timeline that you see robot should be able to do fill in the blank that you're kind of working towards? >> Yeah, I mean in the 6 months to one year frame, our goal is definitely like scaling up our robots and our model and truly getting them to deploy. >> I mean like a task like uh right now it's cutting or opening a fridge. Is there like do you set yourself to by this timeline we should to stay on track be able to do whatever? >> Yeah I think uh in terms of deployments [00:53:33] we're exploring many different industries there isn't a particular task we think we have to nail right because our goal is to build a general model a general platform and we think of each different task as like validation of our platform as a way as a wedge for us to figure out how to build everything better. So today we're doing a lot of that in laundry in restaurants. Uh we have already deployed tens of robots but our goal next year is like overall just deploying hundreds of robots. >> Uh my part two on the last technical question is how do you see the rest of the industry? So fill in for people what other approaches are people taking to world models? Why are they wrong? Um what is different but good about their approaches as well? just what is the landscape of robotic world models in this space and why how have you come to your current >> yeah no that's a great question so I think robotics is the most exciting space today and there's just so many uh [00:54:34] uh you know amazing companies that I truly respect you know they've been doing great frontier work like us and uh it's very exciting and I think what we are seeing today is that there's still many different approaches to robotics like we believe that hardware needs to be built in-house. We need to own the whole uh chain. Uh we believe that scaling human data is extremely important. But there are companies that's betting on just scaling robots data. We're scaling some sort of intermediate like data collection devices, right? All of them are in play and uh I think it's different approaches. We'll just see what happens. I think right now uh it's still a bit early to tell, but we feel pretty good about the way we're building things just because like we as a company culture-wise, mission wise, we're not here to just build a model. We're here to truly get robots deployed. So we deploy as much as possible, work on the hard problem. So we identify problems much earlier on. So then I think that really helps us identify what is the true bottleneck and then solve it. And I [00:55:35] think for uh folks for people who are only working at one part of the problem, it's kind of like fog of war. You don't really see the whole picture and you might be building something that's extremely cool technologically but may not be what people need or want at the end, right? I think there's a whole graveyard of robotics company and I think all of them had very shiny, very cool robots, but uh you know just figuring out what people actually want is actually is very easy to say you know like build something people want. You know everyone says that but it's actually really hard to execute uh botics company >> on the technical side. You have your world action model. Is that standard across industry? Is there other directions people are taking? >> Yeah, I would say that's also not standard. So we're building world action model on top of world model. So there's many companies building world model. You know you mentioned runway they're also getting to adapt adapt video world models. >> Uh there are also companies building uh vision language action model. So that's like taking vision language models and adapting them to robotics. Uh so I think [00:56:38] there's many different ways to go about it. But I think at the end of the day like what we call the models is less important than what we like believe the model needs to be able to do at a first principle level. So I want to explain that a bit. I think for robotics the most important thing is that you need to be able to predict the future. Right. Right. Robotics, any physical system is fundamentally about based on the information you have at the present moment and before what should you do next. So it's really about predicting what comes next. So predicting robot action is like predicting what the robot should do next. But if you think about it, robot is just a part of the world, right? Robot influences the world, but the world also influences how the robot should behave in that environment. So if you think about it this way then the maximal the most first principle thing you can do just predict as much of the world that you can possibly predict right so that's why we believe predicting the future is extremely [00:57:38] important and today the instantiation is world action model and VA only predicts robot actions where some more abstract representation of the world but whenever you're predicting abstract representation of the world you're encoding your own bias into what's important to predict and the bitter lesson tells you let's remove human biases let's just take the most raw representation the camera inputs and just predict all of that right so that's how I think about uh modeling for robotics today >> so just to press you a little bit on this you're you're basically you're saying your approach is better than the VA right because you're you're you're you're emphasizing prediction I fully buy that I like you've sold me on it >> if you were wrong what is the most likely be number two. [laughter] So, someone else is doing interesting question. >> Yeah. So, yeah, there's a caveat in what I've said before and I think that's where I could be wrong. Right. Well, I said that you need to predict the world [00:58:39] as much as possible, right? But what that doesn't say is that does your sensor have everything about the world that you need to predict? I think I touched on this a little bit, right? Like if you train a world model that's purely based on videos, I think it'll be a very powerful world model, but it may not be a sufficiently rich enough world model because you're missing critical aspects of the world like sound like touch, right? So I think what where we could be wrong is like okay I have the best world model on vision vision based world model but tomorrow someone else comes out with a vision plus tactile world model that just like 100x more data efficient just because you know vision is a very high information or like a highly redundant you know input because you know every other pixel they don't change that much you can see you can understand the world without perfect vision but tactile you know other modality might be much denser in [00:59:41] information right so maybe having that predicting that would just allow you to use much less vision so you can get to the same performance with much less compute with much less data right but >> I think you'll be able to adapt very easy >> no exactly there's also the level of how much data >> right so the reason why I bring this up is because I think if vision tactile world model or vision tactile audio whatever world model is the final thing that solves robotics then I think robotics companies will be able to do that better than video generation companies is trying to pivot to robotics the reason is like to understand sensing is actually very complicated like we've been building that since the company started so we know the bottleneck we know how to combine them but for if you're just trying to adapt the video model to robotics action they're like you know if you were wrong if you know video model is not enough they're You don't really have the knowhow to get tactile to work very well, but we have that. >> This is a side question, but I think [01:00:41] it's a quick one. Uh, heat seems very important. Is infrared just another color to you or is that a >> Yeah, it could be another could be another color, another input. Yeah. >> To the model. >> Could be used in in cooking scenario actually. Like I know companies that's using infrared to like know how cooked their food is cuz that's really hard to tell from RGB camera. Right. My uh very naive push back on your last point is uh maybe we don't need to understand hardware cloud code or codecs can just you know v code it in uh there's a lot you can do without understanding but of course >> it's just not in the distribution. >> Uh so so I wanted to bring it a little bit to your personal story or your background. Um we both went to pen. I did not >> I just found out today. >> Uh yes. Yes. Uh I I I don't know my my my big regret is I didn't join the self the darker challenge. Okay. >> Which uh and then I could have been like early whimo or something. Um you were at grasp and then you you've also worked at uh Google and Nvidia. [01:01:41] >> What are the different focuses of let's say academia versus Google versus Nvidia? You've seen all these places, right? And then like that obviously caused you to say like screw all that. I'm going to do my own thing, right? You could have joined Open AI robotics. >> Yeah. Uh so can you just take us through chronologically but also like one thing you learned from each. Yeah. So chronologically, yeah, maybe it's useful to talk about how I got into AI in the beginning, right? So, uh, as a kid, so I spent time in Japan and China. I grew up playing Go, you know, the going alpha go. So when Alph Go came out, I was a senior in high school and I was mind ball because I've been playing this game for many, many years. It's so complicated. Like I think before Alpha Go came out, people were like, "Okay, Go is one of the game that AI will never solve. the search space is like larger than the total number of atoms in the world. how could you possibly search for solutions in that then like you know RL reinforcement learning just like cracked it right and I think that was how I got into AI initially like I was just really interested in like models were systems [01:02:43] agents that can self-improve right so when I got to pen as a PhD student I just really wanted to work on reinforcement learning so I'm co-advised both my advisers were interested in uh at the time one of my adviser was more in robotics he did a posttock at Berkeley robotics lab. So that's how I got into robots uh at pen you know we have grass lab I think one of the best places to do robotics and uh so that's how I got into doing RL for robotics in my PhD and then yeah I did a bunch of research thinking about how to connect reinforcement learning uh with human data how to do reinforcement learning uh theory in a very rigorous way and then I went to Nvidia that's when GBD4 came out at the time we were very interested in just solving uh pen spinning task. I don't know why like uh >> it's the dexter it's the dexterity limit right I mean you can solve Rubik's cubes and then >> time I was working with uh Jim fan yuk which you guys probably also know like [01:03:44] yeah we were just like okay let's try to do dexious pen spinning the initial idea was like learning it from human videos >> right we were trying to watch humans do it but then I realized okay today we don't have a dous hand that can spin pencil at all so how would I even be able to evaluate the algorithm even if learning from human worked at all. So I said, okay, we need to first try this in simulation. Then as soon as I try to set up a simulated environment for a robot hand with a pen, I immediately saw what the issue is. The issue is like like if I had to come up with this simulation environment, the reward function for reinforcement learning to solve pen spinning, that's impossible. Like I wouldn't be able to come up with all the terms that you need to get RL to learn the right behavior. And then I I was just thinking what if a model that can write code can do that. That's what became the Eureka project, right? We basically like in my opinion we had the first like recursive self-improving robotic agent in simulation because we [01:04:44] really like used that to solve a new task that no one has demonstrated before and that that that allowed me to see the potential of using language model using coding models to really accelerate robotics. So then I went to Google deep mind because at the time you know I mean even today they have one of the frontier models in Gemini right so I was looking at how to use Gemini for robotics but eventually I realized like I had a lot of fun doing academic research but there was always a question in the back of my mind like does any of this matter you know like when you're working on this kind of thing in the lab like you know you ask me how do we come up with so many interesting fun tasks you're basically doing your own eval you're grading your own homework like I wanted something more real. I wanted something that could actually be used by uh millions of people in the world and I felt 2024 was when we started Dina like end of 2024 that was the right time that we can take some of these technology to actually putting them out in the real world. So that's why uh I started Dino [01:05:47] with Lynon New York and uh fast forward two years we built our platform our model our robots are in Tinton. Go check it out and uh >> go eat. Yeah, go eat. It's been a journey. And uh but what still gets me really excited is you know like some of a lot of the ideas that I temporarily abandoned when we started Dynam because you know to build a company have to be very practical like I think simulation wasn't ready in 2024 but now with you know like the new frontier models and what we're seeing internally like this might change and uh that's very exciting. >> Yeah. And uh so then also let's talk about company culture. Yeah. um what principles you're you're building with the company with obviously like the the you're you're responsible for the science and the the research agenda. Uh and it it just seems like when we meet the engineers they're they're very engaged and and they have a lot of ownership. They have a lot of motivations. >> I'll say it's it's interesting to see how many side projects people just work on. So some of the demos we've seen it's just the engineers that work on it were [01:06:47] like yeah this seems fun might as well do it. Yeah. >> Um to your point on fun evals right here like 2 feet behind the camera. >> Can't talk about that. >> No, I mean the thing is just the the eval is just there's highlighters for highlighters being rubber banded together, right? It's like who comes up with ah yes we need to test putting rubber bands on high. It's just stationary >> fun stuff. People are very uh you know creative here. It's a beautiful space. >> Yeah. I think culture. Yeah. So let's first talk about you know how we run a company right? Like we think of oursel as like a like a like a eventually like a research plus product company. It's kind of like open AI where you do frontier research but eventually all of them make their way into codeex to chacht. That's how we think about it too. We're not here to just do open-ended like whatever thing that interests us at the time. We're always like product driven like we want to deploy our robots our platform to a new customer. What are the technical problems we have to solve right? We have to solve generalization. We have to [01:07:48] solve whole body control. We have to solve how to adapt quickly. We have to solve how to make the robots follow language. We have to solve dexterity. So once all those problems are very clear then like I mean our engineers are very capable, they're very smart. So they just create instances of the task. They create tasks that would stress test those capabilities, right? Like the the grabbing and drink task is just how we test language following, right? You can just specify what you want. If the model follows intent, follows instruction, you should be able to do it. So that's how we think about like at the top level we identify what are the things we have to solve. Then we have a lot of bottoms up project to like actually drive execution to solve them and uh you know you hire the right people and everyone here is super passionate about robots. You know they've been doing this for a long time. they love it and we just create a culture where it just naturally happens and uh it's it's a really fun place to work. >> What's the split between like uh technical research software versus [01:08:49] hardware side of the company? Yeah, you know, we recently grew so much. So I don't have the perfect number but I would say on the hardware side we have I would say probably like 30% 40% of hardware just because hardware you need a lot more people and then like you know we have like around 10 people working on models then we have like software engineers uh robotics engineer we have a lot of uh very talented people on the business side you know the most fun thing about building a robotics company is like you see everything you know everything under the sun that could possibly be helpful will be helpful, right? And uh so you just get so much exposure working with so many different people from many different backgrounds who are all very passionate about robots and I think it just it just been really eye opening for me as a founder because when I was a researcher at a lab or like a PhD student, you know, I was talking to researchers every day and that was great. That was really good. I still get to do that every day. But now I also get to learn like just from so many [01:09:50] different people from different walks of life. Like some of our uh people who collect data, they had no robotics experience. They were work working retail, working in some other jobs and they came here, you know, they also became very well trained in robotics. And I think that's just pretty amazing things amazing to see. And I think of myself as like a I'm still learning, you know, I think of myself as just always learning in the robotic space and uh I haven't learned. so much as I am now. You know, even though, you know, you think about me now, I'm not a student anymore. I'm I'm here like running a company, managing a lot of things, but I still feel like the day-to-day learning is like just so much. >> Um, and then just to close off, uh, where is Dina going? Uh, two years, five years, what's uh the vision? Yeah, I think our vision is to truly the high level mission is truly bring like you know robots and uh intelligence right into the physical world like actually that means like actually seeing [01:10:51] our robots deployed like thousands of customers even more millions of robots that's what we want to get to. >> What's the what's the name from Dina Dynamic? Is there a is there a story? >> Yeah, there's a there's many stories. So I I I named Dina. There was a lot of uh >> take the credit [laughter] internal different options but uh how the name came to be was there are two things. One is uh so uh touring award winner Richard Sudden who's like the pioneer reinforcement wrote a paper called Dina many many many years ago. It's about basically a Asian that self-improves in a world model and I felt even when we started the company like we weren't doing world model yet but I felt that's how robotics will get solved and now our you know technology technology direction is like converging to that so I felt that was a really neat inspiration also because I worked in reinforcement learning I have a lot of respect for Richard the second thing is that um it's a bit more personal like I [01:11:52] grew up in Japan and uh you know the year I was born there was just like a cartoon show. I don't know if you guys have seen like Ultra Man. >> Yeah, of course. >> Yeah. Yeah. The the Yeah, the Ultra Man that year was called Dina. >> Oh, >> yeah. So, yeah, it's a very subtle thing. Not people know, but I guess now everyone will >> Yeah, >> the team is finding out too. >> Yeah. [laughter] >> But like I I think Richard would appreciate. I don't know if you ever talked to him, but >> No, I've seen him at RL conferences. >> You should just be like, "Hey, this this company >> Yeah, I'll invite him. Come to you." >> Everyone knows Richard for bitter lesson, but you know. Yeah. Yeah. Also bitter or less. >> Yeah. >> Okay. Uh any parting thoughts or calls to action? >> Calls to action. Yeah. I think um I do feel like as an industry robotics is still very demo oriented. Like I have done a lot of demos in the past in my lab. You know, even here like we had to show some demos. I think we need more people that's truly uh mission focused on doing whatever it takes to get robots deployed, get uh robotic foundation [01:12:54] models truly integrated with hardware and seeing them at real customer sites. I think we're are a pioneer on this regard and I want to see more in this field and for the audiences watching if this mission inspires you if you know solving the hardest problems today is what makes you wake up every day and you want to contribute you know we're always hiring and uh yeah looking forward to that but I think just >> Bay Area anywhere else >> uh yes just Bay Area today but you know as company grows never say never to other locations >> yeah but uh that's what I want to say and uh I wanted to see more robots in the world. >> Yeah. Cool. Congrats on everything. Uh I mean this is this is incredible progress for two years. >> Uh yeah, amazing. >> Yeah, I really appreciate it. Thank thank you for visiting us. I hope you know it's it's been fun and uh >> yeah, I would say like yeah I I don't regret doing this onsite like Okay. >> Yeah, we do a lot of in our studio. >> Yeah. >> Like this was worth it. >> Yeah. Yeah. I think for robotics got to do it on site. It just >> for for people listening in, you know, watch the watch the video version. and [01:13:56] we have fun B-roll. >> This is Dina 2.1, our brand new hardware with a physical agentic harness around the world action model Dino 2. So what Dina 2.1 can do is to automate a entire end to end workflow in a real physical industry. So here what we're demonstrating is end to end laundry task in a real laundry room in a hotel. And behind me these are the researchers for work project. They're going to show you something really cool, a quick demo, and going to explain some of the technical details h behind, you know, what made this possible. So, I'll pass it to George. >> All right. Um, I'm just going to show some quick steps in the entire laundry workflow. Just going to start the robot. This will be the task of transferring towels from the washer into the dryer. Let's take a look. [01:14:58] Yeah. So, this is actually an extremely hard task because you saw the robot has to like really maneuver its body to stick the arms in to grab all the towels without colliding with the doors, right? But you see right here, it actually has to be very dextrous and the motion is very smooth because it's a single model that's controlling every single joint in the body. So there isn't like a, you know, classical planning versus acting separation that makes the motion less smooth, >> right? >> Okay, cool. We can also Do you want to simulate it? Okay, we can also simulate uh the robot realizing a drop towel. So I'm just going to drop my towel here. Uh we'll see that how it is able to see that the towel is dropped and then pick it up. So after it's done putting the towel in the dryer, we'll see and it also realizes that this towel is now dirty. So it goes and puts the towel in the dirty basket. >> I think this is a perfect case to [01:15:58] showcase why we need to build a whole body robot like this, right? So it can access different parts all all of the corners of the workspace that it can do everything that a human worker can do. Next, it will close the dryer door and turn it on. And they can also reason based on the tactile uh in in terms of what uh drying mode to put in. >> Yeah. The most impressive part to me is actually this part of uh turning the knob, you know, like actually operating a machine like a human would. And uh yeah, maybe it's worth getting a close shot, right? and making sure it's on the right mode and pressing a button to start the dryer. Right? >> It's also important to realize that this is digital frames. So, you really need more context to realize because when you capture an image, the LED screen also blinks. So, that's why it's important to realize that the robot knows how to start and realizing when the dryer has started. So, it's pretty cool. >> Yeah. So, this is just one part of the [01:16:59] whole laundry workflow. On Tuesday, we're going to drop the entire workflow. The whole workflow will take quite some time as you can imagine. So uh you know for this >> yeah folding stacking transferring basket everything right really like Dina 2.1 is about like a robot for the first time becoming like a whole employee at a work at a workplace doing everything. There's no need for human or for any other machine to feed it you know items or help it with certain step of the task. It's just going to take the whole thing and do it into it. And I think that's a big breakthrough because historically we think about robots as machine right but this is for the first time like I think we can really see like working alongside humans to really like improve and increase productivity physical productivity in the world. Yeah. So we are we're super excited about it. Yeah. Yeah. Go ahead. >> It will be the longest local dextrose manipulation uh de demo. So uh quite proud of the work that we have done and [01:18:00] yeah exciting. >> Yeah. So here's another demo we put using Dynet 2.1. So here the physical agent with our hardware is going to uh grab a drink for you based on whatever drink you select on this screen. So our researchers me and chat will let the project we just give you a quick demo. So in this case uh they selected vanilla shake drink and the robot has received a command. So, it's going to go to the fridge and then now uh use the world action model Dino 2 that's controlling the whole robot to get the right drink for you and then retrieve it. And what's really interesting here is uh this is really to showcase the agent capability. So, the model is not just like doing actions is also following language commands very faithfully and can also navigate back to you uh to fulfill the order. So, all right. So, got the right drink. It's going to nicely close the [01:19:01] or uh and I think it's switching switching. >> Oh, yes. Gates. >> Yeah. Switching mode because here is the navigation part which uh we can just have another highle reasoning model to do, right? also. >> Yeah. Yeah. Anything you guys want to add on what's happening? Anything cool you want to point out? >> Yeah. So, this one has a reasoning layer and an action layer. So, it's a two model um architecture. Underneath we are sending language prompts. Basically, a few sentence instruction to the reasoning model. Um, and then the reasoning model is planning out what it needs to step by step. First go into the fridge and grab something and then come back and then hand it off to a human uh in in sequence and uh oversee the whole [01:20:01] execution. And the lower level um action model is responsible for performing the manipulation and uh the dextrous task. >> Yeah. On the on the action model side um it also needs to be steerable. So you're telling it which of the six drinks you want to pick up. Um and then like our the video action model sort of helps with imagining like uh understanding your intent. So you can imagine like what does it look like when this robot grabs a drink. Um and that sort of helps it really follow uh what you tell it to do. Um and then there's also some like timing aspects to this. It needs to recognize when you've grabbed the drink and and then like release at exactly that point. And you don't want false positives. You don't want to drop a drink on the ground. So, there's all sorts of like little things you just have to be careful about to get a demo like this working. >> Yeah. And the model is uh quite good at this kind of dexterity and uh very robust. So, like Chad was saying, like it's able to just recognize and kind of like imagine what it's going to look [01:21:02] like if it grabs the right drink and then execute.