[One Million AI Engineer Special] How Stanford Teaches AI-Native Engineering — Mihail Eric, Monaco

[00:00:00] Okay, this is a pretty special episode because we're celebrating 1 million subscribers on the AI code uh AI engineer YouTube channel. So, go over there and subscribe if you haven't heard about it. Our guest is Mihil Eric who is not only teaching the modern software developer at Stanford which is going viral right now for basically covering all the AI native development practices. Uh but also he is head of AI in Monaco. So he's not only a teacher but he's a practitioner. Um, and those of you coming to AI code summit next week, um, it's going to be the celebration of all the extreme progress that you've seen in coding in the last year. I'm excited about all the themes of fighting slob, of system one models, and of increasing your ambition for major rewrites of basically everything that you've seen in code. U, there's never been an more exciting time to be an AI engineer. And with 1 million AI engineers, I think it's just getting started. So, let's get into the interview. Okay, we're in the studio with Mihal Eric, adjunct professor at Stanford and head of AI at Monaco. Welcome. >> Thank you. I'm glad to be here. >> You caught our attention because of your [00:01:00] course that you are refreshing for the third time, third time, second time. >> The CS46 146s, the modern software developer at Stanford. >> This particular generation of junior developers of junior engineers of people that are now entering the workforce will I think be the first kind of generation of that new shift. And I looked at the syllabus and I was like, this is everything we cover. So every now and then I like to invite people like yourself who like you cover a broad range of topics and then we can sort of go topic by topic >> and give people the state of the art of each of these things which I I assume you're doing with all your guests anyway. For sure. >> Tell us the history of the course. What made you want to spring out CS146? >> Yes, this is the second time it's being taught and I taught it for the first time last fall. So fall 2020. uh this was pre like opus 45546 days but the even before fall 2025 the course had been about like a year in the making the genesis of the class like November of 2024 and I had sort of had this idea that we're seeing all this stuff happening in coding agents and yes they [00:02:00] were an earlier version of the coding agents but already it felt like there was something drastically changing in terms of how developers were doing their work how quickly we were getting work done all that fun stuff and so at the time I it was all just scattered blog posts scattered like things you could find online if you were trying to really get into the methodology of how to do these things. And so I I had this idea like, hey, well, is there a way to kind of structure a curriculum around this? It's a little bit more formal. Had the idea and you know I my own matter is Stanford and I always had this dream to go back and teach something and and kind of pay it forward. I've always been very deep into education. I talked to Chip who also taught a class uh one of my good friends. Yeah. Chip. Exactly. uh spoke to her who she had taught her I guess designing machine learning systems and so like asked her what that process was like and sort of you know took a lot of inspiration for how from how she designed the curriculum and and thought about you know getting some through this through the school put it together spent about 9 to 10 months this was now fall of 2024 through to like you know mid to late of 2025 got it through the committee uh made a case for the class to exist you know I had to put together [00:03:00] a proposal and everything uh and that took me to fall 2025 when I taught it for the first time which was very well received achieved you know students were really interested in it. Uh there was even I guess like a business inside article that was written about the class last year and then yeah this year a lot of things changed I would say really coding agents autonomous coding agents really picked up in a way that that we had not seen before. Um so it felt like it was time to do this again uh refresh things uh and then here we are. >> Meanwhile you're also head of AI at Monaco which is also a very buzzy startup. >> Yes. Yes we are. We are we we're in the airwaves right now and and having a fun time doing it. Yeah. >> Yeah. So like I I do like that you're also I guess a practitioner. You guys are obviously implementing all these stuff at Monaco. >> Any I guess >> principles or observations about just what you have changed from last year to this year at high level? >> Yeah. Yeah. I mean a ton. I think that when I reflect on what last year's version of the class was, I mean, coding [00:04:00] agents had been around, but we were still in kind of that phase where companies were emerging that were tackling every single part of like the ASDLC. There was like a coding agent for code review, there was a coding agent for, you know, spec design, there was a coding agent for SR, and you know, some of those, of course, some of those companies still exist, but it felt like last year it was a lot more about a broad tour of here's everywhere you can use coding agents. And in a way, a lot of the curriculum last year was was geared toward exposure and this this kind of survey, this kind of like odyssey of sorts through, hey, everything that you knew about the SDLC is now changing because of AI. Um, this year, I think there's some principles that are solidifying, right? The idea of like very careful context engineering, the idea of like really thinking through sort of like design and architecture as a way of handling really complex tasks. And so the nature of my class is it's not intended for the like beginner programmer. that is not it is not a replacement for like traditional computer science in the sense of like algorithms and if in fact a lot of those classes are prerex to what I'm what I'm doing here but I do know that you if you [00:05:00] have a good foundation going now toward applying that and and and sort of using those principles in in production environments in like real world that's something that I think is is now at a place where you know at Monaco we deal with massive code bases you know I help a lot of companies also with thinking through their massive codebase uh designs and things like that and so really it's more of a practitioners course for once the students have graduated or now entering the workforce, how to be the most productive members of of the workforce. Yeah. >> You said you're passionate about education. I feel like students in that phase of being in college are in a pretty interesting spot. Any advice and any take? Should people study computer science right now? Is cloud code and codeex all we need? Like I I want to, you know, hear your thoughts. Yeah, I uh you know there's a lot of anxiety from students as you can probably imagine and uh and I I think a lot of what I'm trying to communicate this time around is that the anxiety a lot of it still is here and there was anxiety last year as well but I want to make the case that a [00:06:01] well-trained AI native engineer and I know that's like a buzzy word but let's say we we'll we'll we'll we'll we'll dive into maybe what aspects of that that are most interesting a like well-trained native engineer can be some of the most you know the most useful asset to like any company around and so and you know this is the first week of class actually and like week one you know first lecture hey I really believe that every company needs you but the floor is much higher right so you're entering as like your first job out of school but it's not sufficient to just be like a you know an L3 and like a Google term if you're going to use like that that kind of leveling system there it's like you have to generally be better than you know almost at the senior level going to this so like what what can we how do we get there more quickly right and so I'm trying to say that there is still a core bit of like foundational software engineering that is important to have some of these core values of of sort of how software systems are built and the trade-offs of those systems. But then beyond that we go and you know that's where I think you as the humans should still be valuable rather than and the rest of the implementation details you can kind of defer more often. M yeah [00:07:04] this is something I've said before which is almost like the controvers I it's like the kind of thing you say to be spicy but I almost like I I do genuinely kind of believe it which is most students are cooked some of them are going to be super valuable I I don't think it's a very popular thing Stanford Sam kids are going to be fine right there they're from Stanford but like what happens to like the rest of America >> right right that is an open question >> weird >> situation cuz I feel like a lot of education is slow to move like I'm even curious in how >> how do you determine where do you put the end date to curriculum in a course on you're teaching a Stanford modern software like that's changing everything >> you cheat you just get the people from the the coding agents to come and speak right >> oh yeah like yeah have them be the yeah that's a funny guest >> it's a funny well yeah I mean you know that was uh okay so so on on the guest lectures point I think that was um I mean that was also last year um that was also the case so so that was by design because I look I I came in right and I'm saying I'm a practitioner as as you [00:08:05] point out Swix like I'm a practitioner I do this thing on my you know my day-to-day I design these systems you know we we build these things at Monaco and have seen that many times before and so that that being said I don't want to claim that I have all the answers today and that would be that would be like very you' be a lot of hubris to to make that claim. So the structure of the class was always here's one practitioner's sort of aggregated set of insights from working in the field from like talking to a lot of people in the field but then there's people that are like literally on the frontier you know seeing things even beyond that and invite them to sort of put their their their proposals forward their view of the the world forward and that's why the guest >> I think it's really impactful for students to see big names uh people actually doing it like hear from them directly be able to like ask them of course you know they want to intern they want to get jobs it's it's good to have guest lectures >> no I think what he's saying so okay I I'll sort of rephrase and uh no so okay there you know we've also interviewed an [00:09:05] >> uh who teaches the Coachella course right >> right which is like here's all the names like it's only the names right whereas like he's actually also producing you know here's cloud code in 200 lines of code right and which is no one's telling you to do that no one no one's going to come speak to you do do that but like yeah that when you want to teach fundamentals in a framework that you can use and a working understanding. This is the kind of thing you do. >> That's right. That's right. Yeah. Yeah. Exactly. And and I but I do think there's a balance for for this kind of material to to your point is there is a you can't it's not 100% solidified. I mean by by design by design it's a feature and maybe a bug of some some kind but a big portion of the curriculum did change from last year right and so that the there's like a dynamism to the whole development of the curriculum. But I'm trying to at least put forth that there are some things that I believe are kind of sticking around and yes prompting coding agents they've gotten smarter. What does that mean in terms of the new ways of dealing with them? But then still Boris is also going to know something that I don't know, right? So let's let's have let's have him also come to the table and share his learnings. >> Um one of the fun things I noticed this time around is a good bit of the portion [00:10:06] is on open source contributions. >> Yeah. >> Uh what's the thinking behind that? You have people >> Yeah. Yeah. This is um this is one of the things that I'm honestly kind of most proud of and and most excited about. Uh I mean this about to be week two of the class. Yeah. So this is this is the very gracious open source partners that I spent the greater part of this summer just kind of talking to and and getting them kind of excited about and and and interested in in sort of this offering. Basically the idea was so last year when I taught the class we had like nine assignments and then we had like a capstone open-ended project, right? The nine assignments were basically me sort of like coming up with these pedagogical exercises like oh here's a repo like use an AI code review tool to to you know post a code review and then see what why is it good why is it not good so relatively pedagogical honestly pretty easy I think for a lot of folks they could have just like one-shot it and I thought okay this is not that helpful at this point agents are around I mean they they have every student that I surveyed you know on Tuesday not a single person really types by by hand anymore they're like everyone's using coding agents there's [00:11:06] no there's and this is you know like All sophomores, juniors at at Stanford. >> Do you do you survey them? What's the market share? >> Was that like what what tools they use? Okay. I haven't yet, but that's a good call. Yeah. Yeah. I I will >> be spicy. Stanford prefers. >> No. No. Well, I think through like the the university, I think they have some the educ like most of the the the providers have like an educational program, right? So, >> I mean, yeah, they're they're very generous on plans for students, >> right? Right. So I think by default even before the class there's like you know claude license I imagine cursor license there's like probably a cognition license as well. I don't know you might know that better than I do but what I was able to also just acrewue on top of that was hey like if I don't want people to be bottlenecked by like the $20 a month subscription like I want you guys to go if you want to like token max something just to see what's possible then then you should and so and I'll come back to the the partners but the the additional thing I was able to to crew through instead of talking to folks was like additional credits for students. So like factory is also getting involved. The recognition folks are getting involved in open as well. But anyway, so so so for the the open [00:12:06] source partners, I I sort of like was like, okay, one of the things I'm most concerned about when it comes to educating the next kind of generation of of students is that if everyone is saying kind of to Six's point, like they're cooked like you know what's going to happen to them there. There there's almost like this sense of like them being left behind. And I think that mentorship is like a really important kind of piece of just upleveling everyone and making sure that there isn't this like massive gap that folds that that occurs between you know the senior people that have already been in the industry before coding agents and know what's up and they adopt coding agents and they become amazing and then the juniors that are just like okay now what do we do? So in a sense one of the the kind of goals was hey coding agents are really good. I want to put you guys in situations where you're dealing with complex code bases large code bases at scale. We're going to work with some of like what I think are among the best like AI native repos, you know, that all do AI focused things or either coding agents themselves or building tools in the service of agents, you know, in the modern world. I want you guys to get involved with these projects. I want part of your assignments to be actually contributing to these projects. In the course of being involved with those [00:13:06] projects, you will actually be talking to the maintainers. They will be reviewing your code. They will be helping you build mental models of like good software. In that way, there's effectively this like passing on of like the guard of true good software design in a way that you know I don't think most students get the opportunity to do. And then there's the benefit of course that like I'm sure some of them will get you know poached and picked up as part of the doing well in those assignments. >> Yeah. One of the interesting things you know for for what it's worth get a bunch of hate if I don't say this. I think you say cooked and versus cracked for dramatic effect but you know like it's just nice to to state extremes. Um, sure. Sure. >> Yeah. So, so maybe we should go into the sort of detail of the curriculum or is there anything else you want to explore at a top level? >> Oh, no. I mean like that that's uh very help. Yeah. All that's there. >> Okay. >> Can I I have high level questions about students using tools, best practices, are you pro it like individual tutors? Um I'm assuming this will also be picked up by students. So you know before we [00:14:06] dive straight into the course just anything you have guidance on there for >> like in terms of what >> yeah you said you're passionate about education right like there's also a stigma of one side is how how do teachers professors use it and then how can students best use these tools that's not just >> throw your assignments in >> it can kind of oneshot it. How do you still learn like if you have anything there? >> Yeah. Yeah. And I think that there's I still have some assignments thrown in there which I think are not intended to be the hardest things in the world but are meant to be more about like concept concepts like more conceptual things like you know what does a skill mean sort of like the maybe more of the the tactical of some of these tools skills like you know agent ready code bases like agents MDs things like this. So there's a certain amount of just hey if you haven't heard these terms kind of get up to speed on them. I do think most of the learning though in terms of the kind of putting the tools to the test will happen both in in the OSS and then also in the like the final project where it's like very open-ended and they'll have to just kind of explore something interesting. >> Cool. Okay. >> Yeah. >> So, so maybe Yeah. I just want to talk [00:15:07] about let's go through 1 2 3 4 10 >> uh and then we'll dive in deeper. So like breath first basically. >> Sure. Sure. Yeah. >> So we start with internals and coding agents. You teach people how to write their own cloud code clone. It's very helpful that it's now open source and then >> yeah yeah yeah so uh week one it's all internals of harnesses you know kind of build a cone agent from scratch uh week two is more okay now assume that there's how do we start thinking about like context engineering at scale u so beyond just vanilla prompting uh and then I think coupled with that is just um in in doing that thinking about like tool design MCP you know people I think for a while MC was getting a lot of hate I'm still pro MCP I see it all the time and so I think that there's At this point it's a standard that's existed and I think it will continue to exist. So you know talking about that effective tool design uh that's week two. Uh week three I would say is more about just like skills and like what does it mean to like extend beyond simple single purpose tools to more like procedures and that's where we'll talk about skills and you [00:16:07] know that kind of starts leading into like the automation of agents. Um week four is is a lot of just more of like the foundational pieces of how can you customize an agent? Um this is things like agent config files alla agents MD cloud MD hooks um you know kind of sub aents like there's a lot of different mechanisms that exist in modern harnesses uh which you can use for different purposes so I want to expose students to that agent ready code bases this is probably one of my favorite topics it's like and this is especially relevant for folks that are going into like industry and thinking about code at scale is you know essentially what does it mean to have a repo that's ready to be ingested consumed effectively by an agent and so thinking a lot about you know the trade-offs of designing of kind of service level boundaries but also docs and and verifiability and things of that nature. Um so that's week five. Uh yeah week six sort of agentic code review. Um and here I think what is interesting is thinking also kind of extending from week five week five around codebase like agent ready code bases going to week six is more about [00:17:07] like okay in order to like code review is now becoming a bottleneck which I don't know that students unless they you know have done a lot of some of them are master students so they've been in the industry but a lot of them have not seen this but for most people that are using code agents today we're all experiencing something of a new bottleneck in code review so how do we unblock that or sort of unbottle the the bottleneck a little bit um what mechanisms are there there and so verifiability and you know things of that nature and how we can extend that into the code review process week seven uh security I mean obviously I think there was a point so this was in last year's class as well and this this I wanted to reinforce I mean with any autonomous system taking a lot of actions we're seeing a lot of this year especially a lot of things getting hacked um you know not least of which are just some of the the core pieces of software from from versell to you know uh gateways LM gateways entirely so I want to spend some time just like talking about what new attack vectors exist exist in security and and you know how we can mitigate some of them. Uh week eight is background agents. So [00:18:08] this is uh what I want to start touching on here is you know how do you like basically run agents in many different like product service areas. Uh not just in what was traditionally an IDE but also having things that are like based on triggers in your codebase in your repo at you know during CI/CD. uh things that get triggered even when you know based on events things like you know S sur style bots. Um how do you even build a system like uh I think this past year was when a lot of companies started releasing their posts like ramp inspect um there was Spotify has their own version of this you know I think Stripe Minions so all these folks were doing really cool write-ups of like their internal info they were building to do this I wanted to touch on that uh get people exposed to what that is. Yeah, this is um yeah, this one is uh more on like what happens when you're in in a team in in the modern world and sort of like the core pieces of of team level infrastructure. So everything from uh LM gateways to you know MCP portals which is like MCP at scale and then just also how to think about the evolving [00:19:10] roles I would say. So this is maybe a little bit like a soft thing. It's like what what what is now, you know, and the and the the vanilla maybe like not super nuance thing to say is like everyone's a builder, but I want to like go a little bit deeper beyond that and like sort of like what thing is actually getting subsumed, what what roles are actually um either evolving or emerging that didn't once exist. Um what should students think about in terms of defensible skill set that's maybe wasn't true before? Um and so a lot of more like team dynamic type type considerations. And then yeah, this is uh it all kind of builds up toward the toward the factory which I think is top of mind for now um for a lot of folks today. Once we're we're building factories, where do humans still exist in all that? What is the you know where are the new choke points that that are intentionally there around around human involvement things like intent things like uh setting up a verifiable systems and what happens in the limit when maybe 80 90% of code is fully driven by just agents uh where we we kind of filled into all that. So yeah, >> nice solid 10 weeks. Uh I was going to say like you know some of these are more [00:20:11] established patterns than others and I I think people need to know that like not all of this is settled science. >> Sure. >> Uh you know but it is everything I'll endorse everything as like things that people talk about and you should know you just we know the open problems we may not know the answers. >> That's absolutely right. Yeah. Yeah. And I mean think even um I mean you probably have your own version of this. I'd be curious to hear what you think is is still not as defensible but or not defensible but not as set in stone but you know even like the the software factory are we talking about like dark factories are we talking about like these sort of half lights on factories like I think even there's a spectrum there. Yeah. And there you know there's the people that are on the on the very frontier like like the warps and the factories that are that are like building factories and selling them as services and you know even they only claim like 30 some% of even that of of like fully you know genetically driven contributions. So what does that mean in the limit? It's it is hard to say and >> and there's people who went very very far dark factory and then came out saying I was wrong guys right so like uh we covered that on the um >> just be aware that you know not everyone [00:21:13] they they probably believe the things they saying at the time but they have the right to change their mind and they do >> and even Boris recently tweeted that like yeah I am often wrong >> like and like everyone is like because he created clock code is like treating his word as gospel and I'm like no well sometimes he disagrees himself >> yeah And and that's okay. I mean, like this isn't politics, right? This is we're we're still scientists. We're engineers. We're we're subject to I don't claim to be dogmatic, but you know, I just want people to experiment. I think that's like a principle as well. You have to have principles earlier. Just like experimentation. I mean, like you just have to be ready to and often times when I sort of like give sort of these lectures or or talk to people about this stuff, they're like, how do I there's so much going on. How do I stay on top of this? And it's like, you know, there's something as simple as just like experimenting for 10 minutes a day. Like just spending some 10 minutes doing something that you didn't do before. If that's a traditional way that you were solving a problem, there's now a new tool you want to try or a new like framework or something and just spending maybe even doing the exact same problem just with that new kind of thing is is is alone will cover most of the gap. >> Yeah. Uh I will call out so some of your uh for example you have cognition on the [00:22:14] week that you're talking about code review. >> Uh I actually did work with silus on devon review. >> Yeah. Yeah. >> Pretty good I would say like you know it's hard to benchmark these things. There is a review bench. No one uses it. Um, but like qualitatively Devon review is pretty good. >> Well, we use you guys. That's also Yeah. So, so Monaco does use Devon. Oh, and that's that's I mean and Sus actually spoke last year and so I mean Sus is great. >> It's just super super smart. But um yeah, we we use you guys and so I was like okay among among the providers of of kind of code review tools that we and we sampled a lot of them. Um we actually did kind of converge on Devon for a lot of use cases. >> Besides actually more model side these days and then actually it's Walden that you want. >> Yeah. Yeah. Yeah. I Yeah, I was talking to Walden as well. I will also shout out you know from factory because uh you know talking about agent readiness they've been all about that they build signals tooling on that do you guys use that opinions? >> Yeah. So so we do use agent readiness of our own of our own kind. I mean, Eno is is awesome as well. Like I I think they were super early on this on this train and I think when he codified that in his talk, I guess at AIE, what was that like last I forget when it was, but one of [00:23:15] his Yeah. Europer um just fantastic talk and I think it was kind of in a way ahead of its time and so I was like there's only one person that I could see being like the most most appropriate person to talk about this where I like really want to give them the spotlight and they're super excited. >> Yeah, it was funny. I was at COG. we were talking about this stuff and like we're like okay we need to form some opinions about it and then like factory like shipped a full product on it that week that we were talking about it I was like okay you won >> like I mean like everyone kind of agrees on the same things >> and it's just a question of like do you want to turn it into a dashboard >> yeah yeah yeah >> because it's a dashboard >> right right but you know there's there's value and like I think they did well from a marketing perspective jumping on like the the radius report and I've definitely spoken to you know I I will regularly kind of pitch that to other people and and say like yeah you should run reports of your own ind and um you know of course now people have open source versions of it but but I think the the principles are super important. >> Yeah. Uh do you want to go piece by piece or do you have any other things? I I can go piece by piece. >> Yeah. Okay. So the first one right uh you focus on cloud code but then you also have pi as open source sponsor. [00:24:15] What's up? Okay. So so okay this one is maybe a little bit more marketing but but okay um the one thing I do want to highlight and I've made that I also made that clear last year but I'm not a I'm not endorsing any tool. Okay. like like I I don't I I I think that is uh the wrong way to approach something like this. Um there are tools that come and go and when you know transparently when when the class was taught last year cla code was not as popular. It wasn't as like rarely accepted. Boris was only at 4,000 Twitter followers and now he's at you know whatever million. I don't even know if he is but you know like he's at a lot of followers. Um so so Cloud Code has really like emerged on the scene. Um I think >> I think yeah I think we we also underestimated Cloud Code when we had Boris and Cat on the podcast. It was like 2 months old. I hadn't really like I used it once. A query cost me 50 cents. I was like, who's going to use this Totally. >> It was still in beta for a while. But I think it's good on them for how they built it before the model was ready. Like you hear so much build for where the model will go. >> Totally. >> Super ambitious. Like models were not that good. You couldn't feel it. And then ah we see we see that it's where it came >> that it's Yeah. It's funny because when [00:25:15] he gave his talk last year, I remember um you know it was like a very high level sort of set of slides but I remember one of the slides was literally like built for where the model is going to be 6 months from now. And this was like October of last year. And I can see why he would have made that point if if he knew in 6 months where like Opus was going to be cuz that was also what I think was a huge unlock where I would argue was one of the reasons why the curriculum changed as much as it did, right? Where there was certainly this huge arms race around true like autonomy in these agents like longunning agentic coding agent tasks. But yeah, so so you know back to your points like less on I'm not endorsing cloud. I mean you know cloud code I of course I use it and and use other things as well but but here I was more about like the what does a modern coding harness look like. I wrote a blog post that actually uh yeah this one where this was yeah earlier this year where I sort of broke down some of some similarish principles. The TLDDR was that you know you could sort of through like pretty much introductory algorithms level understanding you could just define like essentially a gentic loop that does the bulk of what you would call a coding agent. Um, and of [00:26:15] course there's like bells and whistles. There's no sub agents, there's an MCP and none of that stuff. But if you want to see something that you talk to and then it literally will edit your file and you can do pretty sophisticated things with it. You, it doesn't require more than 200 lines. So I, you know, a lot of what I try to do in education is to try to like peel away the stuff that's like the secret sauce and show that like really at its core. I mean, these are still students and you want to teach conceptual principles. There's nothing more to this than in Stanford there's this class called 106b. It's like the introductory introduction algorithms. You don't need more than one 6B to build something like this. And so there you go. There it is. >> Given that you already taught this class, any push back, any questions, any weirdness that you observe from the students >> on >> what do they not get? What do they ask seem to cluster questions around? >> H you mean like now that this like this particular concept or just in general? >> Uh this concept >> of of like >> of like of like building it from scratch and then like doing doing what you did because this is basically what you needed for pi. >> Right. Right. Right. Exactly. Right. Yeah. Yeah. Right. So, yeah, it's uh it not really I think they're I haven't [00:27:16] received any like push back on on like this idea at all. I think they're actually more just most of them are consumers, right? Most of most of the students do just like end up using the tools. I don't think that they get a lot of exposure um if any certainly in the course work around peeling that onion back, right? And I think um so this was the week of the harness. So, so Tuesday first lecture, you know, logistics, blah blah blah, all that stuff. But then first half is you know the first true like bit of curriculum is the 200 lines of of of like a of a harness and then yesterday when I had sort of the follow-up to that is like we actually took cloud code and then you know you have these like proxy things these like proxy tools where you can intercept the request >> proxy. >> Yes. Exactly. Exactly. Right. to use that uh use that tool intercept requests and responses look under the hood see what's happening there and then then like okay I did say you have clawed code in 200 lines but really okay here's like actual clawed code and so what I literally did is I took Opus 55 so this like you know model 2 days old and was like this is I intercepted those requests here's exactly what's going on here's the prompt here's the system prompt here's the plan prompt here's the [00:28:17] compaction prompt and then we just like went through all of that and sort of showed it to students so now they really really have like a state-of-the-art harness that they can understand Yeah, >> it's nice to get those three levels like students are very much consumers. Here's the baseline agent in a loop and then the real one. I like how you have a video walk through of your blog post as well. Do you see students are on like extremes of some people are power users. They consume a lot. For some people, you could assume that this is kind of very straightforward if you know how agents work, right? >> How do you kind of balance that in the classroom setting? like do you tailor towards the average? >> Yeah, I think it's a bit self- selecting because the curriculum is all here, right? So, the curriculum has been available since I mean, you know, at least it's like a few weeks and and I guess I announced it a few weeks ago, but it had kind of been in the works for like probably at least one to two months. So, if you looked for it, you could probably find it. Um, and I feel like the reality is if students are not interested in this, then they won't like they'll they just won't show up and you [00:29:18] they will they will choose not to show up. What I'm finding that is interesting is there are still definitely students that show up but then there's also just a lot of folks from you know broader Stanford postocs at other departments there's a lot of like interdisiplinary sort of interests where folks that are in you know somewhere in pathology or like you know like medicinal sort of fields or or even like random other engineering fields that aren't pure CS that are coming in and also interested in in kind of getting exposed to this. also a lot of uh GSP students, MBAs um that also uh want to I guess learn how to not have a technical co-founder or whatever the motivation there is. But >> yeah, so I I think by the way if anyone hasn't done the MITM proxy exercise, you should do it. Yes, >> you'll be shocked. Uh cloud code has the longest system prompt of all the coding agents. Something like 20,000 tokens. >> Yeah. Yeah. Although uh yeah, that is true. That is true. Yeah. Yeah. Yeah. Although they've also reduced the size by 80%. >> Yeah, I know. But then they grow again and then they cut and they grow again. >> It's it's a seesaw. >> Yeah. Yeah. No, I think it's interesting. You'll be surprised like [00:30:18] what goes in there that actually sometimes you don't use. >> Sometimes it actually hints at the internal workings of some of the tools. >> Um anything that you found there that could be cool to talk about? >> Yeah. No, I mean in fact yesterday when I was talking about this, it was a little bit of this like one of the things that I thought was most interesting about Claude Code back in the day and I think part of why it really blew up in terms of popularity was their focus on simplicity, right? So in the early days they're among the first to really lean into this let's fewer tools do more, right? Lean into SQL, lean into Bash, lean into like the things that are that are powerful like Swiss Army knife tools and have that be a lot of the driver of capability. Um I think they did a fantastic job leaning to that. I think that's why they they were as popular as they did. I think that when I presented something like the system prompt, it was like well there was a different system prompt about 2 months ago. It is now they have reduced in complexity and it turns out that now what was once part of the system prompt things like how to use even git, right? Or how to like set up tasks or things of that nature are no longer part of the system prompt. It's actually expected to be more kind of captured in the models [00:31:18] like the model weights itself. Um so they've kind of baked in a lot of those learnings more at the model level. And that just raises interesting questions around, you know, the model versus the harness, right? And how do you kind of think through that trade-off? What do companies invest in? What should you invest in to really push the the the frontier of your own capabilities? And so I I think we still even as a as a community, we seesaw between the two a lot. Um >> I think you don't want to fault them for growing a system prompt and then cutting it. At some level, it's also how the model was, right? Uh weaker older models, you need to give them examples of this is good, this is bad. And then obviously you cut that away at some point, right? And that's just part of learning what has been done, right? You can apply the same towards if you're using a local coding agent with not the strongest model, >> you might need to add these examples, right? >> That's right. That's correct. Yeah. Yeah. And then and then of course there's interesting AI learnings to fall out of that which is why you know as the models get better when are you over constraining it by adding those examples, right? And then how you pull away the over constraining uh and that's [00:32:18] I think part of the learning as well. Okay, week two, you are going to talk about context engineering. Repit I haven't seen uh but there's many versions of spectriven development. Um >> what do you would you highlight to a broader audience about uh what you're putting in here that people should know. >> Yeah, this is like the you know >> one hour version of the course. >> Yeah, exactly. Like please condense everything. That's where this is going this >> that's fair. Yeah, that's >> Well, no, it's it's also a trailer, right? It's like, well, if you want more sign up next next year or next fall for courses, you know, this is your 1 hour. >> This is like this is like almost like the quibby of the class. You know what I mean? Whether it's it's it's too long to be a trailer. It's too short to be the class. Is this like what what I guess people will part if it's a trailer by the next time you teach it, it'll change again. >> That's true. That's true. That's true. That's true. Yeah. Well, yeah. So, okay. So, good call it. It's funny. Um the I think that that the main thing that I just want to reinforce here is uh and I see this even when I'm like interviewing folks uh the very few people in industry [00:33:18] even progress beyond like just vanilla prompting right there's like a vent form like hey do me the thing right like do do the thing and and that certainly as the models get good that can carry you a certain distance but inevitably you know and any folks folks I speak to that are come out of like a Google or some place where there's a massive like code base the reality is you have to be a lot more thoughtful about how you're like ingesting in context, right? And so thinking through what are some of those principles and those techniques around how to like point, you know, and I sort of like, you know, the rapid framework that I'm sort of describing here is is functionally similar to like Dex's RPI, right? Or something of that nature where you have this like research, you kind of have a proposal, you sort of a spec and then and then you do like your implementation and then you kind of tie it together with some testing and whatnot. So what I just wanted to reinforce here is that like you should have this step especially when you're entering unfamiliar code bases when you're trying to uh sort of infuse into the context something that an agent needs to understand about a very large codebase. There's value in something that is a research phase and then in the case of this particular sort of [00:34:18] framework that I'm putting forward it's like also then suggest some proposal where there's a few alternative ways of solving a problem once you've done the research. And so this is where I sort of also invite the human in the loop to sort of be the decision maker on one appro one approach versus the other. And then you enter the plan which is kind of the spec. All to say the the the TLDDR is like there is a way to do this that actually does scale out to code bases that are you know millions of lines long. Um and it's not just the vanilla prompt. And I think that's what I want to highlight. If you're going to go work in these big code bases you should use something a little bit more a little bit more involved. >> Anything to add on context engineering? I think that the probably the weirdest thing about how I think about context right now is when you split out to a sub aent >> because sub aents can benefit from inheriting your entire context of the main thread or not. Uh sometimes there's a regular disagreement on on that and then sometimes you want to condense what the sub agent does and researchers put [00:35:19] it back onto the main thread and it's very hacky. I don't like anything I'm doing. So I don't have any answers. All I know is like >> uh when you want to fan out work. It is not just about you have a big spec and you you you know you do RPI you do R repit whatever. It is not one agent anymore. It is about fanning out work and make sure they all have the right context and they all have the right guidance. If you have a review agent reviewing all the other works, you have make that guy have have everything and you always go through this phase of expanding contracts like, "Oh my god, these guys don't have enough context. I'm going to give them more context." Okay, great. And then they're going to overtune to some random one-off comment that you made a while ago and then make some really stupid mistakes cuz you didn't mean it that way, but like took it as the law, >> right? >> Something like that. So, I don't know what the word is I'm saying, right? This is like multi-agent management for context. >> Yeah. Yeah. Yeah. Yeah, that that's absolutely right. There there's like some aspect of uh this is maybe the part that's more maybe among the most art [00:36:20] than science, right? It's like some combination I think to your points. Yes, the fan out for the exploration, the fan out for like the subtasks and then and then the consolidation, compaction, whatever you know you might want to call it. I think being thoughtful about refreshing of the context, understanding when things are getting a little bit too bloated. I think you know Dex had this concept of like the dumb zone and sort of like being a little bit more thoughtful about the dum zone. >> He's good at these equating things. >> Oh yeah, I think little diagrams. >> The little Yeah. Oh yeah. Why did you have him on? >> Uh yeah, well I think he's I mean he helped definitely informed a lot of the the curriculum here and so he's just fantastic I think thinker and all this. Um no I think the diagrams are huge. >> That's useful. >> I would add um breadcrumbs for people that are trying to look into this. It's good to see best context engineering preferences like on your sub agent inheriting cloud managed agents has a version where sessions are separate and it's not inherited. This used to be a big talk topic before reasoning models when agents didn't run as long. Um we had shorter context windows. Everyone was very very managing that precious [00:37:20] context window, right? Uh Lingchain used to write a lot of blogs about this and good context engineering. All this stuff is good for people to know. I think in the student context as well, you can expect the average student to take your class cuz it's a great class and they'll know all these things. So you should know them know them better as well. Um this is a fun space for research as well. One of the papers research papers presented at AIE. They looked at the delta in the cloud code system prompt. What got changed and they like um they they plotted a whole bunch of changes in performance. They tried to reverse engineer it. It's just good research that you can do. There was another one that also looked at context window management and how it affects performance downstream. They weren't the best papers, but like it's quite accessible and it puts out good blog posts, so people that are trying to work on it, you know, I just recommend it's a it's a good area you can easily poke into. >> Yeah, it's a good shout out >> agent skills CLA. >> Yeah. Yeah. I think all this is uh so a [00:38:20] little bit of an extension from single single kind of quote function tool calling like MCP, right? Using interacting with like third party external services and then extending into like what you would call like procedures, right? like how do you automate some set of actions over and over again that you then formalize in the form of like a skill MD or something like that. So I think just making that a little bit more uh giving people exercise in in codifying some of those procedures. I think what I'll also sort of like include here is is a few few kind of fun skills that that I think are are good to use like I think Matt Matt Pox like grill knee is like a good one. Uh and then I there's ones that I also like from more from the educational perspective like Jeffrey Litz uh he has like explained diff which I think becomes especially interesting once you start talking about code review because you know the whole >> understand the code >> understand the code. Exactly. Yeah. you got to have to understand the code and and I don't know if you guys have used that but effectively like you run you know you have your your your kind of the the the diff you've put in your code and then you run the skill and at the end of it you have this not only like kind of a visual representation of what's happened so you can you know kind of skim it and [00:39:20] understand what's what's actually at the core of it but then what I really liked was this almost like quiz at the end of uh at the end of it where you go through and sort of like really reinforce whether you understand what's happening and I think that that level of interactive sort of back and forth I think is is quite nice. Yeah, there's there's a couple blog posts in this domain that I really like to bring up. Uh there's one I think recently which is like kind of there's no situation in which turning your brain off is a good idea like you know and and I think these skills actually force you to turn brain on and when everyone wants to use coding agents turn your brain off. They're just like make me an app and get read my mind and guess get everything correct without me telling you about it. >> That's right. But like a lot of these skills are mostly in the genre of like no no like ask me more questions please. Like >> I think there's a lot more output than people are used to. Like I can type in a oneline prompt and get tens of thousands of lines of code and all structure and >> like look at my productivity guys. >> Um but you know more effectively you [00:40:20] should know what you're working on. >> You can really be put out of touch out of the loop. You should spend more time thinking about what you're actually doing. Uh doing the work and getting the code written is much easier than it was. >> Yeah. Yeah. The understanding there's no no excuse for for sure. >> Sorry. What is a web skill? >> Yeah. So I mean effectively like skills that interact with with like the web itself. Computer use >> computer use. Yeah. Things like this. >> Anything else? >> Uh I mean mainly things that involve like pulling pulling information from the web. I think it's >> Are there best practices on skills that will interact with the web versus >> I >> I think this is more of like a a bit of a a survey or kind of a demo of stuff that I found that was kind of interesting. I think it was it might have been out of was it like a Versel? I'm trying to remember now. I'd have to look back at the the full like reading materials, but there was like one site that I found where I think they maybe it was it might have been browser based actually. They might have like done >> do you remember this? >> Yeah. Uh well, it would be them that this thing. It's just I was like did I miss something? I think this is like a big part of the skills community. Good. [00:41:20] He's being educated. >> Yeah, I think I they like made some sort of because I guess they were Yeah, I mean browser base obviously like >> they like browsers. >> They like browsers and um but yeah, they they had like a good I think website where they showed some of these interesting capabilities. >> Seems legit. CLI is a fun one. Do you have a lot of students or this is their first time in a terminal in a CLI or is it kind of the norm now? >> Uh no, I don't I don't think so. Uh I think that given that most of them again uh last year I would say that some of them had probably still remember the days of of finger on keyboard uh typing uh the this is um most I think students have like interacted with cloud code at this point. >> I mean I I'm curious the split of people that are now just using the apps right cloud code also has a desktop app so does codeex. Codex is primarily an app now. You're kind of not in the CLI not in the terminal. It's a thing that puts off a lot of average people. But >> I'm just random note. >> Yeah. U I would say, you know, I think there's also another thing about skill [00:42:21] that are skills that are bundles of skills where they they sort of cross reference each other. >> Yeah. Like the plugins, right? >> Uh like plugins. And then there's also I guess the art of writing a skill, which is a thing that I've also focused on for AIE. >> I think a lot of these uh are focusing on skill consuming. Um, and you have all these influencers like Matt Pog and what have you, they're like, "Yeah, this skill will save it will solve all your problems." But no, you should write your skills yourself. >> And actually, when you try to write them naively, they're usually bad. And actually, it takes a little bit of skill to write a skill. >> That's absolutely right. That's absolutely right. >> What are Swix's best tips on writing skills? >> I have a skill. I have a public skills repo. I went through definitely a phase of psychosis of like skill everything. >> No, no, no, don't give us skills. How do you write good skills? >> Yeah. Exactly. Exactly. So, I have a skill for cutting skills. So I I I like there so every like uh cloud code and codecs they all have skill creation skills right. So then there's I have a skill cutter skill that cuts it down because they they're all it just documents every way in which those [00:43:22] default skills that are created are bad. >> So you always run this after any skills created to cut it down. >> Right. >> Um so >> like this. >> Yeah. It's really annoying because sometimes there is slop and like but there's good in the slop and then you have to like sort of whittle down what they're what they're >> right. Right. Oh, interesting. As you're saying like even the slop is it's still in the language of the agent. So that the slop means something. It's like it's like a it's Yeah. Okay. Okay. Got it. Got it. >> It's fine. Everyone has their own version. I used first there's very easy to like vibe check a skill, right? You just call it run it a few times and then I distill these preferences into a skill that's like this is what I like this way. doesn't like and it it adjusts over time and then you know it get automated. >> The problem is that yeah you're calling it but the the actual the worst part is when a skill triggers when you're not expecting it to like you wrote it one time to fix a thing and then like now you're working on a completely different thing and it triggers. Why? Cuz it's badly written because the trigger says like oh call me whenever blah blah blah [00:44:23] happens and you're like okay read's good and then you do a different project and it and it triggers and you're like what the hell's happening? You know, there's a there's like a few people that are just like, "Okay, I have a bunch of skills. I keep adding on sometimes when I know some people like Eugene Yen, they are pretty particular about how many skills they have. They prune them. If not used, they clean them every few weeks." >> Yeah. >> Yes. So, uh, so yeah, this is like a a huge problem if you're if you're sort of in that skills world. I think this is actually why I suspect, uh, they're good. There are very smart people at over in anthropic that want skills to die. >> Uh, which is not a popular statement. No, no, they they won't say this publicly, >> but it just feels sloppy. This feels sloppy. And I know it's I I can see my own skills people being sloppy. It doesn't feel >> uh AIA pill, >> right? You know what I mean? >> Yeah. >> It's it's not it's not what AGI was supposed to look like. It's like everyone having a giant bunch of markdown files. >> I mean, I think part of it is there's a [00:45:23] gardening problem. Yeah. some sense. >> A lot of people actually don't look at their markdown files, right? Like the the UX of it of just chatting what you want, creating a skill and chat, using it, seeing output. >> There's a lot you can there's a lot of value you can get without ever looking at your markdown. Don't don't over complicate it at some level for yourself. Uh you can get a lot of value if you do stuff repeatedly. Just be like, look at the last five times we did this. Make me a skill. And you can iterate it with chat. There's a good side of it as well, right? >> There's another thing. Um there's I guess another way to look at it as well cuz there's a there's the use of the skill as just this procedure that the agent kind of runs whenever to you know do like a code review or whatever. But then in the case of something like Devon, right, where we've codified skills as a way of of sort of representing our best practices, right, and then making and exposing that to the agent, the code review system to then, you know, institute the APIs the way that we like to define them. Institute kind of the service level boundaries the way we define them. That's like an [00:46:23] alternative way of sort of representing like org level knowledge and maybe not so much this like arbitrary thing you want the agent to do so you don't have to write the prompt each time. And so I like I like that. I feel like that positioning might that that purpose might exist longer than just u you know these kind of random procedures. >> I just Googled Monaco AI skills and it's giving me the the the country. >> Hey the country >> one day. One day. Well, we're going to beat them on AEO. That's for sure. >> Can you just buy the country? >> Well, here's the thing. Okay. On this we we uh we're about to buy No, we're not about to buy the country. But but but um you know we we did uh we did get into a little bit of flack when we uh bought the monaco.com domain. I will say that uh we did get uh sued by them. So >> what >> yeah there's there's like a whole public there's a whole public thing about this where yeah so they didn't like that but but here we are and uh we won. We won. So >> okay then we're keeping it in. I was about to say this is spicy but you won. >> Yeah. Yeah. No no it's [00:47:23] >> Is that why we is that why we see the flags flying around? >> Because it's like a victory. It's It's a victory. It's our victory. That's funny. >> For for those that are in SF, just look up and, you know, so for for those who are on audio, we're showing this on YouTube. Uh it's you fly a plane around. I think you're still one of the few that actually do this on a regular basis. Sam Blonde came up with it. $6,000 a day. Uh, I don't necessarily think it directly causes me to Google what Monaco is, but that all of Monaco's ads are just Monaco dollar sign, >> right? >> Like dollar sign and then X number. >> It starts the conversation, right? All the the whole point of the ad is what the hell is Monaco? >> Yeah. And and you know, you could you could like that's a debate to be had, but but in some sense like what is the purpose of marketing if not to like get attention, right? And and yes, maybe there's one level beyond this, which is if somebody So, you know, I'll go to events and people like, "Oh, you guys are the ones with the planes. You guys are the ones with like the big number on the billboard, blah blah blah, 193, you [00:48:24] know, million, whatever it is." Um, and and that but they're like, "But what do you do?" And then but that's that's but that's but maybe that's the whole point, right? Is that I just needed someone to come and use it as an icebreaker. >> I will say it's what I said as you walked in as well. I was like, "Ah, the planes." >> There you go. And now, what do you guys do? Well, well, let me tell you about what we do. So, so you know, as someone who's been in those meetings deciding what we do on the the Devon billboards, we spend so much time on the words and maybe we should just be like Devon, which is so dumb. But like, >> okay, you know, look, look, I I think Sam had another post uh this past week where he was just talking about like, you know, I mean, Sam is effectively like our our, you know, marketing sort of mastermind here. And uh yeah, I think there's a little bit of history behind behind the planes as well, but but the TLDDR is that >> his whole shtick is like I don't have all the answers. You just got to try things. And I think a lot of this is we just try things and and if it sticks, great. If it like did we know this was going to blow up as much as it did? I mean, not really. But but now that it did, it's an idea. [00:49:24] >> Yeah. Exactly. >> And minimalism, start small, go high instead of start big, go small. >> Uh way easier to start small, go big. >> Yeah. Exactly. This >> also, you know, it's 6K a day. Like it's it's very much just try things. I think I think people >> cheaper than billboard. Cheaper than a billboard for sure. >> Yeah. >> And it's fun. >> Yeah. >> Not not much harm done. >> But you know it does it's um but you know the kind of simplicity here that that's worth mentioning is then when we're trying to like one up the plane we're like you know what's better than one plane? Two planes. >> Two planes. Three. >> My way to rocket. >> Well okay I also say I've seen >> I had that idea. Uh I I would like for that for that to happen as well at some point but uh we'll see. This is the next SpaceX launch. We just see Monaco >> for the ship to start. >> There you go. There you go. Got to got to warm up to our our SpaceX friends. >> Okay. I'm happy. >> You don't want to >> Well, I mean, we're we're on Monaco. Is it worth talking a little bit about Monaco? Do you leave it to the end? >> Yeah. Yeah. Um >> because this is like an intermission between the the topics. >> You decide. [00:50:24] >> Okay. We're We mentioned Monaco, right? Like let's let's give I I I think it's I think it's a little bit annoying to like mention what the hell is Monaco and then not answer it. Sure. Can you answer it? Sure. Well, I hope so. Yeah. So, um >> this is the country. So, it's a small little uh >> There you go. There you go. Okay. That this is Monaco. This is the Monaco that that I know and love. Uh yeah. So, we are uh you know, we're we're series B company uh here at Basin SF. We are are building effectively like an all-in-one sales platform. Uh fully native agents doing all the fun stuff. uh or at least the stuff that humans don't like to do but but that agents are do like to do because they don't really have opinions but do in a less airprone fashion so the sales people and sales teams can focus on doing what humans are good at which is you know closing deals >> so presumably Mark Beni off doesn't like you guys >> probably not >> but he's like also honestly a very AI native person very aggressive and agent force and everything [00:51:24] >> what are you doing that he he doesn't do or he cannot do >> well okay so so right now we or uh we started the the company in a sort of like the the market that we went after was not enterprise from the get- go, right? We sort of said like we're going to start by like if our goal is to build the all-in-one, then we need to kind of build this in an iterative fashion with with a group of customers and we need to like grow with them into this. It's much harder to pull someone away from from Salesforce than to say like hey if we meet you at the point when you are still just a Google spreadsheet tracking pipeline and we sort of show you a you know more AI pill sort of answer to the problem then we can grow with you and in a way it kind of follows the same the same uh model of like you know BS or all the like YC companies where you kind of sell to your batch mates you kind of grow with them and and you then you go up up up market and so we I would argue are the you know the the I would I would claim the among the best if not the best sort of like startup focused uh sales platform and so all of our customers are like early stage startups at this point probably through to like series A is is kind of where where we've primarily been focused kind of seed to series A and [00:52:24] then now we're going more up market and we've kind of learned a lot about the all-in-one in the last year and a half of doing that >> and you are the head of AI. >> Yeah, that's right. >> What's some cool AI stuff you guys have built? >> Yeah, I mean uh we have built our own agent harness for for sales use cases. So all this I mean to call back to the to the coding harnesses uh taking a lot of inspiration from from the systems like the codexes in the clouds of the world and sort of building bespoke tools that are very sales focused bespoke workflows that are all kind of geared toward GTM use cases. Um so we built our own version of that that powers pretty much every single thing that that powers the Monaco platform. So from you know uh scoring of scoring of TAMs uh scoring of ICPS and and develop defining of ICPS uh through to pipeline evolution through to like all the mundane kind of things that maybe a salesperson doesn't want to do which is like oh update my meeting you know premeating notes update my summaries like all these things. Um you know we have agents powered powering all those workflows. So we we've done that. [00:53:24] It it's it's kind of comes comes out of the box. Um and uh yeah, so long long agents long horizon agents for long horizon agents and short horizon agents some cases for for for GTM use cases >> and it's pretty domain specific built on foundation models. Would you do any of your own models for this? >> Yeah, it's a good call out. Uh because so far I mean I think this is generally true of a lot of like vertical AI companies is like you kind of start where you sort of use the frontier models like as third party services to see what's possible cuz they are like the very frontier and then I think we've seen this I guess in the past week with the whole like Harvey sort of where they like the plus 50% to minus 50% margin kind of question. I mean know I think irrespective of the numbers I think what is interesting is like at a certain point things do become interesting from a cost perspective you you start to wonder can you do it faster on you know open weight models and then that a lot of that then a lot of what I think the the purpose of of the AI pieces are is just to post train those models build environments for you know kind of to post using RL and things of that nature so um we're kind of in that phase right [00:54:25] now >> fun yeah I've been calling this uh playbook the agent lab playbook but that is kind of my coining of like you know like agents start from the user first and then they uh they they use whatever models you need to to build products but then you can backfill the the model side once you have the user base and then the model labs go kind of go the other way they they start with the big pre-trains and then they try to find users >> um and it's like very you know it's hopefully working as a as a as a split and Monaco would be on the sort of revenue GTM side >> that's right yeah >> um and I think it's interesting uh they would want to serve you while also trying to compete with Mhm. >> Uh as as model labs, >> that is correct. >> One of the things that I've been I've been I I dabble in in in some of this. I think one of the hard problems is actually enrichment. >> Mhm. >> Uh I would say I spent a lot of my time doing that for for basically SEO and like uh research and what have you. And I imagine for you it's actually like a core part of that. [00:55:25] >> That's right. >> Do you find that models are good at it? Uh do you have any vendors that like you like working with? You want to give them a shout out? Whatever. >> Yeah. Yeah. Absolutely. Yeah. So we we have used we were early adopters of Parallel and then I think they're very strong. >> Okay. So just general search. >> Yeah. I mean >> not like Zoom info or one of those guys. >> No. No. No. Um but yeah, we use Parallel. We also use Exa. Um so we'll give give a shout to both of those folks. >> Can you compare them? >> They're both friends. >> They're both friends. They're both friends. They're both folks that uh that we like. >> They'll be very interested if you like one of the >> Yeah. I'll keep it I'll keep it neutral. I'll stay I'll stay Switzerland in this today. But yeah know so so and then of course we do our own version of enrichment as well. So we it's it's like a most of our data systems I mean part of like one of one of I think the most valuable piece of software that we build or most valuable systems is current like our world database like the database of companies enriched companies enriched accounts uh enriched contacts. So like that whenever we are constructing ICPS for folks, we have the most up-to-date information about who you should reach out to at which company at this point in [00:56:27] time. Like I would say that one of our like I think where we are like world class hands down is just demand. Like we are very good demand company. Like we help companies increase their meetings booked in a way that actually >> find buyer at >> find buyer at the right time, right? And it's kind of like combination at the right time. Like there's there's times when like here's an example like if you are um if you are targeting someone that just you are selling to um VPs of sales let's say okay like the the right time to to reach out to them is not when they are like sort of in the middle of some like they've been at the same place for the last like 5 years. if they are now in the middle of like a job like a good signal is like once if they've gone from one job to a new job they're now like entering a new place where they're more likely to be to purchase or be open to a purchase of a new sales platform because they're in the middle of the job transition. So >> going from company A to company B for like sales a good signal of like hey should you reach out to them at this point in time is did you detect that their LinkedIn changed or something like that. How much of this is your expertise put into model and harness versus models are good there's generic best practices like is there there's obviously some [00:57:30] value in both >> um is it just the glue getting it reliable doing it all in one platform doing it well >> it seems like there's also >> stuff that we wouldn't know that you guys are also domain experts in that's baked in >> yeah I think without the former it would it would be like a fully commoditized I mean not fully I mean because there's still a lot of value in the all-in-one. I think like that is something that we definitely pioneered like most companies in the sales space started as just like point solutions and I think they're they tried more of like the YC approach of you kind of land and expand. You kind of pick a niche and then you expand from there. Uh, and we from the get-go, I mean, that's why we stayed in stealth for the greater part of a year and some months, was just building because we knew we want to be the all-in-one from the start. But, you know, that alone is is powerful. It's meaningful. But, I think the real power is the fact that today Monaco as Monaco has like a world-class go to market team, like a truly like world-class go to market team selling Monaco, of course. But we also have the same go to market people that sell Monaco are the ones that are like what we call for deploy account executives. They're FDA. They're effectively like well I say human human sales people that are like the [00:58:30] consultants of sorts that help onboard all of our customers get them like building agentic workflows and are effectively there to like help guide them through every part of their own sales go to market life cycle and so to your question it's the all-in-one yes but it's also the fact that we have these like you know SAM and and you we we have brought on a lot of like former Sam Sam's team members and stuff like that who were like the best people in GTM at Brex and other places so um we're codifying all of that in the platform Yeah, interesting to have you talk very differently than a normal head of AI might also talk to because you're also a former founder, right? And like you're you're also think used to thinking about these things in this way and it's like interesting to it's important to have that in your role. >> Yeah. Cool. Uh I I I I think you you've introduced Monaco enough there and and obviously I think as the Monaco the story develops we can revisit. >> Sure. For sure. Um maybe the bridge back is also what is coding uh or agent native coding or modern software development like inside of Monaco. Um [00:59:31] you know I think you're when you're teaching you're maybe trying to be here's a lay of the land but maybe inside of Monaco you guys are more opinionated on one thing versus the other. >> Yeah. Yeah. No I mean we have so we we made investments in coding agents from basically like day one and we've like learned a lot. we have our own like scars that we've sort of like picked up and and learnings and I think we've now been getting much deeper into sort of sharing some of our learnings. So like in our blog which might be worth sharing here we've been increasingly posting some of what uh maybe not that so that one is more of like a product the one at the very top but this stuff I would say like yeah when AI gets that spread in before like scaling contributions builder buy S. So you know we spent a lot of time looking at different providers looking at vendors and ended up building a lot of things in house. So for us something like a background agent platform we have our own version of like a ramp inspect um that we call monocoder um which is is built on the coder platform um but it is what we use to essentially scale up our engineing product >> like coder.com >> coder.com yeah >> oh >> yeah we actually yeah have sort of [01:00:31] adapted it to some of our use cases and yeah it's where a lot of our agent development happens it's just this really well-designed piece of software for essentially isolating of of sort of like um development environments scaling out to multi- aents um you know being able to run these long long horizon agentic coding agent tasks. So Monicoder, shout out to John Tracy and and and Co for for building that out and and then yeah like from there we've also built our own versions of like a you know like like a slack bot which is call but um you know effectively how is like this part AIS part like customer success part like uh kind of QA engineer because it it it lives in our Slack but it serves all the purposes of helping us debug things and put up PRs when things are broken but also used by our own sales people to find out how the product works, find out like how to get to the bottom of something. And so there's like interesting questions we've had to answer around the different personalities that HAL has u depending on whether it's talking to like a saleserson versus like a develop developer. And so systems like this that [01:01:32] we built out over time a lot in the last like year or so. Uh shout out to Yuna for for Hal and and John Tracy as well. >> So uh you're building a lot in house. >> Yeah. >> Uh you also use other tools. >> We do use other tools. Yeah. So what is the line? >> Yeah. So >> by >> was that Yeah, the builders by it. It's a good question. Uh and we >> I think there's this um uh like a thing a general principle that that I generally which is like you build to learn and you buy to keep or sorry you buy to learn and you build to keep. Yeah. The other way around there. Yeah. So so you you buy to learn and you build to keep. And I think that's generally true here where in the early days when we were still trying to figure out like you know get our feet under us a little bit we were you know code review is an example one of them where we spent a lot of time looking at different code review platforms. It turns out today we still use Devon because we've actually still find a lot of value in how the systems been built. So you know shout out to the cognition team. Um but then along the way we did try out some like S sur tools and you know we tried some sort of like offtheshelf maybe more like background agent platforms decided that it was just [01:02:32] given our infrastructure given how we had set up our own cloud platform it just made more sense to kind of build this out internally of course we're not dogmatic we we are open to revisiting them if if something comes out but here I felt I think usually we ended up building in house when we felt that there was no like mature offering out there so code review is maybe an interesting example because we did do a whole survey of code review tools like maybe over a year ago and at the time >> you can shout out like code rabbit right >> yeah code rabbit and and them and you know that there was also like code graphite graphite graphite for a while before they got acquired um like ellipsus and which like a y company before the fact so but there's a bunch right I mean so so there were a lot of folks that we did try and they were okay but I don't think they had like crossed the threshold of like they're immediately useful yet and so then we kind of like we sort of like put it aside a little bit this was you know I'm saying like well over a year ago then we kind of went back to this and sort of organically I mean, again, shout out to to the to the Devon team and and cognition folks where we tried it again. I think um I had sort of like gotten that somebody had told me like, "Hey, you should try this and it seems like [01:03:32] it's gotten a lot better." It might have been actually during Silus's talk during the class last year and it was just good and it it was organically like like >> purely vibes, right? Like this is there's no way that you >> there was Yeah. This this was like I mean Okay. Okay. Here here's what I'll say. When we did that first evaluation the first time around where we didn't really pick any vendor um and we were like there's no vendor that we're super happy with. I did do like an analysis where I looked at our PRs basically did sort of like a you know set of pie diagrams like here's what this is solving. Are we are we happy with the kind of the trade-off of verbosity and how often it's posting with utility. Um and so there was it was more statistically driven than just vibes. The second time around I wasn't going to push hard on it. The first time I was kind of like hey we should adopt an AI code tool. Second time I was like no I'm not going to do that again. Um but then organically people were just like, "Hey, Devon is like good. Like can can you like add me to this?" And so today people started just like getting organically added. So this is a vendor where you know one of the cases where we're still leaning to the vendor. You know we'll see in the long term what that looks like but um everything outside of that we just didn't feel like maturity was there. >> Yeah. And it's not painful enough that [01:04:32] you would like rip out Devon and write your own. >> That's right. >> That's interesting. Yeah. >> So good good talk for Devon who I didn't even know. Uh you know for the record I'm only an adviser. I'm not that involved anymore. But like it's interesting to to hear about it externally because internally uh there is a certain amount of let's call it um the the bubble where like on Slack you're you're like you only hear positive things cuz people only want to circulate positive things. Uh but like you have no dog in the fight like you just choose the best tool. >> I mean if you want to get a Slack free credit that's totally that'd be cool too like that. Wouldn't I would say no one would say no to that. >> I think Silus I think Silus will hook you up. >> Okay. So, so, so there's that, right? Um, and then, uh, and that informs all your teaching work and, you know, I think that's that's really good and we can bring in more of this as, as you like, uh, because I think this is this lived experience is very good. >> Yeah. >> Um, uh, I'm happy to go back to agent ready code bases or board like so so week four and five >> kind of uh, you know, matching. [01:05:33] >> Yeah, they they blend a little bit. Yeah, >> blend a little bit. That's right. whatever you want to whatever strong beliefs you have introduced in Monaco. Yeah. Uh and recommend people to start with. >> Yeah. No, for sure. Yeah. Week four and five I would say they they are a little bit um four is maybe a little bit more like uh conceptual foundations and ways to do this cuz there are a lot of ways to customize agents. And then I would say week five is where I really kind of like lean into all right let's think about this and apply this in practice on like a true codebase. And so here I will definitely, you know, give a shout out to to the factory team and and uh and want to just highlight the value of something like a readiness report. I I think that the main piece that I want to uh reinforce something like agent ready code bases is to me like a precursor and a bit of a building block toward everything to come afterwards. When I say everything to come afterwards, I mean like people always got, you know, for a while and I think people still are like super hyped on like multi- aents like paralyzing agents, like tens of agents doing all kinds of things. It's like you actually need to set a foundation in order to do that well. And the foundation is how verifiable is your [01:06:33] codebase? How like well structured is your codebase? How understandable is it by an agent? And so I think like agent readiness to me is like almost almost like piece one of the next four to five weeks where maybe until like week eight or nine where where I spend more time talking about like broadly uh you know AI software teams and like kind of or global structure but definitely building up the the building blocks for for uh background agents the software factory. I think this is where it all kind of gets interesting from from that perspective. >> Yeah. One thing that I observed you talking about for Monaco, but it's not here is the S sur side. When you close the loop, um you actually want to have the agent read your logs for builds and then put ideally read your analytics for live production. >> That's right. That's right. Yeah. Um not as not explicitly, but it's a really good shout out. So, I think that I'm going to weave that in in like two places. One is in the background agents uh week and then also in the software factory week. So absolutely right. Um I do it's a really good call out this idea of like trigger/eventbased closing of the loop. Um which you know now is one of the one of the probably two or three [01:07:34] entry points when people talk about agents kicking off tasks in a software factory. Uh that's absolutely right. That'll come in there. >> Yeah it's interesting that people call it trigger or event based which I guess that is the actual technical implementation of it. >> It just seems too lowle almost. Mhm. >> Uh but so so the trigger is usually a chron job, right? Like let's be real. >> Sure. Sure. Sure. >> Uh event based is usually a chron job that like surfaces an event and then it's event based. But it's it's all crons. >> Yeah. Yeah. That's right. That's right. Yeah. Yeah. Yeah. Or like web host or something like that. But yeah, that's absolutely right. Um yes. And and you know the technical term of course because at the end of the day Yeah. Uh being sort of like a CS person by by by trade and and yeah, we we called it those things. And so I guess that's I fall back on that terminology. Yeah. And have any AI sur companies made an impression on you? >> So, okay. I mean, like I think the Resolve folks are doing well there. The Resolve is doing well. I would say Night Shift is also doing quite quite a good job there in terms of broadly like [01:08:34] background agents. I'm a big fan of kind of like they're greylock backed. I think they're doing doing well in this area as well. Yeah. I I would say that where I think SR is an interesting use case and this might back to the build versus buy question. I think what's interesting about it is like where do when does it become like where does where does an S sur an as like really see the most value? It's typically when you have like a tremendous number of like service dependencies and these like massive sort of like systems and tons of Kubernetes and Helm files like you know just like tremendously large Kubernetes infrastructure Helm that's like files kind of strewn everywhere around. I think that's why um resolve has done really well in like the enterprise, right? Where you go and you sell to Ford, you go and sell to like a Walmart and you kind of say, "Hey, you have your service level dependencies are crazy." Um and you see these these like, you know, these diagrams of how things are set up and then they kind of go in and they're like, "Well, we we'll we'll try and make sense of all this." I think that like at at smaller scales, it's usually still manageable that you can kind of roll your own. But, you know, I will say that for the enterprise, I think they've done a good job. >> Yeah. for for people who have never [01:09:34] worked at this scale because it took me a while to uh to to to to see this it is on the order of let's let's call it tens of thousands of repos more repos than developers. These are the situations where like you don't know what code is running >> any anywhere in the company cuz probably the guy who wrote is no longer here >> right absurd and like by the way it's still low bearing. >> That's absurd. That's yeah that's that's the scariest thing that that's absolutely right. It's it's like it's and and then you're you're the person, you know, you're the person who's on call and you're interacting with a system that goes down where that person still isn't there and there's like old out of date runbooks and you're like, "What the hell is going on here?" It's a scary scary thing. And yes, you're at the at the enterprise level, it's like it's a whole whole different beast. Yeah. So, so like the the the kind of like college hacker, they you know, one person per repo, they've never totally no appreciation. But like I think I think if you just give that idea like you drop them in like a simulation of like okay like day one on your job this happens what now? >> Yeah. Yeah. So so another shout out I'll [01:10:34] I'll give here. So the resolve folks they actually last year they they made an assignment for the class where they effectively created like a sandbox for students to go and then the sandbox was like strewn I mean the within the sandbox there was like many many services that were doing their various things and there was even like bugs in the services uh within that sandbox and then as part of the assignment the students had to like go and actually uh like use the resolve agent and kind of like probe and play the the human on call person probing the system discovering what's happening Um, but >> they've got an interesting interesting website. >> I do not It's been a while since I've seen this. >> Yeah, this is very interactive. >> Okay. They're they're being a pager duty now. >> Yeah. Yeah. Check this out. Yeah. But also this like this this uh video I was uh >> you know you were shouting out interesting interesting but okay. >> Yeah. So okay you know we never get a chance to talk about this side of the world except for conversations like this. any other segments because I think AI SR is maybe too broad or even too [01:11:35] narrow uh depending on how you look at it right like too broad meaning well there's several subdivisions of AI sorry that's interesting >> um or too narrow being like well actually that's not even the biggest problem that we have >> so take it however you like >> yeah uh you mean like specifically for for like Monaco or just like broadly for like organizations thinking about this >> broadly because that is our audience >> take it >> yeah I I think that it again it's it's a function of how your scale Right. And I think that for for the enterprise, they there are a number of issues that aren't always I almost feel like the answer is not even always just more agents. Uh and and since we're still actually on the week five agent ready code bases, like when I talk to companies of like a certain scale, the the things that are holding them back are really more just like low-level like DevOps CI/CD stuff, right? They're like their their merge cues are getting backed up. They it takes too long to like stand up an app. because there's like actually more foundational things that have almost nothing to do with that like throwing in one more S bot or Allah you know a bot wouldn't be the answer to the problem because there's like more systemic [01:12:35] things that they need to figure out and so uh >> so that brings us to we >> yeah exactly right so that's that's where we get into yeah exactly right yeah there we go to the merge Q and beyond um yeah this is a yeah a love letter of sorts to to modern gentic code review um yeah I think that the the the biggest thing that maybe happened between last year and this year was it suddenly became very clear that code review is a bottleneck especially once you people got really really AI pill and and there's the velocity on generation became so high that you know you could in theory like put up 30 plus PRs a day uh and then what like what's at what point do you decide that you either have to institute yourself and put yourself in the middle as some way to to gate and review things in the traditional way but then you're you're literally the constraint you're the choke point of the velocity and that's not good right And then what was the point of generating at scale? Or you say, well, how do we maybe get the best of both worlds where we're instituting ourselves enough that we have trust in what's getting merged, but we're also uh able to kind of like merge [01:13:36] at scale, but we're also not kind of like free freewheing it here and anything goes. So a lot of I think what I want to touch on here is just that that middle ground, right? which is like there are there is a class of problems that if you've put in thoughtful design into your agent readiness, you put in thoughtful design around your verifiability. there's a class of problems that you can feel comfortable you know some percentage of the time essentially like auto approving and what has to be true from a kind of a design perspective from like an architecture perspective to just auto approve and when do you have to insert yourself and say actually here I still want to review this carefully and even if you do review it carefully are you still doing line by line review or you doing something else right >> so yeah some of this >> we one of our hottest topics of this year uh we have a guest writer program for latent space uh this is from Ankit from Aviator and he has a whole thing on killing the a review because I you know that that that is like maybe the final boss >> uh of of that's that's a bottleneck. He has a few >> uh gates by which he does this. I don't know if you you've talked to Vankit or >> Yeah. No, I've read this post. Yeah. Yeah. Yeah. I haven't spoken to him [01:14:36] personally, but I' i've definitely read >> mostly right any criticisms? >> No, I think this is right. I mean like broadly right. Um I guess you mentioned one thing like the last boss. Um maybe not I think there's two choke points. There's the review and there's probably like the intent or kind of the thing at the top, right? like this is >> and there's some there's some stats as well that that I like to pull up. You know, everyone's stats is different for sure. For sure. >> But like clearly more people are merging. Clearly the number of reviews are not increasing in line with >> uh with merge. >> That's right. That's right. And if you look at any modern uh sort of company that's that's that's really invested in this and thought about this carefully and some people have already done posts about this and you I'm sure we'll do posts about this as well. Um but I think recently like the amplitudes and um I think Cloudflare has has done a good one. I think Augment has also done a good one where they talk about the design of their AI code review system. Um these are a bunch of really good if you want to share some of those links. I think they're fantastically written posts. Um yeah, so shout out to I actually um shout out to to Cloudflare because I think like um very like kind of like you know in a sense like legacy company or like you know bigger company [01:15:36] but I think they've done a fantastic job in terms of just their their content around the AI the AIS DLC. Um so I think they have a good post there but TLDDR um yes I think this is true. I think that there's we're still fi finding that sweet spot. What is auto approvable? What is not really auto approvable? And if it is not auto approvable is even in that world of something that has to be somewhat human reviewed, what do we review? Are we still doing the line by line or are we doing some some combination of the spec and you know I think this yeah again a whole separate question there as well. But are people keeping uh are you guys keeping the spec up to date at all times? Uh the spec is no way, right? >> Yeah. No. Well, I mean, okay, in a few different places, right? So, like there's a spec that as it exists in like a linear issue and there's a spec as it exists in like the context of a PR description, right? So, you can like you can have your peer description be the place where you like describe more or less the entirety of the implementation in a more condensed format. And as long as that's the most that's the most upto-date thing representing this point in time of the codebase, I think that's [01:16:37] sufficient actually. So, we're thoughtful about like our pure descriptions. We make them be not just like, you know, a one paragraph. It's more like it is effectively like a spec of what has happened in the PR and there's even like kind of line um file level pointers to things where things have changed. So you not only get the high level of what's happened but even the kind of the the construction of the files and sort of the service level boundaries that you're touching. So you know I think that's where we we've invested quite a bit. So I push back on that because to me of course it's you know bare minimum it should be internally consistent with the PR spec. >> Sure the no what I meant is the spec for the feature the codebase because that is the summation of all PRs that that is like if your if your PR spec got it wrong or missed something it's going to be the delta between what the whole codebase is supposed to do versus what your PR is trying to do. >> Mhm. So, so I I mean I don't expect you to have an answer because like I really stress about this myself. I try to instruct models to do it. They're bad at it. >> That's it. >> So, so the humans are the the spec gods [01:17:37] of like well five weeks ago I did this other thing and now you're reversing it. >> Have you seen the um the post from Basis? Like they have a they had a tweet tweet little thing uh tweet article. I thought they did a great job talking about um like different sort of file artifacts. they they they track via git and some of them are like the specs the intent of what is to be and the other is like what they call like the canonical state of the code which is like a little bit more maybe what you're describing which is implicit knowledge implicit intent that also has to be codified somewhere right so that part that's all version controlled and I think they found that to work for them so that might be another thing to share as well >> is trying to pull it up this is basis the the accounting company >> the accounting company exactly right yeah we're um we're having Mitchell at uh my New York conference soon talk to him about that I like that they are coming out with this. My solution for this is um I don't know if like how ready I am to talk about it cuz I may never ship it. Uh is something that is replacing git in order to capture all of this. [01:18:37] >> Uh so I call it a program database uh where it captures all traces and all and and raifies specs to something that that we that we can ignore and keep uh and it also controls the skills that are attached to it uh as well. Because I think like basically what git is only focused on code and then you just need a source of truth that is uh not only uh everything else outside the code including the the implicit what what do you call implicit assumptions or like the constraints the intent the >> other stuff uh but also multi- like highly concurrent because git is git by intro by design introduces mutxes everywhere which sucks >> uh so so you need like super concurrency. Have you seen um uh it get Butler is it? There's like Yeah. The guy who used to be CTO Ian I think is his name. >> I forget his name but yeah >> Scott. Scott Scott Scott. Um so yeah so I think there's there's a lot of people >> going after this. Um and I think there's the there's the question of like really what is the ideal representation of tasks? Yeah. >> And I think the winner [01:19:38] >> probably not me linear >> really >> cuz like that's where the the task are. >> Yeah. Uh and like clearly they have decided to become a a coding agent company, >> right? >> And well, you know that they have all the source of truth there as far as you as long as you buy into linear. >> Yeah. Yeah. That that is true. That is true. I I wonder I I have like this somewhat like kind of held belief that I think a lot of companies are going to evolve into some version of a coding agent company which is any company that has like an interaction with B2B software where you're collecting something about the user or some like product insight. I think you can also close the loop because ultimately like what a coding agent is the purpose of coding agent is to act on some adventure thing that's happening where users involved or customers involved and then use that to like fix ideally some aspect of the product right I think what linear is doing I imagine the same thing like an amplitude could do as well right because they're also getting insights about user analytics user problems I imagine even like you want to really stretch it you could even have like a decagon or serum do something similar where you know here's what's broken go close the loop >> yeah all right so set all those things [01:20:39] up for agent ready codebase and set up code review. Security is week seven. It's not week one. >> Students, >> what what do you mean security is not important? >> I make fun because I always say like uh you know at Amazon job zero is security and they always say this. >> Yes. Yes. >> Anyway, but yeah, security review at the end. Okay. >> I was at Amazon and I don't remember that but I I believe >> this is the thing Andy Jesse says. >> Okay. Okay. Okay. I see. I see. And that was I guess he I I was there partly during Jeff's time and then partly when was transitioning to Andy. So, but yes, I I think um this is a thing a topic last year as well and it's it's also one this year and I think that there I um Isaac also spoke last year. I find that he's an incredibly thoughtful thinker uh thoughtful thinker, you know, a thoughtful thoughtful leader in sort of the space. I think Sam Grup is an interesting company because they've been around even before sort of agents, right? I mean, the the OSS repo was exist well before kind of traditional SAS kind of defense mechanisms. Um but I think they've done a good job of pivoting and or not pivoting augmenting [01:21:39] their offering into this space. Um you know I think it's it's um it's an interesting time right uh there's just a lot going on where these these attack vectors are now kind of coming to the surface. there's a lot more slop and AI generated code and I think maybe one of the core insights that I want to drive here is just the idea of like dependency kind of dependency tracking and supply management basically like supply chain right um you have to be careful about like your dependencies and and pin things appropriately and don't just arbitrarily hit npm install and expect the chain to go >> yeah so potentially the most secure codebase is one with no dependencies >> yes right that's correct you you look at the dependency you're like cool dep cool implementation pro, but I'm just going to make my own. >> That's absolutely right. That's absolutely right. And and that's an interesting insight because it's sort of to the limit. And you know, I think it's it's probably easier said than done, but if you believe that with code now being commoditized, code being easy, cheap, why not just build everything in house, right? Like that like of course that's and that would be the safest thing because you've seen everything, you [01:22:39] control everything instead of, you know, depending on Axios or whatever. But, you know, we'll see how many people actually act on that. But that is the safest thing. I agree with you. No, I so that statement is dangerous because then you would just make your mistakes instead of making relying on other people to discover mistakes. The whole point of open source is that you know what is it like uh sunlight is the best disinfected. >> It's like linville's thing and it's it's like well yeah you will be very slow to discover your own custom software dependencies cuz you only have your users and they won't report it to you. But if this this thing has 100,000 users worldwide, someone will find something >> and then you can depend on it more, right? >> H >> yeah, this is this is there's no basically I I don't I'm trying to be a voice of like reason and you cannot be absolutist in both and uh you I I kind of trapped you there. I tal I talked to you into the talk track of >> vendor everything. >> Right. Right. >> And there is a bug there. >> Yeah. Yeah. Yeah. It's an interesting one. I I wonder how much that that how much is that I I buy some aspect of that [01:23:40] but how much of that is impacted by the fact that everything I I have this belief that like most OSS repos if not all of them will just become soft for factories that are effectively agent run in the future. what happens then where like >> I mean even um like even if you look at warp right like warp which is a monocoding agent of course they're also you know selling a software factory product but but they are running their warp open source warp terminal as a software factory and that almost suggests that there's a part where yes humans are still involved but increasingly less so certainly not every single person is reviewing every single thing that comes in to the maybe the same exact level that we did in the past does that make this code base more or less vulnerable to securityurities it's a bit hard to say Um, I think that's just where there's a little bit of nuance that I think we're still and even the folks that I know that I that work in OSS are trying to figure out exactly what that stable point is. >> Yeah, there there's a couple of related arguments and then I'm I'm happy to to keep going like security is like a non-stop like huge huge huge huge surface area thing. There's one argument from Malta of Verscell who's like okay guys let's take every single like high [01:24:42] depend highly dependent on software that's written in C C++ port it to Rust and it's like we'll be faster more secure uh and like it it won't probably take like $6 million and but you you you'll get it done and and then there's there's another argument for this this also links to pacing the frontier from the the open source from the model labs who are like okay let's slow down so that we or like we'll do Daybreak, we'll do what's glass wing, we'll make sure everyone's patched and then we can let loose on rampant acceleration. Uh so I think like there there's there's there's an inherent belief that the models will fix all security >> and I think it's just unproven to me because what if they introduce new bugs >> and they're just different bugs? >> That's right. That's right. That's so so I I mean well I'm looking for push back. I'm looking for like well no you're you're you're an idiot. Like obviously your models are way better at this. >> Yeah. Uh I I guess like that's almost like in the limit there there's the [01:25:42] question is like is is a model do we believe a model will create perfect software right? If a model is creating perfect software then then you're done. Is there such a thing then then that's done right? Then then we have then we're done here at all. Like then then we're just going to go home and and and chill on a beach for the rest of our lives. >> I think there's also the question of is it good enough to first give it for security? to let people patch then just delay the release and do it after or is there a problem of >> oh no it'll be much better at finding vulnerabilities you can full send test it for longer even if you've patched or like >> how long right you can give it for 3 months someone can try to exploit it for 6 months the problem's not solved right >> right right yeah >> yeah my whole problem with like and again like this is a whole rant that we don't need to go into but like you know you name 100 bless organizations that like okay these are like priority they they have access to a to a cyber model that uh you've just created 100 attack vectors for me to go compromise and like they're by the way they're pretty easy to compromise. >> Yeah. Yeah. So I don't know anyway um [01:26:43] that above our pay grade >> is fun to get juicy in you know security like >> I mean security is right now very juicy right like and like I think actually the security folks love it because they like they usually don't get that much attention and now they have like a ton of attention. >> Is that true? like I I mean and that they don't get attention. I find that um I find that at least from like a VC perspective, people love VCs love investing in security companies and have like pretty good >> like a whiz or something like has done very very well. >> Whiz is arguably whiz and socket. Uh I literally I had this conversation with VC two days ago. Apparently whiz and socket the only security companies in the last 5 years that have worked. >> What about like abnormal? >> I don't know what it worked for a VC point of view of >> I see from like from like an economics perspective. >> Yeah. Interesting. No, not not for me to say, right? Obviously many like great players in in in that field, but it's it's a tough tough industry, I think. And so basically like the best time but I think it's not controversial. Best time for any security company is when an high-profile incident just happened. [01:27:43] >> Yeah. >> Right. Obviously like then obviously you're going to sell a bunch of security. Totally. >> Yeah. So so I I think I think that's a fair thing like people people should be aware. People should know the the basic mitigation mitigations. I do think like code scanning is important, but like I I just want to give people the the idea that like no models introduce bugs that you wouldn't make as a human and like if you're not checking those bugs then >> it's not even that they make bugs, it's that you have less awareness of what's being written, right? Like when >> Yeah. reviewing everything. >> Yeah. Yeah. Forget bugs. Models are pretty good. They'll have on average, I would say, less bugs. The problem is also that you don't even know what you're putting out like >> Yeah, >> there's a lot of that. >> Yeah. And that's I think we have to be thought about mitigations and and like the and there's sort of a combination of security and and security as part of code review as well, right? I mean think this is also like doing those checks I think is important. >> Yeah. So so I think um maybe I would say one more thing here uh or or check your [01:28:43] check my intuition on on this thing is that uh not all parts of the codebase are created equal. the parts of the codebase where security is very important is data storage >> um and uh billing. So let's call it that data billing uh off and off is obviously related to all those things and those are basically don't vip code and then everything else go nuts. >> Yeah. >> I think that's absolutely right. That's absolutely right. >> And and and it's really funny how like okay well if you don't know what the difference between offend and off Z is please learn it. And then you know if you if you don't know if you have never looked at AWS IM which you have please learn it like this like these are basic like conceptual things to understand that you can prompt for once you have the language you can prompt for it. You don't have the language you're Totally. Totally. Yeah. Yeah. To an extent I even extend that to like just general like certain certain cloud platform of things right. I mean there was the I forget which company got had that hu incident huge incident where like it was just like a DNS issue that [01:29:43] brought down was it do you remember this? Was it >> DNS issues? You need to narrow it down. >> Yeah, I know. I know. There was a particularly prominent one. Was it >> There have also been many prominent DNS issues. >> Okay. >> But list them off. >> It's always DNS. It's always Yeah. So DNS is more of like a reliability uptime. >> Sure. Sure. >> Uh you know security I think people like did you lose money? Did you leak customer information? >> Yeah. Yeah. Yeah. Yeah. Yeah. That I would Yeah. Probably put that above that for sure. Only DNS because of uptime. Yeah. General up time. >> Yeah. >> SLAs and stuff like uh background agents. Uh now you got your buddy from Coffee here, but I mean your guests aren't necessarily correlated with that week, right? >> Yeah. No, I mean like the the nine one was a little bit of a scheduling thing where I've thrown in, you know, Elad and and I guess I'm isn't with definitely week nine. Elad I was trying to get him at the end, but I think it was just timing wise couldn't work it out. Um so yeah, it's it's a bit of a weird one where I don't I guess I'm not lecturing at all week nine, but but we're coming back to those topics in week 10. So it's a little bit staggered in a in a bit of a bizarre way there. Yeah, week eight. Um, Rajesh who has been leading a lot of [01:30:44] like the infra stuff at uh Cloud Infra uh DevX and and beyond at Cloudflare which as I earlier said was I think they're doing a great job here and definitely their adoption for a company of their size and a public company of their size is is fantastic and then their content I think is also really good. I mean this is fundamentally about all the ways you can use like asynchronous background agents. Um, I think I'm also going to spend quite a bit of time talking about, you know, the Stripe minions, the Spotify, I forget the name of their version of this. Um, the ramp inspects, the, you know, things of that nature that I think are all kind of showing what does it look like when you scale this out. >> Tag Instinct. >> Yes. >> Muse. Muse. >> Yeah. I mean, I guess less on consumer, more on like, you know, software teams using them at scale. >> They're all coding agents under the hood. >> True. >> Everything is a coding. >> Yeah. Like Grockbot is one of those interesting things where like very very much presented as consumer e uh it's it's Grock under it's you know it's it's cursor under the hood. [01:31:44] >> It's the cursor team. It's >> no no for sure. Like so I I've seen the like you know cursor working in in the like the dot dot dots but but but they are not they don't have like a codebase integration right. >> They do. >> Okay. I haven't used that. >> I've only been using for like open table reservations. >> Exactly. Okay. >> Uh which I think that's the point of what they're trying to do. >> I see. Yeah. as in like >> they're trying to be like uh well you can use it for consumer stuff but here's the extra power if you know how to use it well uh because because it's actually cursor under hood >> I think this goes to all coding agents right like cloud code cloud design uh people using codecs not knowing they're using codecs right a random code interpreter window pops up and you're just asking jets a question at at some level there's coding agent under the hood right >> yeah so so I'll give you another sort of practical example Uh so I I run all my SEO and AEO inside of codeex and it has a codebase even though like a traditional marketer would just check a address or something because I write all these scripts for doing all the all sorts of crazy stuff. Sure [01:32:44] >> and and also I need to store data and also I need to vis visualize data and like pretty soon you go like as long as you're doing anything on a ongoing basis that has like maybe a lot of steps or a lot of data you need a code base. Hm. That is an interesting take. >> I think models also like building code bases. >> No, but but that could be a temporary thing. >> I guess maybe my my point is more around the interface, right? Like will are you going so like are you going to go so far to say that a grobot will have a place where like the power user will be able to see the code under the hood? Is that or is it maybe they just jump to cursor? I guess is is that what you're saying? >> Well, exactly. Because I'm like otherwise then then you basically cannibalize cursor. Like if if in the limit what you're saying is Grockpot is the cursor then you don't even need like a cursor if all under the hood is just becoming >> it's the same team. So they right now they don't care but yeah it's all Grockpot. >> Yeah. Yeah. Yeah. >> I I I buy that. I mean I do accept that even things that seem outwardly just like non-technical use cases under the hood are technical can be like written [01:33:44] and and designed and implemented in code like that that much which is absolutely true. Right. The question is just more like is a developer is that their single point of entry? >> Yeah, >> I think it's not even developer, right? Like you think of a regular person doing marketing that is using cloud code or just claude in the app now is working in a codebase that they would never know, never touch, but >> there's a codebase manager. >> Mhm. >> I think cloud tag is a good one for uh background agents. It's a big way anthropic works and you know they make it. You should see how they work. >> Yeah. Yeah. Good job. Uh, okay. AI native team. Uh, we've talked a lot about these topics of like gateway. Uh, we just put a routing episode out today and MCP portal. I would say MCP portal is like the least >> bullish on MCPS. >> I'm bullish on MCP. There's not bullish or bearish. It's just is it just is a part of the landscape now, right? >> Um, that's a statement. There was a whole MCP is so back. MCP is dead. People like extreme >> MCP portals is a unproven business. Uh I [01:34:46] I thought about starting one. Uh I actually pitched a guiding topic on on potentially starting one and then I lost interest. I lose interest in a lot a lot of things. But I uh what's what's in there? Uh so okay so what I'm thinking here is if this is um if this is the week design uh more kind of geared toward scaling this out to like what happens when you take some principles and a lot of this was informed by like conversations I had with folks that were you know leading and native teams and teams at uh at respective companies and so here uh I think that something MCP portal really only becomes interesting once you're at a certain scale where you effectively have to have a single point of entry to do things like governance security auditing cost optimization I'd be curious to hear what you see as the future or maybe lack thereof. But it to me is if you believe in like an open router or you know some place like a single point of entry for for many things to happen and connect to and be effectively you know governed. I feel like there's a world where this also continues to exist. >> Yeah. My my only tweak on this is that this folds into the LM gateway. [01:35:47] >> So they are the exact same platform >> that I will give you. That I will give you. I can I can accept that that actually you can can agree with. Yeah. Uh and and then then by the way it also folds into the other thing of um what did I say was was uh offz you know permissioning and all these things and billing so like all to the limit all infra converges to the same exact problems where they have the box in the middle and then you have the box in two sides uh you know god box will solve everything right and it's true only if you take uh if you understand the job of the god box and so separating mcp portals and gateways means you don't fully accept that they are uh you know the the same layer of problem and in a very similar way I also find that people don't understand database querying uh to have a lot of the similar things where they will separate filtering from ranking from uh or or just like general recommendation if they're the same problem. >> Uh and so like then you just com because like when you take all these things you combine them in the same layer then you [01:36:47] can sort of cross depend on on things in useful ways. If there are different layers then they cannot. >> Uh yeah I I I actually will not disagree with this. I think um maybe the yeah so slightly pedagogical I think that the this breaking up of like separate terms versus one. I think I it is absolutely correct that the underlying architecture is the the god box where one thing goes in and there's a bunch of other stuff that is some third party some internal some first party that all kind of exists uh on the other side of the god box. So yes I agree with this actually. I I have a I have a I have a spicy uh spicy tweet. I Oh god, where where is the godbox tweet? Oh yeah. So this is the M MX to M plus architecture. Uh so anyone working in infra needs to understand uh congratulations, you now have a job in infrastructure or you want to be an infra engineer, you need to understand that your entire job is making God boxes and then taking them apart. So either your problem is either you don't have enough you don't have a good gut box or your problem is someone [01:37:48] else's god box you don't like so now you have to take it apart and make your own. >> That's funny. That's very funny. I like that. That's very funny. Until the heat death of the universe. Wow. That's funny. >> Which which is a this is a a restatement of like there's only two ways to make software which is you bundle them, you unbundle them. But like what are you bundling? It is the things that go together hopefully. And then like maybe over time people's opinions shift as what what goes together what should not go together. >> Right. Right. >> So so like you know there there is that that the whole thing you know maybe uh Monaco should be a more of a point solution. Maybe Monaco should own everything. Right. And like there's the question of like in some ways you're bundling for uh the kind of startup segment that you serve. In some ways you're unbundling from Salesforce because Salesforce that and then that's how you win market share from Salesforce. You unbundle from Salesforce. M hm that's interesting. Yeah, that's that's an even broader statement. I mean connecting it to broadly all of vertical AI or uh and now like bundling unbundling I mean that's like the whole like Craigslist and [01:38:48] Airbnb you know the Tinder and all those are separate >> Tinder >> but you know like personals basically >> never thought about thought about it that way but yes uh you know one one of those things so we just shipped the open router pod today right >> and um open router huge success lots of people tried to build gateways none of them had success with open router open router only builds gateways. So there's an example of like this. they disagree with me that the the the router layer should also do ops >> because Arise is doing ops um and observability and and what have you and and they and like the MCP gateway they don't have that >> and they they just they only handle model routing >> and so like I I like I am I love being wrong like that because they're like we are the point solution and we only do this one thing really really well and uh that's our that's our engineering focus which is good. It's also the marketing focus which is nice. >> Mhm. I I think the um this becomes I think that the what you've described of like the one one-stop shop for all the [01:39:50] things is ideologically what makes sense but I think what ends up happening in practice and you know part of it is probably also marketing is that especially when you look at the enterprise is like this is broadly under infra like broadly under under platform but then there's separate teams owning different parts of the stack and I think part of that consolidation at the at the scale of an enterprise if we're going to say MCP portal and gateway someone has to come together and say like now we're there's a single end point that you're hitting both of these things and that fundamentally is like a hard thing to get everyone on board with. Um, so it often times like more like political and and even like people problem rather than like a from a software pure software design perspective this makes a ton of sense. >> Yeah. >> But that's not it right like software systems sometimes are bottled like by people problems more than than pure software. >> The way my my framing of this is typically you know how you ship your org chart is well I forget what the the name of this principle is. uh you need Conway's law, you ship your orchart, right? >> Uh you buy your org chart like like like who like when you start a startup or you you make a tool, who in the company is your are you best friends for? >> And if it's for everyone, it's for no [01:40:50] one. >> H that's interesting. That is interesting. >> So so like a conscious part of uh latent space and and AIE is like okay there's this category of software engineer that really really really cares a lot about AI. Is it going to make it their day job? Let's only make it for them. >> Right. Right. Right. Right. Right. Not not a bundling of engineering. Never going to be the dev everyone. >> Yeah. Look, we talk about databases, talk about security, but like there's an there's an AI angle to it. >> Mhm. >> Yeah. We'll see. That makes sense. >> Uh, one thing I'll just like shout out as well related to the to the point is uh I think role of the future that is increasingly important. U platform engineer. I think this is like a huge one, >> you know. Okay. more lore uh temporal pre my job before all this I would have uh yeah so we wanted to own platform engineer if I hadn't started yeah I would have done platform engineer uh because that is the role that decide that decides all of this mission critical stuff and then the kids over in applications can do whatever they want [01:41:50] on top of our platform but but you know the big boys do platform >> yeah and I think in the modern day they own they own cloud they own devx they own security ops so I think it's it's all the things >> like everything where you can actually like lose lose the company or like it would make or break like yeah platform guys and then product guys can do you know adjust the CSS that >> that's you're starting to fight here I mean I'm not the CSS adjuster but but I I I appreciate everyone >> I mean it's a fun time what's what's the future I feel like all the guys can adjust all the things um >> yeah yeah yeah so there's there's a few aspects here okay so so there's a little bit of spillover from from the previous week so yeah coding agents big teams some of the concepts but then you We'll touch on the software factory. Uh I want people to to really start thinking about this this world of of this machine that builds the machine and where if at all are people still in the loop there. You know today I think the answer is some some combination of intent and kind of [01:42:50] review or maybe the places like so intent where you're defining the specs and what needs to be built and and review where you're you know doing some amount of your own checking of the system. And of course this also this is I think that bottleneck is going to be loosened up before even the intent piece does. But for now those are the two choke points I think in in a fully automated system. Um so yeah I just want to give people exposure that for one of the OSS partners actually uh warp they're doing this thing as part of their contribution to the OSS program where they're going to essentially make a version of warp that is running the factory and students will just be like building the factory contributing to the factory and keep it running >> like work in our factory. >> Work in our factory. Exactly. And then they're going to find things that are interesting there and pull it back. >> This is one of those things where like I'm not sure obviously factory itself is very very keen on this. >> I feel like everyone wants to build their own. >> Yeah, I I think that's that is it is a good question and I think they're, you know, of course entitled to kind of try and sell the products they do. I think it's a hard system to own its entirety. [01:43:50] It's a it's a lot. It's it's a lot. >> Decide everything for me, please. >> Yeah, it is a lot. And I'm I'm trying to you know I haven't like gone too deep in understanding like the how they approach their GTM here but I suspect you would have to do a peace meal right you would have to be like hey we first your coding agent oh then also like you basically land and expand right I mean we talk about this bundling on bundling piece you don't start by saying here's your whole factory there's no way you could get like a Ford on board with that you know what I mean where you're going to go on and just plop the whole thing on on their desk um that's at least how I would think about it is you first be their their coding agent of choice you you you build up some background agent for them you you own some of their S sur pieces and then you're like well now we can just weave it all together if you define everything in code or something like that like you know configs. There was a fun blog post on this from OpenAI. We had >> our buddy Ryan. >> We had Ryan on the podcast. Basically, they spent five months making an internal tool with zero lines of manually written code. They kind of walk through one. It's called harness engineering. So, you can learn harness engineering. But, you know, the [01:44:51] learnings of there was kind of a slump or slower at first and then you offload and you had peer review. It's at this point it's from February but it's you know a vision of a software factory right like you are full sending um letting agents fully write everything and how to do that well >> yeah absolutely yeah I mean some version of this is going to exist uh to SP's point I do wonder about the the sales piece here like the actual business of this that's an interesting question but from a principal perspective I think this is the the this is one version of the future and then beyond that I think there's going to be some amount of dabbling into and speaking about like okay what does it mean to yeah what does it mean to be a developer in in the future right like are we people talk about taste people talk about like you know understanding of systems and architecture and yeah like software taste I want to like go a little bit deeper here on maybe starting to like at least put a some answers to the question around this >> yeah uh so this is your answer to where where AI software engineering goes next [01:45:51] >> yeah so I mean the factory is is like the tactical but I think beyond that there's like a there's a question around like the role right? Like what does the role evolve into? Um if if the factory is is running and you know full steam ahead doing its thing, the conveyor belt is is turnurning out code. What what do we do in all this? Right? And I think we I think you and I we had this discussion with Jerry at the dinner once upon a time a few weeks back with Dex and Jerry around around the factory. But yeah, I I I think that there was this part of at least the discussion points there which I I'm pretty much on board with is like increasing levels of abstraction and then somehow there's someone still even a CEO of a company has to have enough oversight understanding of like different constituent parts to be able to make a judgment on direction right to be able to course correct to be able to set metrics and things of that nature. I think there's a version that translates of that. There's a there's a version that of that analogy that translates to like someone who oversees the technical. We call it a manager of agents. We call it a you know different terms. They're all buzz buzzwordy. But um in some some respect someone who understands product requirements, constraints, technical constraints kind of loops through all of [01:46:51] those and and and knows how to adjudicate. >> Yeah. Um I I will say that uh in the chair before you, we just had uh anthropics uh head of product and head of engineering uh for the the platform. Uh, and I was like, when are when are your jobs going to combine? And they're like, not yet, you know? Uh, and so if Entropic is not doing not there yet, you should not feel like it's it's imminent. >> Good, good, good. Um, do they say how long or are they just not yet? >> Not yet, but who knows? And I I definitely there was definitely a lot of like, okay, yeah, we're I'm doing her job, she's doing my job. Like, yeah. >> I think it's also interesting perspective for students, right? you said there's a lot of sophomore undergrads like >> you got two three years to go to plan out what you want to be doing. Um >> there's different ways to look at it from that, right? You're not straight in industry. >> The thing that I care uh I'm thinking a lot about is well so one part of this discussion that resonates more with me is the loop uh stuff where like um okay you you started writing prompts now you [01:47:52] no longer write prompts. You write prompts that write prompts. That's right. >> Right. And then then you just keep going up. And I I think to me the sort of recursive concentric circles nature going outward but also going inward >> is something that builds the factory. I think you build up to the factory not build down but I think a lot of people will eventually start building down like you start from the top level like what is my goal and then you figure out the systems. Um that currently creates a ton of slop and you should not do that but someday you will. >> Mhm. >> Yeah. Yeah. >> So yeah. So, so in so instead of so you know that would be my question to to prompt you um is uh do we do we uh what is the sort of prime question that people exist to solve like that is your that is your thing right so in in let's let's say an engineer today um you are measured by your PR merge rate PR contribution rate >> well two years from now that's going to be ridiculous okay well what's your main [01:48:52] thing >> right I don't know. >> Yeah, >> money. >> People always go to money too quickly. It's something in between money and >> right PRs, >> customer satisfaction, seesat scores from your features. >> That even that's like a product decision, not really a code decision, right? Like like I don't know like >> but but if the but if the engineering is evolving into like a more of a PM engineer like product engineers, you know, hybrid, then they then I think that's you know that's one answer the question of like what does the role evolve into? I think it it goes without to say that um I would today recommend to students spend more time thinking about holistic systems not just your like single part of the the engineering puzzle but how that ties in with and you know and this is like in practice ends up happening like students for their capstone projects I encourage them to go out and like in fact what I give them as the seed of ideas for final projects is like the the YC request for startups. I'm almost like go build a company like you know what I mean like go go build something interesting. >> What did people build last year? Uh I mean a variety of things like someone was like um someone came in the E department uh was in the E school and E was studying E and they were like they [01:49:53] built effectively a translator for like parsing of u like manuals to that have like um like memory addresses in in C and then converting that into like uh like code that they could then like interpret and even like visualize. So so stuff like specific to their their use case. someone who was like a designer and is now actually at Versel and she like had you know like take a picture of my like design notebook and generate me an app based on what I take a picture of. Right. So literally she was like for her demo I was like here you go snaps a picture of like some random like scribbles that she was doing and she got a fully fleshed out Versell app and from that and I think because of that probably got a job at Versell doing that doing doing some version of this. >> Oh nice. >> Um that's a high bar actually Versell uh very known for their design engineers. >> Yeah. Yeah. And I think she's now working on Eve from I remember. Ah, no Eve in your >> EVE. Eve is an OSS or like OSS partner. I mean like like they one of the the ways is partnering with us is that they are having students build with Eve. Yeah. >> Yeah. So the this is the concept of meta harnesses, right? Uh harnesses of harnesses. Uh you can always go one level deeper. Um but I I do think like [01:50:55] it makes sense at least for me uh in the same way that model routers wrap model APIs. You you need to wrap harnesses as well if you're going to between them. >> Agree. Agree. I think at some level students should also stay grounded in like there's a lot of stuff happening at the edge. There's also a lot of companies that are still like single call in production playing around haven't tried it. Um you know there there is a lot of that. So there's there's still time for it to work. But I think your point on taste is good. have taste, try stuff like uh whether each company has their own version of a software factory, uh you're building tools to help companies do it, like staying on top of it, actually using the tools, playing with it, like you'll you'll figure it out along the way, you know? >> Yeah. And I think the the 100% agree with all that. I I mean, first to your point that are a lot of companies still in the prompt chatgbt or some version of, you know, chat GBT maybe do they just got a cursor license and are just now getting people on boarded? um that is way more true than we believe [01:51:55] certainly here in the valley. Um that is I can I can say 100% confidence. Um but yes, I think for the students being exposed to this is just at a certain point this the fact that they know this and are on the edge is what will eventually allow them to also be some of the more interesting and more valuable contributors when they go to maybe places where that maybe aren't up to snuff in terms of being on the frontier. >> Yeah. uh what one of the this this kind of progression type talk >> still very influential about a year old Steve Yaggi >> talking about the the levels of psychosis uh he's all the way through level 10 now um where he's he's given up and he's moved to a farm but uh yeah like you want sorry I'm making fun of him uh he built Gas Town >> yeah I remember >> all these things moved to Gas City now he's burned down the city and said I'm building my game and no more Gas Town I thought that I mean I I remember talking to him at at the engineer fair uh earlier this year and uh wasn't he his whole like thing like the only useful thing that was built at Gas Town was Gastown, >> right? That that's a horrible failure. [01:52:56] Horrible. >> Totally. Totally. >> Um and and so like people should be aware that like psychosis is real and if you're not holding yourself accountable to delivering real products, you're just screwing wrong with LMS. >> Yeah. I this is like a maybe I mean relevant to but I wonder how much of I mean I played a little with Gastown very involved system maybe a little ahead of its time um but I'd also wonder if they were like bottleneck a little bit by the by the marketing and that it was a very difficult thing to gro from a terminology perspective like there's just a lot going on I think the Gas Town marketing also is one of the things that propelled it right like >> what was right before Gas Town there were like >> the like his his Yeah Beats he did unless you're talking about someone else's No, no, there were like there was the whole open claw, there was something else, and there was Gas Town. And the whole manifesto also really helped spur it into the conceptual. The people that would never use Gas Town, but learned about Gas Town cuz you can read the manifesto like there there's a marketing angle to it. >> Yeah. I mean, so so I I but I do so not [01:53:59] necessarily someone to take literally, but direct like like directionally correct, you know, and and so I think he's directionally correct on this one. And so if you're at the ID stage, it's time to move into power user stage. If you're a single agent, you probably time to move to multi- aent and blah blah blah. Uh and you probably should try to build your own orchestrator, right? And that's that's all that arguably uh this is not even this is like kind of out of date now. There there is more to the software factory beyond this. Uh but this this article predates the software factory. I I do think you know one of those things where like uh I'm open to as an employer I have this problem in my company today which is I do not want to hire anyone who hasn't been through psychosis and come out the other end >> because I am currently paying for somebody to go through psychosis >> and that's very expensive from his salary uh very expensive from tokens and I'm getting nothing out of it and he doesn't get it like my my feedback doesn't get through >> and so like uh it's almost like I would encourage your students to have psychosis and then come out and go like, "Okay, what was wrong with me?" Right? Cuz like that's what Steve just went [01:55:00] through like a very public two-year journey of this. Yeah. >> It's weird on the student side. There's like a lot of uh industry you think you're being productive. You're not is probably the student version, right? You're notion charting and making all these optimizations personal s and it's like dang I went through the psychosis but nothing has changed. Right. Right. >> So go through it. Go through it. >> Right. Yeah. Uh I haven't seen this. I've seen a version of this from Boris actually. obviously more geared toward cloud use cases, but I remember he made like a cloud artifact where it showed I think more at the team level, but I think he had four stages, not eight stages, but it kind of rough roughly followed the same progression of like no adoption to through to you know I guess >> only four is good. This is this is too many levels. Yeah. Well see we'll see if uh V can pull it up but other than that you know zooming out have we you know discussed all the things that's one oh that's one okay there we go. It is an architect. Okay. >> So, it's gated, assisted, parallel. What's the last one? >> Supervised autonomy and AI native. Okay. >> So, it's zero index. So, it's five levels, right? >> All right. [01:56:00] >> Start at zero. >> Yeah. Wow. Actually, it's pretty good. I need to uh read through this in more more depth. >> Use a native. There you go. >> I would give a shout out to dynamic workflows. This is one of those things that is a sub aent pattern that the open guys are also exploring. um that is clearly fitting and uh and so it's basically you're writing a harness on demand for the thing that you're trying to do. >> And so uh this is maybe like a lot of sub agent conversation is just you're copying the the parent agent >> but you actually should be writing way more code and then like running that as a as a as a sub agent and that is what the enthropic people have chosen to call dynamic workflows. >> Okay. Uh but you know >> yeah we we've built a version of this as well. Yeah, it's it's a good pattern >> for Monaco. >> Mhm. Yeah. >> What does it do? Yeah. I mean on the fly sort of like harness sort of use cases for GTM. Yeah. >> Oh, that's great. >> Yeah. >> Um >> I will note that when Boris wrote this, it's for team adoption for people that [01:57:02] are listening. And also he says, you know, Enthropic is on stage three. We're still trying to get to stage four. >> Wow. >> He's like, I am at stage four. I'm the guy. But, you know, it's it's not as simple as Steve's. like Steve is a very good personal do this do this do this uh this is a much more if you're running a team and you want to AI pill your team and not just be a user of one you know anthropic is still at stage three don't >> don't be like yeah >> okay I mean you should teach this this is this is great this is this got all the stuff that this is everything that we talked about >> there you go >> I I can't believe I haven't come across this >> um >> good resource yeah >> very very good okay that was a wild tour once one semester of Stanford in 1 hour or two two hours. Thank you for doing this. Any parting thoughts? Any calls to action? >> No, this was uh thanks for I mean ton of fun. It's it's been a you know nine months in the making. I'm excited. This is end of week one. A lot to do, a lot to learn. I'm excited to see what happens with especially the OSS partners and just getting students like deep in [01:58:02] in the code of of some complex code bases. Um having them build some stuff. So no, it'll be fun. Keep you guys posted. >> You're doing another version next year. You you need credits for students. If OSS people want to get involved, they can reach out. >> Sure. >> Because credits matter. Do they want it? >> What? >> Do they actually want the credit? The >> students or they want you you have credit card. I don't think a lot of students want to spend. >> I want Yeah. credit card. But >> yeah. No, I I I think look I go like I remember you know it was like you know there's a certain number of credits for sure but then on top of that he's like look if they want to go hog wild with this like let me know and we'll do have these like one-off experiments where they can just like scale up to like a thousand plus agents. This is from factory. >> Yeah, you know factory. >> No, those kind of people are valuable. So this is Wilson from cursor, right? Who is like I'm going to write a browser, right? >> Those people who are ambitious and like can actually execute, those are very valuable and that's basically a a recruiting call. >> That's right. That's a good point. Yeah, I but I I do think like students should should like understand how to be ambitious because snapping a photo I [01:59:04] mean respectfully snapping a photo and turning into an app is not that ambitious in in the this day and age. Uh writing a browser is and there and like there actually is a lot more projects like that. >> Yeah. Yeah, I think that's right. Yeah, it's all about raising the floor. I mean I think yeah last year it was uh it would you know you know credit to the students of of last year but uh yeah this year I obviously we're going to expect we're more >> that's a fail now. >> We're going to expect more. I mean, yeah, but that's but that's that's the whole point, right? Like we're leveling up, right? Last year it was like, yeah, hey, have you heard of prompting? Like, here's what it is. And I spent some time talking about like, you know, Kshot prompting as one of the lessons. And like, I don't even think you guys even think about that now. So, >> I think there's a there's a version of thinking you stick out in your cohort of classes. And then there's also a version of you're also in early engineering roles of a bunch of people in industry, people still trying to get jobs. and snap snap uh you know not to call out one demo but you can you can do a lot to stick out for sure. I I'll give uh students a freebie uh of a week 11. Uh make benchmarks, make benchmarks and evals. Uh this will get you uh a good [02:00:05] one going to get you in the door of every frontier lab in every front every top agent lab. >> Good good engineering blog posts with some rigor on anything interesting. There's a lot of benchmarks or low hanging fruit there, but the engineering blog posts do really well, right? You you went deep on one thing. >> Yeah. Um and probably also there's like big a lot of money to be made in that as well. So, not just uh in the doors, but if you make that into businesses, you you can uh you can really really cash in on that. >> Yeah. People don't know like Sweetbench was like undergrad project. >> Yes. Yes. Yes. That's right. From from Princeton, right? Yeah. Yeah. Yeah. >> Great. Okay. Well, fantastic. He has.