⚡️ The 2026 Software Factory Infra Stack — Zach Lloyd, Warp.dev

[00:00:00] Hey, it's me again. I have heard a lot of positive feedback about our episode intros, so I'm going to keep trying to do them as far as time allows. Today, we're going to dive into Warp with Zack Lloyd. Um, Zach has been working on war for a while and uh it's pivoted uh into what it is today, which is basically the definitive software factory uh that is also open source. Um, and he has this whole definition I think which uh has done really well. his his basically his talk at uh AIE has been going viral as one of the top talks of all time and I think effectively everyone is trying to set this up for themselves. They have a solution for you. Um but it's also you know kind of an open spec that is becoming table stakes for what you're going to want. Uh and I think you want going to want cloud, you're going to want a control plane. U you're going to want uh some other sort of basic security features and team features that uh are you take for granted. So, we're releasing this episode as part of a preview for AIE code. AIE code is the last uh AIE flagship of the year. It's [00:01:02] going to be in San Francisco where we're gathering all the top agentic engineers and talking about software factories among other things. System one models do is coming back and uh raising your ambitions. So, Tariq and Jason and Brandon from Artcraft uh is doing his first conference talk ever. If you're interested, um, Zach will also be giving, uh, a great keynote on software factories and sort of recapping the year and previewing what's next. Join us and it'll be lots of fun. See you. [music] We're in the remote studio catching up with Zach Lloyd for more. Welcome. It's good to be here. Thanks for having me. Zach, you've been going viral on AIE. What's up? What's the What's You've hit a [laughter] nerve. You've hit a nerve. people are really interested in software factories and that is a area that I've been focusing on that Warp has been focusing on. I think we have some you know strong kind of interesting takes on what a software factory actually is and what it means for being a software [00:02:02] developer. So yeah that's and that's people have been into it. Your H1 on the landing page says open infrastructure for cloud software factories. I think maybe if since you spend more time than I do talking about telling people about factories, what do people think is a software factory and what do they actually need? When I talk to companies or developers, most of the world is trying to build some sort of internal software factory or set of software factories. what what I think it means or maybe I'll just define what it means to me is like a agentic system in the cloud where you're defining what are the roles of various agents across building software for your team or for your product and there's a certain amount of autonomy in that system. So it's like the agents can do things on their own and that system to me should be something that's very measurable, very closed loop which over time you as a team can like tune, measure, benchmark and just ultimately get confidence that as people switch to this way of doing [00:03:03] aic development that they're doing it in the most scalable and efficient way. And what do people come to you with that perhaps is a common misconception, right? like they're like, "Okay, if I if I do whatever, if I buy whatever, I'm going to come out the other end with like this amazing money machine." Yeah. So, so when people come to us and they're trying to build their own software factory, like often they'll be like stringing together different automations, sometimes people are conceiving of software factories as just like just like a not a not a specific thing, but it's just like we're using agents a lot to ship software and we're calling that our internal software factory. And what we try to to do to make that more concrete is be like no no no you should actually think of a software factory as a very specific thing like you should define your factory in code you should make sure that the factory has all the workflows skills MCPs everything is like it's like a concrete thing that you should be able [00:04:03] to run at some point in time and measure how it's doing cuz people have all sorts of kind of random ideas around what these factories are and so for for warp factories which is our platform. It's like well we say no no no it's actually it's a real thing. You can orchestrate it. You can measure it. You have one factory per product surface. And we try to like take it out of the realm of just being like this nebulous uh it's a collection of like arbitrary automations and interactive agents into a a thing that's an actual the way to think about a software factory or the way that we think about it. What you're looking at here is is Warp Software factory platform is a factory operates on some product surface. So imagine you're building like your mobile app or you're building you know your server app or whatever and the factory itself is operating on a set of repos and it has a set of skills it needs. a set of MCPs and then it has like a a workflow associated with it and this workflow you know roughly matches what you know your team is probably already [00:05:03] doing to make changes to that codebase. So there could be a step where you're doing triage. It could be where you're doing specking implementation. The important thing is like there's some kind of defined workflow and that that workflow itself is defined in code. So for us, we basically have a YAML file which says, okay, these are all the repos that you can be using. These are all the MCPS, all of the integrations that are hooked up. So you basically have this whole thing defined someplace. The way to think about using the factory is that there's actually sort of two personas. So there's someone who is building the product. They're doing product engineering. They're shipping features through a factory. And for those people, the way they use the factory or the way that is through their existing tools. So they could be slacking the factory to put work into it. They could be using a task tracker like Linear Ajiraa. They could be interacting with the factory through [00:06:03] GitHub. They could be calling it via API or MCP. And the metaphor is like you're putting a work item into the factory. And when it goes into the factory, that starts like a predefined workflow. And that workflow is defined in code. Does that make sense so far? >> I love defining things in code. >> Yes. So, okay. So, you have this whole thing set up in code and then once you have that in place and so the again metaphor is like someone starts a task in Slack or in linear or through an automation like you know there's a server error. It all goes through one system is the kind of key thing. And once you have that working you basically get a whole bunch of benefits that flow out of it. And so one of the big benefits is that wherever these agents are running and you're sort of seeing a real-time view across Warp's internal team, how are we using our own factories, you get a full trace of what everyone is doing with coding agents across any product that has a factory [00:07:03] for it. And this is cool not just for like oh I want to see what people are doing like I want to see what Sophic is doing which is cool but it also starts centralizing all of the agent conversations and agent artifacts in all of the work that are is being done by coding agents across your team. And so when I talk about like having a closed loop system these agent conversations these agent traces are like the key aspect of how this system improves. And so for instance, just to look at one of these agent traces, Sopic is working on our mobile app here. Sopic was doing this in Slack. So I could pull this up in Slack if I wanted to. But like the thing is actually generating it has a full record of the conversation, has a full record of all of the these are computer use screenshots that are coming out of building this thing. You can see a full record of exactly what the different pieces of the orchestration agent were doing. And so you have this [00:08:03] but you have it not just for one thing you have it for every single thing. So on your team today what we when I talk to companies for the most part they don't have something like this because most of their engineers are like individually using cloud code in their terminal or codeex the codeex app or something like that in their own local setups but they want to get to a world where they have this centralization. When you have the centralization, you can basically get all this interesting data on how your aentic coding is working across your company. And you could think of this data along different dimensions. So you could think of this data as like okay what's our team's output in the world of using coding agents. And so that's things like you know traditional Dora metrics, how long does it take to ship something? It's things like how automated are you? And so we have a sort of metric around like for every PR that gets merged, how many times did a human have to do something with it, what's your throughput, it's metrics around cost. [00:09:05] And so we're using this like constantly. And when we talk to companies, this cost thing is very, very important where you want to be able to tune your factory with the right models, the right skills to basically bring this cost per PR down by time. You want to be able to slice the cost by person. You want to be able to slice it by automation by by where in the SDLC are you spending money. The final thing that's that's really cool once you have this sort of closed loop system going is that you can actually look underneath the hood at how well your coding agents are doing along different dimensions. And so you can think of this as like online evals or we call it scorers. And these can go from very general things. like are your agents being too verbose? Are they being efficient? Is the code quality good? We got work to do on this. >> Do I as a user define this or do you provide this as a platform? >> Yeah. So both. So we provide out of the box a set of ones that we think are like generically useful but then you know as [00:10:06] we work with companies you know they may have specific failure modes for their agents that they're interested in tracking. They may be like, "Hey, is the agent making a specific type of thing that is like change that is like breaking our build?" They may want something, you know, very specific to their workflow. So again, this is all defined in code and all customizable. And what you do is like when you set up one of these, what's actually happening behind the scenes is that there's some set of runs that you, you know, when I say runs, I mean like historical agent conversations that have been stored that an agent then goes and like retroactively grades. And so this is a good example of one where it's grading for redundant tests, which is like a this is a common failure mode if you're using coding agents. Like sometimes they write a bunch of tests that you don't need. They write it in the course of like figuring out how to like implement something and then they leave it in the code. And so this like bloat your [00:11:06] codebase, it wastes tokens. And so you know say you wanted to check for that you you know you basically create one of these evals and just to show you like a little more about what the eval looks like. The eval is again think of it as LLM as a judge. So you're picking a a judge model. You're you're then seeing okay like where do these run? What's the prompt for judging it? and then a sampling rate because you might not like these are agents that are running. You might you got to be aware of the cost of them. But when you have this set up, think of it then as like you have all these smart agents that are looking across your team as at how all your your coding sessions are going and they're saying, "Okay, sometimes there are agents that are writing too many tests, right?" And so you get this record and this understanding of where you should look to sort of improve the way that your team is using coding agents. So far so good cuz I want to show a couple other things just to like make it more specific what you can do with this. Overall good. I'm I'm a little bit [00:12:07] wondering if this if basically there's different levels of instin of of Warp where there's like this is the CTO level what you're showing us right this you're you're running the company you've got a bunch of coders this is like the the people who use this it's partly the engineering manager or the CTO but it's also like very commonly like the platform team but it's not your like day-to-day engineer typically but like when we work with companies you know they'll often have an internal team team that's been trying to like it's like the DevX team or the platform team and they're trying to figure out how to optimize coding agents. So what I'm showing you here yeah is like the tool set that they are using to be like how do we get the most out of cloud code at our company or codeex or something like that that part is clear. >> Yeah. Cool. Yeah. Yeah. And so like a couple other things that I would just want to emphasize, not to make this just like completely about warp factories, but like the general principle here for anyone who's trying to build their internal factory, whether it's on warp [00:13:07] factories or not, is like you want to get to a spot where everything is sort of stored, everything is observable, everything is measurable because that's when you can actually start to like apply engineering rigor to how you make coding agents better. You know, you probably you see this on Twitter all the time. Everyone is always tweeting about like you should check out this new model. You should use this new harness. What's better? Is it pi or cloud code or you know cursor or whatever? And so what our opinion on this is and what the the world sort of needs is like out of the the realm of vibes for understanding these coding agents and actually into the world of measurement. And so we think that this is the direction that companies are going to want to go. They're they're going to want to like put in place a system that's like Warp Factories, whether they actually use Warp Factories or they build their own, which collects all this information and provides all this way to see how your coding agents are doing. I guess one [00:14:07] other thing I like kind of take for granted that's part of the system is like you can create all these different automations around it. So like I I mentioned people work in Slack but people are also setting up uh different trigger based ways of putting work into the factory. Once you you have this view that I showed earlier of uh like scoring how your factory is doing. You can also set up another kind of loop of agents and everything is kind of loops these days where those agents are looking over they're looking for failures in your scores like they're looking for things that aren't going well and trying to suggest changes to your system to make it go better. And so like a really simple example here, we call this self-improvement is like, okay, our self-improvement agents were looking at how well our triage step was doing. So we have a triage step and they're saying, okay, we have 54 failure cases of this particular form where it would [00:15:10] have been better if our triager had been able to spawn a sub aent to do its investigation. And so what they're able to do and like you could go look at like why it thinks this if you want but the cool thing is like because you have your whole factory defined as code and these are coding agents they can essentially make a pull request against the factory definition to try to improve the factory itself. And so this is like a like a full learning loop that's built into the system to try to actually selfoptimize. And like we don't have this yet, but the next thing that we want to do with this is like actually for any suggested change like this, you could go back replay the runs and make sure that this actually would have improved them. So the possibilities of like how you can engineer this stuff get pretty cool once you have the right primitives in place. And I'll show one last thing and then we could sort of maybe talk more generally about the whole approach here. But the other thing that is um you know I think companies should strive for that we find [00:16:11] very useful internally is like the ability to actually do what we call benchmarking. And so benchmarking in this context, in the factory context, means essentially the ability to like take any coding agent that's run in your system, you can sort of add it to a benchmark suite, which means like, you know, you might curate a set of coding agent runs that were related to like doing front-end coding or something like that. And then once you have that, you have the ability to sort of replay those same tasks against a matrix of different configurations. And so, you know, I'm a big believer in not just allowing you to vary the model, but also to vary the harness. And so, you know, in Warp, we have our own harness, but we actually will let you benchmark against Cloud Code or Codeex or Pi. You can pick the model. And when you do something like this, what you end up getting is a result that looks very similar to like [00:17:11] the published results for SWE or Terminal Bench, but on your company's own real runs. For instance, we're looking at like a sort of cost verse uh correctness trade-off for a certain set of things here. And like you want to be in the upper leftand corner. So you want like lowest cost, highest correctness. These are these models are now slightly older because the world's changing so fast. But the idea is basically, you know, we could see, okay, GPT56 soul here was like 95% correct on this task set, but it cost six bucks on average. And GLM 53 flash was 93% and cost two bucks. And then the cool thing about this is you can take this information and feed it into a model routing strategy or harness routing strategy honestly that's defined specifically for your own workflows in codebase. And so the you know if I had again to zoom out the really big picture thing that I think is so cool and powerful here is like you're getting out of the realm of guessing and you're getting into what we [00:18:11] call factory engineering where it's like you are taking a real engineering mindset. you're like, how do I we have this amazing new primitive of of coding agents? How do I know as a team, as a company that we're actually deploying them in the right way? And so, you know, we're trying to build this infrastructure and tools for companies uh to do this. And like this approach is really really resonating with a lot of companies that have been trying to build, you know, their own their own versions of this. So, that's like a quick tour of like what do I mean when when I talk about factory? I can I can sort of stop talking for a sec. I do think that you know the the the power power move I guess is to sort of thought lead so hard that everyone ends up trying to build warp even if they don't use warp. >> Yeah. >> So you know you have on your landing page something I I like in your talks you've done a few of these as well. Uh something I really like also is any surface right? It really has to be any surface. It has to be factories code. um [00:19:13] you have to have the eval dupe like what is the whole like people calling this the ADLC I don't know if you use that term it looks like you use SDLC still you know all that stuff this is how I would think about it again whether it's warp or not if it's I would obviously you know warp warp is this is what we're fully focused on we have an awesome platform for it but just in general what I would think about if you are trying to build something in this space is a I would view it as like a stack so as infrastructure at the bottom of the stack. I think it's really foundational and important that you define your factory declaratively as code. So again, the analogy is like terraform here. You get the same benefits but actually more. So the benefits you get from defining it as code is like you get a versioned view of like what was your we SDLC, AI DLC, PDLC, whatever DLC you want. You get a version view of that and then you get the ability to create branches off it [00:20:14] which you can test. You get the ability if you change it to revert it. So all the obvious things about source control that are great and then you get the added benefit that's extremely cool where because it's in code coding agents can suggest improvements to it. So to me this is like the foundation of a self-improving software factory setup is define your factory declaratively. The next step up in this stack as you think about architecting it is someplace to gather all of the information that agents are producing. Because what what you want to be able to do is examine that information, look for failure modes or replay those same past tasks with different variables to sort of I don't know scientifically is the right word, but from a measurement standpoint really optimize your system. So that's the mindset. So you need the data to do that. You need the agent conversations [00:21:14] and you need the agent artifacts. you need a record of all of the interactions from all of the tools to get to a result. And so what I mean by that is like you might have a a factory conversation that starts in Slack and then continues into linear and then continues into GitHub and there's a code review comment and that whole interaction is going to result in some piece of merged code. In fact, it might even result in something gets shipped and then as a server crashed and you want the ability to like retroactively go back with an agent most likely, but you could do it as a person and and diagnose how could that have been better. What was the failure in it? You also need this part will seem fairly obvious, but you need a way of getting data into this system. So, you need some way of plugging in MCPs. And then we find that as companies get bigger, they want different access for different parts of their organization. And so they want things like what MCPs can the marketing team access versus the sales team versus the engineering team. And so [00:22:15] like again you you'll find there's a bunch of complexity if you're depending on how big your organization is if you're trying to roll something like this out. You really want a compute layer. You should think of that as a separate thing. And so there's there's obviously a lot of agent sandbox providers at this point. So if you're going to use a third party one, I would make just make that pluggable. We use namespace as our primary one. Um, >> they launched it at AI Europe. Namespace is awesome. But this should be pluggable. I think that's really important. So if you wanted to use Daytona or modal or E2B or whatever, like I would architect it. So this is a pluggable component of it. I would make sure depending on your requirements that you can do different platforms. So again, by default, you know, I think you'll probably want to use like a Linux type sandbox, but for computer use, like the way that we test like iOS apps is you're going to want to have the ability to run a Mac VM and actually like have it run the simulator and click around. We have a very open approach to inference. And this is like this is very [00:23:16] much with our ethos of trying to be super duper flexible. So we find that a lot of companies right now will have pre-existing commits with Anthropic or OpenAI or Google that they want to burn down. And so we're we provide like a very big variety of like you can plug in your own inference provider or you can buy inference through us. You can even plug in your own harness which I think is a pretty big deal. So you know you you can orchestrate things like PI like I mentioned earlier basically any ACP compatible harness and then you want this improvement layer and so this is some of the stuff that I showed in the demo where it's like uh the real power of this whole system is is the ability to improve it and so just to give some stats at warp when we first started using the factory approach it was costing us about $80 per PR because we were using like opus where we shouldn't have. Our triage agent was using like a [00:24:16] really highowered model. You know, we were doing a bunch of redundant work with our the shape of agents. Like we were doing too many sub agents relative to just having fewer agents. And so we basically did a whole bunch of internal tuning and testing using these built-in features. And we were able to get the cost per PR down to like 20 bucks or 15 bucks, something like that. you know, we can measurably improve the throughput of our PRs. Over the last month, it's gone up by 40%. We measure like the revert rate and it hasn't gone up. And so, I really feel like a lot of confidence in this general approach of like observe, measure, tune. Finally, you need some sort of like orchestrator. Sometimes people talk about the orchestrator like it's the whole platform. I think it's a piece of it. What I mean by orchestrator is like you need some system that will take the factory definition code and then actually move work along the different stages calling out to the right models loading the right skills etc. You need something that will fire [00:25:17] the right automations. And then finally, you need a bunch of access points. And again, our philosophy with this, which you can I could see people have different philosophies. Our philosophy is to sort of bring the factory into a company's existing knowledge work tools. So, think of like, you know, the most obvious one is Slack. And like we're doing like 60% of our work is starting in Slack, but sometimes it makes sense to start in the task tracker. Sometimes it makes sense for for a developer who is more comfortable using like cloud code in their terminal to do a bunch of work there and then submit a spec into the factory. And so you know our philosophy is to try to to make this whole thing APIdriven and MCP driven so that the factory itself is like a piece of kind of headless infrastructure where it has all these different entry points. So I think this this stackbased view of it when I've shown it we talked to companies seems to land very well as like okay yes this this makes sense. This is a new thing in the like [00:26:19] developer infrastructure stack. It kind of sits one level back from like where CI/CD is but I think most companies will have something like this for software development. >> Yeah that has a lot of layers actually. I've never actually seen it spelled out like like you have. I wonder because I think most you know advice cognition and I've se you know cursor what all these words they all try to do some bundle of like the ver like from bottom to top and there's a little bit of bundling and bundling with all software and so there's a critical question of like look I'm trying to do my job here as CTO of a you know B2B SAS like [laughter] do I need like all these like six layers I think it's like who's this for maybe is is the kind of the question. So when we talk to companies that where this resonates the most is it has to be a company that has some some amount of scale like this is if you're like a few people honestly this is this is [00:27:20] overkill. >> Don't do this. Use the cloud code sub or the codec sub or whatever and just you'll go faster. You probably don't need >> you could use this. Yeah. Same for like any like larger team stack, right? Like you at some point you switch from like fire, yolo, whatever to we actually need a platforming system here to accelerate. >> Yeah. So you have to be at a certain scale. We start seeing that maybe around like 5 to 10 engineers would maybe be the floor where this makes sense >> and then all the way up to to very large companies, they want something like this. the bigger the company, the more that this requires, you know, implementation work. Like I'm sure if you, you know, you're working with Cognition or or I don't know if com cursor does, but you know, to get something stood up like this within a company's own infrastructure, it's not going to be purely plug-andplay. There's like, you know, the sort of complexity of a team's internal developer setup just grows with size within like but [00:28:23] there's a bunch of companies where you know maybe like it's like a 10 to 100 engineers fast growing like growth stage startup where it is pretty self-s serve actually where it's like you just if they're using like the happy stack of like Slack you know linear Jira >> Hang on did you call that the happy stack? I call it like yeah I that's like our I don't no uh no no no shade to to teams or whatever like sure but like there's like you know like a set of integrations that I find very common at startups where you can plug something like this in and get like immediate immediate value without too much setup. I don't know if how much you're working like this, but if you're working with like Devon or something, you can kind of get some of the same experience where when you're working in a some an environment that's natively multiplayer, so like Slack for instance or linear, there's all these benefits that you might not realize just by having like this central system of record where you can have, you know, for like the Slack [00:29:23] integration for instance, it's like you can get some flow of information coming in through like a server crash channel and um through a user feedback channel, you can just feed that directly into the factory to have it do the work. The factory can loop in multiple people on your team depending on who knows the right part of the codebase. And so, you know, I don't think you have to be that big for this for this to make sense, but it definitely doesn't make sense for a for a small team. I mean, it really makes the most sense for a company that's like, I want to I want to put in place the right infrastructure so that as models and agents continue to change, we have a system that we're building upon. There's a lot to go on here. The the one followup question and then I'm going to have you plug your AI talks and just generally where to where to follow up with you. If you what is the most steelman criticism of setting up all this like when you encounter criticism or push back what should people know as like here here is like what people are [00:30:23] running into. >> I'll answer this in like there's a certain classic customer where I don't think it makes sense to like go on to this particular thing and that's like >> yeah and beyond the size thing. Yeah. where we have challenges right now are companies with very very bespoke infrastructure or companies with uh like our our kind of not a good fit customer for this for instance would be like Uber or ramp like they're building their own thing they have very large internal developer teams but apart from the size thing I'm thinking like I mean there's definitely like the for some people they should it's just going to be more economical to just use the subs like for small very small teams. But that comes back to the size >> actually. >> Yeah. >> I don't want to actually omit the subs stuff because I think a lot of a lot of people go straight to API pay by API and that is between >> five to 50 times more expensive than just getting a whole bunch of subs. Do [00:31:23] people do that? Is is that like so against terms of service that like just don't bother or like >> talk to like a lot of small companies we talked to where they're like oh this system seems amazing. We want this but we're on these like collection of $200 subs that we're rotating through. I'm like rotating like 17 different subs, right? And then you can't imagine that's allowed. But I I don't the truth is I just don't really know. But like they they are and it's it's like and it makes like honestly it's like if you can do that that's going to be the cheapest way but you you you do lose a whole bunch of other benefits if you do that. Uh and it's not a scalable thing and if your company scales to a certain point they're you know you're definitely not going to be allowed to do that but it's like it it makes sense. >> Yeah. I'm just you know I'm just saying like it's the extent of subsidy like you get uh for like a $200 plan you get 8,000 of API right? >> Yeah. It's crazy. That's not that's not money to sneeze at. >> No, no, no. And it's like it's one of the I mean just from like a business [00:32:24] perspective, it's one of the reasons that you know I I don't want to be in like the pure coding agent business because you you just can't good luck fight. >> You can't you can't build a competitive coding agent on top of a thing that costs so much more money. And like that's just the reality of it. I mean Curser was brilliant. they did it and then I think that they had a perfect exit and became a model lab because I think they fully realized the economics of it. And so yeah, the the cool thing about Warp Factories is like we're actually not we're not trying to pitch like a token reselling product. We're trying to pitch basically an you know a coding agent optimization platform and a coding agent like productivity platform. So we we're we're not in that game with this product, which is a strength. And like that game is I I know there's still a few independent coding agents out there. I don't know how they're going to how they're going to do it. It's basically cognition and open code. That's it. Oh, and and Factory, of [00:33:24] course. Um let's wrap up with like a little plug for your talk. People will come see you and we we'll live stream it and all all that, but uh give us a general arc of the the sort of series that you've been developing and where where you're going to talk about next. What the talk is going to focus on is what it looks like to be an engineer in this world of uh increasing factory engineering and like what what are what are your roles? What's your job going to be? What are the two types of engineering, product engineering, factory engineering? And then what is this new discipline, if you want to use the word, of factory engineering actually look like? What are the tools? How do you as an engineer build the right skills so that you're engineering the system that builds the software and not just building the software directly? >> That is a perfect recap and overview of if people want to talk to you. Uh where should they find you? Uh if you're hiring, what are you hiring for? Anything that's like a call to action. >> Yeah. So come check out warp.dev. So most stuff is there. Uh we're hiring engineers. We're hiring on our go to market team. We're hiring on the data [00:34:25] team. We're hiring a designer. So hiring up for a whole bunch of roles. If you're interested in like my my musings on factories, I I put out a lot of content on my Twitter now. So, Zach Lloyd tweets. Uh so, come come follow that and I I'll keep sharing how I'm thinking about it. We'll do. We'll do. All right. Thanks so much for the time, Cole. >> Cool. Thanks for having me. >> [music]