How to kill your SaaS: what we learned replacing a $43K/yr tool with AI agents

[00:00:00] Every few weeks, someone on my timeline declares that AI killed SAS. Then someone else replies that you can't vibe code your way out of a real business problem. Back and forth, back and forth. So over, so back. So at Space, we decided to stop arguing and actually try killing a real life SAS and evaluating it on real business evals. [music] This is a special episode recording on that experiment. So let us know what you think. And you literally put it into an LLM and you walk away for like 2 hours, like 3 hours. you just get the entire thing pixel perfect build. >> Um I'm normally the person who has to set up session board and so I was the one actually having to suffer through our actual session board. I can't ask someone who's like you know at a multi-million dollar company to refresh their cash because something we did broke. [music] So quick context, I run two businesses, the Inspace and AI engineer. An AI engineer runs conferences for the AI [00:01:01] industry. We're looking at event software like session board to manage our speakers when we buy a bunch of software uh all over from tens of dollars to hundreds of thousands of dollars to fit our various events and platforms and needs. Um for this particular piece of software that we needed, they quoted us $43,000 a year for four events. for three grand is a lot of tokens. And when we buy a software off the shelf and file a change request, it often takes something like 3 months to 6 months or never for them to actually ship something that we want to use. We've run seven AI engineer events and we know how bad this category of software is. For years, I've had it written into our literal company founding contract. I can literally show you that nobody could build internal software without my explicit approval because it is so easy to get not invented here syndrome with the general running of a business and it's so easy to get distracted into building commodity software that you can just buy [00:02:01] off the shelf. This year I changed my mind. I think agents are finally good enough for us to have personalized software and I'm putting my real money to the test. The other thing that we found in terms of buying software is SAS is usually a bundle of things. We use maybe a third of what this particular vendor was selling us and there's no discount for the stuff that we don't use. So on a whim basically I put up a challenge called kill my SAS. Anyone can sign up and I give them the specs the jobs to be done of this particular piece of software and the hardest thing to get a real eval from real buyers who are aren't interested in how you implement things. They just want to get done. And that's basically my events team. They're not technical. They don't care what stack you use, what agent you use, what model you use. They don't care if it's open source, closed source. They just care whether they actually buy it and use it. So ultimately, 72 people or 72 teams ended up submitting. Most of them are still live and open source, and we'll link to all of them. So go poke at them yourself. So why is this harder than it looks? I want to use us as a case study for [00:03:02] exactly the kind of routine mid-tier business software. You know, it's not Salesforce. It's not something that you vibe coded in a day. It's something in the middle where there's billions of dollars of just crappy software that regular everyday people are using. Um, our software is like still relatively complicated. I think it's easy for people who don't do or don't have any exposure to this kind of business to appreciate why it's hard to replace. Um, so for example, here are some of the jobs to be done. Speaker submits a talk. Reviewers score the talk. The review committee accepts the talk. Now we need a speaker portal to upload a headsh shot and slides. Speakers get reminders if they haven't completed some tasks. We place them in a room on the schedule. We move the schedule. They change their title. They have to show up correctly on the public agenda. All sorts of stuff. They have to add co-speakers. They are sponsors. Uh we have to do it all again for invited speakers and sponsors and attendees and all these stuff. And one [00:04:03] individual person can have one or two or three roles at once. You might be a speaker and a sponsor and an attendee or you might be a organizer, a volunteer, a vendor, god knows what. Sometimes we just make roles up on the fly just because of that particular situation. Event people are very riskaverse and for good reason. If software pushes us a week past the deadline, people show up at the wrong place or the conference just doesn't happen. All of that has happened to us before. we have to pay rush fees or union fees or stuff that just is just super ugly under the hood and attendees just take everything for granted as just happening because they pay for a ticket. And nobody I know loves their conference app. So here's how we ran the contest. Builders got a brief that I wrote, a walkthrough video of a real product and 42 screenshots that I sent inside of that brief. And I put up $10,000 for the winner. And since nobody builds this on a $20 cla whatever plan, uh we actually offered up to $500 in token reimbursement for anyone who made a serious attempt because there were a few uh half-ass attempts. The [00:05:05] final call basically goes to my real events team. So it's not me picking and that's what uh is basically a semi-blind eval. [music] >> Here's what they kept tripping over. There were roughly three things. The first was portals. speakers, invited speakers and sponsors each need their own portal and most submissions only built one kind of portal that was not very flexible. >> So, [clears throat] like immediately off the bat, I noticed a bunch of quality of life improvements, so I appreciated that. Um, you were one of the few that actually was able to like parse out the difference between a form and a portal, which I loved. Um, the one of the biggest things that I'd say was the detractor, and don't get me wrong, again, this is amazing for everything you've done, was we have multiple portals. So we have for instance the CFP portal, we have a sponsor portal, we have like an invited speakers portal. So having the ability to have an admin, those multiple different portals and set them up in different ways. That was also a feature we have in session board, but it was one of the few things I didn't notice in yours that everyone else had. [00:06:06] >> The second thing that people tripped up on was effectively drag and drop. One of the more complex elements of a UI that is hard for models to get across. Does the basically what we tested was does the section actually move when you drop drag and drop it and can you tell whether it's confirmed or updated and if if it's ambiguous or not if uh to in in any form um because obviously you can't have a single person in two sessions or double book a single room at a single time slot uh for multiple sessions >> but I did like the tools tips um it did tell you how to do the setup and how to to do it. The problem was like uh the tool tips was not a um it felt like an overlay over this should have done the thing that it it was telling you to do like oh click here to to do a session or click the top to do a session. I'm like why can't I just click the button to add a session there. Um so um I really did like your your clean drag and drop with schedule. Um the one of my criteria was [00:07:08] can you do drag and drop with the schedule cuz that made me like be able to say hey did they actually follow the instructions. [gasps] The third was adding a talk without pretending to submit it through the CFP form which you need for an invited speaker or a VIP. None of these three were well specified on our docs or session boards but you do only find them immediately if you use the product like a real person. During the challenge, we also published our own AI judge. It was not meant to pick the winner. It's just a sanity check for people to get their applications up and running. Basically, an RL environment [laughter] without the RL part. Uh just a sort of a environment for people to heel climb. Uh so basically our eval was we had Claude logging in as each row and clicking through 20 scenarios that we wrote as normal user flows. You can see a sped up version here. and basically using cloud again to grade it. So you can get flexibility on what you name the button, where you put it. Uh if you ever run a [00:08:09] competition like this, uh I highly recommend it. Record yourself clicking through the full uh application, turn it into scenarios, release them so people can hill climb. Um and I thought we we felt it was pretty popular. So we ran 56 apps through these evals. 40 of them had enough coverage to rank. Top seven landed within 2.3 points of each other, which is basically saturation. And I think it was very successful overall. So hill climbing against an automated eval I think is a very useful initial bootstrap step. Uh so for example we have some of these examples from lectturn to curtain call uh pointing their workers uh particularly at areas where they scored lower and just hill climbing that really quickly. I think that is a good good first task for people doing SAS coding to do and that works for parts you can build independently but we also found that you have to tie them into an overall app and that's where a lot of people fell down they would spend a lot of tokens and they duplicated features with no connection because they didn't really dog food their apps and then they submitted it. Um so really our real [00:09:12] judging after the AI judge was our events team led by Kelsey who is head of ops at AI engineer. um the ratings had a pretty poor correlation with the AI judge. So for example, six of the AI judge top 10 was actually rated weak by our humans and eventually our winner was 19th on the own on the AI judge leaderboard. So it's kind of weak. It's just uh it has like some kind of R squared correlation but not perfect and it was never intended to be. So some of the correlation stuff was on purpose. We let people deviate from the spec based on their own taste. to encourage them to because honestly this is also a hiring exercise looking for people who can code with taste. We want people who understand the job to be done instead of this specific spec and instead of mechanically cloning a product you can actually define what a good user experience looks like and step into the role of our real testers um and for us to subjectively agree with them or disagree with them. So really this competition is not really objective. the rubrics are not well defined, the spec [00:10:12] is not well defined. Um, so really, you know, if there was a checklist, we could publish a checklist. So, we did publish a checklist that can be cloned, but taste is what we really are looking for from humans that are manipulating the coding agents. Uh, and if that's you, we're hiring, of course. Um, one of our builders, Open Stash, actually told us most of what they were proud of build putting into the app wasn't covered by the AI judge at all. And they were right. And we found that because we had humans clicking through every app. >> [music] >> So, how did people actually build these cloned apps? First of all, the good news for the SAS killers, almost everyone shipped something substantial. The medium public repo was 91,000 lines of code, and people ran agents for days on end. Cloud code and codecs were basically tied. And more interestingly, 40 of the 66 uh submissions used both of them. Nearly everyone mixed model families. All three finalists use GPT soul, but so did 50 of the other 66 and so there's no alpha. How they split the work is what I [00:11:13] find most interesting because I don't really do it myself. I tend to keep one model per project so that all the context stays in one place. So I want one codeex thread to cover everything to be able to discover the past stuff that I did. Um I want one claw code thread to also do the same thing for the task that I deem them to be most fit. It turns out that a lot of these builders don't do that. They actually switch roles. And one interesting part of this whole process is that we had the builders commit their prompts and process docs to GitHub. And this is a voluntary uh uh submissions by them. So you can actually read exactly how they did it. Overall, we studied what they submitted and how they use these mixed model uh situations and we found three patterns. One, a smart model decides what to do and then a cheaper model is the slot cannon that writes what was specified. So one example was Cboard HQ. They had claude plan and review while Codeex wrote most of the code. Two, one model builds and a different one checks and usually whichever is better at driving a browser [00:12:14] or testing that's the one that checks. Um I think that right now that that is uh GPT codeex and three no model approves its own work. Green Room HQ ran a cloud team and a Codex team in separate workies and neither can merge until the other one verifies it. And that's just a principle of on policy versus off policy verification. Uh agents tend to be easier on their own work than they are on other people's work. And I think that's reflective of what humans do as well. So overall that is the process of cloning and we data mind and everything. We've we're sharing some our initial findings. Uh but now let's talk about the finalists. We interviewed all three of them and and they were so nice to come in for uh a recorded interview and agree to this recorded interview. Uh and this is the qualitative side of like humans talking to humans about the process and understanding them. Uh also we also wanted to make sure that they were real before we award the prizes. The [clears throat] first thing they all reported in common was how much work [00:13:14] they happened before any code was prompted or got written. They actually did a fair amount of preliminary exploration and specw writing. So Marco has cloned apps for clients before. His rule is to plan everything first. >> If the UX UI is done, the only thing I do is I sit down for 8 n hours and I in the most granular way I plan out the entire data schema, the business logic and like literally pseudo code. Um, so it's kind of funny because uh it's like if you take these tasks that I did like four or five years ago and like the design and you literally put it into an LLM and you walk away for like 2 hours like 3 hours you just get the entire thing pixel perfect built. >> Yazin from Conference started with research. He crawls sessions board's entire help site before writing a spec. >> Yeah. Um so the the first step sort of before uh even you know doing the direct prompting was doing a lot of research. Um, I took the uh Google doc uh I turned it into a markdown file and then I also [00:14:16] ran firecrawl on I think it was like they they crawled the entire the entirety of session boards uh help portal as well as um a few other sites that were kind of similar that kind of popped up in the results and yeah I just tried to get like you know as much information as possible and um >> our third finalist Valentin put most of his effort into guardrails and he turned them into a template that anyone can use. >> So I've made it a template rather than formalizing it. Um in many project I have to reduce this old work and this was a perfect example where I hate swap like everyone and if you have to take over this project you want to be able to write on it like code and so I spent so much time putting the links putting the whatever code reviews whatsoever and now I just made it a GitHub template. So for each new each new repository I will clone from system light so I don't have to redo this whole thing but I don't have a Google doc saying here's what I've done. Yeah, it's fully it's fully [00:15:17] open source. >> Then his agents mapped every flow in the product before he planned who works on what in parallel. >> Huge exploration of swiping everything on the internet. Basically, I just asked Codex, hey dude, you must be able to draw me every single flow of every single feature of this app. And amazing thing is that was a bunch of YouTube videos. And so he took screenshot of frame by frame to be like, oh, okay, here's what happened. And then he had a huge markdown folders of markdown with the expectation of each features. And only from that then the goal was okay well divide that into things that can be done concurrently. Some features might overlap and so you might have to do it sequentially but basically create a plan of what must start how many agents can you do concurrently for each step and then do everything. Once it's done everything um I did the manual process of everything it works like oh wait it's not perfect. Why? Oh, why? Because I see that he's actually never launching the app locally. Okay, then the next step is to redo the whole process, but the app [00:16:17] locally. So, >> Marco fed the walkthrough video and session boards API docs straight into cloud. >> Uh, I literally took the video that uh Sean shared. Um, I uh took the documentation, the API reference uh from session board. Um, uh, I was uh I was I I put it I put it into into cloud. I told it to use convex and x.js and like my preferred stack uh because like obviously speed etc is a is a is is quite important. So the convex just gives you a real-time database. It's a super simple like it will literally give you something that's highly performant and feels good without a lot of work. Um >> so if you look at all of these clips notice that nobody said they typed one prompt and walked away. really the prompt was almost the last step that you sort of lead up to after building up a full knowledge of the the problem and uh mocks and docs and specs and all this stuff. Um most of the work was figuring out what the software actually does and [00:17:17] then they can deviate from there. [music] One other thing that all three finalists did was the unglamorous part. They used their own products as a real user over and over. One of them wrote that the agent was very good at building but quite bad at noticing UX issues. Uh for example, they had an headshot upload feature which saved a photo somewhere but then the public speaker page would uh not even read from the same place because it was just coded independently. Valencian's version of this was a 5minute test. If you can't get through the product smoothly in five minutes, something is missing. >> So I just put myself in your shoes and be like, okay, if I have only five minutes to test the product, what will it be? Um and as long as the five minutes were not smooth, I wouldn't redo the whole process. >> Um meaning okay, something must be missing. Either it's context about the product, either it's context about um the expectation or maybe it's not able to test what the current state of a product is. But my goal again [00:18:18] was never to do anything manually of oh I don't like that or oh I don't like this. I would just for example give okay Zboarding is um why the hell and I would ask him to explore the traces to understand okay what comes from is it because we gave no instruction about boarding and he would tell me something like oh well this was just a title about boarding and boarding is just supposed to board but we have no idea about what's a good on boarding about in boarding okay so I know that in the process it's not only about do an on boarding it's about okay what define a good path for inboarding what define a bad pass and so ask him okay and redo the old step, the old flow, but include a step that is researching what defines a good and boring and >> Yazin just kept clicking through and keeping a running list of fixes. >> And once it was done, so it just sends me back these videos and I have to sort of click through and that's when I realized, okay, like it's actually really annoying to switch like between the different views. So that's why I did the the [laughter] [00:19:19] yeah there there actually wasn't a way to like you had to sign out and sign in and then I'm like where are the username and passwords and then I had to add those. So yeah it was uh uh that that's when I kind of realized there were some obvious things that were missing. So the the sign-in stuff and then um switching portals and then I also added resend supports and you know I kind of realized there were a few UX issues like um yeah just stuff that was kind of obvious and I was looking through it and I was like all right you know the let me fix those. So I just had like a running list just clicking through >> and net care showed up in places we didn't actually ask for. Here's Kelsey on conferencers deliverables page. >> Your tweaks were great. Like I showed this to our content manager and she's like what? That's all in one page. You don't need to click into the individual forms to see the statuses and she's just like that's amazing. >> Yeah. Structurally it was uh it looked okay after it was done. But then I was like you know it it would just make you know so much more sense if the if the [00:20:19] menu is reorganized and there were icons and you know things kind of logically came together. And I think that was one of the things where it it just ended up doing, you know, huge changes to um the the deliverables page and also the abstracts page because that that one looked, you know, it just looks very tablely and then, you know, you're trying to make it seem like an actual UI. >> So now it's time to push back on the idea that we can completely kill SAS with AI. Cloning something that looks like it works is really the easy part. And we've proven that with so many submissions that we're all honestly decent first shot clones, but somebody still has to keep it running. Take bug reports, fix it at 2 a.m. Literally last night, Flo on my team was throwing a dozen Codex agents at our token billionaires program right before it went up with a conference already running at AI engineer. If you break it, you own it, you know, and now a lot of people talk about the other way around too. If you own it, you can break it. [00:21:19] Kelsey put it better than I can. >> So, we have multiple things connecting to each other that are hodge podgeed together with certain APIs that some speak to another. Anyways, we want this all-in-one system of like we want to be able to say registration data goes to the sponsor portal. The sponsor portal can immediately go to our our schedule. Like we're having problems with the peacemeal effect and we know enough to basically say, "Hey, we want better. We could do this better. We need this better." Um, but we don't have time. Uh, so we can't prop the agents. We can't have that, you know, like as soon as something breaks, >> we we need someone to be able to go in, fix it, fix the error of it. >> So that's kind of like why we're having this. >> And here's Kelsey again on what it's like when a sponsor hits a bug and the vendor's answer is to clear your cash. >> Our sponsor came to us and said, "Hey, we have a bug." Like, "Hey, I can't unassign these tickets." And I had to go back to our chat the chat support for this. And then, you know, do all of this. And then they had to go back to their engineers and do all of this. And then their engineers came back to their [00:22:19] chat support agent which came back to me and said, "Hey, can you try refreshing your cache?" And I'm like, I'm not going to tell our sponsor to refresh their cache in order to fix the bug two steps away from this. And so this was one of those like user availability errors of like I can't ask someone who's like, you know, at a multi-million dollar company to refresh their cash because something we did broke. So there's a level of like accountability of I don't care how you fix it. I need it to not break. So it's like that level of accountability of being able to say here's the mental load of the entirety of your >> which is why I told Yazin what I want to build first in terms of building up this new internal software tools team. um we should probably like replace our own software first because I think that is something that will give joy to everybody but and and then uh then we can talk about other stuff but you know I have non-stop ideas for the team uh I just need people who know how to uh to to work with me on on like you know turning those ideas into uh things that [00:23:21] people actually enjoy using. >> Yeah. So like go deep and then um create software that's not sort of you know a small piece of the the the entire slice but the entire thing. I think that would be that would be so interesting like just seeing you know what what the experience could look like if you were to build out the entire the entire vision um in a way that no like you know peacemail software would be able to fulfill. >> Okay the moment you've been waiting for the results. Five apps cleared our bar for something we'd actually adopter adopt or seriously consider buying and three finalists plus two more. There are another 12 projects that we found ideas we'd probably steal even if we wouldn't use the whole app. Obviously all of them open source so congrats to them. Okay, announcing the second runner up with a prize of $1,000, Marco Kramer of TrackStage. We found that he had the cleanest drag and drop complex scheduling UI experience in the field and also a really well doumented repo. If you want to learn how to run an agent fleet, go read it and also check out [00:24:22] Marco's main day job, Cortex, uh, which is kind of a chatbt knowledgebased clone. First runner up, which we're awarding $5,000, Yazin of Conferencer, by far the best clickthrough experience of just intuitively navigating the whole app with every deliverable displayed on one page, which you heard Kelsey get excited about, and also real emails at every step of the speaker life cycle. It made transactional email enjoyable and that's a really hard thing to do and we have some experience with that. Um, and we came away pretty sure that Yasin puts that level of care into everything. But overall, the winner of Kilmeas with a prize of $10,000, uh, we're awarding to Valentin de Matas of Open Rostroom. He built a software factory, debugged the factory, shipped multiple portals, and a schedule that our team actually wanted to use and we'll probably be shipping to all of you guys for 2027's AI engineer conferences. My team is so excited. What we run in 2027 will probably be a mix of open roster and conferenc plus ideas from the [00:25:23] other 12. My team is so excited about this because now we know what event software built in 2026 can look like instead of software built in 2006. And the fact that we can own that software and improve it based on real user feedback. Like my people that are coding the most on the team are not engineers. They're designers. They're domain experts. They're operations people. the people actually doing the work and reporting everything and getting minute level turnaround on their uh bug reports and feature requ. So just to wrap up, that's the overall story of how we killed our own $43,000 SAS that we're about to buy. If you're renting a software that you don't love and will never own, that's actually a reasonable thing to do now. And we have published the road map to do that. cloning that something piece of mid-tier software probably costs a thousand dollars in tokens, let's call it, on subscriptions, and maybe another 10,000 more in AI engineering expertise. Uh, for which all of these people who submit submitted [00:26:23] probably you can hire some of them. But then the trade-off is that from then on you own it, which also means you're responsible for maintaining it. That's not a trivial trade, but also it's not not a negative in a strict sense because agents make maintaining and customizing your own software for your team far more feasible than literally 12 months ago. All right, to everyone to who All right, to everyone who took part in kill my SAS, sorry it took so long to get back. It turns out that evals are hard. We were running two simultaneous conferences. I'm recording this at AI engineer New York and we have AI engineer code next month and we did this on top of our regular day jobs. So the first time around it was a very chaotic process but I think the next time we have learned a lot of lessons and the next time I'm definitely planning this a lot better and not doing this by myself so that we [laughter] can get back to people and and be more structured. Um, our team has gone from skeptical about AI vioded apps to now we're demanding it and excited not about [00:27:23] the cost savings. I mean, cost savings is part of it, but using the cost savings to fund actual software improvements that we've never been able to pay for. we like like literally we're paying I much rather pay for tokens and use my time to build software that um all of my attendees, my sponsors, my speakers actually enjoy using and deserve. So uh if you want to sponsor the next kill my sess um if you want to participate I really appreciate that. Um all the repos we're going to link in the related post to this post and congrats to the winners. See you next time.