High Taste Testers: OpenAI Shipped Everything. What Actually Works?

[00:00:00] I actually believe that like you know if you're building production software if you run the models non-stop overnight you're going to create more code than you can like reliably consume >> use all my existing labels as as eval essentially if it agrees like you know more than you know x% of the time and then you only have to review the parts that disagree essentially I'm still pretty much at read only write with explicit permission is where I Okay, this is the first ever High Taste testers. Welcome, guys. >> Thank you. Thanks for having us. >> So, you've been gathered because you're people of high taste that you test models and products all the time. And we're going to talk about just all the latest and greatest. Uh we we're recording this right after open dev day. uh opening eye is not in the room. We can be as honest and frank as you like as many quad users and also like I don't know all the other stuff news gro whatever. Um so we're going to basically [00:01:01] pair food with questions and think we can get the first course started >> but not squash croettes crispy fried chicken and for Ben bread >> bread >> but like very nicely presented bread I got this from like some kind of trader farmers market type thing. >> Really good. >> Yeah. Yeah. >> Dev day we just released uh we just came out of it. Uh what are people's impressions overall? >> Well, I don't know. Uh it's not really about the events. It's more about the releases. >> Yeah. >> There's Dots, there's 6.1 Soul, there's Ultra Fast, there's like the notion competitor, which is like something that we're excited about. >> Is that actually >> like it's weird. It's >> Yeah, they're really interesting, right? >> Yeah. Uh I don't know. I don't actually >> I posted I posted our like company vision video some like Opus slop which we can talk about later. Uh, but Peter replied late last night. He's like, "Oh my god, dude. We built the same thing." And I was like, "Interesting." Okay. I didn't even realize he was behind that. But yeah, it's like uh I guess it's like open like assistant shape thing of like, "Hey, how do you collaborate on what all [00:02:02] everybody's open call is doing." >> Mhm. >> Yeah. It seems like a shared whiteboard type thing. You can write the markdown, do drawings or whatever. I mean, it's pretty good, but it's like deeply hidden within the UI and it has the same problem. >> Yeah. Like it's the thing that they do for everything where like you can ask Codex to visualize in the chat but no one knows that's a thing. >> Yeah, even I mean even Dots is pretty hidden. >> Yeah, Dots is very hidden. That's weird. >> I I thought that it was going to be a standalone app actually. >> I was shocked that it wasn't. >> My first feedback to them to Tibo was it needs to be standalone app. >> I have multiple things running with Astra with Codex Go stuff notifications. I want a separate assistant. They can look over there but send notification separately. >> My theory is that it's not a standalone app. It's a standalone device. So that's that's where it's coming. Hunk. >> I don't think it's that long term, man. >> Well, longterm in AI time. >> They said end of year. >> Yeah. Three to six months is basically long term at this point. >> Speaking of dev day, Johnny IV was not in this one. I haven't seen him. Last last day, he was there on stage with Sam >> aluminum >> and he promised something's coming. [00:03:03] Where is it, Johnny? Where's Where's my Where's my physical device? >> Is Zach going to beat Open the Eye to a physical device this Christmas? That's going to be very interesting. Yeah, you're already audio. Perfect. >> They have a physical device, right? It's on my face. Muse is coming here. I wonder where dot is coming. That's not just on my phone like we said inside hidden inside the chat app. >> I mean, if you count the glasses, they're already there. I'm guessing open is probably going to do the like home assistant thing like the the Google the Amazon whatever that was called the Echo I think it was. >> So, this is very interesting. And for like probably like at least two and a half years, I've been like why has no one done this? It's very easy. My friends and I actually >> like Smart Echo. >> Yeah. Yeah. like a like Google Assistant or Alexa, but like you know intelligence from the last like 5 years or something >> like a proper like assistant harness type of thing. Yeah. >> And so like my my friends and I built one over a single weekend like a week or two ago using like Live and like it it works like you know there's not like I [00:04:03] don't know why has put it into a home like I would buy it. >> Yeah. I mean Google did it with Gemini but Gemini is just still still still trying to catch up right. It's still Gemini. >> Yeah. And like I I think the difference is like there's like the old way and there's the new like it has like computer use tools like MCP CLI all like that entire harness and no one has really done that although I haven't used Google for a long time. This was my challenge with dot too is uh it I was like I have I have a workflow that we used to run the company and it's a git repo full of markdown files and like scripts that my agent has sort of like openclaw shaped before openclaw was an option >> and I told I told my cloud running in that repo to go configure my use computer use to go configure my dot overnight. I want to wake up with a dot that can do all the stuff in this repo. It's like all the O stuff is really wonky. Like if like I don't know, we have a script that pulls like data from Mercury from our bank account to like write weekly updates on things like this. And like there's no Mercury plugin in chat GPT. So unless I like leave my codeex desktop app open and running so that my dot can talk to it, it can't [00:05:05] actually do any of the off or use any of the scripts that I've like accumulated over the year. >> Yeah. One of the most unsolved things to me about the whole story is like the where is stuff working thing is still completely unsolved. Like I still don't really get how to use they they talked about like you know they had this sort of like um you know uh uh you know mission accomplished like you can close your laptops now. >> Yeah. >> My laptop's still closed because I don't >> Yeah. Yeah. But like I don't I kind of like saw the cloud button. I tried looking at it. I didn't really get how to set it up. up like it wanted me to do all these things and then now my laptop's still open when I and when I close it, it stops working. So like >> but you texted me on the way here you were like should I bring my laptop? I was like I don't know. I think we're just hanging out. You're like cool. I'm going to leave it open at the office so that my agent keep working. >> So it's like not mission accomplished. Like whatever they did was they didn't do it. So I don't know. >> I think it's a hard engineering problem to solve the remote execution and especially for like UX and being able to do that well for for people is pretty challenging. >> Oh this actually it lets me bring up like something that is super recent. [00:06:06] This happened yesterday. Pi durable. I don't know if you guys saw >> um so basically uh a lot of coding agents by default like keep everything in memory and all that state. Um and you have to sort of do a rewrite to externalize all that state so that you can transfer easily from local to cloud and cloud to local. Uh and PI has done that >> and somehow cloud and codecs are like taking a year to do it. >> Mhm. I mean, Mario did say he's been working on Pi Durable, has like rewritten it seven times in the last year before launching it. >> Yeah, but the basic thing is like everything should be in a log. There's a file and then you append to the file and then you pick up where it left off. >> And like technically that's the internal representation for like cloud code and and codecs. They have this big JSONL file for >> what's so hard about sending things to cloud, >> but it's not it's like they haven't committed to the API. It's like we can't we actually built our own coding harness that has this property where it's a like a pendon only log because we didn't want to build against an API that the OpenAI team or the cloud code team will not actually like guarantee that this will not change in the future. >> So there's no like agents MD standard [00:07:08] for that. You're saying it's everybody's like making their own. >> There's a guy Justin U shoulder. I featured him at AIE. He's trying to do standard agent. Um, but until one of the labs picks it up, it's something there. >> Well, there's like a couple trace standards for like the people like working in Harbor and stuff like this, but it's more for like research and benchmarking a way of like packaging up. Okay, what is everything the agent did, right? >> Yeah. Um, I want to go back to what you mentioned before that, you know, PI releases something and then the big labs are catching up. It seems to in the frontier intelligence that's kind of the other way around. The frontier labs intelligence is the fastest one and then open open source is catching up. in the personal assistant space. You know, Peter Steinberger came up like with the thing that blew up and then Hermes agent it and it takes Meta and Open EI and quite a while to like release a product that's mature around this. There's a lot of stuff that tinkers are okay with >> breakages and different memory systems that they set up etc that I think these guys need to bake into something that just works for people. And I got I haven't used dots for a long time. But I wonder if if it works for you guys. If you feel like the memor is there uh that [00:08:08] like it supports well because I think the whole promise is it grows with you. The more you use it, the more memor is there, the more it knows kind of the stuff about you. >> I think the labs just really suffer on product. They're it's been a problem for a long time where they'll get something pretty right. Like the codeex desktop app was so right when they dropped that >> and it has become less and less right every time. Like I most of the updates for dev day in my opinion were regressions. Like adding the secondary extra sidebar on there feels bad. The performance got worse. >> It's it's a mess. >> It's just like there's too many of these icons we're getting. >> Yeah. There's just so much crap over there now. And the performance is getting worse. >> Yeah. I It keeps freezing. I think when the internet is not if it's not getting to the internet, the whole like input field freezes is like it's really annoying cuz I like clicking clicking clicking and like oh my internet's up. That's why I need a standalone app for my assistant. >> Yeah, >> this is my message. >> Yeah, >> that's why I use I'm find Did anyone use [00:09:10] the original dot by the way? I'm just curious. Do you guys remember >> one released by the third party? Uh >> no, no, no. Like like like three years ago. There was a company called the Dot by New Computer. It was like one of the first. You remember? Yeah. Like Jason. >> It was like really good actually. Like it's a pretty good assistant. like it didn't have all the like you know it can send email like that didn't exist then >> but like the form factor was good actually like there was just like a lot of things right there a lot of stuff is getting packed in the chat GBT app and I get why they want it cuz they have a really good distribution through the chatbt app and like etc etc but like >> but it's also forget about it >> that's why they have to hide all the cool stuff is because like if we break it for the 8 million users that are like that's right >> not super AI pill like we have to hide all the things the power trust the power users to find basically I I found I found the maps feature which I Yeah, the maps. >> What maps? >> There is maps in there. >> It has a Google maps clone >> in GBT. >> It's like hidden in like the three dots on the second sidebar. >> Like you ask your dad. >> Yes, that is a real sentence. >> I posted about it and they were like, [00:10:10] "Yeah, it's been there for a while. What do you mean?" And I'm like, "What are you talking about? It's been a while." >> Ironically, I feel like a lot of this is the symptom of using AI to write code and like build products. It's kind of funny. Like it feels like the so much like for the Codex app for example so much stuff just randomly changes all the time which is very like frustrating from a developer standpoint. Like I remember like I was trying to figure out how do I like after they did the chat GBT codeex like merger which also feels like a AI beaver like psychosis beaver dream of like you know like >> you think they should not have merged. I I well I can imagine a way to do it correctly potentially, but certainly like a drop down that changes seemingly nothing about the entire app except like the model selector and >> different modes. Yeah, I was like I I I will often accidentally be in chat GBT and like anyway they just like it was it was like especially the initial version was like quite bad >> and um you couldn't see your chatbt threads on the side. Anyway, it was a whole mess. is still for the record they they know it's not done and they're there's a [00:11:10] vision they want to work on >> but I guess what I'm trying to say is that I mean it both in the positives and the negatives like also the codeex app is like really good and like it like I think it would have been very hard to build prei like with like all the browser use obviously not even just the AI that powers like the app is very complicated but also it's just like yeah things randomly change all the time thing randomly break all the time there's like a lot of like you like it's almost like AI written text like you squint and you're like wait What? Like, >> yeah. Uh, >> it feels so directionless. >> Like, I I've kind of been thinking a lot about this lately where there's these middle ground products that are trying to be something for everybody and end up being for nobody where like I I really like Grockbot. I think Grockbot's in a great state, but it is much more like a professional personal assistant type thing. It feels dialed in for that. But then there's Muse, which is so good for the end consumer and works super well there. But now you have like Codeex getting watered down and chat GBT getting made more complicated and no one's happy. >> No, I mean this is literally like they I remember one of the things I got really frustrated with Codeex about is like they renamed it used to be called fork [00:12:12] chat. They renamed it to like either copy chat or some or it just said copy or something. It was like and I was like why? And they're like well like non-developers wouldn't understand the word fork and I was like oh but like this is I'm a developer. Like I think also I worry about that with dots too, right? which is like they're like, "Oh, it's for enterprise and it's for consumer." And I'm like, "Oh, man. Those are different products." Like those are very different products. >> Oh, there's like enterprise dots, right? >> That's what I'm saying. It was part of the same announcement. >> Yeah. They showed off how a person that's an employee of OpenAI and their dot showing up in the same Slack thread and some people send their dots to kind of check in on other Slacks and and kind of tell them about this. It's a very >> And that's not a that's not released to us yet. No, no, but but there's a very interesting uh vision of the future where we're all going we're going all going to be superpowered by multiple AI assistants kind of in our name and whether or not if Ben sends me a Slack message um would he expect me to show up in the Slack message and answer or my bot for example? I don't I don't know [00:13:12] the answer to that. But also like would you bring your dot would you put would you add your dot to our shared Slack when it has access to like your say like personal email or your IME messages or >> I think I think that prompt injection is largely solved. >> No, absolutely not. What? >> All right. Um >> he's sufficiently sophisticated. >> I can point to a mutual friend of ours Matthew Burman who brought Ply. >> Remember that episode? He brought Ply the the guy who jailbel liberator and he gave him five attempts to break into the new Hermes agent that he had although he had some stuff uh and pl wasn't able to >> well five attempts. Yeah. >> But also I I think there's like prompt injection maliciously and then there's just like >> stuff you're not supposed to see. >> If you look at like instinct like like >> oh yeah muse had an incident like this somebody showed up at somebody's like front door when he wanted to sell something their Facebook marketplace. the muse just like told that person, hey, you should >> That's the kind of stuff I'm talking about. Yeah. Like, oh, accidentally tells your boss you're looking for a new job or like accidentally like there's [00:14:13] just so many things, right? Ben can't make make this meeting like he's like >> he's busy >> interviewing someone. Yeah. Interviewing someone for your job like not doing that but you know it's just like uh scary. Yeah. >> Yeah. So uh okay like any positives? >> So I I think Okay. So I think the pattern of like always on persistent thing that messages other things and kicks off stuff. >> It is the right idea. >> It's the right idea. >> Assistant, I I'm I'm going on a crusade and I would love to to bring all of you with me that uh we used to say 2025 is the year of AI agents. Uh and then we kind of switch in 2026 proactive agents. But I think like the renaming to assistance I think needs to take place specifically because there's a difference between something that kind of you code with AI agent and something that does proactive stuff for you reads your emails and makes a decision on which ones are important to highlight to kind of reduce your load and not bring more of it that schedules stuff for you with like haircut and appointments etc. I think that that's the new pattern that most people that's not developers I know [00:15:15] scary thought outside of our bubble that's what they need. agents for them like is a scary word. Assistance is an easy one. Why? Because they know that all the rich people have a personal assistant schedule stuff for them. And I think that this is what um news is doing well. They're bringing this to >> I want to take it back to So Switch was asking positive things about that. So I'm going to take it back there for a second. >> Dot is or dots. I'm actually not sure which one who you have a dot but the product is dots, right? >> It's the first agent or assistant that I have connected my personal email to. >> Yes. I didn't trust Instinct. I didn't trust any of these. >> I tried I I used Poke. I'm very very paranoid about this sort of stuff. And so I had connected my calendar >> to Poke and that was like 8020 for me like it could it could like set you know you know it could uh change my calendar you know send calendar invites out etc etc but I I didn't go like all all the way to email. There's really three companies in the world for me that I I would trust and like in the order of [00:16:15] like Apple, Google, OpenAI and like there's not a fourth for me. >> Like >> it was it was my first time connecting email as well. I was actually talking with another founder and he's like look OpenAI already has all of my tightest most intimate inner thoughts and things like this. >> I already have my data retention configured etc. Like yeah >> Amazon Facebook >> no Amazon Amazon Facebook are not getting personal email. Yeah. >> Yeah. No. >> All right. I would not use >> this is a good temperature check. I used to me I will say I didn't I also didn't connect Pope until cognition bought them and obviously because I know I know I know I just like I know they take it seriously. >> Of course of course of course make it makes sense. >> Do you guys not think that Meta is taking it seriously >> given the residual >> I saw so I I I I was in the meta the the muse preview and uh they were like hey can you cook up your email and I said no. They were like why don't you trust us? I think do you know where you work? >> No, no, no. They I do I do trust them. It's actually more just like [00:17:15] >> based on all the data we see about you predicted. >> I will never type my my email password into anything else other than Gmail and they will ask you for my email password and I'm like absolutely not. >> They didn't have an OOTH connector at the time. >> Yeah, I see. >> Yeah, I think they need it too bad if that makes any sense. Like I think OpenAI has a lot of data. I think that they are less likely to do crazy with my data. That's like just what I feel. I feel like meta if it was if it meant like some little like a little bit more likely that they were going to win whatever they would just do it and and and pay the repercussions later. >> I don't think that last point. Sorry. >> Can I ask a question? Why do we need the assistance instead of making uh we're all developers. Why can't we make codeex or whatever agent a better experience for assistance? like like it's the same tools, it's the calls, it's >> well I actually I tweeted about this and kind of went a little bit viral. Like I the main agent I'm using right now for like personal assistance stuff is literally just Pi with like I gave it [00:18:16] its own Gmail. It has credit card. >> It has like reminders cron and I have like a very literally I was like codeex clone chatgbt qi like just works like I I I have >> better than dots. So, I didn't I haven't really dialed in any of like the proactivity stuff. And I think I actually imagine that that part is like the really hard part of this sort of stuff. I'm not I don't feel a deep sense of longing for it right now. Maybe I don't know what I don't have, right? And it's pretty useful. I really like it. >> Imagination unlocking like literally executives at the companies, >> they get so busy that they cannot manage their schedule. They have a person doing this for them. >> Like those people cannot go back. like this is a standard thing for humanity like when you go to that next level you will still be more busy but you'll be able to do more stuff when the proactive stuff will just come in. >> I don't know but I bet that we see people start I think someone is going to do a do open source kind of version of [00:19:16] something like that. I'm shocked it hasn't really happened yet. >> Like OpenCL you guys mean or >> no like OpenC and Hermes are not it like they are they are last generation >> because the a big part of why something like Rockbot is so compelling is because it has such a good mobile app web interface all that stuff feels really good to work with. >> You don't have to configure memory. >> You agree then you then you have your answer is because you need a polished product. >> Well I actually >> engineers can build polished products. No, because I actually think that what you're talking about is very correct cuz for my stuff personally, I do a lot of what Ben does where I just have all my stuff hooked into codeex and I can go directly in, use Gmail in there, do whatever I need. the assistant, quote unquote, whatever it is, it really just feels like taking a normal coding agent, putting cron jobs in it, putting maybe some like shared box on it so that it I I made my own like S3 back to Google Drive essentially and like it's amazing. >> Like it's very easy. >> How did they release remote codeex and dot at the same time? It's the same [00:20:16] underlying tool. I >> I will I have a very simple answer for you. I don't want to another problem. So I have two problems. I'm bringing on an assistant to help me solve a problem and now I have three problems. I have to maintain the the infrastructure. I wanted to reduce my load not to increase my load. Like another one like another box that I have to manage it goes down. If I run it on my Mac Mini which we bought for like Hermes's open qu many of us that Mac Mini disconnects and then it's not live. This one simple. Hey are you there? Do this thing for me by >> the future of harnesses is probably something that's shaped like that. Anyway, >> I think that a lot of why ChatGBT did the merger thing for ChatGBT and Codex is longterm it's all going to be CEX cuz they want to RL on one harness, one set of tools, one way of doing things. >> DOS, >> it'll all be the same. It'll all be codecs under the hood. Yeah, >> I think they're not there yet, but eventually it's where it'll end. >> One more thing, big thing in Dev Day, which is also hot topic, uh is decision models, let's call it OB clone. We um had a exclusive podcast with them uh [00:21:16] where they would basically say it's Luna with like a structured output wrapper right now which is interesting but like obviously they're going to improve on that. Um what are you you guys' stakes? You know that's another very very hot topic. >> It is multimodal. Jeff came out in his text only and open the biggest difference is coming. >> I I I know it's coming just okay live live a little bit in the future and just be like okay these are the same like so structured output for with AI and pipelines for a very long time. Uh the coolest thing about Jev is like most of the model providers took away the log probs and I actually like knowing hey what are the percentages of all these outputs not just here's the answer but actually being able to get a little bit more signal about the probabilities and like what were the actual like output probabilities for that token before it actually decided >> you want per token or for decision cuz Jeff gives you per decision. >> Yeah, exactly. That's what I'm saying. It's like you can use GPT40 mini to do almost all of the classifications that Joe can do like for a long time. I mean obviously it's smarter and better and faster and cheaper and whatever it is, but like it doesn't it didn't make anything like technically new possible. It's just the price and the speed make [00:22:17] it applicable for new things. >> I I would say like you could even have done decision probabilities obviously with like structured outputs in you know any other model before. They just like were not well calibrated like >> I I never trust a model to give me like a one to 100. >> That's what I'm saying. That's what I'm saying. You could say 0 to one and like you it it essentially bucketed it into zero 0.5 or one, you know, no matter how much you prompted. >> And so yeah, like it you it feels like true probabilities. I like we use structured outputs like a ton like probably like >> literally ra like >> we've trained our own models to do this like etc. Um I think that I I'm glad that OpenAI is doing it. I feel like you can actually switch more easily once you have a lot of labelled data because you can just sort of say like hey like how much does this match how what I was doing historically like if if you so once you like label a bunch of stuff if something comes out that's like 10x cheaper you can literally just say okay use all my existing labels as as eval essentially if it agrees like you know more than you know x% of the time and [00:23:17] then you only have to review the parts that disagree essentially so very very easy to switch um really really low switching costs here >> and then you can have actually a model like hill climb I don't know. We do like the poor man's Jeepa where I'm just like cool here's a case that doesn't work go at it as an eval and then keep tweaking the prompt until >> these sort of structured things. Yeah, >> that's awesome. >> There's tons of like product UX experiences that >> Jevshaped things are able to unlock as well and I I think it's yes, it'll improve from Luna uh in this current decisions API, but it's like the right decision to release decisions API. >> I actually I would ask like the I think that the product stuff you can do with this is super cool. Like what are your favorite ones that you've seen so far? Cuz I have a couple. I don't know how OpenAI does it uh for them, but approve for me is a is a thing. It's going to be a lot easier with platification models. Um different uh UX changes. I was actually kind of surprised DOTS is like pure just like texted print output and you can get dynamic things generated, but I was surprised there's not any clever creativity around UX or changing what the shape of an experience looks like with an agent. Um I think there's a [00:24:17] lot of possibility. >> What's an example? Is there something that you have in mind? Like what's the closest neighbor that we would know? Let me think about it. >> I think to me the coolest thing with the this shape is that you can stick AI in a lot of weird places that it wasn't before. Like a great example is like on a website you have the command K menu to search for things >> that is often broken and bad and weird. But if you you can just throw that in Jev and it's fast and instant to just predict out of the list where should we go? you kind of had a decision before to make if you're doing just a regular look up in the table in JavaScript and you're loading all the states or you're using AI and then you're waiting and streaming tokens etc. Now you don't kind of need that decision. It's kind of the best of both worlds in that command. >> Yeah, it's like borderline instant. So you can just like rank these 20. Okay, here are the 20 pages you could go to. The user typed these three letters, which one do they want to go to? That ranking will make a lot of weird things feel better. I think it's dope. I am annoyed at the general discourse around Jeb and all of these models because most of the p like the super viral examples are mostly stupid. Yes. [00:25:17] >> Like aggressively stupid. >> I was at a dinner or something. Yeah. Now now that I have Jeb, I can guarantee that the output is in the format I want. I'm like, oh my friend, you have outed yourself. >> They're just they're telling themselves on themselves how dumb they are. >> Can we talk about the the clones? The the cleft that came off >> the cler one is crazy. The numbers are insane. Tell us. >> What do you mean numbers are insane? the like the num there's some benchmark and the Cloudflare one is like the really small one is better like same performance as Jev like way faster and then even their really big one is better performance than Jev and like noticeably faster. >> Something strategic that Yogo did was like don't benchmark us. >> Yeah. >> You know like he repeated I I super cut your chat with Yogo. He repeated this maybe like seven times during the podcast like I don't believe in benchmark he really believes it. What are you going to do? is sort of true though like I think that is really like in this exact like structured classification it is really really really easy to overfit like a specific domain which is like could be fine right the thing is like if you have a super repeatable task and you're just doing [00:26:17] it's like it could just be fine to overfit it like like you can have an overfit classifier that's like fine essentially um if it's super well constrained but I think that the whole selling point of Jev at least is like super diverse data really good at like huge range of tasks etc so it I think probably very hard to actually like faithfully benchmark. >> So it's also hard to know which one to pick. You're saying right clouds are released something but your worked on data for two years and their data is immaculate. How do we tell? >> Yeah. >> Yeah. Which also applies to too. >> Yeah. So uh the rumors are that type safe closed a $10 billion round this week which is insane for some friends of mine. that I think the the core question of our company is if OpenAI and Entrophic and like and and and everyone else has a has something of Jeff uh would you use would you go out of your way to get a Jeff type safe API key or would you just use your open >> I think it's going to run on device like I don't want network Jeff anymore the models getting cheaper >> yeah it's already shipped on right so [00:27:18] there's a web GPU already Jeff is a thing you can run it maybe not to the level of Jeff quality but again we don't know how we tell I think class I have this very deeply held belief that classification is actually really really really really hard and it looks deceptively simple because it's like you're returning like a single you know either like zero or one you know in some ways but I think that that like literally binary classification is one of the hardest problems in the world like uh you know there's this really funny paper from open AAI a long time ago where they were like you know hallucination is just a binary classification problem and all these people were like great like open AI solves hallucination it's like no it's either like true or not true and that's like really really hard. So, I think they'll definitely be like on device versions, but I think they'll also be like big. I think that's like the whole thing that they did really well is be like essentially like big, you know,ish or like very smart generalized classifiers. I think that like you said like would I get a type safe API key? The answer is like I actually use it through the versel AI gateway. So, I didn't get one. So, I don't really care. >> Yeah. >> So, like I don't you know I don't know if that's good or bad for them but [00:28:18] >> use Jev not CL not the >> because they have their brand name right now. You have open AI Jeff and like it it's fine. It's good enough. >> I just trust it. >> There is a thing about they created this technology >> so they probably understand about their primitives are great. I don't know about open I haven't gotten access yet on them on the wait list. So I'm not sure they have the right primitives yet and I'm really hoping they will align to the same primitive so I be able to switch easily like kind of open router type thing. But they they are the experts in not only Jev the model but also how to think about Jev. Do the yoga released this thing about like how to build a hardness. At some point he said, "Ah, we thought about building a harness with Jev. We decided not to do it. Here's open source how to like add this to your harness." >> Very very smart. Yeah. The I really like the like main guy at Jev. He's been so good about like no this is a dumb way to use it. Sorry. >> Yes. >> He's awesome. >> Main guy. >> He's really good. >> He's so passionate also about this like one specific thing that brought to the world and I find no he has other things coming. I'm sure [00:29:18] >> that's the thing. I think like I think it'd be very silly to like look at a company that just like was the first one in a very long time to change like the way that people are using models and be like that's the like that's it for them like you know >> do you feel the the changes of the same magnitude as JP was? >> No. >> Do you feel the changes of the same magnitude as reasoning models was or one? >> Not quite but but like it's it's it's get it's getting there like you know it's like it's like 30% of that or something like for us. Yeah, I I think that it is especially as it gets faster. It's just Yeah, it's going to be I think there's a lot of I think it has not fully permeated yet. Um but we'll see it. >> My my hint for people listening who because we did the podcast is we actually left hints as to what the future models would be. Uh if you're smart enough, you can figure out the >> or if you have Jeb, you can run every sentence of that podcast through Jeb and say, "Hey, is there a hint there?" And you'll get an answer within a second, less milliseconds, right? You're going to get an answer. Yeah, I'll say I I do [00:30:18] think that it Jeb is still actually ridiculously underexplored. Like we're have a lot of stuff for cooking which we'll share later. >> So you just like you're throwing on everything. >> What? >> You're throwing it yourself on. >> So don't you like you train your own models? >> Yeah. Yeah. We actually still use it for that for like for what we call signals. We still use our own models. This will be more obvious like in uh we have some very very fun stuff coming in like the next uh over the next few weeks that that uses Jets. So we'll share. >> Yeah. We have a we have a thing that we were just play testing this week of like a aentic memory pipeline and it's like a mix of like it's like back and forth between Jev for structured output and then like a soul 5 5.6 or a soul 6 uh model for like more structured output and then Jeb for like sorting and reranking but it's like look at all my agent traces and like tell me what to add to my cloud MD. It's like as a building block in these systems and basically like we tried it with a bunch of smart models and putting Jev in there like reduced the cost by like 10 or 20x. So did it improve accuracy? >> This is what I say cost is very very easy to to reduce on classification. What I will say is that um in all of our [00:31:18] evals it is like pretty well aligned with like opus 55. >> Yeah. Yeah. With no reasoning. >> Yeah. >> That's the crazy thing. Um and I think if you like think about it deeply, there's like no obvious reason why classification needs as much reasoning like all the time as it does in order to get good answers. And like also if you use like any of the frontier models to do classification without a ton of reasoning it they're they're abysmal and really bad at classification. Yeah. >> Um and so >> you need a few shots >> and so that's the thing that even like speedwise is not even >> so it's a smart move for open to his decision. I I asked around I asked why and I I really wanted to joke with Sam Alman to say hey did you because yoga mentioned on your podcast right that like he came with this idea to Sam and Sam said oh that's a great idea you should work on it. So I really wanted to joke with Sam like did you wait for your buddy to like release his thing before you can come into the category. Uh what I the answer I got between the lines not to quote anybody specific is that there's only like Jeff types have proved the category exists and there's market fit for it and then open well well our [00:32:20] developers would like our offering in that area as well. So that's why they released. >> Yeah. >> And by release I mean weight listed. It's not out yet. It's not public. >> Yeah. >> Okay. We're going to move on to the main course. We are we have sumac beef tenderloin tataki style pansered um halibet and creamy lemon risoto >> sweet. >> Okay, models, right? We got the We got the Astra, we got the Soul, we got the Opus, we got the Sonnet. What are people using? What are your picks? Go. >> Mainly Astra ultra like ultra on the fastest mode possible. >> How much can you burn? Like I was able to burn down an entire weekly reset in about 20 minutes with that. It's really easy. >> But you know, >> I I think it's worth it. I think it's worth it. Um my party line has been I think Opus like quad models have a soul. Like obviously GPT models do not have a soul. Like I think you can like GPC by design. >> Yeah, by design. >> Almost by design. Like fantastic engineer, zero soul. Uh but like Opus [00:33:22] 55, Fable, they have souls. But I use Opus 55 a lot for anything that's like writing related even adjacent to writing cannot use any of the physics models which is really interesting. But >> Aster improved it a bit but it's still >> Sam Sam talked about this in very openly. He said we know our models are not good at writing. We're like working to catch up on this. >> I mean up until it was 5'5 though I felt the exact opposite. I was like I will never use a clock. >> Well after 45 you mean they went to 48 48 for example was a mess. >> You have to go all the way back to 46 to get something. 46 was the last like great oppus and 55 is 46 >> smarter. >> Yeah. Like I run AI summaries every day and the default model was 55 until Opus 55. >> Oh, we got like user like our customers got mad like they're like I can't read and they were sending screenshots of like and and it's like yeah you're right like we when we switched to 46 to 48 it was like >> do you work on imprompting at all? >> Um we like default >> well we do we do graduated. Another interesting thing about the Quad launch was that they basically came out and shhat on all previous versions of Cloud. [00:34:22] >> Yeah, >> that always happens >> because the researcher like apologizing for Opus 5. >> They they were making memes of like it just got slowly worse and then like 5 is the thing now we fix it. >> And they never admitted it until now. >> It's weird, right? >> Very different for Anthropic. I don't think they've ever done anything like that before. >> Mhm. >> Yeah. It's >> But like did so uh this is why like I actually do want to convene like it's obvious why though. No, no. I didn't I didn't I didn't I didn't actually feel it that much. >> You didn't feel that? >> Oh, really? >> Yeah. So, so like that's my problem. >> They just weren't using high chase. >> Like they don't use opus. No one at Anthroic is using Opus. >> That's actually very true. Like they use Fable all the time. It's either they're mainly just using mythos problem. >> I don't think they're using release models. They're using unreleased models. They're play testing all the time. They have unlimited and super fast as well. >> Yep. >> Which is a completely different experience than what users get. So please just don't don't see it yet. >> Can you mention your your uh Tying thing that you they told me last night? >> Oh yeah. Okay. There's one thing I want to say before that which is that I think on the topic of what anthropic uses [00:35:23] internally. I really thought that you like I think you're right. They do just use the fable or mythos or whatever class. And I thought that opus sauna all those would just be forgotten children forever especially after how bad Opus 5 was. Was shocked that they like brought it back with this >> one and and such beautifully as well. It's a great model. model. It's a really good model >> and and so now I feel like there is an offering from Entropic which is almost 5.5 and an offering from Open EI which is a soul 6.1 right >> um that make it so it almost you don't even need to reach for Astro Fable you like for the daily work stuff >> definitely for for most people I think those models are enough and reaching for those that like finish your quarter real quick really needs the big big big like refactor everything review like huge things or like yeah difficult tasks. So it's very interesting to switch. >> Has anyone tried Sonic by Fire? >> Yeah. >> It is the best model that you shouldn't touch. Like it's great for Opus. Like Opus can use it as sub agents and [00:36:23] they're like cheap and pretty capable but >> right. >> Yes. S is decent writing. OP is still way better. >> Opus is way better. >> Um but Silent has like it's the little brother, right? It has a little bit of that soul. Has a little bit of that like oh okay. you can understand things between between the lines that you kind of would expect a is like >> analytical and would not but also >> using that when you have a subscription on topic it's kind of like why oppus is almost limitless that's like a new thing also with the crazy stuff >> it's really hard to finish OPU 5.5 now >> oh no I used my first reset ever on cloud code the this week >> for oppus 5.5 >> before 5.5 it's like finally good enough that like I am in that like I haven't done this in a long time because I believe that like you know if you're building production software if you run the models non-stop overnight you're going to create more code than you can like reliably consume and ingest right now. >> And now like well I had like 10 different prototypes that I'm working on and now I'm like okay cool I actually like am doing the thing. I was like okay 10 p.m. you start all the night agents and then like 9:00 a.m. you you log on [00:37:25] and do the thing. >> No, just honestly like a lot of it on like medium or high. >> Medium high. >> But I what I'll do is I'll do like a a poor man's Ralph loop where I'll just like cue up a bunch of prompts of like do this. Okay, now make a better compact. Now do this. Now read the plan. Now keep making it better. And then compact. And so those will like queue up and run overnight rather than like having a So I kind of like it's like a Ralph loop, but it's bounded. You're only queuing five to 10 messages. >> You're playing you're playing chess and you premove every >> like I'm like like hey the design's awful. Like redo it. Uh all the architecture is way too complicated. I literally >> not stocks but one of the previous things that got solved. That's how the guy did it. He's like you can do this. You can do this to favor. You can do this. This is the best use of I've ever heard from bun >> like a lot of my version of this by the way in case it helps people is I have a detach mode um skill basically where um you it's kind of like goal but instead of like running the thing in in the foreground you run it in the background you like spin up the CLI version with the agents and you run it in background >> just tell it to go run or whatever [00:38:25] >> because then I can run 40 of them parallel. >> Yeah. >> Yeah. And then you can you can monitor the the the the the logs uh and then also uh pause everything improve the logs and then rerun. So that is full overnight. So like I asked Codex people about this whether codex can spin up different threads of codex that you can actually see and talk right and they can talk between each other kind of like bots that's nice but also they have sub agents as a as a primitive and so the codex people multiple ones said the sub agents are significantly better and faster because of the cache optimizations and how the model treats sub aents versus different sessions. So I guess switch like to to your thing >> because this is the fork sub aents in the new codeex harness, right? Yeah. How many sub aents do that you have? >> When I had 48 in my toml file, uh, Astra burned through my limits in like 2 hours. So I decided to clean that back to default. >> I keep it at like 15. But yeah, you can't just in the config be like, hey, give me more sub agents. And you can just tell 6.1 to do it. Go do this. It'll do it. >> If you if you have >> I never What's What other config things should I be doing? Cuz I actually never [00:39:25] really touch that config. >> How would I say this gently? If you have unlimited access, 1 million tokens I think is a good idea. >> No, I just set my autoco compact to 400k yesterday. >> No, but the default is like 275. No, for um forex. >> I wish you could disable compaction and codeex. Yeah, >> like I wish that it would just truncate tools or trunk like literally anything besides the summarizing. >> I wish they would use Jev for the tool calling and just like give the model back when it needs everything. >> I don't I don't know. I don't know about the >> It could work. Oh yeah. >> But compaction always compaction not always. Okay. >> So you just went you want different kinds of compaction as an option. >> No. No. I just don't want a summary compaction. Like I I just like most like a lot of the time it's just like there are a bunch of like images or a bunch of like whatever that's in the previous messages. >> You just want to strip out all the biggest. >> Yeah. Just strip out the like that >> cloud code used to do that internally. I think they I don't know if they're still doing it. Um cuz their got >> compaction is really weird actually. It's like uh the weird thing about quad [00:40:25] codes compaction is I feel like a bonus forget who it was. >> It thinks it's a new person like it'll be like hey like I saw what you were working on before. >> I've had this happen multiple times. Yeah. >> Momento moment like >> are you using like the native built-in compaction? >> Yeah. Yeah. >> Huh. >> Do you notice this or not seen that? >> It's been is I haven't used this since the 4. Like 4.0 days. So it's been a while. >> I've loved opening eyes compaction. Uh been using it the last couple weeks and >> it's slow though. It's so slow. I mean, if you're running 20 things, it >> does it matter with two months? >> Yeah, it does. >> Okay. Yeah. Do Do you actually watch your trajectories? Cuz I I I do and I feel like I'm doing it wrong. You know what I mean? >> No. No. I watch it. >> I don't I don't stare at them, but if it says something, I'm usually going to scroll up and be like, "Okay, what kind of >> racing actually like often times like cuz I especially when you're doing like heavy tasks, you'll like catch it going off so bad. >> You see the thinking, you're like, "Nope, here's what I actually >> And I always have a skill for having a side thread that watches the main thread's reasoning." Yeah. >> Uh and then just goes like >> that's the project or chief of staff pattern, right? That's cloud. I didn't [00:41:25] get projects yet, but Corser has project where like you have the one thing that manages everything and basically it kicks off sub agents for all the time. Uh that's great in Corser. I can I used um that's great in in Grabbot. I use the cloud like that like hey don't do work in the main thread at all. I don't want you to your context your context should be my conversation with you and not execution. execution should happen elsewhere and you get things about that. >> Oh, I do it the opposite. I'm doing things and then I have conversation about the thing on the side. >> Oh, wow. >> The sub thing because that that is ephemeral. >> If you're trying to do like a super heavy longunning loop, that is definitely the best way to do it. The only way I got asked her to burn down some insane problems I couldn't get any of those models to do and the only way I got it to work is put it in ultra, give it tons of sub agents and tell the main thread your only job is to send off sub agents. Put a timer on them. If they stop or get lazy, just keep push them. Just it is the pushing machine. >> It's kind of like um this pattern in bigger companies when you know developers rise to the level of their incompetence as a manager. But at some [00:42:26] point the companies really want the developers to keep developing while managing other people. I this is the opposite for me. I I don't want the main guy to code to run tools. I just wanted to tell other folks in the best way possible and then review and bring it back. The second it starts to >> that is dots, right? That is the idea of dots. >> Um I mean you can slash command that >> that's all. I don't think that's all bots are. I think that's my personal assistant. >> How I use that. >> It does shell stuff out like it can fire off codeex threads >> which I found it's very nondeterministic how it does that like more aggressively than I would like it to be. >> It doesn't I don't trust it. >> That's what I'm saying. There's just something really broken with the whole model of like where are the where execution's happening. >> It's so confusing >> and like codecs and dots and like Yeah, it's like really confusing. Okay. So, keeping on the model discussion, uh we talked about the the two big ones. Um other models that you guys have been trying thinking about. >> Where's Haiku? Where's >> the one year anniversary is coming up like next week? >> Hiku of Haiku 45. [00:43:26] >> Haiku 45 is almost one year old. It's like in birthday. >> Luna is not having the small fast kind of nonjuds. Luna is great for speed for like small stuff. >> Terra, where's Terara? What happened? >> I love that they had um the lineup they were like Terra Luna, Terara Soul. It's this beautiful thing like opening eye pixels and then they were like, "Okay, now it's six Astra. Wait a minute. 6.1 Soul and you're looking for Luna." Like and then you're like, "What happened to Terra?" They're like, "That's out now." And you're like, >> and it's one week after six one week later. I thought the opening I finally got their together with the naming saw terra and Luna was beautiful and easy to understand. Screw it up. >> No one used terra. >> No one used. >> So then like saying like oh this is like you know ex you know like six terra wouldn't have done anything to this one cuz you're like >> so I think they did six soul because the brand was good. >> Exactly. That that's all it was. Yeah [00:44:26] >> should be soul. Soul should be Terra and >> and Luna should stay the same. >> They should have the model with Terra wasn't >> but no open AI needed a fable class model. >> Yeah, they needed something better than the normal. They couldn't call it soul. >> Um Sam refers to the next models as whatever next hugest celestial body there is. That's how he mentioned kind of the next model. >> It's hard to beat. >> Nova supernova. >> Got to be Galactis. >> Nebula. >> Yeah, Nebula. >> Uh Nebula. I I I I may be the only person I know that uses Muse Spark. >> I use Mark. I love M Spark. >> Oh my god, you're such a muse show. Okay. Yes. Yes. >> Um >> it's use code. Yeah. >> First of all, it's 1 million context multimodel, right? Other models don't give me that. >> What do you use it for? What do you use? >> Speed and like where like coding or newsletters? >> No, no, definitely not coding. >> Definitely not coding. The cheapest until like Jeff came out and started to rethink a lot of stuff. I would just like chuck huge chunks of text out especially on the contributor >> room. Yeah, but but but multimodal as well. So you can you can send it videos [00:45:27] and say, "Hey, when did Ben say this thing?" For example, and not transcription wise, it can hear. So it can it can hear like the you know the claps for example. >> Gemini also does that. They've done that. >> Yes. Gemini >> Gemini is kind of behind on that. >> Wait. Okay. So you think Muse Spark is better than anything Gemini has? >> No. M Spark is just more more useful and significantly cheaper. M Spark is cheaper than than Gemini just >> including the the train on that train on me daddy mode. >> Yes. the contributor mode. >> Oh, that is so cheap. >> Cheating on me, daddy. >> Yeah, >> and they will. Don't worry. >> Uh, also Muse Spark is what's running Muse as well. And it's it's good enough for most assistant tasks, I think. >> Yeah, it works. >> Um, Grock, >> yeah, it's so bad. >> Grog 4.7 was a disappointment. Oh my god. I was like, all right, the cursor folks will give Elon the data he needs to train the models. Finally, >> walk me through. Yeah, you guys are I don't see, you know, someone releases, you download it, and what do you what do you try and then like what is your >> what's your bench? [00:46:27] >> Well, there's like there there's two sides of where I think they screwed up on this one. Is one the pricing. They were so hyped to say that it's like a better model but the same price. And that's just flat out true because if you increase the if you go over 200k in the context window, you double in price. If you turn on fast mode, you double in price. And then if you do both at the same time, you nearly triple in price. It's like one one and a halfx. So it really ends up being a very expensive model. And then the thing that they're really struggling on is their token efficiency is awful. It will do so many steps, so much reasoning. It's awful. And it just kind of >> it doesn't I don't know what purpose it serves because it's not >> it's not the fast model that's now Muse. It's not the cheap model that's Muse or Luna or whatever. And it's not the capable model that's Opus. Like Opus is not that efficient, but it's so capable it's fine. and you're paying Opus prices on Grock. The only thing that it does feel good in is Grockbot because they did RL the model on Grockbot and that's kind of what I feel like happened is they just sacrificed it to Grockbot. [00:47:28] >> Grockbot and the Grock tool itself. So Grock is not only the model, Grock is also the product Grock.com and inside Grock inside XAI that has access to real-time data from Twitter which no other model has and that gives it like superpowers in research. >> Has anyone used Grock in X.com? Yes. All the day. >> Every day. It's the best research model. >> Why? >> I I I try using it all the time and I it it always says it doesn't know my username. >> They fix it. There was a bug for like a month and a half. I hated this. It was so ridiculous. >> I would be like, "Hey, pull up the tweet where I said this. It's like, hi, I don't know who you are." >> And like, >> you know what's funny? >> A month. Yeah. >> The the built-in integration within X didn't know who you are. Grog.com knew who you are because you're logged into that person. It was literally a bug. >> The the most useful thing with it though >> to say about that. Like >> maybe they use Brock to build. >> Yeah, they need raindrop. They need rain. >> Have you tried the um I the gro build [00:48:28] CLI is actually a great CLI, but the most useful thing for it is you can have other agents call it as a sub agent because it has native X API capability for unlimited for free. >> Instead of an MCP to Twitter, you're just like if you need Twitter stuff, go. It's literally just have it call that to go get some posts or whatever you need. It'll just do it. >> That's that is good. >> I I don't know if it's unlimited. Um, you know, >> yeah, I that's fair. I don't because they released this. >> It does search. >> I've used it before, >> but because they released this, they released the O connector to X and you know, Pi and Hermes and everyone basically can use the same thing without even their their tool. You can just like add it and do O and then it just works. >> I don't Is the tool exposed when you do that? Like I it I don't think it adds the MCP to like PI or whatever when you do that. >> Uh you you can search Twitter without paying for X AI X API tokens. >> Oh yeah, you can well in build CLI you can but I don't know if like that tool I don't think just magically gets exposed when you sign in with >> the search Twitter tool. >> Mhm. >> Yes. >> I mean it's >> let's check this after this. [00:49:28] >> Yeah. I'm actually I'm so >> I definitely I know that like my Hermes agent used to do the bird CLI that Peter time released with OpenClaw and then uh because it was not it was frowned upon to use and they were starting banning folks that used like the cookie via browser thing. Uh I switched to that via the O and it works. >> Yeah. >> Oh, I could totally see the O working. It's just I don't know the tool. You would have to give it the tool somehow. >> Okay. Other model stuff. Uh DeepS um I I don't know if people are are deep into that. like basically open models versus closed like how do you how are you guys feeling about the general landscape? >> So I have four DGX Sparks. I when GLM 53 Flash came out as soon as I saw the size of the model I was like oh okay so these are going to spike in value. So I instantly just texted my dad and had him go to MicroEnter back in Ohio and get the Sparks for me before they went up in price. So I actually got a pretty good deal on them. I've made money off of compute which is stupid. It should not be appreciating in value. But regardless, the point is I've been using V4.1 Flash on the Sparks, and it's [00:50:30] really cool. It actually can get stuff done. It's very fun and kind of useful for any hill climbing problem where you can grind it down with verification or whatever. It is not useful for like true day-to-day work. Like, there is such a chasm between the frontier from OpenAI and and literally everything else that I I don't think it's all that viable at the moment. There was a moment where open source models were like six month months behind and then when Opus came out the 55 the latest one in 6.1. So like the frontier like shot straight up again and open source is not not there maybe a year. >> Yeah. No Opus felt super sold out. I run this benchmark called slop code bench which is out of like a lab in University of Wisconsin and I haven't looked at the new data but the the biggest thing I saw that was like it's less like okay this is a like perfect benchmark of everything cuz I only ran a subset but it was like for the you know 10 problems we did opus 5.5 wrote three times as much code as it was at the time it was like sonnet 5 and opus 4.8 They both had about 10,000 lines. This thing wrote [00:51:30] like 30,000 lines and literally half of it was tests. Like oh more than like 51% of the code was tests whereas the other models wrote like 10 or 15% code is tests. Really excited to publish the Opus 5.5 stuff. I don't have all the data together yet, but like >> yeah, it felt like clotted just like flattened out. It was just like like the Fable was an interesting thing, but none of their like daily use models were. And suddenly it's like okay, the new conversation is >> the new is the new hot >> to me. model capability kind of like the generations quote unquote is there's these like lines of capability that everything almost like comes and approaches and then like doesn't quite cross like they're almost as they get right up next to it and they can't get past until you break through that ceiling and then you get to that next plateau of capability and then can catch up. But I feel like at the end of the summer before we got Astra in 5.1, Fable 5.1, everyone was kind of getting up towards that ceiling of the last generation of sub agent orchestration, longunning tasks, cap very capable of solving a lot of problems, but didn't quite have that power and discernment [00:52:32] that are now in the new models. And there's just a certain something that is a step function above that now 6.1, Soul, Astra, and Opus 55 and Fable 5.1 to me are the only ones that are in this like new category. Now, >> what can you name the category? >> Uh, for me, I think the big ones on this new generation is like really good swarming behaviors, the ability, it's it's hard to put a strict word on >> anything other than trust. Like there's a certain level of trust that these new models have. Yeah. Also, there's always some like meme thing that that has to go viral when the new generation comes in, like the blender slop or the video. >> Oh, yeah. It's actually a great point. Yeah. >> Yeah. That's just this marketing, right? Yeah. >> No, I agree with you. Like, if you think about, you know, people are connecting their email like to models now, like giving the ability to especially send email. You could have done that a year ago. >> No. >> Or two. I mean, you could have >> it probably would have done most. added a tool that would like took in text and sent an email, but like no one in their right mind would have done that [00:53:32] on their actual email. But now you would >> and not not cuz all the emails would >> I said no one in their right mind. >> I was very specific now. >> How else would you get a helpful personal assistant if it doesn't know you? >> No, I'm saying a year ago. >> A year ago. >> Yeah. >> Yeah. I I will say I will just keep my same qualifications. Like I got so many emails. In fact, we had our Azure rep, like our company Azure rep. >> Oh, >> we got like started getting like o messages from their open call like, "Hi, I'm this person's open call like on our like person like our dedicated support chat, we're like we get these from people who I knew personally." >> Yeah. Yeah. Yeah. I know. Yeah. So like >> when I saw again I think on the record when I saw that ply the liberator cannot liberate uh Matthew Burman's open claw >> I and then I saw a bunch of benchmarks they're doing about prompt injection specifically and the latest models Astra I think it started with Astra and Opus 5.5 they have benchmark jailbreak prompts and like prompt injection benchmarks uh these models do not trust [00:54:34] anything that comes outside of kind of the system prompt and you cannot inject the into that level. So they have a very high hierarchy of like uh their constitution which is operained on right and then the system prompt which the creator developer gives and then the content and they always wrap this with like very this is from here to here this is like an email this is whatever it's really hard to jailbre >> with a prompt injection. >> Yeah I'm again I'm I'm prompt injection is definitely not the only thing I'm worried about like I think there's a lot of other ways to cause your life a lot of harm that is not other people trying to do it. Yeah, the models actually trying to be helpful is much more dangerous. >> And I I I also think that there's still I I guarantee that there are ways of doing pro I don't even know if prompt injection is the right word, but like as soon as you have your model responding to emails and you want it to give information some of the time, it is going to mclassify whether that is because someone's intentionally trying to get to mclassify or not like what it should or should not share. And I think I I I I believe that to be to be true. I [00:55:34] I will ask for evidence from all of you like have you seen this on like you follow a lot of people they try a bunch of >> stuff instinct is like a very you know someone asked their friends instinct for the uh Netflix 2FA code from their email I just gave it >> instinct I think uses open source models I don't know don't quote me on this but I don't think they can scale right now >> maybe like maybe later on but in the beginning they were different >> it's very much like I have mine set up such that it will like give me a permission prompt before it's yours >> uh Grockot I use Grockot for all this Yeah, >> with your emails connected. >> They are connected and I do use them there, but it is not allowed to send anything until I can see the actual draft and then hit send myself >> in this little expander thing that you need to scroll to the right to see the the worst UI ever in reviewing what it's about to send. >> No, the the one in Grockpot's great. It just shows as a message. Yeah, it's just it's like a normal text. It's great. But the what I have noticed though is like it's rare, but I'd say like one in five times there is details and things in there that I do not want to have. weird like it's a support email and [00:56:34] it'll be like hi I'm this person's grob let it fire and forget >> so okay so but but let's draw a line here right so Ben was talking about a year ago people will not connect their emails there is a line between readonest >> emails I was saying sending >> oh okay I oh I didn't realize this I don't send willy-nilly I review what it sends >> yeah I'm talking about I was talking about sending uh reading is like a different thing yeah I mean if you're especially a year ago are a lot more closed off, less like, you know, I think now people are just connecting everything. I mean, it has access to like their work stuff, your personal stuff, your, you know, you can post on Twitter for you, has your email, you know, it's kind of >> here's a problem that I'm noticing after using a lot of these tools. I got to wonder if people will just choose one and just stick with it because there is a attention spent cost at switching between those tools. I'm now at a point where I don't remember which conversation I had with which assistance. >> I know. I agree. and uh and importing their memories one to another and that's not good enough because their memories kind of move in different ways, right? If you think about them as clones, the [00:57:35] clones start living different lives at some point and this one knows about this thing, this this one did this thing and you're like, "All right, is it use case based? Do I go to Grock only for work stuff and then use for personal management stuff?" How do you guys deal with like that? I I would love to know. I I don't have a good solution. I have a solution. It's not good. I've been trying to rip things out like the the idea of like an MCP gateway. Well, one of my friends is building a company around this. It's called Executor where it's basically just a central it's a big MCP gateway that has all my connections, my Gmail, my Google Drive, notion, etc. And that lives on one of my Linux boxes at home and I have a Cloudflare DNS proxy so that I can get to it on any device. And now whenever I set up a personal assistant, I do one MCP connection. I log in and then that now it has all of my tools and all those tools include like the skills to use them, the prompts to use them, the config for each of them. So my like core connections layer, which is the really obnoxious part to set up >> is centralized in one place and all my devices share it. So if I change something there, it's changed everywhere and the actual like interface for [00:58:36] working for things ends up being pretty lightweight. Like I >> integrate with RagBot. You said like Ragbot shows you what it's about to send over email. Will that generalize and instead of their Gmail connector, it will go to your connector and still show you the preview. >> It works pretty well. There's a couple things that I need to work out on it. I do there are some connectors per thing that I do still use natively like the finance connector. I really like the finance connector in Grockbot. It's useful, but like that I'm not going to do over a direct MCP. >> Just to note that he said a year ago people will not connect their emails even for now. You're talking about taking your bank data. >> Well, read only and write read only is the line here. Like I'm still pretty much at read only write with explicit permission is where I am. >> The only one I want I want right for is I wanted to be able to create I have people who are very sophisticated about every single vendor gets a different like virtual credit card. I'm way too lazy to do that. But if I could tell my agent like hey go make a virtual card on Mercury and go for us. >> That's link.com. That's their wallet. That's exactly what it does. You give your agent that skill. It generates a virtual card for every purchase that [00:59:37] your assistant does and then you approve it. Huh? >> Through marketing. >> No link from site. Oh, okay. I need >> Yeah, >> but they're all still charging my one card, so I can't set like individual limits or set the expirations. >> No idea what it does. I still haven't been able to figure out. >> I don't want to give my agent a card thing. >> I just want a different card for AWS versus like G Suite versus etc. >> All righty. For dessert, we have lemon blueberry cake and dark chocolate Kreml mousse. I use Instinct for things that don't need my data like just like going and communicating with businesses and helping me book like events and things like this. And then my personal assistant is still a git repo with a ton of scripts and like the whole brain the whole memory system is marked down and I just launch CL codecs in there when I want to do stuff. >> Mine is pretty much Grockbot for all like work administrative type stuff but then I do still do a lot of the management through codeex itself. I'm using poke lindy and instinct at the same time and it's hell >> all dry message. >> I don't uh Yeah. Yes. Yes. Uh and Slack. [01:00:37] Uh so I really like the the bridge into Slack. >> Yeah. I I'm realizing how silly this sounds as I say it, but um so dots, you know, very recently. Uh Devon seen that a bunch >> as a as an assistant and not as agent. >> Can you show us both? >> Yeah, us both. >> Codeex, my own thing that I built that has access to like my own little Google Drive setup. It's just like essentially sacked. >> There's like zero trust on anything. >> Good. >> Email. I'm very like yeah like like I think that you know you get off code. You get 2FA codes in some cases to your email. I'm even for read I'm I'm scared of it. >> And then there's a long tale of like other like assistants like Orchid instinct etc that Orchid or I was orchid. >> Yeah. >> Like 2 million views on Twitter, right? >> Yeah. Yeah. >> This was the one where the guy was like planning an anniversary date or something. >> Yeah. >> Yeah. Oh, it was actually good. And um and Muse too for like marketplace, shopping, like it has um I think it's had actually really good uh browser use actually. Like Muse is >> surprisingly decent. [01:01:37] >> Surprising. >> Yeah, >> for me I use uh codeex and I actually don't like do anything else really. I just connect to codeex and have all my tools and al alternative harnesses that we're building as well. But um that's kind of all I do. I have a Hermes agent that I mostly keep up to date because there's a lot of stuff there that I built. Then Grogbot around 30 of them doing different things which I kind of like like the pattern of closing one bot to do a specific thing and they talk and then uh recently muse and dots probably muse the most. Oh, and instinct. Yes, instinct also. >> Wait, so how do you section them? >> I that's a problem that I want to bring to the table. Yeah. Uh so Gro is mostly for like work stuff like personal company management. I have like a chief of staff over there, a PM. I have a linear connector between all of them that they all talk through linear to ticket so they can tag each other. That seems to be apparent that's working. And then uh Muse for one off Muse seems to be the best when I talk to it to do something just like record the voice [01:02:37] message like hey what about my hotel? How far away is it from this place or that other place or when do I need to live for that place? That's m like immediately. And also just send all three of them stuff that don't have independency or don't know how to pronounce the word. If there's a task that all three of them can can do without causing harm, I would send to all three of them just copy paste just to see who's better, who's faster. >> Use other >> actually for the Grockbot users. I'm curious like do you ever have it dispatched to like Devon or Codex or anything like that or no? >> Grockbots have cursor integration and it's amazing. So, Grogbot can work as a chief of staff for you and then he can can kick off a cursor cloud agent for with any of the well now they're not getting Astra, right? They're they're only getting the the soul models, but it can kick off like like Opus 5.5 and you can tell it hey like don't do this yourself on your computer. Kick off a cursor cloud agent with Opus 5.5 to do this. >> It's significantly better than obviously the Gro 4.7 that's on the line there. Uh so that's the pattern to like get it to [01:03:37] do actual stuff. Even writing you can ask about to always write via cloud thing in cursor. >> I feel like I have enough code like I feel like between codeex and devon like I'm kind of good right now. >> Yeah, cursor good but it's such it to get anything good out of it. It's so much work. Like you have to set up the cloud environments for every project. You have to get the environment variables. Make sure the scripts are good. It's easy for the agent. Like it's so much work. >> It's also like it loves developing. So if you ask it to write something, it will send a PR. I don't need a I just need an answer. Write this in. But it has a pull and he loves a good pull request. Man, >> I think I one thing I love about Devon that I for some reason maybe cursor has fixed this now, but like uh I saw like almost every company fall in the same trap is like Devon like by default is just like it's just a computer, you know? It's like like an agent on a computer and like you know you can like create a repo, you could like you know find like you know use an existing repo can work across repos like just a computer. So it's just like super flexible in that way. >> Yeah. And so I never have to I like [01:04:38] every like wild idea I ever have I like that's where I go. And whereas like um not not Codeex cuz I think it'll probably die like as soon as I close my laptop and like not anything else because it's like oh like which repository do you want to work in? It's like I >> figure it out. You have the GitHub connector, right? >> Sometimes you don't even want to be in a repository. You just want like a thing. you just have an idea and if it's trying to work within a repository it's like oh well I'll add it to this page and I'll like I just like want to see this like try out this new model try out jab try out like whatever it is it's so nice to just have a sandbox >> then I think this is like the difference also between assistant and like the coding agents that were built for coding they're trained to do the diff work they're trained to write a markdown file they connect compare it to something shove it in a doc somewhere that doesn't execute write tests whatever I mean just tell me tell me like write the thing and and give it back and I think that like all of them are that >> the whole system prompts like well I can't do that. What do you want to code today? I don't want to code. I just want you to be you unh my model for me. Thank you. >> Any things uh other than the these are [01:05:39] the common names any things that are maybe way lower under the radar that you want to you think deserve more recognition? Uh, this isn't on personal assistance, but I do think it is very relevant. Is 6.1 soul is much much better than I think most people think it is because of just weird branding stuff that's happened. Six soul happening last week and now 6.1 soul happening this week. It's weird. I agree. It's an incredibly good model for an incredibly good price. It is worth looking at and driving. >> It's just like to compete with Opus 5.5 basically. >> It is literally just kneecapping OS. I heard between the lines don't don't want to name sources that it wasn't named to be soul and then last minute marketing staff decided that like hey we need to like make up on whatever happened. >> No the the headline was that they had an extra and he pulled it >> for a personal assistance. >> Uh it doesn't have to be personal assistant anything underrated whatever you're enjoying whatever you think is cool. >> For me it will be personal assistance if you don't mind. Uh because I think it's an incredible unlock for regular people. We're developers all of us. We do stuff [01:06:40] with these models for a long time. High taste testers will like test between the model. The normal person does not care. What personal assistants give to normal people is just like time unlocking. Doing stuff that otherwise would take them some time. And I think uh the labs now understand this. And I think that this is why they're all pivoting towards the the simpler use case. They also get the benefit all of them of how much token this will cost per user because most people still use charges Google. They type in a query, they get a response. Most people don't run agents 24/7. Now with personal assistant suddenly they are hooked on that like sweet sweet uh streaming of tokens that that most lab like enjoy but they also get productivity in time. And so I think it's cool to just not also talk about the products but use cases because I think in the limit most of them do the same things. Most of them can do booking. Most of them can reply to your emails and scan them. But I think it's very interesting to talk about what use cases can people do. I I can give you one that I did that I noticed that an assistant can do better job than like a coding agent before. Uh I woke up one [01:07:42] weekend with my kids early to take them to a karate class that was closed. I arrived there and it was closed and I got so pissed that now in the 2026 proactive assistance era, there's nothing there that alerted me like, "Hey, you're going to a class that's closed that doesn't know my schedule." So I built a weekend plans bot for kids and and for my wife as well that every like as the weekend comes up goes and searches all the local places all the places I would never go to websites like ah cheap and fast or like good and expensive it just plans and says hey if you want here's a bunch of things for oh and by the way the stuff that you're meant to do is close that within as well that like is a good use case now I'm like I know that that corner is covered >> I think that uh Google AI search overview or that whatever that thing Super underrated. >> Really? >> So underrated. Every question that I have that I want a fast answer to, I Google it. >> Wow. >> I do too still actually. >> Like it's really fast. It's like it's like it's like 300 milliseconds or something. by the time the page loads [01:08:43] the answer is there and like chatbt it's like I'm always in like the max reasoning setting change it and then like but then if I change it then if I make a new chat it's on that stupid setting >> so like I'm like it's just more it's like 10 30 seconds you get an answer >> you can keyboard shortcut to instant uh which is still there >> I should I should look up how to do that or whatever that >> but actually the way I do it is recast I don't know if you Yeah, recast with AI. You just like, you know, command enter and then type in your thing. Enter. Done. >> Oh, is it good? >> It copies it to your board. >> What does it use? >> Whatever you >> Whatever you want. So, I I set it to whatever. >> Okay. Well, actually, maybe I actually >> It's interesting you said this thing about like asking Muse like, "Where's my hotel and stuff?" Cuz I've actually started using Instinct instead of Google. Like, I was traveling and I got off a train at midnight in Amsterdam. I'm like, "Where's the nearest train station?" And like, "Can I pay with a tap?" And it's like, "Yep, you can tap your card and like go I go make two rides out of the door." And I'm like, it's not probably not as fast as Google, but it's much less overwhelming if I [01:09:44] just want like a yes or no or a very short answer. It's it's very nice for and I'm not going to open the chat GPT app for all the reasons that you just said. >> No, actually everything that all you guys are doing is actually like suboptimal for your best experience. >> Yeah. >> Which is one thing should have all your searches ever. >> Exactly. >> So like you shouldn't route. Routing is bad. >> Yeah. >> But we also we you can't be high tester without testing. Yeah, >> you have to test around to know like the differences in other stuff. The thing that I'm super excited about is the hardware integrations. So, Grogbot now exists in a Tesla once they release it a little bit, but like the ultra whatever tier now sits Grogbot in your car. >> It's in It's amazing. I I've tried I have a Tesla. >> Really? Not Grock. Grogbot specifically. >> No, you just like you do the voice to text with Grock and you're like, "Hey, go tell this bot to do this thing and be like sending it to the bot." And then it will tell you, "Oh, the bot responded with whatever. car drives it. It's so good, >> dude. The car drives itself and does stuff for you. It's like what kind of beautiful future we're living in. I'm super excited about this. That's so [01:10:45] integration into hardware, I think, is is is something that excites me. Muse own glasses and use devices like the charm thing. >> I I I'm going to make a prediction. Small prediction. You guys will see if it's going to work. The charm thing that that Zuck showed, like the little Tamagotchi Muse lives in is going to get sold out immediately. >> Oh my god. >> Like immediately. I think I have plans in my agents to buy like six of them. Every agent is going to buy one. Uh because it's like if you think about people who are new to Judge BT or whatever and like new to assistance and they get on board on news that's a perfect Christmas gift just a perfect hey here's the thing >> charm it's a Tamagotchi >> but also like I don't know my my feeling on some of these things is like why why the hell has this not like happened yet? Why why is it not on the Apple? >> Oh you want to get into the why Apple is not in the race? >> Not I know why Apple is not your phone and everything else won't be your phone either. >> Yeah. >> Same. you don't it's just I I actually firmly believe that like the number of things to charge is saturated and like some people are able to uh they're responsible enough to handle a watch but [01:11:45] like it's actually a pretty small number and I'm not like the reason why I don't have Apple Watch is that I would always forget to charge it. >> This is a solve problem. Just bring those little battery chargers. >> No, it's not it's not solved cuz then I have to like remember and you're like the solution is another thing to charge to get to charge the thing does not work. $15. Buy four of them. Put it everywhere you're >> It just It doesn't work for me. It doesn't work for you. >> Uh cool. We're almost at a wrap. Thank you. Thank you guys so much. Uh actually, uh this is for you. Uh what's one thing that people can do to you do for you? How do they find you, give you money, whatever? >> The one thing that people can do for me >> or to you >> is or or to me to me as a m I would always like appreciate a massage. A random on the street approach me, start a massage, it's going to be fine. I want people to unlock their imagination about what's possible with AI systems. I think we're moving into a new era where a lot of people will get a lot of benefit from AI finally. I think it's very important to counter doomerism as well because there's one thing to see oh it's coming for us on the news and completely another thing that oh it actually booked [01:12:46] me hairstone and saved me like five minutes once people feel that tactile like helps me saves me time they will definitely move towards oh AI is actually like helpful I think it's very important all the companies move there so the one thing they can do for me is unlock the imagination try uh AI assistance and if they want to know which ones we talk about this on my podcast called Thursday I every Thursday >> okay I at RC and uh we're doing a lot of models uh that are going to be pretty good and open source and coming out soon and uh and I'm also building harnesses over there. Checking me out on Twitter at Allisonology would be uh good as well. >> So in addition to being a you know a ship poster on Twitter, I also run a very serious AI company that makes agents more reliable. We work with like Fortune 100 healthcare, a bunch of Fortune 500 companies and some of the like fastest growing AI uh new startups. So um give it a try. It's uh free but for most like most people like if you're under like a thousand runs a day. >> What's it called month? >> It's called raindrop. Sorry detail. >> Raindrop. >> Yes. [01:13:46] >> CTO. >> Yeah. So um I have a YouTube channel and a podcast. If you just look me up on YouTube as Ben Davis or Nerd Snipe, you will find me there. And yeah, >> uh I work at a company called founder of a company called Human Layer. We're building the software forge. I guess we're saying software forge instead of software factory now. The forge of the future >> because factories fail. >> Yeah. >> Well, not all of them. >> Factory reference. >> There are certain things. >> Oh, no. Dude. >> Oh, no. >> No comments. Uh, >> no. No. Your AI talk actually went like super viral like >> Well, it was misinterpreted because everyone saw it and said like, "Oh, you're the don't like software factories guy." I was like, "No, no, no. there are things that you could do that would cause them to fail and I'm going to tell you what those things are so you can you know not do them but um anyways yeah I'm Dexory on Twitter check out human layer and uh yeah welcome to the future I guess I don't know >> I know >> well until the new future comes in 3 months >> can do a personal reference for Dex I [01:14:47] think um met Pacock one of your guests also uh he said there's like a few folks whose skills he trusts and Dex was one >> talking about markdown skills I'm not like skilled in real life >> I'm talking about markdown skills >> I think if I give you an You can you can >> Yeah, I I submitted Maya for consideration next time. >> Uh anyway, this has been a great uh first episode. Thank you guys for being part of it. This has been High Taste Testers. The thing you can do for me is like and subscribe. >> There we go.