1 00:00:02,319 --> 00:00:04,960 X as well is how you were part of the 2 00:00:04,960 --> 00:00:07,440 team that built this internal Google bot 3 00:00:07,440 --> 00:00:10,240 that was Chad GPT but a year before Chad 4 00:00:10,240 --> 00:00:10,880 GPT 5 00:00:10,880 --> 00:00:12,480 >> that caught up like you know wildfires. 6 00:00:12,480 --> 00:00:14,160 It felt more than a research [music] 7 00:00:14,160 --> 00:00:14,638 project 8 00:00:14,638 --> 00:00:17,760 >> for Codeex you built it in Rust and at 9 00:00:17,760 --> 00:00:19,519 the time the model was not on 10 00:00:19,519 --> 00:00:20,559 distribution for Rust. 11 00:00:20,559 --> 00:00:23,278 >> Turns out it was quite clear that Rust 12 00:00:23,278 --> 00:00:25,199 as a language would actually be quite 13 00:00:25,199 --> 00:00:27,199 good for agents fairly quickly if we 14 00:00:27,199 --> 00:00:29,039 decided to put some effort into it. When 15 00:00:29,039 --> 00:00:30,879 someone joins the Codeex team, what do 16 00:00:30,879 --> 00:00:32,399 you tell them? How do things get done 17 00:00:32,399 --> 00:00:32,719 here? 18 00:00:32,719 --> 00:00:34,159 >> The thing that they hear the most about 19 00:00:34,159 --> 00:00:35,359 when they have a question is like, "Have 20 00:00:35,359 --> 00:00:38,079 you asked Codex?" It still surprises new 21 00:00:38,079 --> 00:00:40,079 starters that you can basically ask it 22 00:00:40,079 --> 00:00:40,640 anything. 23 00:00:40,640 --> 00:00:42,558 >> Building was the fun part, but then 24 00:00:42,558 --> 00:00:43,840 maintenance was the painful. 25 00:00:43,840 --> 00:00:45,679 >> So maintenance is really sort of like a 26 00:00:45,679 --> 00:00:47,520 tax that you pay over time just to keep 27 00:00:47,520 --> 00:00:48,960 things running. But where I think it 28 00:00:48,960 --> 00:00:50,320 changes is like a lot of it is just 29 00:00:50,320 --> 00:00:51,359 going to be automated. 30 00:00:51,359 --> 00:00:52,799 >> So you still have the concept of code 31 00:00:52,799 --> 00:00:53,198 review. 32 00:00:53,198 --> 00:00:55,359 >> The role of code review is changing. And 33 00:00:55,359 --> 00:00:57,198 the role of code review now is like I 34 00:00:57,198 --> 00:01:02,799 think 35 00:01:02,799 --> 00:01:04,799 AI coding harnesses today. But how did 36 00:01:04,799 --> 00:01:06,799 it all start? Many of you will know 37 00:01:06,799 --> 00:01:08,319 today's guest Tibo [music] from his 38 00:01:08,319 --> 00:01:10,239 generous and pretty frequent Codex usage 39 00:01:10,239 --> 00:01:12,560 resets. He was also there when Codex as 40 00:01:12,560 --> 00:01:14,159 a product started and has led the 41 00:01:14,159 --> 00:01:16,719 broader Codex team since. Today we cover 42 00:01:16,719 --> 00:01:18,560 how Codex started and why it was built 43 00:01:18,560 --> 00:01:20,719 in Rust and made open source. How 44 00:01:20,719 --> 00:01:22,719 coderies are changing inside the Codex 45 00:01:22,719 --> 00:01:24,719 team and open AAI. What it means when 46 00:01:24,719 --> 00:01:26,239 maintenance and rearchitecting are 47 00:01:26,239 --> 00:01:28,159 getting ridiculously cheap. what the 48 00:01:28,159 --> 00:01:30,239 merge of codeex into chat GPC looked 49 00:01:30,239 --> 00:01:31,118 like [music] and the many 50 00:01:31,118 --> 00:01:32,719 underappreciated engineering challenges 51 00:01:32,719 --> 00:01:34,239 of this project. If you want to 52 00:01:34,239 --> 00:01:36,159 understand how teams inside of OpenAI 53 00:01:36,159 --> 00:01:38,159 plan review and ship software, this 54 00:01:38,159 --> 00:01:40,478 episode is for you. This episode is 55 00:01:40,478 --> 00:01:42,719 presented by Turbopuffer, a ridiculously 56 00:01:42,719 --> 00:01:44,400 scalable, fast and cheap hybrid search 57 00:01:44,400 --> 00:01:46,478 engine built on top of object storage by 58 00:01:46,478 --> 00:01:47,920 an engineering team that I've really 59 00:01:47,920 --> 00:01:49,359 grown to like after spending time with 60 00:01:49,359 --> 00:01:51,438 them. Turbopuffer is the tool that 61 00:01:51,438 --> 00:01:52,959 companies like Entropic, Notion 62 00:01:52,959 --> 00:01:54,719 Cognition, and Harvey all use to connect 63 00:01:54,719 --> 00:01:56,640 their AI products to massive amounts of 64 00:01:56,640 --> 00:01:58,718 unstructured data. When I've talked with 65 00:01:58,718 --> 00:02:00,799 engineers who use Turbopuffer, the theme 66 00:02:00,799 --> 00:02:03,040 that always comes up is reliability and 67 00:02:03,040 --> 00:02:05,200 performance at scale. The reasons for 68 00:02:05,200 --> 00:02:06,399 this have everything to do with 69 00:02:06,399 --> 00:02:08,719 Turbopuffer's architecture. Turbopuffer 70 00:02:08,719 --> 00:02:11,038 uses only object storage for state and 71 00:02:11,038 --> 00:02:14,318 MVME SSDs with memory cache for compute. 72 00:02:14,318 --> 00:02:16,159 Data in Turbopuffer is organized into 73 00:02:16,159 --> 00:02:18,639 namespaces. You can think of a namespace 74 00:02:18,639 --> 00:02:21,120 as a database table or a search index or 75 00:02:21,120 --> 00:02:23,199 an S3 prefix depending on the world you 76 00:02:23,199 --> 00:02:25,680 come from. When a namespace is not being 77 00:02:25,680 --> 00:02:27,360 queried, it stays on cheap object 78 00:02:27,360 --> 00:02:30,000 storage with no associated compute cost. 79 00:02:30,000 --> 00:02:32,318 When a namespace is active, Turbopuffer 80 00:02:32,318 --> 00:02:34,318 pulls it up into hot caching tiers, so 81 00:02:34,318 --> 00:02:36,959 queries are very fast. This design 82 00:02:36,959 --> 00:02:38,479 fundamentally makes it effortless to 83 00:02:38,479 --> 00:02:39,840 scale to hundreds of millions of 84 00:02:39,840 --> 00:02:41,680 namespaces. If you're building a 85 00:02:41,680 --> 00:02:43,919 multi-tenant AI product, every user and 86 00:02:43,919 --> 00:02:45,759 their agent can have their own dedicated 87 00:02:45,759 --> 00:02:48,318 search index without any overhead. And 88 00:02:48,318 --> 00:02:50,159 each namespace can hold hundreds of 89 00:02:50,159 --> 00:02:51,759 millions of documents without any 90 00:02:51,759 --> 00:02:53,680 special configuration. You can scale 91 00:02:53,680 --> 00:02:55,919 Turbopuffer virtually without limit. And 92 00:02:55,919 --> 00:02:57,519 the performance, reliability, and 93 00:02:57,519 --> 00:02:59,680 operating model all stay the same. If 94 00:02:59,680 --> 00:03:01,598 you need to connect AI to lots of data, 95 00:03:01,598 --> 00:03:03,680 Turbopuffer should be your first choice. 96 00:03:03,680 --> 00:03:04,239 Check it out at 97 00:03:04,239 --> 00:03:07,039 turbopuffer.com/pragmatic. 98 00:03:07,039 --> 00:03:08,959 Tibo, welcome to the podcast. So good to 99 00:03:08,959 --> 00:03:10,959 have you here. Thank you for having me. 100 00:03:10,959 --> 00:03:11,919 It's so good to see you again. 101 00:03:11,919 --> 00:03:13,439 >> It's good to do this. Last time we did 102 00:03:13,439 --> 00:03:15,519 it in person. Now, now we're doing our 103 00:03:15,519 --> 00:03:18,800 video. First, I I wanted to ask you, how 104 00:03:18,800 --> 00:03:21,840 did you get into tech? When did you 105 00:03:21,840 --> 00:03:23,919 first know that you want to work with 106 00:03:23,919 --> 00:03:25,039 computers? 107 00:03:25,039 --> 00:03:27,840 >> It's a good question. It was a long long 108 00:03:27,840 --> 00:03:31,680 time ago. Um my my parents actually 109 00:03:31,680 --> 00:03:34,318 decided to move out of Brussels where I 110 00:03:34,318 --> 00:03:37,759 was born and just just thought it was 111 00:03:37,759 --> 00:03:41,519 great to um just buy a small house and 112 00:03:41,519 --> 00:03:43,199 refurbish it. But it was in the middle 113 00:03:43,199 --> 00:03:46,479 of the middle of a village with not much 114 00:03:46,479 --> 00:03:49,039 going on. I think there was like roughly 115 00:03:49,039 --> 00:03:52,239 200 people living there. Not many that I 116 00:03:52,239 --> 00:03:54,560 felt like I wanted to talk to or you 117 00:03:54,560 --> 00:03:57,280 know could make friends with. And so I 118 00:03:57,280 --> 00:03:59,759 kind of got stuck uh this is like very 119 00:03:59,759 --> 00:04:01,680 early like 8 8 years old. I kind of got 120 00:04:01,680 --> 00:04:03,759 stuck cuz like you know computers and 121 00:04:03,759 --> 00:04:05,120 you know it's like early days of like 122 00:04:05,120 --> 00:04:07,280 the the for me the internet and you know 123 00:04:07,280 --> 00:04:09,759 that was my way to learn about things 124 00:04:09,759 --> 00:04:12,318 and so just the rest is just like you 125 00:04:12,318 --> 00:04:14,158 know came from that. Um I sort of like I 126 00:04:14,158 --> 00:04:16,079 owe it to my parents to you know have 127 00:04:16,079 --> 00:04:17,839 moved into the middle of nowhere and 128 00:04:17,839 --> 00:04:19,439 then you know I had no choice but to get 129 00:04:19,439 --> 00:04:21,839 interested in computers. Once you you 130 00:04:21,839 --> 00:04:23,680 finished high school like you went on 131 00:04:23,680 --> 00:04:25,199 and went to university, right? Actually 132 00:04:25,199 --> 00:04:26,800 studying it properly. 133 00:04:26,800 --> 00:04:29,759 >> Yes. Uh I I studied mathematics, applied 134 00:04:29,759 --> 00:04:32,399 mathematics at university. I I went 135 00:04:32,399 --> 00:04:36,160 there quite quite early. Um and so I I 136 00:04:36,160 --> 00:04:38,319 graduated early as well. Like I thought 137 00:04:38,319 --> 00:04:41,279 for a long time um that I would actually 138 00:04:41,279 --> 00:04:43,360 not make it and that I would drop out. I 139 00:04:43,360 --> 00:04:46,800 had like small companies and small 140 00:04:46,800 --> 00:04:49,680 consulting business like uh like while I 141 00:04:49,680 --> 00:04:51,918 was studying uh I was like working for 142 00:04:51,918 --> 00:04:54,639 banks. I was working for I was like um 143 00:04:54,639 --> 00:04:57,120 very interested in supply chain and 144 00:04:57,120 --> 00:04:59,279 applied mathematics problems and I sort 145 00:04:59,279 --> 00:05:00,800 of like selling that and learning a lot 146 00:05:00,800 --> 00:05:03,279 through that. Eventually ended up in the 147 00:05:03,279 --> 00:05:05,600 startup world in Belgium. I did that for 148 00:05:05,600 --> 00:05:08,000 for a little while and then moved to 149 00:05:08,000 --> 00:05:10,240 London to work uh initially at Google 150 00:05:10,240 --> 00:05:12,478 and then Deep Mind and then now you know 151 00:05:12,478 --> 00:05:15,199 moved uh to be here at OpenAI like this 152 00:05:15,199 --> 00:05:16,560 California. I love the California 153 00:05:16,560 --> 00:05:18,560 weather. We can talk about that. Uh it's 154 00:05:18,560 --> 00:05:19,519 been very good. 155 00:05:19,519 --> 00:05:21,839 >> Right after university you started you 156 00:05:21,839 --> 00:05:23,839 you've founded a startup right? You had 157 00:05:23,839 --> 00:05:25,199 the startup bug in you or the 158 00:05:25,199 --> 00:05:27,038 entrepreneur entrepreneurial bug. 159 00:05:27,038 --> 00:05:29,600 >> Yeah. So this this startup was all about 160 00:05:29,600 --> 00:05:33,038 uh pharmaceutical uh supply chain um 161 00:05:33,038 --> 00:05:35,279 looking at the supply chain for uh 162 00:05:35,279 --> 00:05:37,279 clinical trials and like try to optimize 163 00:05:37,279 --> 00:05:38,959 and decide like hey you know should you 164 00:05:38,959 --> 00:05:40,720 produce more medicine where should you 165 00:05:40,720 --> 00:05:42,000 send it where should dispatch it like 166 00:05:42,000 --> 00:05:44,478 how do you avoid waste and through that 167 00:05:44,478 --> 00:05:46,959 making clinical trials more efficient 168 00:05:46,959 --> 00:05:49,038 and this was using traditional like 169 00:05:49,038 --> 00:05:52,079 nonML techniques um like traditional 170 00:05:52,079 --> 00:05:54,160 like more optimization 171 00:05:54,160 --> 00:05:56,800 solving Monte Carlo simulations these 172 00:05:56,800 --> 00:05:58,959 kinds things stochastic multi-stage 173 00:05:58,959 --> 00:06:01,120 optimization problem really and we also 174 00:06:01,120 --> 00:06:03,519 applied it on steel industry and we 175 00:06:03,519 --> 00:06:05,918 applied it to electrical grid as well in 176 00:06:05,918 --> 00:06:08,240 Europe um it's like anything that sort 177 00:06:08,240 --> 00:06:09,839 of had the shape of like an optimization 178 00:06:09,839 --> 00:06:11,279 problem we sort of like get interested 179 00:06:11,279 --> 00:06:13,199 in and you know to this day like this 180 00:06:13,199 --> 00:06:14,720 this this company still exists and I 181 00:06:14,720 --> 00:06:16,079 think they do some of the most 182 00:06:16,079 --> 00:06:18,079 interesting work uh still but it's 183 00:06:18,079 --> 00:06:20,000 changing a lot uh you know with with 184 00:06:20,000 --> 00:06:21,279 modern AI for sure 185 00:06:21,279 --> 00:06:22,720 >> but it's interesting because you kind of 186 00:06:22,720 --> 00:06:24,720 said like oh yeah that wasn't ML it was 187 00:06:24,720 --> 00:06:26,000 just the traditional stuff and then you 188 00:06:26,000 --> 00:06:27,839 go into like Monte Carlo simulation and 189 00:06:27,839 --> 00:06:29,918 optimization and this algorithm I get a 190 00:06:29,918 --> 00:06:32,160 sense that you kind of just went deep 191 00:06:32,160 --> 00:06:33,680 right that it was like okay like here's 192 00:06:33,680 --> 00:06:35,839 a problem space like how can I use 193 00:06:35,839 --> 00:06:37,360 mathematics stuff that I learned stuff 194 00:06:37,360 --> 00:06:38,879 that I didn't learn to just go deeper 195 00:06:38,879 --> 00:06:40,959 and deeper do I sense that correctly 196 00:06:40,959 --> 00:06:42,478 >> yeah that's that's why I was obsessed 197 00:06:42,478 --> 00:06:45,038 with applied mathematics is just really 198 00:06:45,038 --> 00:06:47,439 this idea of you have theoretical 199 00:06:47,439 --> 00:06:49,038 mathematics or you have theoretical 200 00:06:49,038 --> 00:06:51,038 science and physics and like there you 201 00:06:51,038 --> 00:06:54,079 just you you do it because there's 202 00:06:54,079 --> 00:06:55,600 something to be discovered and something 203 00:06:55,600 --> 00:06:57,038 beautiful about it and it's all about 204 00:06:57,038 --> 00:06:58,959 patterns and pushing the frontier but 205 00:06:58,959 --> 00:07:00,319 you don't necessarily always know like 206 00:07:00,319 --> 00:07:02,160 how you're going to apply it and then 207 00:07:02,160 --> 00:07:03,759 there was like the real world right this 208 00:07:03,759 --> 00:07:05,120 like you know there's all these cool 209 00:07:05,120 --> 00:07:07,519 problems that just lie around and I was 210 00:07:07,519 --> 00:07:08,959 like very interested in seeing like you 211 00:07:08,959 --> 00:07:10,800 know how can I make the world better and 212 00:07:10,800 --> 00:07:12,639 so like how do I apply like you know 213 00:07:12,639 --> 00:07:14,959 sophisticated mathematics you know to 214 00:07:14,959 --> 00:07:16,800 just optimize the world around me and 215 00:07:16,800 --> 00:07:18,319 that was like a lot of the thesis behind 216 00:07:18,319 --> 00:07:19,038 that startup 217 00:07:19,038 --> 00:07:20,720 >> yeah and then after a startup you ended 218 00:07:20,720 --> 00:07:23,038 up at Google and first at Google London 219 00:07:23,038 --> 00:07:25,598 it was in 2015 and I remember 2015 220 00:07:25,598 --> 00:07:27,038 Google is a really really competitive 221 00:07:27,038 --> 00:07:28,879 place to get into like maybe as 222 00:07:28,879 --> 00:07:30,800 competitive as open AI is today in terms 223 00:07:30,800 --> 00:07:33,038 of the industry or terms of prestige. 224 00:07:33,038 --> 00:07:35,439 You worked on maps initially and then 225 00:07:35,439 --> 00:07:37,360 you moved over to deep mind. Can you 226 00:07:37,360 --> 00:07:39,120 talk a little bit what what you worked 227 00:07:39,120 --> 00:07:41,199 on and and then why did you move on from 228 00:07:41,199 --> 00:07:42,879 a already really interesting space that 229 00:07:42,879 --> 00:07:44,560 you clearly loved you know like 230 00:07:44,560 --> 00:07:46,079 optimization logistics and all these 231 00:07:46,079 --> 00:07:46,560 things? 232 00:07:46,560 --> 00:07:49,038 >> Yes. I I didn't I didn't start on Google 233 00:07:49,038 --> 00:07:52,560 Maps. I started on a project that was 234 00:07:52,560 --> 00:07:55,759 meant to make the web faster. uh and 235 00:07:55,759 --> 00:07:58,879 make to me to uh make websites faster 236 00:07:58,879 --> 00:08:01,199 especially on mobile. At the time you 237 00:08:01,199 --> 00:08:02,639 know Google was kind of so like seeing 238 00:08:02,639 --> 00:08:05,519 the transition from desktop to mobile 239 00:08:05,519 --> 00:08:07,519 and like more and more traffic going to 240 00:08:07,519 --> 00:08:09,199 like you know mobile phones and so like 241 00:08:09,199 --> 00:08:10,720 wanted to get ahead of that. So funded 242 00:08:10,720 --> 00:08:13,199 like a number of a number of initiatives 243 00:08:13,199 --> 00:08:15,199 and projects. Um I was working on one of 244 00:08:15,199 --> 00:08:17,519 them. This was like really really fun 245 00:08:17,519 --> 00:08:20,959 because it was a small group um within 246 00:08:20,959 --> 00:08:22,959 actually the ads organization. It was 247 00:08:22,959 --> 00:08:24,240 meant to sort of like you know offset 248 00:08:24,240 --> 00:08:27,120 the the loss uh for the ad revenue loss 249 00:08:27,120 --> 00:08:28,560 because of the shift of traffic to 250 00:08:28,560 --> 00:08:31,439 mobile and worked on it for roughly two 251 00:08:31,439 --> 00:08:34,240 years. Um and then it was cancelled and 252 00:08:34,240 --> 00:08:35,918 although it was like the most fun I've 253 00:08:35,918 --> 00:08:38,320 had on, you know, solving hard technical 254 00:08:38,320 --> 00:08:40,639 uh challenges. I learned a lot from not 255 00:08:40,639 --> 00:08:42,240 having product market fit, not having 256 00:08:42,240 --> 00:08:43,599 the right users, not having the right 257 00:08:43,599 --> 00:08:45,839 feedback loop, not trusting your product 258 00:08:45,839 --> 00:08:47,440 manager when they say the project is 259 00:08:47,440 --> 00:08:48,879 going well when in fact it's not going 260 00:08:48,879 --> 00:08:50,320 well at all. And then you know one day 261 00:08:50,320 --> 00:08:52,720 it's just like this VP flew in uh from 262 00:08:52,720 --> 00:08:54,399 California and then it was just like oh 263 00:08:54,399 --> 00:08:55,679 yeah it's like you know we're canceling 264 00:08:55,679 --> 00:08:57,919 this project um you know unfortunately 265 00:08:57,919 --> 00:09:00,240 you only have you know hundreds of users 266 00:09:00,240 --> 00:09:02,559 and this is clearly not Google scale and 267 00:09:02,559 --> 00:09:04,399 then uh it's unbelievable but people 268 00:09:04,399 --> 00:09:07,600 were surprised um and I think there's a 269 00:09:07,600 --> 00:09:09,919 lesson there um that I that I carry with 270 00:09:09,919 --> 00:09:11,759 me of course is you know just always 271 00:09:11,759 --> 00:09:14,799 always question always go to like always 272 00:09:14,799 --> 00:09:16,879 you know deeply think about the impact 273 00:09:16,879 --> 00:09:18,480 that you're having but also like the 274 00:09:18,480 --> 00:09:19,759 importance of the overall project that 275 00:09:19,759 --> 00:09:21,919 you're contributing. And then I moved 276 00:09:21,919 --> 00:09:24,000 into Google Maps. Google Maps was super 277 00:09:24,000 --> 00:09:26,080 fun. Worked on reviews. And then after 278 00:09:26,080 --> 00:09:27,839 roughly a year, I couldn't ignore like 279 00:09:27,839 --> 00:09:30,559 Deep Mind. It was just it was this 280 00:09:30,559 --> 00:09:32,639 special place. Uh headquartered in 281 00:09:32,639 --> 00:09:34,799 London. So many great things were 282 00:09:34,799 --> 00:09:36,080 happening. This was like really the 283 00:09:36,080 --> 00:09:38,000 early days, you know, with rumblings of 284 00:09:38,000 --> 00:09:41,039 um things like AlphaGo and they just 285 00:09:41,039 --> 00:09:42,958 seemed to be doing extraordinary things 286 00:09:42,958 --> 00:09:44,958 and you know, just really tackling the 287 00:09:44,958 --> 00:09:46,480 very very hardest problems that you can 288 00:09:46,480 --> 00:09:48,240 tackle. And like with my background I 289 00:09:48,240 --> 00:09:50,159 was obviously drawn to that started 290 00:09:50,159 --> 00:09:52,720 there like I worked on uh a lot of like 291 00:09:52,720 --> 00:09:54,320 the research infrastructure research 292 00:09:54,320 --> 00:09:56,159 tooling. This is a theme that I carried 293 00:09:56,159 --> 00:09:57,839 on for almost a decade and it's like 294 00:09:57,839 --> 00:10:00,000 this is very much also the like how I 295 00:10:00,000 --> 00:10:02,559 approach things is how can I build 296 00:10:02,559 --> 00:10:04,879 tooling and products that help make 297 00:10:04,879 --> 00:10:06,958 others more efficient and bring a lot of 298 00:10:06,958 --> 00:10:08,639 utility to them. Initially I was doing 299 00:10:08,639 --> 00:10:10,720 this for research and then like over 300 00:10:10,720 --> 00:10:13,519 time you know I got like into thinking 301 00:10:13,519 --> 00:10:15,278 about things in a much more like more 302 00:10:15,278 --> 00:10:17,200 general and general and general way you 303 00:10:17,200 --> 00:10:18,799 know eventually like you know ending up 304 00:10:18,799 --> 00:10:19,759 where where I am now. 305 00:10:19,759 --> 00:10:21,839 >> Yeah. And a fun story that you recently 306 00:10:21,839 --> 00:10:24,559 shared on X as well is how you were part 307 00:10:24,559 --> 00:10:26,879 of the team that built this internal 308 00:10:26,879 --> 00:10:29,120 Google bot that was you know if you want 309 00:10:29,120 --> 00:10:32,159 to say similar to Chad GPT but a year 310 00:10:32,159 --> 00:10:34,720 before Chad GPT can you can you talk 311 00:10:34,720 --> 00:10:36,958 about that? That's a that that is a new 312 00:10:36,958 --> 00:10:38,399 story. I haven't heard it before. 313 00:10:38,399 --> 00:10:39,679 >> This was part of deep mind. There were 314 00:10:39,679 --> 00:10:41,120 like multiple efforts as well. There was 315 00:10:41,120 --> 00:10:42,559 like brain as well that was separate at 316 00:10:42,559 --> 00:10:44,559 the time. They had their own efforts on 317 00:10:44,559 --> 00:10:46,078 large language models, but it was 318 00:10:46,078 --> 00:10:47,120 definitely something that was being 319 00:10:47,120 --> 00:10:49,679 explored. It was not the main thrust of 320 00:10:49,679 --> 00:10:51,360 of deep mind. Deep mind was like very 321 00:10:51,360 --> 00:10:53,839 much worried um and and busy like 322 00:10:53,839 --> 00:10:56,000 thinking about grand challenges and 323 00:10:56,000 --> 00:10:59,360 games and you know thinking about RL not 324 00:10:59,360 --> 00:11:01,839 in the language sense. And so there was 325 00:11:01,839 --> 00:11:04,320 like this group um that was pushing on 326 00:11:04,320 --> 00:11:06,720 large language models and you know 327 00:11:06,720 --> 00:11:08,159 thinking about you know think what what 328 00:11:08,159 --> 00:11:10,720 if what if large text corpuses are 329 00:11:10,720 --> 00:11:13,759 everything. What if uh you just pushed 330 00:11:13,759 --> 00:11:16,000 language to its maximum and you just 331 00:11:16,000 --> 00:11:17,519 scaled language models like you know 332 00:11:17,519 --> 00:11:19,440 would that be enough to get to general 333 00:11:19,440 --> 00:11:21,278 intelligence? That was like a hot debate 334 00:11:21,278 --> 00:11:24,000 at the time and then one one group 335 00:11:24,000 --> 00:11:26,480 decided to just really push on that and 336 00:11:26,480 --> 00:11:28,480 then it it felt really natural like you 337 00:11:28,480 --> 00:11:30,639 know as I was building tooling you know 338 00:11:30,639 --> 00:11:32,559 with others for for research is like you 339 00:11:32,559 --> 00:11:33,759 know obviously you're like hey you know 340 00:11:33,759 --> 00:11:35,200 what can we do with this model like how 341 00:11:35,200 --> 00:11:36,799 do we present it you know to the 342 00:11:36,799 --> 00:11:38,559 researcher like how can they sort of 343 00:11:38,559 --> 00:11:40,320 like you know debug the inputs outputs 344 00:11:40,320 --> 00:11:41,759 and eventually you sort of like end up 345 00:11:41,759 --> 00:11:43,919 with you know like a chat system. So we 346 00:11:43,919 --> 00:11:45,679 built that internally. We had a lot of 347 00:11:45,679 --> 00:11:47,519 fun. Uh initially the models were like 348 00:11:47,519 --> 00:11:48,799 you know kind of like almost like a 349 00:11:48,799 --> 00:11:50,320 little bit absurd like you know not very 350 00:11:50,320 --> 00:11:53,679 coherent. Uh not super useful but it was 351 00:11:53,679 --> 00:11:55,839 a lot of fun um to sort of like tinker 352 00:11:55,839 --> 00:11:57,440 with them that caught up like you know 353 00:11:57,440 --> 00:11:58,720 like wildfires like you know this 354 00:11:58,720 --> 00:12:00,078 application is just sort of like you 355 00:12:00,078 --> 00:12:02,799 know everyone uh was kind of like u 356 00:12:02,799 --> 00:12:04,958 sharing little conversations within deep 357 00:12:04,958 --> 00:12:07,360 mind. It felt more than like a research 358 00:12:07,360 --> 00:12:09,919 project or like a research a project for 359 00:12:09,919 --> 00:12:11,440 researchers. And so then there was this 360 00:12:11,440 --> 00:12:13,440 desire over time to like launch it as an 361 00:12:13,440 --> 00:12:15,120 external product. But Deepine was just 362 00:12:15,120 --> 00:12:16,799 like not set up, you know, there was 363 00:12:16,799 --> 00:12:18,879 like the the right way to launch 364 00:12:18,879 --> 00:12:20,320 products at Google. There's like, you 365 00:12:20,320 --> 00:12:21,759 know, the whole machinery of like, you 366 00:12:21,759 --> 00:12:23,600 know, how you do that. Um, you know, the 367 00:12:23,600 --> 00:12:25,839 whole like blessed production stack. You 368 00:12:25,839 --> 00:12:27,839 obviously very very optimized over the 369 00:12:27,839 --> 00:12:30,480 years to do things well. Um, but also 370 00:12:30,480 --> 00:12:32,480 very very hard as an environment to 371 00:12:32,480 --> 00:12:34,480 truly innovate. And then I I I wanted to 372 00:12:34,480 --> 00:12:36,480 ask what made you you know look around 373 00:12:36,480 --> 00:12:38,559 or or or maybe even consider open AI but 374 00:12:38,559 --> 00:12:40,399 I feel you partially answered this 375 00:12:40,399 --> 00:12:42,240 question just just putting myself back 376 00:12:42,240 --> 00:12:44,078 into your shoes like you're you know if 377 00:12:44,078 --> 00:12:46,799 it's it's 2024 or 2023 you're inside of 378 00:12:46,799 --> 00:12:49,278 Google who are publishing amazing papers 379 00:12:49,278 --> 00:12:50,639 doing really good research you're doing 380 00:12:50,639 --> 00:12:52,240 super fun stuff right that pushing the 381 00:12:52,240 --> 00:12:54,399 limits of what what's been done before 382 00:12:54,399 --> 00:12:56,159 it's inside a company where you already 383 00:12:56,159 --> 00:12:58,078 moved you know for people who are 384 00:12:58,078 --> 00:12:59,600 feeling kind of comfortable or good 385 00:12:59,600 --> 00:13:01,200 about where they are right now which I 386 00:13:01,200 --> 00:13:03,120 imagine you must have been like what 387 00:13:03,120 --> 00:13:05,679 made you still explore all right like 388 00:13:05,679 --> 00:13:08,000 what else might be there? Yeah, I was I 389 00:13:08,000 --> 00:13:11,519 was very comfortable uh at my it's it's 390 00:13:11,519 --> 00:13:15,839 a good place, but really I had a I had a 391 00:13:15,839 --> 00:13:18,720 desire to, you know, meet great people, 392 00:13:18,720 --> 00:13:20,399 but also join a mission that I truly 393 00:13:20,399 --> 00:13:22,958 believed in and that, you know, I felt 394 00:13:22,958 --> 00:13:25,200 like the people were true to the mission 395 00:13:25,200 --> 00:13:27,360 and cared deeply about impacting the 396 00:13:27,360 --> 00:13:30,559 world in in a very in a deeply positive 397 00:13:30,559 --> 00:13:33,120 way, but also in a direct way, not not 398 00:13:33,120 --> 00:13:34,320 being like, oh yeah, it's just like, you 399 00:13:34,320 --> 00:13:35,759 know, we just do this work over here and 400 00:13:35,759 --> 00:13:37,360 then it's like it's the job of someone 401 00:13:37,360 --> 00:13:39,200 else to figure out you know how to how 402 00:13:39,200 --> 00:13:41,200 to make this useful. It's like I wanted 403 00:13:41,200 --> 00:13:42,799 to join a group where you know like all 404 00:13:42,799 --> 00:13:44,240 the parameters were sort of like 405 00:13:44,240 --> 00:13:45,839 considered together where you know 406 00:13:45,839 --> 00:13:47,278 research and product were like really 407 00:13:47,278 --> 00:13:48,958 co-designing. Open was just like 408 00:13:48,958 --> 00:13:52,000 crushing it. Uh I thought chatbt was 409 00:13:52,000 --> 00:13:53,440 like you know taking off. I was like I 410 00:13:53,440 --> 00:13:55,759 was uh I met a couple people from OpenAI 411 00:13:55,759 --> 00:13:57,360 and then I was like wait what you know 412 00:13:57,360 --> 00:13:59,679 you only have like 20 people working on 413 00:13:59,679 --> 00:14:03,919 Chad like that is that is an insanely 414 00:14:03,919 --> 00:14:05,440 small number that that must be like 415 00:14:05,440 --> 00:14:06,958 extremely empowering like you know how 416 00:14:06,958 --> 00:14:08,720 does that work? How do you manage to 417 00:14:08,720 --> 00:14:11,839 maintain you know a product with that 418 00:14:11,839 --> 00:14:14,000 level of scale um and with that level of 419 00:14:14,000 --> 00:14:15,759 autonomy with you know only only 20 420 00:14:15,759 --> 00:14:17,679 engineers and then you know as I kind of 421 00:14:17,679 --> 00:14:19,120 dug and dug and dug and it's like it was 422 00:14:19,120 --> 00:14:21,039 just a an amazing group of people 423 00:14:21,039 --> 00:14:23,198 amazing mission you know super talented 424 00:14:23,198 --> 00:14:25,679 super driven and like it was it was like 425 00:14:25,679 --> 00:14:27,519 drew me in and then I joined 426 00:14:27,519 --> 00:14:30,559 pre-ereasoning uh efforts immediately 427 00:14:30,559 --> 00:14:32,480 like typical openi fashions like I 428 00:14:32,480 --> 00:14:34,639 joined it was like oh yeah you know like 429 00:14:34,639 --> 00:14:35,919 there's this thing going on like you 430 00:14:35,919 --> 00:14:37,039 know we're going to launch reasoning 431 00:14:37,039 --> 00:14:38,639 models like you know it's like some new 432 00:14:38,639 --> 00:14:40,078 paradigm and then you know start 433 00:14:40,078 --> 00:14:41,600 sprinting on that and like you know like 434 00:14:41,600 --> 00:14:43,839 a month later like the company launched 435 00:14:43,839 --> 00:14:46,958 01 uh 01 01 preview and that was 436 00:14:46,958 --> 00:14:48,799 exhilarating to be part of I wanted to 437 00:14:48,799 --> 00:14:51,278 be part of like a place that moves fast 438 00:14:51,278 --> 00:14:54,078 cares about impact would be in tune with 439 00:14:54,078 --> 00:14:56,320 the world and you know just really 440 00:14:56,320 --> 00:14:58,720 listen um and sort of like that's also 441 00:14:58,720 --> 00:15:00,240 to me like you know what I've carried 442 00:15:00,240 --> 00:15:02,240 with me like when when building codecs 443 00:15:02,240 --> 00:15:04,320 when building products is like having a 444 00:15:04,320 --> 00:15:06,879 community listen to the community just 445 00:15:06,879 --> 00:15:09,120 really focus on like a really intense 446 00:15:09,120 --> 00:15:11,600 feedback loop uh and then building 447 00:15:11,600 --> 00:15:13,519 something that is just like you know you 448 00:15:13,519 --> 00:15:15,440 just really want to care about it and 449 00:15:15,440 --> 00:15:17,360 like you know care about the utility of 450 00:15:17,360 --> 00:15:18,720 it that it provides to the world 451 00:15:18,720 --> 00:15:20,159 >> and then of course you started to work 452 00:15:20,159 --> 00:15:22,078 pretty quickly on codeex. So you joined 453 00:15:22,078 --> 00:15:25,120 in 2024. Can you take us back what the 454 00:15:25,120 --> 00:15:27,278 thinking back there when you joined was 455 00:15:27,278 --> 00:15:31,278 about AI or LMS and and code? I I know 456 00:15:31,278 --> 00:15:35,120 there was this ASWE um effort back then. 457 00:15:35,120 --> 00:15:37,120 The we talked about it in the deep dive 458 00:15:37,120 --> 00:15:38,399 as well that we did in the pragmatic 459 00:15:38,399 --> 00:15:39,679 engineer the autonomous software 460 00:15:39,679 --> 00:15:40,480 engineer 461 00:15:40,480 --> 00:15:42,720 >> AS3. Yeah, that's what it's it was 462 00:15:42,720 --> 00:15:45,519 pronounced internally. Um we we don't we 463 00:15:45,519 --> 00:15:46,240 don't 464 00:15:46,240 --> 00:15:48,000 >> have an A3 effort anymore like you know 465 00:15:48,000 --> 00:15:50,399 it's it's it's codeex. Um but really for 466 00:15:50,399 --> 00:15:53,039 me it was I I joined I started building 467 00:15:53,039 --> 00:15:55,600 infrastructure for research like my a 468 00:15:55,600 --> 00:15:59,039 lot of what I did before was large scale 469 00:15:59,039 --> 00:16:02,399 um data storage analysis and then tools 470 00:16:02,399 --> 00:16:05,039 to understand training runs. I I I did a 471 00:16:05,039 --> 00:16:07,278 lot of different things over my years. 472 00:16:07,278 --> 00:16:10,879 Um, but it was always about building for 473 00:16:10,879 --> 00:16:13,039 others and making them faster and just 474 00:16:13,039 --> 00:16:14,480 really caring about, you know, 475 00:16:14,480 --> 00:16:16,879 fundamentally doing that well and then 476 00:16:16,879 --> 00:16:18,399 through tooling and infrastructure 477 00:16:18,399 --> 00:16:20,240 making new things possible. And so when 478 00:16:20,240 --> 00:16:22,240 I joined OpenAI, I was like with the 479 00:16:22,240 --> 00:16:25,440 same idea and then with the re with with 480 00:16:25,440 --> 00:16:27,039 one preview and like you know some some 481 00:16:27,039 --> 00:16:28,720 of some some of the later models, it was 482 00:16:28,720 --> 00:16:31,360 very clear that we had to use the models 483 00:16:31,360 --> 00:16:34,320 themselves to help us go faster. And so 484 00:16:34,320 --> 00:16:35,839 I just really got obsessed with this 485 00:16:35,839 --> 00:16:37,919 idea of what were the limitations, how 486 00:16:37,919 --> 00:16:39,839 were we going to use those models for 487 00:16:39,839 --> 00:16:43,600 research itself. So got together with um 488 00:16:43,600 --> 00:16:45,278 other folks in research. We started 489 00:16:45,278 --> 00:16:47,198 training models. We started building 490 00:16:47,198 --> 00:16:49,759 little agents. Those were truly the 491 00:16:49,759 --> 00:16:52,078 precursor to to to Codex. And like this 492 00:16:52,078 --> 00:16:53,919 was like we were training internal 493 00:16:53,919 --> 00:16:57,039 models to be very uh proficient on the 494 00:16:57,039 --> 00:17:00,639 Python codebase of OpenAI and then uh 495 00:17:00,639 --> 00:17:02,480 very proficient with you know having 496 00:17:02,480 --> 00:17:05,599 like good good taste in architecture, 497 00:17:05,599 --> 00:17:08,000 good taste in you know like code style. 498 00:17:08,000 --> 00:17:10,720 It was like Python only and then the 499 00:17:10,720 --> 00:17:11,919 idea was like you know we would sort of 500 00:17:11,919 --> 00:17:13,599 like use that to build infrastructure 501 00:17:13,599 --> 00:17:15,439 very quickly and you know help 502 00:17:15,439 --> 00:17:17,679 researchers code faster as well. uh and 503 00:17:17,679 --> 00:17:18,798 then you know and then we would move 504 00:17:18,798 --> 00:17:21,519 faster and then over time when you just 505 00:17:21,519 --> 00:17:23,199 kind like push that and simplify it to 506 00:17:23,199 --> 00:17:25,279 its core you're making a lot of uh you 507 00:17:25,279 --> 00:17:26,880 know we found like you could make a lot 508 00:17:26,880 --> 00:17:29,038 of progress very quickly and then learn 509 00:17:29,038 --> 00:17:32,079 very quickly and then Greg and Sam are 510 00:17:32,079 --> 00:17:34,480 you know people with they're immensely 511 00:17:34,480 --> 00:17:37,519 supportive and also uh Greg was very 512 00:17:37,519 --> 00:17:39,839 adamant that you know we would we would 513 00:17:39,839 --> 00:17:41,919 not just focus on ourselves but we would 514 00:17:41,919 --> 00:17:44,558 also focus on benefiting uh the world 515 00:17:44,558 --> 00:17:47,440 and so he just sort of encouraged that 516 00:17:47,440 --> 00:17:48,960 we would be thinking about this not just 517 00:17:48,960 --> 00:17:51,200 as a tool for OpenAI itself but also as 518 00:17:51,200 --> 00:17:52,400 something that we would actually make 519 00:17:52,400 --> 00:17:55,519 into a product and this is when uh we 520 00:17:55,519 --> 00:17:57,599 merged this research effort with this 521 00:17:57,599 --> 00:18:00,079 AS3 effort um and we started building 522 00:18:00,079 --> 00:18:02,240 one thing and then that led to a sprint 523 00:18:02,240 --> 00:18:04,400 which was like the initial cloud codecs 524 00:18:04,400 --> 00:18:06,240 that we launched which didn't really 525 00:18:06,240 --> 00:18:08,000 have PMF because it was like a little 526 00:18:08,000 --> 00:18:10,400 bit too high friction and then we also 527 00:18:10,400 --> 00:18:12,880 launched the codeex CLI and we continued 528 00:18:12,880 --> 00:18:15,359 to push but it was always this idea of 529 00:18:15,359 --> 00:18:17,919 >> hey how do we get models to really help 530 00:18:17,919 --> 00:18:19,359 here. You mentioned that first you 531 00:18:19,359 --> 00:18:21,200 started to build this model to train on 532 00:18:21,200 --> 00:18:23,519 the Python code and and actually help 533 00:18:23,519 --> 00:18:25,519 build in for better but then you made 534 00:18:25,519 --> 00:18:27,440 this interesting decision where for 535 00:18:27,440 --> 00:18:30,798 codeex you built it in Rust and at the 536 00:18:30,798 --> 00:18:32,798 time the the model was not on 537 00:18:32,798 --> 00:18:34,480 distribution for Rust right it wasn't as 538 00:18:34,480 --> 00:18:36,400 good as in Rust and it was in Python or 539 00:18:36,400 --> 00:18:39,519 Typescript why did you make that kind of 540 00:18:39,519 --> 00:18:41,359 a decision was was it kind of like did 541 00:18:41,359 --> 00:18:43,519 you expect that it'll catch up or or you 542 00:18:43,519 --> 00:18:44,880 figure that performance is more 543 00:18:44,880 --> 00:18:46,079 important or because it was very 544 00:18:46,079 --> 00:18:48,480 counterintuitive most of the other 545 00:18:48,480 --> 00:18:50,160 harnesses built were actually not built 546 00:18:50,160 --> 00:18:51,519 in Rust. They were built on distribution 547 00:18:51,519 --> 00:18:53,038 on TypeScript or Python or something 548 00:18:53,038 --> 00:18:53,759 else. 549 00:18:53,759 --> 00:18:58,640 >> Yes. From first principles like we very 550 00:18:58,640 --> 00:19:01,519 early on we were thinking about the 551 00:19:01,519 --> 00:19:03,679 product interface and the agent as 552 00:19:03,679 --> 00:19:06,960 different things. So it was very 553 00:19:06,960 --> 00:19:10,720 important um to build 554 00:19:10,720 --> 00:19:14,640 the core of the agent in a way that was 555 00:19:14,640 --> 00:19:17,759 robust, that was secure as well, that 556 00:19:17,759 --> 00:19:20,319 was um you know engineered for 557 00:19:20,319 --> 00:19:25,200 efficiency and scale and having worked 558 00:19:25,200 --> 00:19:27,839 through projects over the years that go 559 00:19:27,839 --> 00:19:30,319 from hey this is a fun thing to like, 560 00:19:30,319 --> 00:19:32,319 hey, we need to scale this to the scale 561 00:19:32,319 --> 00:19:35,599 of like the largest data center um is 562 00:19:35,599 --> 00:19:38,240 the ear decisions early on are like 563 00:19:38,240 --> 00:19:41,119 really turn out to be quite important as 564 00:19:41,119 --> 00:19:42,720 long as you don't sacrifice too much of 565 00:19:42,720 --> 00:19:44,640 the velocity and so it's like it's a 566 00:19:44,640 --> 00:19:46,160 it's a trade-off but we had very 567 00:19:46,160 --> 00:19:48,558 prolific and amazing Rust developers our 568 00:19:48,558 --> 00:19:51,279 internal models were not bad at Rust um 569 00:19:51,279 --> 00:19:53,519 and then you get a lot of uh validation 570 00:19:53,519 --> 00:19:55,119 as well at compile time it's like you 571 00:19:55,119 --> 00:19:56,480 know statically verified and all these 572 00:19:56,480 --> 00:19:58,798 things and that is great for agents too 573 00:19:58,798 --> 00:20:01,519 so turns out you know it was quite clear 574 00:20:01,519 --> 00:20:04,400 that you know Rust as a language would 575 00:20:04,400 --> 00:20:06,880 actually be quite good for agents fairly 576 00:20:06,880 --> 00:20:08,880 quickly if we decided to put some effort 577 00:20:08,880 --> 00:20:10,880 into it. But primarily we were focused 578 00:20:10,880 --> 00:20:12,720 on correctness and we were focused on 579 00:20:12,720 --> 00:20:13,759 efficiency as well. 580 00:20:13,759 --> 00:20:15,359 >> Interesting. So you're saying you know 581 00:20:15,359 --> 00:20:17,279 it's it's worth in your case it was 582 00:20:17,279 --> 00:20:19,279 worth thinking ahead of where you want 583 00:20:19,279 --> 00:20:20,880 this thing to be and for example thing 584 00:20:20,880 --> 00:20:22,400 like a language choice. Obviously with 585 00:20:22,400 --> 00:20:24,400 agents you can rewrite a bunch of stuff 586 00:20:24,400 --> 00:20:26,160 and easier than in the past, but it's 587 00:20:26,160 --> 00:20:28,720 still like you can say save yourself 588 00:20:28,720 --> 00:20:30,880 reworking by putting in the right I 589 00:20:30,880 --> 00:20:33,599 guess scaffolding or or or well the you 590 00:20:33,599 --> 00:20:35,519 know the the baseline of of what you're 591 00:20:35,519 --> 00:20:36,480 building on right 592 00:20:36,480 --> 00:20:38,079 >> I think we could have been successful if 593 00:20:38,079 --> 00:20:40,240 we had written it in Typescript or you 594 00:20:40,240 --> 00:20:41,919 know maybe even Python and then it would 595 00:20:41,919 --> 00:20:43,279 have fine and then you know we would 596 00:20:43,279 --> 00:20:45,359 have rewritten it at some point but 597 00:20:45,359 --> 00:20:48,558 having a very clean separation between 598 00:20:48,558 --> 00:20:50,880 the agent itself which can exist 599 00:20:50,880 --> 00:20:54,159 irrespective of the product. Um it was a 600 00:20:54,159 --> 00:20:55,919 very important principle and if you 601 00:20:55,919 --> 00:20:58,319 write everything in the same codebase in 602 00:20:58,319 --> 00:21:00,480 the same language it's like inevitably 603 00:21:00,480 --> 00:21:02,319 you're going to be a little bit sloppy 604 00:21:02,319 --> 00:21:04,558 and um you're going to intertwine things 605 00:21:04,558 --> 00:21:06,240 more than you should and then it's going 606 00:21:06,240 --> 00:21:07,919 to prevent further innovation after 607 00:21:07,919 --> 00:21:10,079 that. And so that was that was very 608 00:21:10,079 --> 00:21:11,679 important like the rust boundary in a 609 00:21:11,679 --> 00:21:14,159 sense like was very useful for that. One 610 00:21:14,159 --> 00:21:15,599 interesting decision that you made which 611 00:21:15,599 --> 00:21:17,759 is unique across all of the major labs 612 00:21:17,759 --> 00:21:20,558 is having this built-in open source 613 00:21:20,558 --> 00:21:22,640 right the CLI is open source the SDK and 614 00:21:22,640 --> 00:21:24,480 the app server are all open source when 615 00:21:24,480 --> 00:21:26,880 and why did you decide that it's not a 616 00:21:26,880 --> 00:21:28,480 given especially you know there used to 617 00:21:28,480 --> 00:21:30,798 be jokes about open AI having things 618 00:21:30,798 --> 00:21:32,720 closed but this this is actually the 619 00:21:32,720 --> 00:21:35,038 opposite where like this is open whereas 620 00:21:35,038 --> 00:21:37,038 like some competitors would would ship 621 00:21:37,038 --> 00:21:39,200 closed source harnesses which again I I 622 00:21:39,200 --> 00:21:40,640 I think it's very easy to understand why 623 00:21:40,640 --> 00:21:41,919 you would want something closed source 624 00:21:41,919 --> 00:21:43,440 why did you want it open source There 625 00:21:43,440 --> 00:21:46,720 was something really cool about the idea 626 00:21:46,720 --> 00:21:50,240 of having the code open source because 627 00:21:50,240 --> 00:21:52,000 fundamentally what you're building is 628 00:21:52,000 --> 00:21:55,200 you're building a coding agent. And so 629 00:21:55,200 --> 00:21:57,759 we were sort of like thinking about well 630 00:21:57,759 --> 00:21:59,038 if you have that you know you're 631 00:21:59,038 --> 00:22:00,720 obviously going to point it at itself 632 00:22:00,720 --> 00:22:03,359 and you know maybe you know you can 633 00:22:03,359 --> 00:22:05,200 build a community of you know 634 00:22:05,200 --> 00:22:07,038 contributors that use it to improve it 635 00:22:07,038 --> 00:22:08,558 and then you know you can learn a lot 636 00:22:08,558 --> 00:22:11,119 from that. Also, it felt at the time is 637 00:22:11,119 --> 00:22:13,599 like, you know, very clear to us that if 638 00:22:13,599 --> 00:22:15,279 we were going to be successful, open 639 00:22:15,279 --> 00:22:18,079 source itself would change and the role 640 00:22:18,079 --> 00:22:20,079 of code itself would change and so being 641 00:22:20,079 --> 00:22:21,839 part of that community seemed important 642 00:22:21,839 --> 00:22:23,679 instead of divorced from it. I think, 643 00:22:23,679 --> 00:22:25,200 you know, it's it's it's hard to solve 644 00:22:25,200 --> 00:22:26,720 problems if you don't sort of like 645 00:22:26,720 --> 00:22:29,839 witness them yourself. Uh, and then the 646 00:22:29,839 --> 00:22:32,079 the other thing was just it still feels 647 00:22:32,079 --> 00:22:34,079 like early, but it was very early at the 648 00:22:34,079 --> 00:22:37,200 time. Um, it felt like we would have 649 00:22:37,200 --> 00:22:39,679 some ideas for how to solve things well. 650 00:22:39,679 --> 00:22:42,319 Um, and we were co-designing these, you 651 00:22:42,319 --> 00:22:44,159 know, with with with with the training 652 00:22:44,159 --> 00:22:47,038 and and and and the research and it's 653 00:22:47,038 --> 00:22:48,720 it's all about expressing like the 654 00:22:48,720 --> 00:22:50,480 capabilities of model in like the most 655 00:22:50,480 --> 00:22:53,038 flexible and the best way, but also we 656 00:22:53,038 --> 00:22:55,759 didn't have all the answers and sort of 657 00:22:55,759 --> 00:22:59,279 being very open about, hey, this is what 658 00:22:59,279 --> 00:23:01,440 a good harness looks like. This is how 659 00:23:01,440 --> 00:23:03,279 we think about it. We did like a couple 660 00:23:03,279 --> 00:23:05,279 of like very technical like deep dives 661 00:23:05,279 --> 00:23:06,880 and blog posts and we talked about it a 662 00:23:06,880 --> 00:23:08,880 lot and we thought you know hey it's 663 00:23:08,880 --> 00:23:11,279 just like the world is vast out there's 664 00:23:11,279 --> 00:23:12,720 like you know crazy smart people it's 665 00:23:12,720 --> 00:23:14,159 like you know we're we're going to get 666 00:23:14,159 --> 00:23:16,960 inspired by other open source project as 667 00:23:16,960 --> 00:23:20,720 well and so let's just make this a level 668 00:23:20,720 --> 00:23:23,599 playing field and sort of like encourage 669 00:23:23,599 --> 00:23:26,720 a lot of tinkering um and exploration at 670 00:23:26,720 --> 00:23:28,480 this stage. Now this has been now you 671 00:23:28,480 --> 00:23:31,119 know like a a year later a year and a 672 00:23:31,119 --> 00:23:33,359 half later which is a very long time in 673 00:23:33,359 --> 00:23:35,038 right now in this AI time frame but 674 00:23:35,038 --> 00:23:38,079 looking back or taking the experience 675 00:23:38,079 --> 00:23:39,919 what are the benefits you've seen the 676 00:23:39,919 --> 00:23:41,200 kind of engineering benefits the 677 00:23:41,200 --> 00:23:42,720 engineering team's benefits from being 678 00:23:42,720 --> 00:23:44,720 open source and just honestly what are 679 00:23:44,720 --> 00:23:47,119 things that are kind of hard about being 680 00:23:47,119 --> 00:23:49,038 open source right like there must be 681 00:23:49,038 --> 00:23:51,200 downsides like just try trying to get an 682 00:23:51,200 --> 00:23:52,880 honest take on both sides 683 00:23:52,880 --> 00:23:54,319 >> yeah there there there are definitely 684 00:23:54,319 --> 00:23:56,720 downsides it it it comes at a cost 685 00:23:56,720 --> 00:23:59,599 Right. Um the the benefits are it's 686 00:23:59,599 --> 00:24:01,440 almost something to build in the open. 687 00:24:01,440 --> 00:24:04,319 Uh it's awesome to have like a small a 688 00:24:04,319 --> 00:24:06,880 small repo as well. Like whenever we 689 00:24:06,880 --> 00:24:09,038 hire uh someone and they join the Codex 690 00:24:09,038 --> 00:24:10,798 team, it's like they've seen the repo 691 00:24:10,798 --> 00:24:12,720 before. They've they've looked at PRs. 692 00:24:12,720 --> 00:24:13,038 They're like 693 00:24:13,038 --> 00:24:14,159 >> onboarding is done. 694 00:24:14,159 --> 00:24:15,440 >> You know, it's it's it's done. Yeah. 695 00:24:15,440 --> 00:24:16,880 It's like in onboarding is just like you 696 00:24:16,880 --> 00:24:18,720 use Codex to look at the repo, you know, 697 00:24:18,720 --> 00:24:20,159 with you and you ask them questions, but 698 00:24:20,159 --> 00:24:21,919 it's like it's not it's not a secret 699 00:24:21,919 --> 00:24:23,440 issue that you can get productive right 700 00:24:23,440 --> 00:24:25,119 away. We get a lot of good 701 00:24:25,119 --> 00:24:27,119 contributions. Although we get like you 702 00:24:27,119 --> 00:24:30,000 know an a tsunami of like random stuff 703 00:24:30,000 --> 00:24:30,720 as well. 704 00:24:30,720 --> 00:24:32,558 >> Obviously you and everyone else right 705 00:24:32,558 --> 00:24:34,079 open source is changing. I think this is 706 00:24:34,079 --> 00:24:35,679 one of the examples. 707 00:24:35,679 --> 00:24:37,919 >> That's right. And then to to me it just 708 00:24:37,919 --> 00:24:39,359 and and to a lot of the team it just 709 00:24:39,359 --> 00:24:40,960 brings a lot of energy to just be part 710 00:24:40,960 --> 00:24:43,440 of the community and like be directly 711 00:24:43,440 --> 00:24:45,440 contributing um not just saying that we 712 00:24:45,440 --> 00:24:46,640 care about the community but actually 713 00:24:46,640 --> 00:24:48,319 doing things that you know you can see 714 00:24:48,319 --> 00:24:50,798 it's it's costing us effort right. U we 715 00:24:50,798 --> 00:24:53,599 don't have to do it. the the downsides 716 00:24:53,599 --> 00:24:55,278 are, you know, it's it's separate from 717 00:24:55,278 --> 00:24:57,599 the rest of our code. Um so, you know, 718 00:24:57,599 --> 00:24:58,720 sometimes we have to draw like 719 00:24:58,720 --> 00:25:00,640 artificial boundaries and, you know, 720 00:25:00,640 --> 00:25:03,038 work across multiple repos. Um when 721 00:25:03,038 --> 00:25:04,720 we're working on something particularly 722 00:25:04,720 --> 00:25:07,119 exciting, um and uh you know, we're 723 00:25:07,119 --> 00:25:08,880 building it in the open then, you know, 724 00:25:08,880 --> 00:25:10,720 at times we find that uh you know, 725 00:25:10,720 --> 00:25:12,480 others copy it, you know, before we have 726 00:25:12,480 --> 00:25:15,919 the time to release it. And it's like 727 00:25:15,919 --> 00:25:18,480 it's just a little bit sad. Um, but also 728 00:25:18,480 --> 00:25:20,319 it's like it's part of the game, you 729 00:25:20,319 --> 00:25:21,839 know. It's it's like you're building in 730 00:25:21,839 --> 00:25:23,119 the open. It's like, you know, that's 731 00:25:23,119 --> 00:25:24,319 that's sort of like the contract that 732 00:25:24,319 --> 00:25:25,519 you signed is like, you know, you can 733 00:25:25,519 --> 00:25:27,519 copy it. Uh, we have a very permissive 734 00:25:27,519 --> 00:25:29,599 license as well, but it does sting a 735 00:25:29,599 --> 00:25:30,640 little bit when you're working on 736 00:25:30,640 --> 00:25:32,558 something and you're like, you know, and 737 00:25:32,558 --> 00:25:34,720 then uh the the third thing is just just 738 00:25:34,720 --> 00:25:36,400 like everyone else is like, you know, we 739 00:25:36,400 --> 00:25:38,159 are overwhelmed with, you know, random 740 00:25:38,159 --> 00:25:39,679 contributions and, you know, we have to 741 00:25:39,679 --> 00:25:42,159 deal with that additional tax. Um, but 742 00:25:42,159 --> 00:25:44,558 then that pushes us to, you know, also 743 00:25:44,558 --> 00:25:46,558 like try and solve for it, right? which 744 00:25:46,558 --> 00:25:47,359 I think is good. 745 00:25:47,359 --> 00:25:48,960 >> And on top of the open source, one thing 746 00:25:48,960 --> 00:25:50,960 that surprised me about Codeex, and I 747 00:25:50,960 --> 00:25:51,919 didn't even know about it until 748 00:25:51,919 --> 00:25:54,960 recently, it's not tied to the OpenAI 749 00:25:54,960 --> 00:25:58,000 models, you can use other models with 750 00:25:58,000 --> 00:26:00,480 Codeex, you know, like putting myself if 751 00:26:00,480 --> 00:26:02,960 in a vendor's shoe, it it might not be 752 00:26:02,960 --> 00:26:05,359 very obvious because again, all the 753 00:26:05,359 --> 00:26:07,359 other vendors I look at when they do a a 754 00:26:07,359 --> 00:26:08,880 CLI, it's kind of use it with our 755 00:26:08,880 --> 00:26:12,640 models. again what made you decide to be 756 00:26:12,640 --> 00:26:15,119 this permissive about you know using or 757 00:26:15,119 --> 00:26:17,839 allowing to use your harness with with 758 00:26:17,839 --> 00:26:18,880 other models? 759 00:26:18,880 --> 00:26:24,000 >> It it felt quite natural if if you are 760 00:26:24,000 --> 00:26:28,720 part of this community and building an 761 00:26:28,720 --> 00:26:30,640 excellent coding harness is like why 762 00:26:30,640 --> 00:26:33,119 would you couple it to your model? That 763 00:26:33,119 --> 00:26:34,798 that that felt like sort of like quite 764 00:26:34,798 --> 00:26:36,798 disappointing to make that decision. So 765 00:26:36,798 --> 00:26:38,960 it didn't it didn't feel right. Um and 766 00:26:38,960 --> 00:26:40,880 in general it's like I think you know 767 00:26:40,880 --> 00:26:42,558 it's like I kind of try to make 768 00:26:42,558 --> 00:26:44,400 decisions that I'm like yes you know 769 00:26:44,400 --> 00:26:46,000 just like I can just sort of like 770 00:26:46,000 --> 00:26:48,240 explain it you know it is correct. It's 771 00:26:48,240 --> 00:26:49,759 the same reasoning with you know it is 772 00:26:49,759 --> 00:26:51,919 open source in the first place. It would 773 00:26:51,919 --> 00:26:55,200 have been trivial for anyone to fork it 774 00:26:55,200 --> 00:26:57,119 and then add support for another thing. 775 00:26:57,119 --> 00:26:58,558 But then but then you're just 776 00:26:58,558 --> 00:27:00,079 encouraging people to just like you know 777 00:27:00,079 --> 00:27:01,519 go and use that fork and then now 778 00:27:01,519 --> 00:27:03,440 suddenly you have overhead and the only 779 00:27:03,440 --> 00:27:05,519 reason you have a fork is because you 780 00:27:05,519 --> 00:27:07,839 know you wanted to change like 10 lines 781 00:27:07,839 --> 00:27:09,599 of code to add support for like another 782 00:27:09,599 --> 00:27:11,839 model provider that feels very silly. So 783 00:27:11,839 --> 00:27:13,359 like you know why not just support it in 784 00:27:13,359 --> 00:27:16,079 the first place. The other thing is we 785 00:27:16,079 --> 00:27:18,558 benefit a lot from like being able to 786 00:27:18,558 --> 00:27:20,798 just give optionality. So, you know, 787 00:27:20,798 --> 00:27:22,798 it's like maybe today, you know, you you 788 00:27:22,798 --> 00:27:26,480 you love using OpenAI models um and you 789 00:27:26,480 --> 00:27:28,720 know, you're super productive with them, 790 00:27:28,720 --> 00:27:30,159 but like tomorrow there's a new model 791 00:27:30,159 --> 00:27:32,079 that comes out, you want to try that. 792 00:27:32,079 --> 00:27:34,558 Why force you to go and completely 793 00:27:34,558 --> 00:27:36,319 change your setup just to try a new 794 00:27:36,319 --> 00:27:37,839 model and then we benefit from the 795 00:27:37,839 --> 00:27:39,599 feedback that we didn't get uh which is 796 00:27:39,599 --> 00:27:40,558 like, you know, maybe there's something 797 00:27:40,558 --> 00:27:41,839 that you liked about that model. Maybe 798 00:27:41,839 --> 00:27:44,079 it actually didn't work well. But it's 799 00:27:44,079 --> 00:27:46,880 sort of like being nice to our users and 800 00:27:46,880 --> 00:27:49,278 to the community is like you know feels 801 00:27:49,278 --> 00:27:51,679 like the right thing to do here. Um and 802 00:27:51,679 --> 00:27:53,759 then you know we also you we also try 803 00:27:53,759 --> 00:27:55,679 like other models right so and you know 804 00:27:55,679 --> 00:27:57,759 we try them in the same harness and you 805 00:27:57,759 --> 00:28:00,240 know it's just all all good and then 806 00:28:00,240 --> 00:28:02,798 this is all also often like um this 807 00:28:02,798 --> 00:28:04,640 optionality is very important to 808 00:28:04,640 --> 00:28:07,599 companies that we work with. 809 00:28:07,599 --> 00:28:08,960 This is something that, you know, we we 810 00:28:08,960 --> 00:28:10,079 absolutely lean into. 811 00:28:10,079 --> 00:28:11,359 >> This last point, I think, you know, as 812 00:28:11,359 --> 00:28:13,278 any serious company, you want to have 813 00:28:13,278 --> 00:28:15,038 optionality and you want to use a tool 814 00:28:15,038 --> 00:28:16,880 that gives you that optionality. 815 00:28:16,880 --> 00:28:18,398 >> But I kind of appreciate it cuz I feels 816 00:28:18,398 --> 00:28:20,000 to me like it's kind of honest like look 817 00:28:20,000 --> 00:28:23,519 like it forces the whole company to be 818 00:28:23,519 --> 00:28:25,200 to compete the best in everywhere in the 819 00:28:25,200 --> 00:28:26,960 model layer and the harness layer with 820 00:28:26,960 --> 00:28:28,640 open source with with choosable models 821 00:28:28,640 --> 00:28:30,079 and it kind of like doesn't doesn't 822 00:28:30,079 --> 00:28:31,359 allow you to like kick back and say 823 00:28:31,359 --> 00:28:32,640 like, "All right, we're done. We we can 824 00:28:32,640 --> 00:28:34,159 we can we can hang back for a little bit 825 00:28:34,159 --> 00:28:34,798 for now." 826 00:28:34,798 --> 00:28:37,759 >> Yeah. I want us to win users by having, 827 00:28:37,759 --> 00:28:39,038 you know, the best models, the most 828 00:28:39,038 --> 00:28:41,278 efficient models, the best product, and 829 00:28:41,278 --> 00:28:42,640 then, you know, if we do all of these 830 00:28:42,640 --> 00:28:43,839 things, it's like we're going to have a 831 00:28:43,839 --> 00:28:45,440 good time. If we sort of like force you 832 00:28:45,440 --> 00:28:47,038 to use the product because, you know, 833 00:28:47,038 --> 00:28:49,278 this one thing, it's just like then I 834 00:28:49,278 --> 00:28:50,720 don't think that will attract, you know, 835 00:28:50,720 --> 00:28:52,558 the the very best people to work on this 836 00:28:52,558 --> 00:28:54,240 product either. And like, you know, it's 837 00:28:54,240 --> 00:28:56,240 like we we're doing our best work here. 838 00:28:56,240 --> 00:28:58,640 We care a lot about the experience. It 839 00:28:58,640 --> 00:29:00,398 should feel delightful, you know, like 840 00:29:00,398 --> 00:29:02,480 it doesn't irrespect of the model that 841 00:29:02,480 --> 00:29:04,798 powers it, it should feel delightful. I 842 00:29:04,798 --> 00:29:06,558 love the idea of winning based on merit, 843 00:29:06,558 --> 00:29:08,558 not based on lockin. And this is a 844 00:29:08,558 --> 00:29:09,839 perfect time to mention our season 845 00:29:09,839 --> 00:29:11,839 sponsor, Entire, who also play by the 846 00:29:11,839 --> 00:29:14,159 same rules. Like it or not, Git is 847 00:29:14,159 --> 00:29:15,759 becoming a bottleneck for modern agent 848 00:29:15,759 --> 00:29:17,679 heavy software development. Devs are 849 00:29:17,679 --> 00:29:19,278 creating more code with agents. These 850 00:29:19,278 --> 00:29:21,278 agents are pushing more code. Many devs 851 00:29:21,278 --> 00:29:22,960 are running more parallel agents. These 852 00:29:22,960 --> 00:29:25,359 are pushing even more code. GitHub is 853 00:29:25,359 --> 00:29:27,119 clearly struggling to keep up and has 854 00:29:27,119 --> 00:29:29,200 frequent outages. So what's the 855 00:29:29,200 --> 00:29:31,599 solution? Entire was founded by GitHub's 856 00:29:31,599 --> 00:29:34,319 last coza and he rebuilt git hosting for 857 00:29:34,319 --> 00:29:36,480 the agentic era from scratch. Entire was 858 00:29:36,480 --> 00:29:38,480 built to be very fast and to have your 859 00:29:38,480 --> 00:29:40,558 repos regionally close to you to reduce 860 00:29:40,558 --> 00:29:42,880 latency allowing for fleets of agents to 861 00:29:42,880 --> 00:29:45,200 push in parallel. Some numbers they 862 00:29:45,200 --> 00:29:48,079 published. Entire can handle 418 pushes 863 00:29:48,079 --> 00:29:50,720 per second. That's up to 89 times faster 864 00:29:50,720 --> 00:29:52,798 than every competitor on the market. 865 00:29:52,798 --> 00:29:54,640 When GitHub is down, you can still keep 866 00:29:54,640 --> 00:29:56,319 working and you don't even need to 867 00:29:56,319 --> 00:29:58,319 migrate from GitHub. You just sign up to 868 00:29:58,319 --> 00:30:00,159 entire and the platform mirrors your 869 00:30:00,159 --> 00:30:02,319 repo. And one more neat thing, have you 870 00:30:02,319 --> 00:30:04,480 ever wondered what prompt resulted in 871 00:30:04,480 --> 00:30:06,880 this specific code being generated? I 872 00:30:06,880 --> 00:30:08,398 find that the prompt and conversation 873 00:30:08,398 --> 00:30:10,079 with the agent carries more information 874 00:30:10,079 --> 00:30:12,159 than the PR itself, at least for me. 875 00:30:12,159 --> 00:30:14,159 Entire captures all the prompt history 876 00:30:14,159 --> 00:30:16,159 with your agent right in the repo easy 877 00:30:16,159 --> 00:30:17,839 to check back and has a pretty 878 00:30:17,839 --> 00:30:19,839 innovative UI to show all of this. If 879 00:30:19,839 --> 00:30:21,278 you're looking for Git hosting that 880 00:30:21,278 --> 00:30:23,038 works even when GitHub is down, head to 881 00:30:23,038 --> 00:30:24,798 entire.io/pragmatic, io/pragmatic, 882 00:30:24,798 --> 00:30:27,119 install the CLI, and mirror your repo 883 00:30:27,119 --> 00:30:29,519 with a click. I've already done it. Oh, 884 00:30:29,519 --> 00:30:31,119 and did I mention that it works with any 885 00:30:31,119 --> 00:30:33,519 agent and it's open source? I'd also 886 00:30:33,519 --> 00:30:34,960 like to mention our season sponsor, 887 00:30:34,960 --> 00:30:37,599 Anticys. Tibo talked about how the 888 00:30:37,599 --> 00:30:39,038 experience of the software you use 889 00:30:39,038 --> 00:30:41,119 should feel delightful. Delightful 890 00:30:41,119 --> 00:30:43,278 includes no annoying bugs. But when 891 00:30:43,278 --> 00:30:45,038 you're using agents to write your code, 892 00:30:45,038 --> 00:30:46,720 how do you avoid chipping bugs? 893 00:30:46,720 --> 00:30:48,558 Reviewing every line of code is becoming 894 00:30:48,558 --> 00:30:49,919 a challenge with the amount of code that 895 00:30:49,919 --> 00:30:52,398 agents generate, which is why antithesis 896 00:30:52,398 --> 00:30:54,960 goes well beyond code review. Antithesis 897 00:30:54,960 --> 00:30:56,640 runs your whole system in a hostile 898 00:30:56,640 --> 00:30:58,880 simulation. This simulation includes 899 00:30:58,880 --> 00:31:01,278 both targeted testing and fuss testing. 900 00:31:01,278 --> 00:31:03,038 By running this simulation, it finds 901 00:31:03,038 --> 00:31:05,519 every bug before your users do. And 902 00:31:05,519 --> 00:31:06,960 because the simulation is fully 903 00:31:06,960 --> 00:31:08,798 deterministic, it doesn't only find 904 00:31:08,798 --> 00:31:10,398 bugs, it gives you a perfect 905 00:31:10,398 --> 00:31:12,398 reproduction of every issue, which makes 906 00:31:12,398 --> 00:31:14,880 it much easier to fix issues. The first 907 00:31:14,880 --> 00:31:16,079 thing I thought when I heard about 908 00:31:16,079 --> 00:31:18,000 antithesis is that automated bug 909 00:31:18,000 --> 00:31:19,679 discovery and fully deterministic 910 00:31:19,679 --> 00:31:21,919 testing sounds like science fiction, but 911 00:31:21,919 --> 00:31:23,759 it's actually hardcore engineering under 912 00:31:23,759 --> 00:31:26,398 the hood. Jane Street, Fly.io, and the 913 00:31:26,398 --> 00:31:28,398 Etscd community ship agent written code 914 00:31:28,398 --> 00:31:30,319 with full confidence because they know 915 00:31:30,319 --> 00:31:32,798 it's been verified by antithesis. To see 916 00:31:32,798 --> 00:31:34,558 more case studies and details, head to 917 00:31:34,558 --> 00:31:36,880 antithesis.com/pragmatic. 918 00:31:36,880 --> 00:31:38,798 And with this, let's get back to Tibo 919 00:31:38,798 --> 00:31:40,880 and why competition between tools is 920 00:31:40,880 --> 00:31:43,200 great. Yeah. And I think as as an 921 00:31:43,200 --> 00:31:44,640 engineer like I always see that whenever 922 00:31:44,640 --> 00:31:46,720 there's competition as someone who's 923 00:31:46,720 --> 00:31:48,640 using tools it's it's always amazing 924 00:31:48,640 --> 00:31:50,798 like I remember like when Microsoft had 925 00:31:50,798 --> 00:31:53,599 with Jet Brains the IDU wars and then 926 00:31:53,599 --> 00:31:55,519 there's the clouds battling with each 927 00:31:55,519 --> 00:31:57,599 other with all the features and now of 928 00:31:57,599 --> 00:31:58,960 course we have the harnesses we have the 929 00:31:58,960 --> 00:32:00,640 models and as a as a as a user it's 930 00:32:00,640 --> 00:32:02,159 great because now we have more more 931 00:32:02,159 --> 00:32:04,720 choice they just develop faster I guess 932 00:32:04,720 --> 00:32:06,480 our voice gets heard a bit better. So 933 00:32:06,480 --> 00:32:08,720 it's it's great to hear. Speaking of the 934 00:32:08,720 --> 00:32:10,480 harness, can you tell me how it works 935 00:32:10,480 --> 00:32:13,679 today in the sense of like when I 936 00:32:13,679 --> 00:32:16,720 start a codeex task, 937 00:32:16,720 --> 00:32:19,278 does it run always on my machine? Does 938 00:32:19,278 --> 00:32:21,759 it choose the cloud? Does it use a 939 00:32:21,759 --> 00:32:25,679 sandbox? And how do I control this or or 940 00:32:25,679 --> 00:32:26,880 know this or how much should I know 941 00:32:26,880 --> 00:32:28,640 about this as as an engineer? 942 00:32:28,640 --> 00:32:33,599 >> Yes. So, by default, it runs uh in uh it 943 00:32:33,599 --> 00:32:37,839 runs sandboxed. Um it everything that if 944 00:32:37,839 --> 00:32:39,839 there is like an an a command that 945 00:32:39,839 --> 00:32:42,159 should run with additional permissions 946 00:32:42,159 --> 00:32:45,359 outside of the sandbox, it will ask uh 947 00:32:45,359 --> 00:32:48,079 you as a user for permission, but 948 00:32:48,079 --> 00:32:50,240 everything every tool execution happens 949 00:32:50,240 --> 00:32:52,398 within the sandbox by default and it 950 00:32:52,398 --> 00:32:55,038 runs entirely on the machine um your 951 00:32:55,038 --> 00:32:59,200 local machine. And this has been the 952 00:32:59,200 --> 00:33:02,480 case for you know more than a year now. 953 00:33:02,480 --> 00:33:04,720 But it is something that is evolving and 954 00:33:04,720 --> 00:33:07,599 shifting where like you can select to 955 00:33:07,599 --> 00:33:10,398 run this um in the cloud which then runs 956 00:33:10,398 --> 00:33:13,679 in like a managed VM uh where it's the 957 00:33:13,679 --> 00:33:17,359 same VM that you get through chatk work 958 00:33:17,359 --> 00:33:19,440 um and you can you can sort of like 959 00:33:19,440 --> 00:33:21,038 inspect it but like it runs in a kata 960 00:33:21,038 --> 00:33:23,278 container it's like a secure environment 961 00:33:23,278 --> 00:33:27,119 and so everything runs inside of that VM 962 00:33:27,119 --> 00:33:28,880 and it doesn't run on your machine and 963 00:33:28,880 --> 00:33:30,558 then the only thing that happens on your 964 00:33:30,558 --> 00:33:33,440 machine is like the the your input and 965 00:33:33,440 --> 00:33:35,599 then the streaming back of the output 966 00:33:35,599 --> 00:33:37,599 and so that you know obviously then uh 967 00:33:37,599 --> 00:33:39,759 is like much nicer on your CPU and and 968 00:33:39,759 --> 00:33:42,480 your machine and you can scale much much 969 00:33:42,480 --> 00:33:44,880 um much more and this is like just a 970 00:33:44,880 --> 00:33:48,319 step um it's going to be much more 971 00:33:48,319 --> 00:33:51,359 seamless in the future um to like you 972 00:33:51,359 --> 00:33:53,200 know use cloud machines and then you 973 00:33:53,200 --> 00:33:54,960 know maybe have a combination of like 974 00:33:54,960 --> 00:33:57,200 partial execution on your laptop partial 975 00:33:57,200 --> 00:34:00,960 execution on on on cloud machines and 976 00:34:00,960 --> 00:34:02,319 really The thing that we're thinking 977 00:34:02,319 --> 00:34:04,880 about that is very natural is as models 978 00:34:04,880 --> 00:34:08,639 just get better and uh more capable. 979 00:34:08,639 --> 00:34:10,639 They can leverage so much more compute 980 00:34:10,639 --> 00:34:13,599 and many more resources than are 981 00:34:13,599 --> 00:34:16,639 available on your local machine and so 982 00:34:16,639 --> 00:34:18,878 it would be a constraint at some point 983 00:34:18,878 --> 00:34:20,960 to just limit execution on your local 984 00:34:20,960 --> 00:34:21,358 machine. 985 00:34:21,358 --> 00:34:23,838 >> Yeah. One thing that is great about it 986 00:34:23,838 --> 00:34:25,199 running locally, and I think the reason 987 00:34:25,199 --> 00:34:27,199 I I love it when it runs locally. Of 988 00:34:27,199 --> 00:34:29,039 course, it's a pain because if I'm doing 989 00:34:29,039 --> 00:34:30,639 some work, it's like, you know, I have 990 00:34:30,639 --> 00:34:32,960 several agents, it's it's eating CPU. If 991 00:34:32,960 --> 00:34:35,280 I want to close my laptop, I I I cannot 992 00:34:35,280 --> 00:34:37,519 kind of leave it like half open, right? 993 00:34:37,519 --> 00:34:39,199 When I was in one of the offices of an 994 00:34:39,199 --> 00:34:41,199 AI company, I I had it half open and 995 00:34:41,199 --> 00:34:42,398 they're like, "Are you running agents?" 996 00:34:42,398 --> 00:34:43,679 I'm like, "Yeah, I have one running." 997 00:34:43,679 --> 00:34:44,719 He's like, "I get it." 998 00:34:44,719 --> 00:34:46,159 >> But the reason the reason I do it 999 00:34:46,159 --> 00:34:48,239 because I have my local tools, I have my 1000 00:34:48,239 --> 00:34:51,199 local Postgress database. I have my my 1001 00:34:51,199 --> 00:34:53,039 this this and that. How are you thinking 1002 00:34:53,039 --> 00:34:54,800 about the cloud is amazing but it 1003 00:34:54,800 --> 00:34:56,320 doesn't have this setup or it's just a 1004 00:34:56,320 --> 00:34:58,320 pain to set it up. are you thinking or 1005 00:34:58,320 --> 00:35:01,119 or are you experimenting with you know 1006 00:35:01,119 --> 00:35:02,880 making the these setups and I'm kind of 1007 00:35:02,880 --> 00:35:05,280 reminded of a topic that we talked about 1008 00:35:05,280 --> 00:35:06,719 prea which is cloud development 1009 00:35:06,719 --> 00:35:09,599 environments and like 2022 23 they're 1010 00:35:09,599 --> 00:35:12,159 hot and then we talked about AI more but 1011 00:35:12,159 --> 00:35:15,039 >> yes I think outside of large tech 1012 00:35:15,039 --> 00:35:17,358 companies like cloud dev boxes really 1013 00:35:17,358 --> 00:35:19,280 never took off because there's a very 1014 00:35:19,280 --> 00:35:22,000 big upfront cost uh and then you need to 1015 00:35:22,000 --> 00:35:23,679 pay like a maintenance cost as well and 1016 00:35:23,679 --> 00:35:25,599 you just you know don't benefit from it 1017 00:35:25,599 --> 00:35:28,000 as like a a solo developer or like a 1018 00:35:28,000 --> 00:35:29,358 small team with the level of 1019 00:35:29,358 --> 00:35:31,199 capabilities that we have in agents now 1020 00:35:31,199 --> 00:35:33,519 is like the setup almost is free, right? 1021 00:35:33,519 --> 00:35:35,358 So like this the setup cost and this 1022 00:35:35,358 --> 00:35:37,199 maintenance cost is like if if your 1023 00:35:37,199 --> 00:35:38,719 agent is capable of doing it, you know, 1024 00:35:38,719 --> 00:35:40,400 it should just do it for you. So for 1025 00:35:40,400 --> 00:35:41,920 example, if you're saying like hey, you 1026 00:35:41,920 --> 00:35:45,119 know, I have like I have my local um 1027 00:35:45,119 --> 00:35:48,960 SQLite or I have a local server and MCPS 1028 00:35:48,960 --> 00:35:51,199 and whatnot. It's like how hard is it to 1029 00:35:51,199 --> 00:35:53,440 actually configure exactly the same 1030 00:35:53,440 --> 00:35:55,920 setup and keep it in sync on a cloud dev 1031 00:35:55,920 --> 00:35:58,480 box? Well, maybe it's not that hard if 1032 00:35:58,480 --> 00:36:00,400 the model just does it for you. And so I 1033 00:36:00,400 --> 00:36:01,838 think we're going to see a resurgence 1034 00:36:01,838 --> 00:36:05,440 of, you know, fully cloud orchestrated 1035 00:36:05,440 --> 00:36:08,719 uh machines which then frees you from 1036 00:36:08,719 --> 00:36:10,000 your laptop, right? It's like one thing 1037 00:36:10,000 --> 00:36:12,000 that we've seen a ton of success with 1038 00:36:12,000 --> 00:36:13,519 with chat work is like it's just 1039 00:36:13,519 --> 00:36:15,838 available on your mobile. I start my day 1040 00:36:15,838 --> 00:36:18,880 just dictating a bunch of tasks into it. 1041 00:36:18,880 --> 00:36:20,719 uh next to the coffee and it just does 1042 00:36:20,719 --> 00:36:22,400 it. It has access to my calendar. It has 1043 00:36:22,400 --> 00:36:25,119 access to my my email. Uh it has access 1044 00:36:25,119 --> 00:36:27,280 to Slack and it's just so awesome to 1045 00:36:27,280 --> 00:36:29,039 just be able to walk around and you know 1046 00:36:29,039 --> 00:36:31,599 get stuff done without having to you 1047 00:36:31,599 --> 00:36:34,400 know carry my laptop everywhere and I 1048 00:36:34,400 --> 00:36:36,000 think it's the same things like you know 1049 00:36:36,000 --> 00:36:37,599 we shipped like Codex remote where you 1050 00:36:37,599 --> 00:36:39,199 know execution is like still happening 1051 00:36:39,199 --> 00:36:41,760 on your laptop but it would be wonderful 1052 00:36:41,760 --> 00:36:43,519 if you know you didn't have to keep your 1053 00:36:43,519 --> 00:36:45,440 laptop open. Can you tell me a bit on 1054 00:36:45,440 --> 00:36:48,159 how in the past how did you improve 1055 00:36:48,159 --> 00:36:50,480 codecs? Because I remember when I first 1056 00:36:50,480 --> 00:36:52,719 used codeex this was one of the early 1057 00:36:52,719 --> 00:36:54,559 versions you know like you could talk to 1058 00:36:54,559 --> 00:36:57,039 it did stuff but for example I said like 1059 00:36:57,039 --> 00:36:58,639 all right make this change and it did 1060 00:36:58,639 --> 00:37:00,559 that change and I had unit tests and it 1061 00:37:00,559 --> 00:37:02,480 didn't run it and then later a few 1062 00:37:02,480 --> 00:37:04,000 months later I don't know exactly when 1063 00:37:04,000 --> 00:37:06,000 it just started to run it automatically. 1064 00:37:06,000 --> 00:37:08,159 were these things did you improve the 1065 00:37:08,159 --> 00:37:10,079 you know the the script that runs you 1066 00:37:10,079 --> 00:37:11,599 know the instructions I'm not sure how 1067 00:37:11,599 --> 00:37:13,838 exactly you call the you know the the 1068 00:37:13,838 --> 00:37:15,838 bootstrapping script or whatever that is 1069 00:37:15,838 --> 00:37:18,000 is it improving the model like as a dev 1070 00:37:18,000 --> 00:37:20,400 how can I imagine you making each 1071 00:37:20,400 --> 00:37:22,719 version better between the harness and 1072 00:37:22,719 --> 00:37:24,480 then between the model and like what's 1073 00:37:24,480 --> 00:37:26,960 the connection between the two 1074 00:37:26,960 --> 00:37:30,719 >> yeah this is a good question so the uh 1075 00:37:30,719 --> 00:37:32,639 the harness in a sense is always a 1076 00:37:32,639 --> 00:37:34,719 little bit ahead of the model 1077 00:37:34,719 --> 00:37:36,400 >> oh really how so? 1078 00:37:36,400 --> 00:37:39,358 >> Oh, what what I mean by that is that you 1079 00:37:39,358 --> 00:37:41,199 you have the model, it's capable of 1080 00:37:41,199 --> 00:37:44,719 certain things, but then you 1081 00:37:44,719 --> 00:37:46,559 set it up with like a couple of crutches 1082 00:37:46,559 --> 00:37:49,358 so that it can actually do the thing um 1083 00:37:49,358 --> 00:37:52,400 to a level of reliability and um in in 1084 00:37:52,400 --> 00:37:56,400 in a way um that is like efficient and 1085 00:37:56,400 --> 00:37:58,480 also with the behavior that you expect 1086 00:37:58,480 --> 00:38:00,719 as a user. And sort of like that's the 1087 00:38:00,719 --> 00:38:02,000 role of the harness, right? is like you 1088 00:38:02,000 --> 00:38:05,119 know provide guard rails like safety, 1089 00:38:05,119 --> 00:38:06,960 make it more efficient, make it more 1090 00:38:06,960 --> 00:38:09,440 like steerable, controllable and then 1091 00:38:09,440 --> 00:38:11,358 the harness usually is also responsible 1092 00:38:11,358 --> 00:38:13,838 for you know what we call like the the 1093 00:38:13,838 --> 00:38:17,838 developer um message which is sort of 1094 00:38:17,838 --> 00:38:20,320 like infected in the context uh at the 1095 00:38:20,320 --> 00:38:23,679 start of of each turn. And so that 1096 00:38:23,679 --> 00:38:25,760 affects obviously like the the purpose 1097 00:38:25,760 --> 00:38:27,599 of that to affect like the behavior of 1098 00:38:27,599 --> 00:38:29,838 the agent throughout uh throughout the 1099 00:38:29,838 --> 00:38:33,358 turn. A lot of what you have is like the 1100 00:38:33,358 --> 00:38:35,440 result of of the harness and the model 1101 00:38:35,440 --> 00:38:37,440 initially like maybe you're like oh it 1102 00:38:37,440 --> 00:38:38,880 doesn't run tests. So you know you have 1103 00:38:38,880 --> 00:38:40,639 to remind it to run tests and then you 1104 00:38:40,639 --> 00:38:43,679 know we train a battle model that is uh 1105 00:38:43,679 --> 00:38:46,079 just you know capable of like better 1106 00:38:46,079 --> 00:38:48,559 reflecting on what is it that you really 1107 00:38:48,559 --> 00:38:50,880 want when you ask for something uh and 1108 00:38:50,880 --> 00:38:52,480 then you know you don't actually have to 1109 00:38:52,480 --> 00:38:54,719 tell it anymore. So over time what we 1110 00:38:54,719 --> 00:38:57,280 see is like the system uh the the 1111 00:38:57,280 --> 00:38:59,440 developer message shrinks and then the 1112 00:38:59,440 --> 00:39:00,800 harness also shrinks 1113 00:39:00,800 --> 00:39:02,800 >> inside of the codeex team. Do you have 1114 00:39:02,800 --> 00:39:04,480 specific goals? Do you say like all 1115 00:39:04,480 --> 00:39:07,358 right now the the codeex as a hardness 1116 00:39:07,358 --> 00:39:10,000 and and model to combine is not very 1117 00:39:10,000 --> 00:39:12,480 good at this or it's kind of doing silly 1118 00:39:12,480 --> 00:39:14,239 mistakes or here or how can I imagine 1119 00:39:14,239 --> 00:39:17,119 how as the engineering team how you're 1120 00:39:17,119 --> 00:39:19,199 working on the next you know version of 1121 00:39:19,199 --> 00:39:21,358 of codeex is because the thing that I 1122 00:39:21,358 --> 00:39:23,679 don't really get as a as a dev is like 1123 00:39:23,679 --> 00:39:25,358 okay there's a model which to me is this 1124 00:39:25,358 --> 00:39:26,880 magical thing which will get better of 1125 00:39:26,880 --> 00:39:28,000 course I'm sure you have some feedback 1126 00:39:28,000 --> 00:39:29,679 channels but you also have the harness 1127 00:39:29,679 --> 00:39:31,039 which is the tools that you're building 1128 00:39:31,039 --> 00:39:32,239 like that's probably what the team is 1129 00:39:32,239 --> 00:39:34,000 responsible for how do set even your 1130 00:39:34,000 --> 00:39:35,358 goals right like in traditional software 1131 00:39:35,358 --> 00:39:36,320 you'll be like we will build this 1132 00:39:36,320 --> 00:39:37,280 feature and you build that feature 1133 00:39:37,280 --> 00:39:38,719 because you know how to do it but it 1134 00:39:38,719 --> 00:39:40,719 feels a bit more fuzzy to me this this 1135 00:39:40,719 --> 00:39:41,679 development process. 1136 00:39:41,679 --> 00:39:44,719 >> Yeah it is and it's why we we co-design 1137 00:39:44,719 --> 00:39:46,480 you know most things and it's it's a 1138 00:39:46,480 --> 00:39:48,480 process where it's a collaboration 1139 00:39:48,480 --> 00:39:51,440 between research and the engineering 1140 00:39:51,440 --> 00:39:53,519 team like primarily building the the 1141 00:39:53,519 --> 00:39:56,719 core the core agent harness. It's always 1142 00:39:56,719 --> 00:39:58,559 uh it's always a question of like okay 1143 00:39:58,559 --> 00:40:01,358 we see today that you know we are very 1144 00:40:01,358 --> 00:40:03,039 good at this but we're not very good at 1145 00:40:03,039 --> 00:40:05,440 this and you know we have a desire to do 1146 00:40:05,440 --> 00:40:07,199 like another thing because it would be a 1147 00:40:07,199 --> 00:40:08,880 very cool products feature and then you 1148 00:40:08,880 --> 00:40:11,119 know we always like look at it it's like 1149 00:40:11,119 --> 00:40:13,039 okay this should this be like a harness 1150 00:40:13,039 --> 00:40:15,039 change or should this be a model change 1151 00:40:15,039 --> 00:40:16,639 and if it's a model change like how soon 1152 00:40:16,639 --> 00:40:18,400 can we have it can we have it in a month 1153 00:40:18,400 --> 00:40:19,760 can we have it in you know three months 1154 00:40:19,760 --> 00:40:21,440 six months and we sort of like work 1155 00:40:21,440 --> 00:40:24,400 through that and then depending on you 1156 00:40:24,400 --> 00:40:26,320 know how soon we can just fix it in the 1157 00:40:26,320 --> 00:40:28,559 model at which level of training. Then 1158 00:40:28,559 --> 00:40:29,920 we might decide to not even do something 1159 00:40:29,920 --> 00:40:32,639 in the harness at all and not and just 1160 00:40:32,639 --> 00:40:34,800 wait for for for the model to to solve 1161 00:40:34,800 --> 00:40:36,320 it. You know, it's it's agents all the 1162 00:40:36,320 --> 00:40:38,400 way, right? So we use agents to analyze 1163 00:40:38,400 --> 00:40:40,320 like a lot of the feedback to like, you 1164 00:40:40,320 --> 00:40:42,159 know, come up with themes, you know, to 1165 00:40:42,159 --> 00:40:44,079 just help us have these conversations 1166 00:40:44,079 --> 00:40:46,639 and decide on priorities. But we analyze 1167 00:40:46,639 --> 00:40:48,800 it across all of coding. We analyze it 1168 00:40:48,800 --> 00:40:50,480 across like all of like know the other 1169 00:40:50,480 --> 00:40:53,199 domains like finance, coms, marketing, 1170 00:40:53,199 --> 00:40:54,880 you know, all the things where our users 1171 00:40:54,880 --> 00:40:57,280 are using these agents nowadays and 1172 00:40:57,280 --> 00:40:58,800 there's like you know subcategories 1173 00:40:58,800 --> 00:41:00,800 within those and then we roughly know 1174 00:41:00,800 --> 00:41:02,159 like you know how well we perform and 1175 00:41:02,159 --> 00:41:04,079 then we're always pushing the frontier 1176 00:41:04,079 --> 00:41:05,519 and there's a thing that is interesting 1177 00:41:05,519 --> 00:41:07,440 is like as we make you know as our 1178 00:41:07,440 --> 00:41:09,280 pre-training model gets better as we 1179 00:41:09,280 --> 00:41:11,199 make the overall model better like the 1180 00:41:11,199 --> 00:41:13,199 whole thing lifts up but then there are 1181 00:41:13,199 --> 00:41:14,719 sometimes things that we pay a little 1182 00:41:14,719 --> 00:41:16,159 bit more attention to. You mentioned you 1183 00:41:16,159 --> 00:41:18,159 know he analyzes agents all all the way. 1184 00:41:18,159 --> 00:41:20,880 Can we talk about the the software 1185 00:41:20,880 --> 00:41:23,358 development life cycle on codeex in the 1186 00:41:23,358 --> 00:41:27,119 sense of whenever a new engineer joins a 1187 00:41:27,119 --> 00:41:29,199 team any team it's like okay how are 1188 00:41:29,199 --> 00:41:31,519 things done here and you know back preai 1189 00:41:31,519 --> 00:41:32,719 it would have been you join the company 1190 00:41:32,719 --> 00:41:34,639 like Uber or Google and they would tell 1191 00:41:34,639 --> 00:41:36,400 you that cool the way it works is we 1192 00:41:36,400 --> 00:41:38,559 have an idea or the PM has an idea we 1193 00:41:38,559 --> 00:41:40,480 make a plan we get together we do some 1194 00:41:40,480 --> 00:41:42,639 estimations we break up the work we code 1195 00:41:42,639 --> 00:41:44,960 the work we do tests we do code reviews 1196 00:41:44,960 --> 00:41:47,199 we release we do feature flags and then 1197 00:41:47,199 --> 00:41:49,358 you know we we were on call that that 1198 00:41:49,358 --> 00:41:51,119 you know that's how it used to be when 1199 00:41:51,119 --> 00:41:53,039 someone joins the codeex team you know 1200 00:41:53,039 --> 00:41:54,880 they they've clearly been contributing 1201 00:41:54,880 --> 00:41:57,358 to the open source part but what what do 1202 00:41:57,358 --> 00:41:59,280 you tell them how do things get get done 1203 00:41:59,280 --> 00:42:01,199 here if they're like a brand total 1204 00:42:01,199 --> 00:42:01,599 newbie 1205 00:42:01,599 --> 00:42:04,159 >> I introduce them to great people um and 1206 00:42:04,159 --> 00:42:06,000 then the thing that they hear the most 1207 00:42:06,000 --> 00:42:07,280 about like when they have a question is 1208 00:42:07,280 --> 00:42:11,679 like have you asked Codex um and Codex 1209 00:42:11,679 --> 00:42:14,079 like is just by default at openi is like 1210 00:42:14,079 --> 00:42:16,159 plugged into everything so it has access 1211 00:42:16,159 --> 00:42:17,679 to slack it has access to all the 1212 00:42:17,679 --> 00:42:20,559 documents, access to all the code, and 1213 00:42:20,559 --> 00:42:23,838 it still surprises uh new starters that 1214 00:42:23,838 --> 00:42:25,920 you can basically ask it anything. Uh 1215 00:42:25,920 --> 00:42:28,239 and it will very often just like come up 1216 00:42:28,239 --> 00:42:31,440 with like a really good response. Um and 1217 00:42:31,440 --> 00:42:34,318 so the easiest way to understand the 1218 00:42:34,318 --> 00:42:36,400 state of a project or who's working on 1219 00:42:36,400 --> 00:42:38,880 something or why a decision was made is 1220 00:42:38,880 --> 00:42:40,400 like critics knows about it all 1221 00:42:40,400 --> 00:42:41,440 internally. 1222 00:42:41,440 --> 00:42:43,679 >> Um and so you just you just use all of 1223 00:42:43,679 --> 00:42:45,679 that. We do a lot of work in you know 1224 00:42:45,679 --> 00:42:47,119 for that reason we do a lot of work in 1225 00:42:47,119 --> 00:42:50,800 public channels. Um we open up uh 1226 00:42:50,800 --> 00:42:52,079 documents with like you know fairly 1227 00:42:52,079 --> 00:42:54,079 broad uh permissions and so that you 1228 00:42:54,079 --> 00:42:55,358 know everyone has access to this 1229 00:42:55,358 --> 00:42:56,880 information as well and so that you know 1230 00:42:56,880 --> 00:42:58,400 your agent can go through things and 1231 00:42:58,400 --> 00:42:59,760 like you know reason through things and 1232 00:42:59,760 --> 00:43:01,280 then you know we have a couple of other 1233 00:43:01,280 --> 00:43:03,440 things that are just really uh very 1234 00:43:03,440 --> 00:43:06,239 helpful for uh team productivity and 1235 00:43:06,239 --> 00:43:07,679 team collaboration that we haven't 1236 00:43:07,679 --> 00:43:09,679 released yet but are are going to come 1237 00:43:09,679 --> 00:43:11,519 like some of it at def day. All of that 1238 00:43:11,519 --> 00:43:13,199 just sort of like makes you very 1239 00:43:13,199 --> 00:43:14,880 grounded and in tune with the rest of 1240 00:43:14,880 --> 00:43:17,199 the team. Um, and allows you to like, 1241 00:43:17,199 --> 00:43:18,639 you know, just very very quickly like 1242 00:43:18,639 --> 00:43:19,920 understand the state of things and and 1243 00:43:19,920 --> 00:43:22,800 and produce things yourself. The general 1244 00:43:22,800 --> 00:43:24,480 recommendation is just like, hey, care 1245 00:43:24,480 --> 00:43:26,400 about the user, uh, care about the 1246 00:43:26,400 --> 00:43:28,880 coherence of the product, care about the 1247 00:43:28,880 --> 00:43:31,039 models and where they're going. Uh, if 1248 00:43:31,039 --> 00:43:32,639 you're doing something and you know 1249 00:43:32,639 --> 00:43:34,639 you're building like this 10,000 lines 1250 00:43:34,639 --> 00:43:36,559 of code crutch to work around the model 1251 00:43:36,559 --> 00:43:37,599 flaws, like you know, you're probably 1252 00:43:37,599 --> 00:43:38,960 doing the wrong thing. So, we have a set 1253 00:43:38,960 --> 00:43:42,639 of principles. Um but it's just really 1254 00:43:42,639 --> 00:43:45,199 um sort of like a team culture and ethos 1255 00:43:45,199 --> 00:43:46,719 at this point and you know it's just 1256 00:43:46,719 --> 00:43:48,800 very much so like carries on you know 1257 00:43:48,800 --> 00:43:50,639 when people join it's just like through 1258 00:43:50,639 --> 00:43:52,000 the rest of the team just like you know 1259 00:43:52,000 --> 00:43:53,599 sort like teaching the ropes and then 1260 00:43:53,599 --> 00:43:56,079 when I have an idea I think it's a good 1261 00:43:56,079 --> 00:43:57,920 idea I I talk it through with codeex 1262 00:43:57,920 --> 00:43:59,119 maybe I talk it through with some my 1263 00:43:59,119 --> 00:44:00,480 colleagues like here's a cool new 1264 00:44:00,480 --> 00:44:02,239 feature I'm going to build as my first 1265 00:44:02,239 --> 00:44:03,838 first contribution or first major 1266 00:44:03,838 --> 00:44:06,719 contribution at to codeex how do I go 1267 00:44:06,719 --> 00:44:08,239 about that obviously I I code it down 1268 00:44:08,239 --> 00:44:10,159 with codeex I obviously test it and make 1269 00:44:10,159 --> 00:44:12,318 sure that it works from there on what's 1270 00:44:12,318 --> 00:44:14,639 the process do you still have the 1271 00:44:14,639 --> 00:44:17,119 concept of code review or AI code review 1272 00:44:17,119 --> 00:44:18,880 of verification of rolling out of 1273 00:44:18,880 --> 00:44:20,960 verifying of stage rolled out you know 1274 00:44:20,960 --> 00:44:22,880 the things because codex itself it goes 1275 00:44:22,880 --> 00:44:25,519 out to millions of people like I just 1276 00:44:25,519 --> 00:44:28,400 crossed a big 20 million active user 1277 00:44:28,400 --> 00:44:30,480 mark but if it's chat GPT then it also 1278 00:44:30,480 --> 00:44:32,480 goes out to like even a lot bigger 1279 00:44:32,480 --> 00:44:33,599 number of people 1280 00:44:33,599 --> 00:44:36,559 >> yes um but it's it's surprisingly like a 1281 00:44:36,559 --> 00:44:38,480 similar process whether you ship on 1282 00:44:38,480 --> 00:44:39,838 codeex or chat 1283 00:44:39,838 --> 00:44:41,519 even though Chibd goes out to like a 1284 00:44:41,519 --> 00:44:43,679 billion a billion you know active users 1285 00:44:43,679 --> 00:44:48,480 and growing you can ship a PR um you 1286 00:44:48,480 --> 00:44:49,760 know you can make a change and you know 1287 00:44:49,760 --> 00:44:51,199 get it shipped like the next day or like 1288 00:44:51,199 --> 00:44:52,960 even the same day um and it just goes 1289 00:44:52,960 --> 00:44:56,239 out to a billion users and it's fine we 1290 00:44:56,239 --> 00:44:58,800 just really instill a sense of ownership 1291 00:44:58,800 --> 00:45:01,679 and care so you're like people are very 1292 00:45:01,679 --> 00:45:03,199 empowered to make changes even large 1293 00:45:03,199 --> 00:45:05,119 changes the general thing that is being 1294 00:45:05,119 --> 00:45:07,280 asked is like sort of like evidence that 1295 00:45:07,280 --> 00:45:09,199 it's going to be wellreceived D evidence 1296 00:45:09,199 --> 00:45:11,599 that is like a worthy addition uh 1297 00:45:11,599 --> 00:45:13,280 evidence that you know it's like it is 1298 00:45:13,280 --> 00:45:15,039 worth maintaining over time but also 1299 00:45:15,039 --> 00:45:16,400 like the cost of maintenance is like 1300 00:45:16,400 --> 00:45:18,960 just really as you know gotten done 1301 00:45:18,960 --> 00:45:21,440 significantly as well. So we we we think 1302 00:45:21,440 --> 00:45:22,880 about these things slightly differently 1303 00:45:22,880 --> 00:45:24,480 than you know say like two years ago or 1304 00:45:24,480 --> 00:45:26,079 3 years ago. The other thing as well is 1305 00:45:26,079 --> 00:45:27,440 like you know we automate as much as 1306 00:45:27,440 --> 00:45:28,880 possible. So like a lot of like the 1307 00:45:28,880 --> 00:45:31,358 process of like code review and deploys 1308 00:45:31,358 --> 00:45:33,199 and you know catching regressions is 1309 00:45:33,199 --> 00:45:34,639 like you know all of that is like pretty 1310 00:45:34,639 --> 00:45:37,199 much automated. Uh and so like you know 1311 00:45:37,199 --> 00:45:40,318 you get to just focus on just really the 1312 00:45:40,318 --> 00:45:42,400 idea and you know how it's going to help 1313 00:45:42,400 --> 00:45:44,800 our users and you care about you know 1314 00:45:44,800 --> 00:45:46,639 the coherence of it all and so like the 1315 00:45:46,639 --> 00:45:49,280 overall power of the agent um and making 1316 00:45:49,280 --> 00:45:50,880 things better and like we don't we have 1317 00:45:50,880 --> 00:45:52,639 a long long list of things that you know 1318 00:45:52,639 --> 00:45:54,318 we sort of like aspire to do and haven't 1319 00:45:54,318 --> 00:45:55,920 gotten to yet. And then there's like the 1320 00:45:55,920 --> 00:45:57,920 sort of like the northstar direction um 1321 00:45:57,920 --> 00:46:01,519 which is a delightful simple to use 1322 00:46:01,519 --> 00:46:03,679 personal AGI that you know knows 1323 00:46:03,679 --> 00:46:05,358 everything about you like you know that 1324 00:46:05,358 --> 00:46:07,440 it needs to know has access to the right 1325 00:46:07,440 --> 00:46:08,960 resources can take like you know 1326 00:46:08,960 --> 00:46:10,559 sometimes risky actions on your behalf 1327 00:46:10,559 --> 00:46:11,920 but then you know you get like the push 1328 00:46:11,920 --> 00:46:13,358 notification and then you know you can 1329 00:46:13,358 --> 00:46:15,119 verify that and it's like a thing that 1330 00:46:15,119 --> 00:46:17,119 you know you deeply understand as a user 1331 00:46:17,119 --> 00:46:19,119 but also it knows about your schedule it 1332 00:46:19,119 --> 00:46:20,480 knows about your goals it can be 1333 00:46:20,480 --> 00:46:21,760 proactive and it should be like 1334 00:46:21,760 --> 00:46:23,760 extremely natural it should be something 1335 00:46:23,760 --> 00:46:25,280 that you can control through like 1336 00:46:25,280 --> 00:46:27,440 natural language, voice, you know, like 1337 00:46:27,440 --> 00:46:28,880 maybe it should understand, you know, 1338 00:46:28,880 --> 00:46:30,239 your emotions like if it has like a 1339 00:46:30,239 --> 00:46:31,760 camera feed, it should be the most 1340 00:46:31,760 --> 00:46:33,280 natural thing on earth. It's like it 1341 00:46:33,280 --> 00:46:35,199 should not be like a thing with 10, you 1342 00:46:35,199 --> 00:46:36,239 know, different buttons and 1343 00:46:36,239 --> 00:46:38,800 configurations. It's like AGI should be 1344 00:46:38,800 --> 00:46:40,239 simple to use. Now, 1345 00:46:40,239 --> 00:46:42,318 >> you kind of mentioned just briefly the 1346 00:46:42,318 --> 00:46:44,880 review the code review, but I wanted to 1347 00:46:44,880 --> 00:46:46,559 go back to it. you you worked at Google 1348 00:46:46,559 --> 00:46:48,880 on a on a product used by you know like 1349 00:46:48,880 --> 00:46:50,239 hundreds of millions which is Google 1350 00:46:50,239 --> 00:46:52,079 maps and Google is very well known for 1351 00:46:52,079 --> 00:46:54,318 their culture of very strict code 1352 00:46:54,318 --> 00:46:56,079 reviews. They have I think two layers of 1353 00:46:56,079 --> 00:46:57,280 code reviews. There's a language 1354 00:46:57,280 --> 00:46:58,559 correctness review and they they've 1355 00:46:58,559 --> 00:47:00,000 taken I think they've really perfected 1356 00:47:00,000 --> 00:47:02,079 it across the industry for for a long 1357 00:47:02,079 --> 00:47:03,679 time and they do believe that it it 1358 00:47:03,679 --> 00:47:05,920 works and they they use it. How do you 1359 00:47:05,920 --> 00:47:07,358 think that part is changing specifically 1360 00:47:07,358 --> 00:47:09,358 the human review? Because for a very 1361 00:47:09,358 --> 00:47:12,239 long time until maybe a year or two ago, 1362 00:47:12,239 --> 00:47:13,838 I would have said you code review has 1363 00:47:13,838 --> 00:47:15,440 all these benefits that knowledge 1364 00:47:15,440 --> 00:47:17,760 sharing the second pair of eyes removing 1365 00:47:17,760 --> 00:47:19,440 the bus factor because now someone else 1366 00:47:19,440 --> 00:47:20,960 understands and when that person is out 1367 00:47:20,960 --> 00:47:23,519 that they can jump in conversations are 1368 00:47:23,519 --> 00:47:25,358 happening about architecture not just 1369 00:47:25,358 --> 00:47:29,199 not just the code but now there's you 1370 00:47:29,199 --> 00:47:32,559 know there's a lot more code uh and what 1371 00:47:32,559 --> 00:47:35,679 was the value of code review? What in 1372 00:47:35,679 --> 00:47:38,159 what cases? And so on your team, because 1373 00:47:38,159 --> 00:47:41,039 you guys are so ahead of this, where do 1374 00:47:41,039 --> 00:47:43,358 you see humans still being or developers 1375 00:47:43,358 --> 00:47:45,199 being involved in the review stage 1376 00:47:45,199 --> 00:47:48,318 valuable? And and where is it fine? Did 1377 00:47:48,318 --> 00:47:50,639 you find it find it fine to uh hand it 1378 00:47:50,639 --> 00:47:52,480 off to an agent? 1379 00:47:52,480 --> 00:47:54,719 >> Yeah, the the the role of code review is 1380 00:47:54,719 --> 00:47:56,639 changing. One one of the early projects 1381 00:47:56,639 --> 00:47:58,719 that I did on Codex was like working 1382 00:47:58,719 --> 00:48:01,119 with research on developing a code 1383 00:48:01,119 --> 00:48:05,119 review model um that was 1384 00:48:05,119 --> 00:48:08,880 going to be to a level where it can spot 1385 00:48:08,880 --> 00:48:12,239 mistakes in logic and reasoning. uh to a 1386 00:48:12,239 --> 00:48:14,400 degree where it would require humans 1387 00:48:14,400 --> 00:48:17,280 like you know multiple like potentially 1388 00:48:17,280 --> 00:48:18,960 multiple hours to capture the same level 1389 00:48:18,960 --> 00:48:20,719 of mistake because it requires like 1390 00:48:20,719 --> 00:48:22,559 really digging like you know three four 1391 00:48:22,559 --> 00:48:24,400 levels deep into like the dependencies 1392 00:48:24,400 --> 00:48:26,239 and like you know understand that maybe 1393 00:48:26,239 --> 00:48:27,920 the documentation actually was wrong and 1394 00:48:27,920 --> 00:48:29,199 like the implementation of like this 1395 00:48:29,199 --> 00:48:30,480 third party dependencies like different 1396 00:48:30,480 --> 00:48:31,920 from what you expected and so therefore 1397 00:48:31,920 --> 00:48:34,880 your invarants are not upheld um and 1398 00:48:34,880 --> 00:48:36,000 these things it's just like you know 1399 00:48:36,000 --> 00:48:37,519 unless you're an expert in that library 1400 00:48:37,519 --> 00:48:39,599 you wouldn't know uh and therefore you 1401 00:48:39,599 --> 00:48:41,280 have a bug and so we developed like 1402 00:48:41,280 --> 00:48:43,358 these uh code review models and you know 1403 00:48:43,358 --> 00:48:44,960 we we we released them and now they're 1404 00:48:44,960 --> 00:48:47,280 like the same level of like capability 1405 00:48:47,280 --> 00:48:50,079 and like ability to spot these mistakes 1406 00:48:50,079 --> 00:48:52,239 by doing like you know deep verification 1407 00:48:52,239 --> 00:48:53,519 are like just part of the mainline 1408 00:48:53,519 --> 00:48:55,119 models like when we benchmark them it's 1409 00:48:55,119 --> 00:48:57,039 like they're like super human in code 1410 00:48:57,039 --> 00:48:59,039 review and this is not just true for 1411 00:48:59,039 --> 00:49:00,559 correctness. This is also true for 1412 00:49:00,559 --> 00:49:02,318 security for example where you're 1413 00:49:02,318 --> 00:49:03,838 capable of like reasoning across like 1414 00:49:03,838 --> 00:49:05,199 you know very very complex things and 1415 00:49:05,199 --> 00:49:06,800 then you know coming up with like hey 1416 00:49:06,800 --> 00:49:08,239 you know you have a critical security 1417 00:49:08,239 --> 00:49:10,159 vulnerability here which is now 1418 00:49:10,159 --> 00:49:12,559 mandatory across like all of OpenAI pull 1419 00:49:12,559 --> 00:49:14,400 requests like we block pull requests 1420 00:49:14,400 --> 00:49:16,639 from merging if you know we flag them 1421 00:49:16,639 --> 00:49:19,199 with like a a security issue and this is 1422 00:49:19,199 --> 00:49:21,440 like all automatic and the role of code 1423 00:49:21,440 --> 00:49:23,440 review now is like I think it was always 1424 00:49:23,440 --> 00:49:25,119 about correctness it was always about 1425 00:49:25,119 --> 00:49:26,559 you know ensuring that things worked but 1426 00:49:26,559 --> 00:49:28,639 it was also sort of like a little ritual 1427 00:49:28,639 --> 00:49:30,480 for information exchange and you know 1428 00:49:30,480 --> 00:49:31,838 bringing people on the same page and 1429 00:49:31,838 --> 00:49:33,199 like you know encouraging like a 1430 00:49:33,199 --> 00:49:35,440 discussion which ideally would have 1431 00:49:35,440 --> 00:49:36,960 happened before but sometimes it just 1432 00:49:36,960 --> 00:49:38,239 only happens like around the code 1433 00:49:38,239 --> 00:49:39,920 because once it merged it just actually 1434 00:49:39,920 --> 00:49:41,760 runs in production it's doing stuff 1435 00:49:41,760 --> 00:49:43,519 >> and then you have to maintain it so 1436 00:49:43,519 --> 00:49:45,519 there's like this social aspect to it as 1437 00:49:45,519 --> 00:49:47,358 well it's I think all of it is changing 1438 00:49:47,358 --> 00:49:49,599 like the correctness the cyber the the 1439 00:49:49,599 --> 00:49:51,039 security is like I think that will be 1440 00:49:51,039 --> 00:49:53,838 automated really what we see and I see 1441 00:49:53,838 --> 00:49:56,639 is there's a sort of um really 1442 00:49:56,639 --> 00:49:59,199 discussion around the intent that takes 1443 00:49:59,199 --> 00:50:01,199 place around the poll request. It's like 1444 00:50:01,199 --> 00:50:03,280 what are you even trying to do? Um and 1445 00:50:03,280 --> 00:50:04,960 is that a right thing to attempt to do? 1446 00:50:04,960 --> 00:50:06,318 I think you can have that discussion 1447 00:50:06,318 --> 00:50:07,760 outside of the poll request. It doesn't 1448 00:50:07,760 --> 00:50:08,880 have to be around code. 1449 00:50:08,880 --> 00:50:11,039 >> So, so maybe this helps crystallize the 1450 00:50:11,039 --> 00:50:13,119 you know like where a discussion needs 1451 00:50:13,119 --> 00:50:15,519 to happen versus versus where we we did 1452 00:50:15,519 --> 00:50:17,280 it because maybe we didn't have the type 1453 00:50:17,280 --> 00:50:18,639 the type of tooling that we have right 1454 00:50:18,639 --> 00:50:20,480 now. Yeah, I think this is going to 1455 00:50:20,480 --> 00:50:22,159 change and it was like a forcing 1456 00:50:22,159 --> 00:50:23,519 function because you know you have to 1457 00:50:23,519 --> 00:50:24,639 have that discussion where like it's 1458 00:50:24,639 --> 00:50:26,000 good to have that discussion before you 1459 00:50:26,000 --> 00:50:28,239 merge it and it becomes production code 1460 00:50:28,239 --> 00:50:31,199 but I I think there are other ways to 1461 00:50:31,199 --> 00:50:33,039 have these discussions and you know 1462 00:50:33,039 --> 00:50:34,400 design things together and make sure 1463 00:50:34,400 --> 00:50:36,960 that the intent is good uh and then the 1464 00:50:36,960 --> 00:50:38,480 code doesn't matter as much 1465 00:50:38,480 --> 00:50:39,838 >> and it's interesting because when I 1466 00:50:39,838 --> 00:50:41,519 think back of all my code reviews like 1467 00:50:41,519 --> 00:50:43,039 of course I have like memories where 1468 00:50:43,039 --> 00:50:44,880 like it was great we we had a good 1469 00:50:44,880 --> 00:50:46,559 discussion or I learned something really 1470 00:50:46,559 --> 00:50:47,920 interesting but a bunch of times 1471 00:50:47,920 --> 00:50:50,318 honestly It was such a pain in the ass. 1472 00:50:50,318 --> 00:50:52,400 Like I I was trying to get my stuff. 1473 00:50:52,400 --> 00:50:54,719 You're paying. Hey, could you remove my 1474 00:50:54,719 --> 00:50:56,559 code? And like, no, right now I'm busy. 1475 00:50:56,559 --> 00:50:58,480 No, I really need this to unblock me. 1476 00:50:58,480 --> 00:51:00,480 And then you context switch. And then I 1477 00:51:00,480 --> 00:51:03,280 feel it's always been like good and bad, 1478 00:51:03,280 --> 00:51:05,599 right? So I I feel whatever we do there 1479 00:51:05,599 --> 00:51:08,079 will be always upsides and and downside, 1480 00:51:08,079 --> 00:51:09,760 but there now they're just moving. So I 1481 00:51:09,760 --> 00:51:12,480 guess one upside is as an engineer you 1482 00:51:12,480 --> 00:51:15,440 might have to not give your attention to 1483 00:51:15,440 --> 00:51:17,440 just kind of basic stuff that doesn't 1484 00:51:17,440 --> 00:51:19,599 need your input per se. Yes, it saves 1485 00:51:19,599 --> 00:51:22,800 time and I think progressively what 1486 00:51:22,800 --> 00:51:25,280 we're going to see is also like you have 1487 00:51:25,280 --> 00:51:28,079 an agreement on you know the box and the 1488 00:51:28,079 --> 00:51:29,920 overall contract of what it's supposed 1489 00:51:29,920 --> 00:51:31,679 to do and then you know what is inside 1490 00:51:31,679 --> 00:51:33,440 the box as long as you have like strict 1491 00:51:33,440 --> 00:51:35,599 guarantees in terms of resource 1492 00:51:35,599 --> 00:51:39,039 utilization, data access, um security, 1493 00:51:39,039 --> 00:51:40,480 these kinds of things. It's like what 1494 00:51:40,480 --> 00:51:42,880 happens inside the box is you know it 1495 00:51:42,880 --> 00:51:44,480 could be literally anything. it's like 1496 00:51:44,480 --> 00:51:46,239 don't really need to care and like 1497 00:51:46,239 --> 00:51:47,760 really what you need to agree on is like 1498 00:51:47,760 --> 00:51:49,199 what does the box actually do and what 1499 00:51:49,199 --> 00:51:50,719 are the invariants that must be 1500 00:51:50,719 --> 00:51:52,480 satisfied and I think that is then 1501 00:51:52,480 --> 00:51:54,239 worthy you know having like a really 1502 00:51:54,239 --> 00:51:56,400 good conversation on you know maybe 1503 00:51:56,400 --> 00:51:58,800 assisted by by your favorite uh agent 1504 00:51:58,800 --> 00:52:00,559 but then once you have that and you have 1505 00:52:00,559 --> 00:52:01,838 that understanding it's just like 1506 00:52:01,838 --> 00:52:03,519 changing anything within the box is like 1507 00:52:03,519 --> 00:52:05,358 you know doesn't require for discussion 1508 00:52:05,358 --> 00:52:06,559 and it's like you know just really 1509 00:52:06,559 --> 00:52:07,760 preserves your attention 1510 00:52:07,760 --> 00:52:10,480 >> the cost of maintenance has gone down 1511 00:52:10,480 --> 00:52:11,920 you know maintenance is always such a 1512 00:52:11,920 --> 00:52:13,838 hot topic whenever we build thing uh 1513 00:52:13,838 --> 00:52:15,199 inside of all these companies like 1514 00:52:15,199 --> 00:52:16,719 Google, Uber, even startups like 1515 00:52:16,719 --> 00:52:18,880 building was the fun part but then 1516 00:52:18,880 --> 00:52:20,400 maintenance was the painful and that's 1517 00:52:20,400 --> 00:52:22,079 when we learned like okay it was not 1518 00:52:22,079 --> 00:52:25,280 we're building it etc inside of codeex 1519 00:52:25,280 --> 00:52:28,000 and open AAI what do you see maintenance 1520 00:52:28,000 --> 00:52:30,719 becoming cheaper changing in terms of 1521 00:52:30,719 --> 00:52:32,800 instead of what you're building what the 1522 00:52:32,800 --> 00:52:35,519 ambition is the I guess custom tooling 1523 00:52:35,519 --> 00:52:36,480 those kind of things 1524 00:52:36,480 --> 00:52:38,719 >> maintenance is really like sort of like 1525 00:52:38,719 --> 00:52:40,559 a tax that you pay over time just to 1526 00:52:40,559 --> 00:52:42,800 keep things running and it's It's it's 1527 00:52:42,800 --> 00:52:44,400 it's always been necessary. It will 1528 00:52:44,400 --> 00:52:45,679 continue to be necessary, but where I 1529 00:52:45,679 --> 00:52:47,199 think it changes is like a lot of it is 1530 00:52:47,199 --> 00:52:49,280 just going to be automated. So, you 1531 00:52:49,280 --> 00:52:50,960 know, it's like okay, you have you have 1532 00:52:50,960 --> 00:52:53,280 this third party dependencies like you 1533 00:52:53,280 --> 00:52:55,280 need to upgrade the version number. It's 1534 00:52:55,280 --> 00:52:57,440 like a like 1535 00:52:57,440 --> 00:52:58,559 [laughter] 1536 00:52:58,559 --> 00:53:00,639 you can fully automate this you know um 1537 00:53:00,639 --> 00:53:03,199 if you have good change log and you know 1538 00:53:03,199 --> 00:53:04,880 and the code is well documented and like 1539 00:53:04,880 --> 00:53:06,239 you know and and the model can just like 1540 00:53:06,239 --> 00:53:07,440 reason through it like you know you can 1541 00:53:07,440 --> 00:53:09,440 just like blast through your codebase do 1542 00:53:09,440 --> 00:53:11,920 it uh in a couple of hours and you know 1543 00:53:11,920 --> 00:53:14,159 previously you would have like punted on 1544 00:53:14,159 --> 00:53:15,440 it because it's not the most fun thing 1545 00:53:15,440 --> 00:53:17,760 to do but it's actually really important 1546 00:53:17,760 --> 00:53:19,280 for your business. So it's like really 1547 00:53:19,280 --> 00:53:20,480 important for your project, you know, 1548 00:53:20,480 --> 00:53:22,239 especially for security vulnerabilities. 1549 00:53:22,239 --> 00:53:24,079 You want to stay up to date, right? You 1550 00:53:24,079 --> 00:53:25,440 want to apply, you know, all these 1551 00:53:25,440 --> 00:53:26,960 patches. 1552 00:53:26,960 --> 00:53:28,639 Um I think that's just going to be fully 1553 00:53:28,639 --> 00:53:30,719 automated. So a large part of like 1554 00:53:30,719 --> 00:53:32,318 maintenance, it just kind of comes for 1555 00:53:32,318 --> 00:53:35,599 free, right? And then um I think it's 1556 00:53:35,599 --> 00:53:37,679 it's awesome to also think about before 1557 00:53:37,679 --> 00:53:39,119 like you know when you wanted to just 1558 00:53:39,119 --> 00:53:41,519 completely react re you have to do like 1559 00:53:41,519 --> 00:53:43,519 a new architecture because you're trying 1560 00:53:43,519 --> 00:53:46,000 to make space for like a new you know 1561 00:53:46,000 --> 00:53:47,519 different kind of trade-offs or you have 1562 00:53:47,519 --> 00:53:49,599 a new understanding of like the workload 1563 00:53:49,599 --> 00:53:51,440 or you're trying to fit a new feature 1564 00:53:51,440 --> 00:53:53,039 and like suddenly you realize like your 1565 00:53:53,039 --> 00:53:54,559 current system is just very limiting and 1566 00:53:54,559 --> 00:53:56,639 you need to completely rearchitecture it 1567 00:53:56,639 --> 00:53:58,159 that was like a really really costly 1568 00:53:58,159 --> 00:53:59,358 endeavor right so you know like 1569 00:53:59,358 --> 00:54:01,599 sometimes like multiple years and I 1570 00:54:01,599 --> 00:54:03,199 think this is also like super super 1571 00:54:03,199 --> 00:54:06,318 accelerated now. So you like the cost of 1572 00:54:06,318 --> 00:54:08,719 mistakes uh you know I would say like 1573 00:54:08,719 --> 00:54:10,800 you know is going down but then at the 1574 00:54:10,800 --> 00:54:13,519 same time the good old rules I would say 1575 00:54:13,519 --> 00:54:14,719 of software engineer of like you know 1576 00:54:14,719 --> 00:54:16,239 having good abstractions like really 1577 00:54:16,239 --> 00:54:18,000 help like you know is going back to this 1578 00:54:18,000 --> 00:54:19,838 like having the box with invariance like 1579 00:54:19,838 --> 00:54:21,599 you know if you sort of like draw the 1580 00:54:21,599 --> 00:54:23,358 right shape you're going to be able to 1581 00:54:23,358 --> 00:54:24,800 change things much more quickly within 1582 00:54:24,800 --> 00:54:26,239 the box and like not affect the rest of 1583 00:54:26,239 --> 00:54:28,719 the of the services or the rest of your 1584 00:54:28,719 --> 00:54:30,239 infrastructure 1585 00:54:30,239 --> 00:54:31,358 and I think it's important it's 1586 00:54:31,358 --> 00:54:32,960 important to design for very quick 1587 00:54:32,960 --> 00:54:35,039 iteration and I remember when I talked 1588 00:54:35,039 --> 00:54:37,280 with Peter Shamberger that was before he 1589 00:54:37,280 --> 00:54:40,159 joined OpenAI but about open claw and 1590 00:54:40,159 --> 00:54:41,760 how he thinks about it like you know he 1591 00:54:41,760 --> 00:54:43,199 told me that he doesn't read the code 1592 00:54:43,199 --> 00:54:45,199 but he kept thinking about like I could 1593 00:54:45,199 --> 00:54:46,719 see that he's holding the architecture 1594 00:54:46,719 --> 00:54:48,239 in his head and he was telling me how he 1595 00:54:48,239 --> 00:54:50,000 rearchitects a lot and he thinks about 1596 00:54:50,000 --> 00:54:53,280 how to make it modular how to allow 100 1597 00:54:53,280 --> 00:54:55,599 contributors to each build their thing 1598 00:54:55,599 --> 00:54:57,358 without stepping on each other's toes. 1599 00:54:57,358 --> 00:55:00,159 So I'm hearing what you're saying that 1600 00:55:00,159 --> 00:55:04,000 this this care this this planning this 1601 00:55:04,000 --> 00:55:05,920 this structuring has become maybe just a 1602 00:55:05,920 --> 00:55:08,400 lot more important to like which which 1603 00:55:08,400 --> 00:55:09,679 which was which which was something back 1604 00:55:09,679 --> 00:55:11,440 in the day you know it was like the 1605 00:55:11,440 --> 00:55:13,199 architect or the staff engineer or 1606 00:55:13,199 --> 00:55:15,119 experienced folks were doing this thing 1607 00:55:15,119 --> 00:55:17,119 and other engineers around them were 1608 00:55:17,119 --> 00:55:18,400 kind of building this you know smaller 1609 00:55:18,400 --> 00:55:19,679 parts but it sounds like now all 1610 00:55:19,679 --> 00:55:22,559 engineers need to be aware of when 1611 00:55:22,559 --> 00:55:23,599 you're building your software right and 1612 00:55:23,599 --> 00:55:24,480 plan for it 1613 00:55:24,480 --> 00:55:26,480 >> yeah and and the GP models are getting 1614 00:55:26,480 --> 00:55:27,838 better and better at this as well of 1615 00:55:27,838 --> 00:55:29,358 like you know thinking about long-term 1616 00:55:29,358 --> 00:55:31,119 maintenance and like good architecture 1617 00:55:31,119 --> 00:55:32,800 and like this is like a natural sort of 1618 00:55:32,800 --> 00:55:34,880 like next step right. It's like not just 1619 00:55:34,880 --> 00:55:36,559 about code quality in the sense of like 1620 00:55:36,559 --> 00:55:38,239 oh is this code clean within this file 1621 00:55:38,239 --> 00:55:39,838 but like you know is this like is the 1622 00:55:39,838 --> 00:55:41,599 architecture actually correct to reduce 1623 00:55:41,599 --> 00:55:43,119 maintenance burden over time and like 1624 00:55:43,119 --> 00:55:45,760 you know make space for like future u 1625 00:55:45,760 --> 00:55:47,920 product or feature extensions or changes 1626 00:55:47,920 --> 00:55:49,199 and just really this act of like you 1627 00:55:49,199 --> 00:55:51,679 know engineering over time that's uh 1628 00:55:51,679 --> 00:55:53,199 kind of like something that models are 1629 00:55:53,199 --> 00:55:54,639 starting to become capable of like 1630 00:55:54,639 --> 00:55:56,079 thinking about very well I think it's 1631 00:55:56,079 --> 00:55:57,679 just kind of fascinating to understand 1632 00:55:57,679 --> 00:56:01,920 that the software that we're building is 1633 00:56:01,920 --> 00:56:03,358 just going through the life cycle much 1634 00:56:03,358 --> 00:56:04,639 much faster. Right? Like you know before 1635 00:56:04,639 --> 00:56:06,400 you had you know you were you were 1636 00:56:06,400 --> 00:56:07,838 scaling you were starting it you know 1637 00:56:07,838 --> 00:56:09,599 maybe as like a small team of you know 1638 00:56:09,599 --> 00:56:11,679 yourself maybe a couple of engineers and 1639 00:56:11,679 --> 00:56:14,159 then you would add engineers like slowly 1640 00:56:14,159 --> 00:56:15,920 and then you know maybe after a year you 1641 00:56:15,920 --> 00:56:17,119 know it's like if it's very very 1642 00:56:17,119 --> 00:56:18,719 successful you would have 50 engineers 1643 00:56:18,719 --> 00:56:21,119 on it or like a 100 engineers on it. You 1644 00:56:21,119 --> 00:56:22,798 would have time to see it coming. you 1645 00:56:22,798 --> 00:56:24,159 would have time to see like you know the 1646 00:56:24,159 --> 00:56:26,318 humans on board and you sort of like you 1647 00:56:26,318 --> 00:56:27,599 know you can think about the 1648 00:56:27,599 --> 00:56:29,280 documentation all of that stuff but now 1649 00:56:29,280 --> 00:56:30,960 it's just sort of like that explosion of 1650 00:56:30,960 --> 00:56:32,559 like you know suddenly you have like a 1651 00:56:32,559 --> 00:56:34,400 100 agents contributing to this thing is 1652 00:56:34,400 --> 00:56:35,599 like you know that can happen like you 1653 00:56:35,599 --> 00:56:38,318 know in a weekend and so you know you're 1654 00:56:38,318 --> 00:56:41,119 just going through it at you know major 1655 00:56:41,119 --> 00:56:42,639 major speed compared to before 1656 00:56:42,639 --> 00:56:44,719 >> okay but how do you and and the folks at 1657 00:56:44,719 --> 00:56:46,318 OpenAI like deal with this like does it 1658 00:56:46,318 --> 00:56:49,280 not mess with your mind like you know 1659 00:56:49,280 --> 00:56:50,719 what I mean in the sense of like you you 1660 00:56:50,719 --> 00:56:52,559 you've been in this business for quite 1661 00:56:52,559 --> 00:56:55,519 some time now like like decades or or 1662 00:56:55,519 --> 00:56:57,838 well over and there was a pace that we 1663 00:56:57,838 --> 00:56:59,358 kind of got used to and obviously is 1664 00:56:59,358 --> 00:57:02,079 it's now a lot faster but how do you get 1665 00:57:02,079 --> 00:57:04,318 your head around the fact that a it's 1666 00:57:04,318 --> 00:57:06,559 faster b the stuff that you've been 1667 00:57:06,559 --> 00:57:09,039 doing a year ago right now you're not 1668 00:57:09,039 --> 00:57:10,559 doing because now the model is is good 1669 00:57:10,559 --> 00:57:12,559 at it and you know like how do you kind 1670 00:57:12,559 --> 00:57:14,000 of reconcile that because there I'm sure 1671 00:57:14,000 --> 00:57:15,199 there's stuff that you've been really 1672 00:57:15,199 --> 00:57:17,039 good at uh related to software that now 1673 00:57:17,039 --> 00:57:18,719 you can hand off to the agent do you not 1674 00:57:18,719 --> 00:57:20,159 get a little bit of sting you know we 1675 00:57:20,159 --> 00:57:22,400 talked about it's stinging for your 1676 00:57:22,400 --> 00:57:23,838 features to be implemented open source. 1677 00:57:23,838 --> 00:57:25,199 But it can also sting that I've been 1678 00:57:25,199 --> 00:57:26,318 really good at like I don't know 1679 00:57:26,318 --> 00:57:29,519 refactoring or or or right now it might 1680 00:57:29,519 --> 00:57:30,960 be architecture but maybe the model will 1681 00:57:30,960 --> 00:57:32,400 be really good at that and now I'm like 1682 00:57:32,400 --> 00:57:35,199 uh okay damn like I'm glad but also like 1683 00:57:35,199 --> 00:57:36,480 uh it would have been nice for me to do 1684 00:57:36,480 --> 00:57:36,880 that. 1685 00:57:36,880 --> 00:57:38,639 >> Yeah, I think there's like a craft 1686 00:57:38,639 --> 00:57:42,239 aspect to it. Um which occasionally I 1687 00:57:42,239 --> 00:57:44,000 still you know pull up an editor and 1688 00:57:44,000 --> 00:57:45,838 like write some code. Um, and it's just 1689 00:57:45,838 --> 00:57:48,960 like it it it it feels nice and and and 1690 00:57:48,960 --> 00:57:50,798 it's sort of like I have fond memories 1691 00:57:50,798 --> 00:57:53,519 of like late nights sitting [laughter] 1692 00:57:53,519 --> 00:57:56,798 in uh in Vim and you know, just like 1693 00:57:56,798 --> 00:57:57,838 >> cranking it out, 1694 00:57:57,838 --> 00:57:59,920 >> you know, drinking 1695 00:57:59,920 --> 00:58:03,358 um Coke Zero and uh yeah, just not 1696 00:58:03,358 --> 00:58:04,480 having to think about anything else 1697 00:58:04,480 --> 00:58:05,920 other than like the problem in front of 1698 00:58:05,920 --> 00:58:10,480 me. But really, I think it's um it's all 1699 00:58:10,480 --> 00:58:12,559 about being in the flow and and solving 1700 00:58:12,559 --> 00:58:14,719 problems. And what I find is like you 1701 00:58:14,719 --> 00:58:17,119 know folks here and also like everyone I 1702 00:58:17,119 --> 00:58:19,280 talk to is just like adapting very 1703 00:58:19,280 --> 00:58:21,760 quickly and I think if you if you have a 1704 00:58:21,760 --> 00:58:24,639 mindset where it's all about code is a 1705 00:58:24,639 --> 00:58:27,039 tool to solve problems and you can solve 1706 00:58:27,039 --> 00:58:29,199 so many more problems. It's like before 1707 00:58:29,199 --> 00:58:30,880 you wanted to benchmark something and 1708 00:58:30,880 --> 00:58:32,639 you weren't quite sure where you were 1709 00:58:32,639 --> 00:58:34,719 going to net out. It's like you can just 1710 00:58:34,719 --> 00:58:36,798 do it. it's it's going to take you like 1711 00:58:36,798 --> 00:58:38,880 no more than 30 seconds, you know, to 1712 00:58:38,880 --> 00:58:40,318 launch something in the background and, 1713 00:58:40,318 --> 00:58:41,920 you know, get proper numbers and be able 1714 00:58:41,920 --> 00:58:43,838 to do like a better trade-off. It makes 1715 00:58:43,838 --> 00:58:46,079 you it should make you a better engineer 1716 00:58:46,079 --> 00:58:49,039 if you just really care about, you know, 1717 00:58:49,039 --> 00:58:51,679 the outcome and the system working well. 1718 00:58:51,679 --> 00:58:53,199 And so what it allows us to do at 1719 00:58:53,199 --> 00:58:55,920 OpenAI, it allows us to run, you know, 1720 00:58:55,920 --> 00:58:57,920 our inference much more efficiently. It 1721 00:58:57,920 --> 00:58:59,519 allows us to, you know, get like much 1722 00:58:59,519 --> 00:59:01,199 more like effective compute and, you 1723 00:59:01,199 --> 00:59:03,119 know, deploy that to the world. And so 1724 00:59:03,119 --> 00:59:04,719 like everyone's just like very focused 1725 00:59:04,719 --> 00:59:06,960 on that and solving important problems 1726 00:59:06,960 --> 00:59:08,400 at the speed that was not possible 1727 00:59:08,400 --> 00:59:11,519 before. And like I I haven't yet, you 1728 00:59:11,519 --> 00:59:12,639 know, encountered someone who's like, 1729 00:59:12,639 --> 00:59:14,880 "Oh, that's not that's not good. Um 1730 00:59:14,880 --> 00:59:15,760 that's not fun." 1731 00:59:15,760 --> 00:59:17,358 >> Do I understand correctly that it sounds 1732 00:59:17,358 --> 00:59:19,280 like if you have ambitious problems, if 1733 00:59:19,280 --> 00:59:20,960 you have way more problems than what you 1734 00:59:20,960 --> 00:59:22,798 can solve today or tomorrow or the next 1735 00:59:22,798 --> 00:59:24,960 week, sounds like this is not really a 1736 00:59:24,960 --> 00:59:26,159 problem because when you know you get 1737 00:59:26,159 --> 00:59:27,760 more efficient somewhere, you keep 1738 00:59:27,760 --> 00:59:30,079 going. Which which is a lot of startups, 1739 00:59:30,079 --> 00:59:31,838 right? like startups are always way more 1740 00:59:31,838 --> 00:59:33,760 ambitious than than what they're able to 1741 00:59:33,760 --> 00:59:34,400 do. 1742 00:59:34,400 --> 00:59:36,079 >> I don't we're not we're not out of 1743 00:59:36,079 --> 00:59:38,159 problems for sure, right? So, and and I 1744 00:59:38,159 --> 00:59:40,159 don't think we will be for a while. Um 1745 00:59:40,159 --> 00:59:42,639 we have a long long road ahead of us in 1746 00:59:42,639 --> 00:59:44,480 terms of like mathematical 1747 00:59:44,480 --> 00:59:46,400 breakthroughs, scientific breakthroughs, 1748 00:59:46,400 --> 00:59:47,760 you know, making the world a better 1749 00:59:47,760 --> 00:59:50,079 place like just really 1750 00:59:50,079 --> 00:59:52,159 building for humans and solving the most 1751 00:59:52,159 --> 00:59:54,559 important problems that everyone is is 1752 00:59:54,559 --> 00:59:56,480 facing and just doing it in a a deeply 1753 00:59:56,480 --> 00:59:59,440 human way. that's that's what we're here 1754 00:59:59,440 --> 01:00:02,000 for. Also, just going back to coding and 1755 01:00:02,000 --> 01:00:03,119 like you know these late nights, it's 1756 01:00:03,119 --> 01:00:04,960 like I think there's like it's also like 1757 01:00:04,960 --> 01:00:07,199 maybe like a glamorous version of it. 1758 01:00:07,199 --> 01:00:09,119 Just like I also had very a lot of late 1759 01:00:09,119 --> 01:00:10,880 nights where I was trying to refactor 1760 01:00:10,880 --> 01:00:13,599 something. Um and you know just like I 1761 01:00:13,599 --> 01:00:14,960 would be like three hours deep into the 1762 01:00:14,960 --> 01:00:17,280 refactor and then realize like actually 1763 01:00:17,280 --> 01:00:19,838 this is a dead end. Uh and I must 1764 01:00:19,838 --> 01:00:21,440 restart from scratch and it was like 1765 01:00:21,440 --> 01:00:23,760 very frustrating. Um, and so it's like 1766 01:00:23,760 --> 01:00:25,920 there was like there are like these very 1767 01:00:25,920 --> 01:00:27,519 very fun times, but there's also the 1768 01:00:27,519 --> 01:00:29,119 time where it's like it doesn't compile 1769 01:00:29,119 --> 01:00:31,760 and you're just like, why is this not 1770 01:00:31,760 --> 01:00:33,199 compiling yet? Like 1771 01:00:33,199 --> 01:00:35,440 >> I'm sure you had the time where you you 1772 01:00:35,440 --> 01:00:37,838 go you go later, it's now super late, 1773 01:00:37,838 --> 01:00:39,199 you need to go to bed cuz you need to 1774 01:00:39,199 --> 01:00:40,880 you need to get some sleep and then you 1775 01:00:40,880 --> 01:00:42,480 can't really sleep and you have this 1776 01:00:42,480 --> 01:00:44,798 thing where like you you have some some 1777 01:00:44,798 --> 01:00:48,079 task that is halfway and it upsets you 1778 01:00:48,079 --> 01:00:50,079 sometimes. I remember dreaming about the 1779 01:00:50,079 --> 01:00:52,318 code as well. And I guess one thing I 1780 01:00:52,318 --> 01:00:54,318 don't really have these days when I'm 1781 01:00:54,318 --> 01:00:57,039 working on my uh software for my 1782 01:00:57,039 --> 01:00:59,119 business is I don't really have 1783 01:00:59,119 --> 01:01:01,519 something that is halfway cuz I can just 1784 01:01:01,519 --> 01:01:03,119 tell it do this and then I can leave it 1785 01:01:03,119 --> 01:01:04,480 at a state where it's kind of like you 1786 01:01:04,480 --> 01:01:06,239 know done either finished it's either 1787 01:01:06,239 --> 01:01:08,159 working or it's I have proof that it's 1788 01:01:08,159 --> 01:01:09,599 failed. But it's interesting because you 1789 01:01:09,599 --> 01:01:10,960 know everything's sped up right. 1790 01:01:10,960 --> 01:01:14,239 >> Yeah. I may maybe like I I do have like 1791 01:01:14,239 --> 01:01:15,760 what what a lot of people do and I do 1792 01:01:15,760 --> 01:01:17,119 myself is like you know I have like 1793 01:01:17,119 --> 01:01:18,719 sometimes like bigger questions that I'm 1794 01:01:18,719 --> 01:01:20,960 asking myself like and I you know from 1795 01:01:20,960 --> 01:01:22,880 conversations I've had during the day or 1796 01:01:22,880 --> 01:01:24,719 like I haven't yet you know just like 1797 01:01:24,719 --> 01:01:27,039 had the time to just look into it and so 1798 01:01:27,039 --> 01:01:29,599 I you know I will send off codecs to 1799 01:01:29,599 --> 01:01:31,440 just like look at it overnight and then 1800 01:01:31,440 --> 01:01:33,358 I'm very excited to then wake up and 1801 01:01:33,358 --> 01:01:35,679 look at the results and so you know it's 1802 01:01:35,679 --> 01:01:37,519 always like an exciting morning. Well, I 1803 01:01:37,519 --> 01:01:39,679 feel there's an art to doing longunning 1804 01:01:39,679 --> 01:01:41,280 tasks. And of course, you can use the 1805 01:01:41,280 --> 01:01:44,000 slashgoal which will go and and and run. 1806 01:01:44,000 --> 01:01:45,440 You know, that's also something that was 1807 01:01:45,440 --> 01:01:46,880 recently added like a few months ago, 1808 01:01:46,880 --> 01:01:48,798 right? The /go goal command to codeex. 1809 01:01:48,798 --> 01:01:50,798 >> Yeah. And back to, you know, maybe like 1810 01:01:50,798 --> 01:01:54,239 the harness is a crutch, right? Is uh uh 1811 01:01:54,239 --> 01:01:57,838 slash goal was like necessary to allow 1812 01:01:57,838 --> 01:02:00,318 like no to keep the the model like on 1813 01:02:00,318 --> 01:02:02,880 track on like a singular goal for a very 1814 01:02:02,880 --> 01:02:05,440 long uh period of time. And it's like it 1815 01:02:05,440 --> 01:02:07,119 allows the model to literally run for 1816 01:02:07,119 --> 01:02:09,039 days or or weeks if it's like a really 1817 01:02:09,039 --> 01:02:11,599 really hard problem. But with the new 1818 01:02:11,599 --> 01:02:13,119 generation of models like what we're 1819 01:02:13,119 --> 01:02:14,197 seeing is like you know you don't 1820 01:02:14,197 --> 01:02:15,440 [clears throat] need SL goal anymore. 1821 01:02:15,440 --> 01:02:16,880 You don't need a harness around it. You 1822 01:02:16,880 --> 01:02:18,079 can just tell the model like you know 1823 01:02:18,079 --> 01:02:20,159 hey go and work for a week and you know 1824 01:02:20,159 --> 01:02:21,440 it will actually do it. 1825 01:02:21,440 --> 01:02:23,679 >> Speaking of hard problems and the fact 1826 01:02:23,679 --> 01:02:25,760 that you're not out of them. One of the 1827 01:02:25,760 --> 01:02:27,599 interesting things that you shipped from 1828 01:02:27,599 --> 01:02:30,719 the outside it I would say it it was you 1829 01:02:30,719 --> 01:02:32,318 know as an engineer it was moderately 1830 01:02:32,318 --> 01:02:35,199 interesting uh is the what you call the 1831 01:02:35,199 --> 01:02:38,239 merge which is codeex appeared inside of 1832 01:02:38,239 --> 01:02:40,000 chat GPC and the reason I say that as 1833 01:02:40,000 --> 01:02:41,199 engineers it's kind of moderately 1834 01:02:41,199 --> 01:02:42,480 interesting because we've been using 1835 01:02:42,480 --> 01:02:44,480 codeex like yeah it's there you can now 1836 01:02:44,480 --> 01:02:46,798 open it in the chat GPT app great like I 1837 01:02:46,798 --> 01:02:48,079 just went there and I just immediately 1838 01:02:48,079 --> 01:02:49,920 went to Codex cuz I don't I don't really 1839 01:02:49,920 --> 01:02:53,039 use Chad GBT in the app per se but I 1840 01:02:53,039 --> 01:02:55,440 talk with uh folks at Open AAI and 1841 01:02:55,440 --> 01:02:56,960 people in your team and you know they 1842 01:02:56,960 --> 01:02:58,719 were telling me like there was a lot of 1843 01:02:58,719 --> 01:03:00,559 preparation going on a lot of 1844 01:03:00,559 --> 01:03:02,318 engineering challenges. Can you give a 1845 01:03:02,318 --> 01:03:05,039 sense of how big this project was, what 1846 01:03:05,039 --> 01:03:07,039 you needed to do and why was it 1847 01:03:07,039 --> 01:03:08,880 difficult to pull off and and how you 1848 01:03:08,880 --> 01:03:10,960 know how did codeex and other tools help 1849 01:03:10,960 --> 01:03:12,880 you get it done in in ways that would 1850 01:03:12,880 --> 01:03:15,760 have been hard before cuz since you've 1851 01:03:15,760 --> 01:03:18,159 launched the merge the the numbers that 1852 01:03:18,159 --> 01:03:20,000 you keep sharing of how many people use 1853 01:03:20,000 --> 01:03:22,239 codeex it's like it's going up way 1854 01:03:22,239 --> 01:03:23,920 faster than before. So I assume I assume 1855 01:03:23,920 --> 01:03:25,838 there's a big scale uh problem you've 1856 01:03:25,838 --> 01:03:26,798 solved here. 1857 01:03:26,798 --> 01:03:29,199 >> A lot of things were 1858 01:03:29,199 --> 01:03:31,440 the challenging with the merge is first 1859 01:03:31,440 --> 01:03:33,760 of all 1860 01:03:33,760 --> 01:03:36,079 completely different stacks. Chbt is 1861 01:03:36,079 --> 01:03:37,920 like fully 1862 01:03:37,920 --> 01:03:41,358 uh managed cloud-based like you know you 1863 01:03:41,358 --> 01:03:44,400 run everything on our uh on our systems. 1864 01:03:44,400 --> 01:03:47,280 We we store things like traditional 1865 01:03:47,280 --> 01:03:49,679 traditional way of like building things. 1866 01:03:49,679 --> 01:03:52,239 Um built for scale, built for for 1867 01:03:52,239 --> 01:03:55,920 efficiency. Codex fully local. And so 1868 01:03:55,920 --> 01:03:57,838 the merge is just really like how do you 1869 01:03:57,838 --> 01:04:01,119 get the same uh the same benefits and 1870 01:04:01,119 --> 01:04:03,199 the same capabilities from this local 1871 01:04:03,199 --> 01:04:05,358 coding agent and then build a product 1872 01:04:05,358 --> 01:04:07,280 around it and build it in a way where it 1873 01:04:07,280 --> 01:04:08,880 can benefit like a much much broader 1874 01:04:08,880 --> 01:04:12,960 pool uh of people which is which is also 1875 01:04:12,960 --> 01:04:15,599 why you know all of us joined OpenAI is 1876 01:04:15,599 --> 01:04:18,000 like to benefit like this very very 1877 01:04:18,000 --> 01:04:20,960 broad population across the world and so 1878 01:04:20,960 --> 01:04:22,559 it was like a very exciting journey of 1879 01:04:22,559 --> 01:04:25,440 like figuring out like how do we build a 1880 01:04:25,440 --> 01:04:28,719 cloud version of this that in essence is 1881 01:04:28,719 --> 01:04:30,880 capable of like very very much the same 1882 01:04:30,880 --> 01:04:32,960 things um but is also built in a way 1883 01:04:32,960 --> 01:04:35,119 where you know we can serve it to like 1884 01:04:35,119 --> 01:04:37,199 tens and hundred millions of hundreds of 1885 01:04:37,199 --> 01:04:39,358 millions of users in a way that is still 1886 01:04:39,358 --> 01:04:41,039 like you know efficient so that we can 1887 01:04:41,039 --> 01:04:43,039 include it all the way into the plus 1888 01:04:43,039 --> 01:04:47,039 plan work is essentially like running uh 1889 01:04:47,039 --> 01:04:51,280 the full codeex harness in uh a cloud 1890 01:04:51,280 --> 01:04:52,960 like together with like a a cloud 1891 01:04:52,960 --> 01:04:54,719 computer it's It's a very powerful 1892 01:04:54,719 --> 01:04:56,960 machine actually like people have sort 1893 01:04:56,960 --> 01:04:58,480 of picked up on it and showed like you 1894 01:04:58,480 --> 01:05:00,400 know what you can do like you know if if 1895 01:05:00,400 --> 01:05:01,920 you are creative with the prompt is that 1896 01:05:01,920 --> 01:05:04,159 you know you can get um you can get 1897 01:05:04,159 --> 01:05:06,318 chatbt to like you know train another 1898 01:05:06,318 --> 01:05:08,159 model in there you know 1899 01:05:08,159 --> 01:05:09,599 >> wow [laughter] 1900 01:05:09,599 --> 01:05:11,679 >> there are some pretty wild things uh you 1901 01:05:11,679 --> 01:05:13,838 know you can get it to install blender 1902 01:05:13,838 --> 01:05:16,719 and you know do like 3D modeling it's 1903 01:05:16,719 --> 01:05:18,079 like very permissive it has like 1904 01:05:18,079 --> 01:05:20,159 internet access it's like a powerful 1905 01:05:20,159 --> 01:05:22,559 machine and then codex just works on it 1906 01:05:22,559 --> 01:05:24,400 and this is like what we ship through 1907 01:05:24,400 --> 01:05:27,440 cas work. a lot of system uh challenges. 1908 01:05:27,440 --> 01:05:29,440 We the team did it very quickly. 1909 01:05:29,440 --> 01:05:31,599 Obviously like Codex helped you know to 1910 01:05:31,599 --> 01:05:34,880 make it more efficient look at and and 1911 01:05:34,880 --> 01:05:37,280 build a lot of the infrastructure 1912 01:05:37,280 --> 01:05:39,760 and then you know help resolve a lot of 1913 01:05:39,760 --> 01:05:41,679 the little differences as well that you 1914 01:05:41,679 --> 01:05:43,838 know had been occurring between between 1915 01:05:43,838 --> 01:05:46,559 Codex and CHBT like merging plugins 1916 01:05:46,559 --> 01:05:49,119 architecture um you know merging library 1917 01:05:49,119 --> 01:05:50,719 and like so like really really working 1918 01:05:50,719 --> 01:05:52,559 towards like unified system which is 1919 01:05:52,559 --> 01:05:53,920 really the goal is like you shouldn't 1920 01:05:53,920 --> 01:05:55,119 feel like you know you can do something 1921 01:05:55,119 --> 01:05:57,760 in Codex that you can't do in TAB or 1922 01:05:57,760 --> 01:05:59,039 vice versa like what we're trying to 1923 01:05:59,039 --> 01:06:01,440 build is like one unified product that 1924 01:06:01,440 --> 01:06:03,039 gives you access uh to the same 1925 01:06:03,039 --> 01:06:04,400 intelligence but in the way that you 1926 01:06:04,400 --> 01:06:07,679 want to use it. And so it was very fun 1927 01:06:07,679 --> 01:06:09,440 as well because Codex throughout the 1928 01:06:09,440 --> 01:06:11,519 whole journey also acted as a journalist 1929 01:06:11,519 --> 01:06:13,519 to sort of like document all the steps 1930 01:06:13,519 --> 01:06:15,440 and the debates and the discussions that 1931 01:06:15,440 --> 01:06:17,358 the teams were having. And it was very 1932 01:06:17,358 --> 01:06:19,119 animated debate you know of how we 1933 01:06:19,119 --> 01:06:20,880 should do it and how we should name the 1934 01:06:20,880 --> 01:06:22,798 thing and you know when to introduce it 1935 01:06:22,798 --> 01:06:24,960 in what way and like what to merge into 1936 01:06:24,960 --> 01:06:26,559 what. there were like many different 1937 01:06:26,559 --> 01:06:28,639 permutations considered and so there's 1938 01:06:28,639 --> 01:06:31,039 like a very fun journalistic element to 1939 01:06:31,039 --> 01:06:32,559 it where we like we have a full 1940 01:06:32,559 --> 01:06:36,159 recounting that Codex did over time uh 1941 01:06:36,159 --> 01:06:38,798 and uh yeah it's just it's kind of 1942 01:06:38,798 --> 01:06:41,440 become known as well as like the toggle 1943 01:06:41,440 --> 01:06:43,920 arc of uh OpenAI where you know we 1944 01:06:43,920 --> 01:06:46,639 introduced like the work toggle um which 1945 01:06:46,639 --> 01:06:48,239 there was also a lot of debate around of 1946 01:06:48,239 --> 01:06:49,519 like you know whether this was like the 1947 01:06:49,519 --> 01:06:50,798 right thing and then you know just like 1948 01:06:50,798 --> 01:06:52,719 we kind of grew to to just really like 1949 01:06:52,719 --> 01:06:55,760 it um but over time we're going to merge 1950 01:06:55,760 --> 01:06:57,280 things further. So it's like we're 1951 01:06:57,280 --> 01:06:59,358 really headed into this direction of 1952 01:06:59,358 --> 01:07:01,358 like full unification and you know we 1953 01:07:01,358 --> 01:07:03,920 kind of view this as like um a temporary 1954 01:07:03,920 --> 01:07:06,159 state uh where you know you have like 1955 01:07:06,159 --> 01:07:09,039 you have better stronger capabilities 1956 01:07:09,039 --> 01:07:11,599 when you're in work mode but over time 1957 01:07:11,599 --> 01:07:14,159 we're bringing this you know all all the 1958 01:07:14,159 --> 01:07:15,679 way to like you know everyone that uses 1959 01:07:15,679 --> 01:07:18,559 CHP. And how do you personally use 1960 01:07:18,559 --> 01:07:21,119 codeex? Like what's your what's what's 1961 01:07:21,119 --> 01:07:22,639 your working setup in terms of agents, 1962 01:07:22,639 --> 01:07:24,480 in terms of task, in terms of what you 1963 01:07:24,480 --> 01:07:26,719 what what you manage uh with it. And 1964 01:07:26,719 --> 01:07:28,480 related to this, I asked Peter 1965 01:07:28,480 --> 01:07:30,798 Stainberger what I should ask about you 1966 01:07:30,798 --> 01:07:33,039 and he said like you I I you need to ask 1967 01:07:33,039 --> 01:07:36,400 him how do you deal with the fact that 1968 01:07:36,400 --> 01:07:38,079 you're involved in all these projects? 1969 01:07:38,079 --> 01:07:40,400 Uh your calendar is like Tetris, but 1970 01:07:40,400 --> 01:07:43,679 usually you show up pretty cheerful. 1971 01:07:43,679 --> 01:07:50,559 >> My calendar is fine. Um um and it's just 1972 01:07:50,559 --> 01:07:52,719 I am capable of doing so many more 1973 01:07:52,719 --> 01:07:55,679 things nowadays because I have the 1974 01:07:55,679 --> 01:07:58,559 technology like CEX and I actually 1975 01:07:58,559 --> 01:08:01,599 shifted a lot of like my my my work on 1976 01:08:01,599 --> 01:08:05,679 uh on mobile using tab work where 1977 01:08:05,679 --> 01:08:07,760 whenever I have something that I want to 1978 01:08:07,760 --> 01:08:09,920 take note of I just like fire that off. 1979 01:08:09,920 --> 01:08:12,559 I use dictation a lot. Um, whenever I 1980 01:08:12,559 --> 01:08:14,719 have a question, instead of like writing 1981 01:08:14,719 --> 01:08:17,039 it down to look into later or delegating 1982 01:08:17,039 --> 01:08:18,960 to someone, I just like fire it off in 1983 01:08:18,960 --> 01:08:20,560 charge of your work and I get like a 1984 01:08:20,560 --> 01:08:22,798 report. It has like a whole bunch of 1985 01:08:22,798 --> 01:08:26,319 like custom skills and um custom uh 1986 01:08:26,319 --> 01:08:28,158 instructions where it's now like very 1987 01:08:28,158 --> 01:08:29,679 very tailored to like you know produce 1988 01:08:29,679 --> 01:08:32,479 the kinds of reports and slide decks and 1989 01:08:32,479 --> 01:08:34,079 uh code explorations you know in the 1990 01:08:34,079 --> 01:08:36,798 style that I can consume effectively. 1991 01:08:36,798 --> 01:08:38,238 And so every time I'm like between 1992 01:08:38,238 --> 01:08:39,600 meetings or like you know you'll kind of 1993 01:08:39,600 --> 01:08:40,719 like see me like you know I was just 1994 01:08:40,719 --> 01:08:42,560 like dictating to my phone. As I said 1995 01:08:42,560 --> 01:08:45,279 before it's just like we do a lot of 1996 01:08:45,279 --> 01:08:47,679 work in public channels. We have like a 1997 01:08:47,679 --> 01:08:49,838 lot in Slack. We have a lot in in in 1998 01:08:49,838 --> 01:08:52,319 notion and Google Docs as well. And so 1999 01:08:52,319 --> 01:08:53,759 there's pretty much like there's no 2000 01:08:53,759 --> 01:08:55,759 question really that I feel I cannot ask 2001 01:08:55,759 --> 01:08:57,359 that you know Codex will be able to sort 2002 01:08:57,359 --> 01:08:58,960 of like do at least a first pass of 2003 01:08:58,960 --> 01:09:01,039 thinking through whether it is like 2004 01:09:01,039 --> 01:09:03,679 public sentiment on a feature um looking 2005 01:09:03,679 --> 01:09:06,719 at production logs for you know how much 2006 01:09:06,719 --> 01:09:08,640 usage we have on a certain thing making 2007 01:09:08,640 --> 01:09:09,759 a list of things that we should 2008 01:09:09,759 --> 01:09:11,039 deprecate because they're not getting 2009 01:09:11,039 --> 01:09:13,279 traction uh understanding what a certain 2010 01:09:13,279 --> 01:09:15,119 team is up to. It's like any question I 2011 01:09:15,119 --> 01:09:16,560 have I can get an answer to like you 2012 01:09:16,560 --> 01:09:18,399 know within 30 minutes. And so that's 2013 01:09:18,399 --> 01:09:20,158 how I use it. I use it for everything. 2014 01:09:20,158 --> 01:09:23,759 It's like my personal uh agent in in 2015 01:09:23,759 --> 01:09:26,640 like all the ways. And then oftentimes 2016 01:09:26,640 --> 01:09:28,880 on on weekends as well, I do some like 2017 01:09:28,880 --> 01:09:31,279 code explorations or like I build some 2018 01:09:31,279 --> 01:09:33,039 prototypes and I have fun like sort of 2019 01:09:33,039 --> 01:09:34,640 like imagining the future of the product 2020 01:09:34,640 --> 01:09:37,119 in some ways. And I do that with others 2021 01:09:37,119 --> 01:09:38,960 on on on on the teams. It's not always 2022 01:09:38,960 --> 01:09:42,000 the same team. And it's just like in one 2023 01:09:42,000 --> 01:09:44,960 day I can build things that I sort of 2024 01:09:44,960 --> 01:09:46,640 like I had it in my system, right? It's 2025 01:09:46,640 --> 01:09:48,479 like it's like I woke up one day I was 2026 01:09:48,479 --> 01:09:50,399 just like we should explore what it 2027 01:09:50,399 --> 01:09:52,479 means to build this and then I can just 2028 01:09:52,479 --> 01:09:54,560 sort of express all of that and get like 2029 01:09:54,560 --> 01:09:56,880 something in front of people in a day so 2030 01:09:56,880 --> 01:09:58,158 that they can think through it and 2031 01:09:58,158 --> 01:10:00,319 criticize it and hopefully get inspired 2032 01:10:00,319 --> 01:10:02,479 by it. It's like by no means you know we 2033 01:10:02,479 --> 01:10:04,399 need to ship it but it's more like okay 2034 01:10:04,399 --> 01:10:06,399 I flush it out of my system and then you 2035 01:10:06,399 --> 01:10:07,840 know I go on and like you know do other 2036 01:10:07,840 --> 01:10:10,880 things. So it's just like so I know it's 2037 01:10:10,880 --> 01:10:12,560 such a magical time and it's like so 2038 01:10:12,560 --> 01:10:14,800 empowering. And as closing, what would 2039 01:10:14,800 --> 01:10:17,119 your advice be for a software engineer/ 2040 01:10:17,119 --> 01:10:18,319 AAI engineer, someone who builds 2041 01:10:18,319 --> 01:10:22,158 software who would want to get the skill 2042 01:10:22,158 --> 01:10:24,560 set and the experience to have the 2043 01:10:24,560 --> 01:10:26,238 opportunity to work at a place like the 2044 01:10:26,238 --> 01:10:28,319 Codeex team, like OpenAI or like an AI 2045 01:10:28,319 --> 01:10:29,840 startup. So like you know just become 2046 01:10:29,840 --> 01:10:32,960 this really great builder with with 2047 01:10:32,960 --> 01:10:34,960 these tools because the question that 2048 01:10:34,960 --> 01:10:36,719 comes up is often like should I start 2049 01:10:36,719 --> 01:10:38,960 with the theory? How important are the 2050 01:10:38,960 --> 01:10:40,479 basics? Should I just get really good at 2051 01:10:40,479 --> 01:10:41,920 using the tools? 2052 01:10:41,920 --> 01:10:43,920 >> Yeah. I think there are two things that 2053 01:10:43,920 --> 01:10:46,880 are important is a deep deep curiosity 2054 01:10:46,880 --> 01:10:49,840 for how things work um and an ability to 2055 01:10:49,840 --> 01:10:51,198 like you know train yourself to 2056 01:10:51,198 --> 01:10:53,920 understand things very quickly and so 2057 01:10:53,920 --> 01:10:55,679 it's it's it is the case that things 2058 01:10:55,679 --> 01:10:57,359 will continue to change but people that 2059 01:10:57,359 --> 01:11:00,079 do extremely well at OpenAI are like you 2060 01:11:00,079 --> 01:11:01,279 know people that just sort of like are 2061 01:11:01,279 --> 01:11:05,119 able to like gro uh a system quickly and 2062 01:11:05,119 --> 01:11:07,359 like you know also dive into like a new 2063 01:11:07,359 --> 01:11:08,960 code base and sort of like know make 2064 01:11:08,960 --> 01:11:11,359 sense of it. Um but obviously like all 2065 01:11:11,359 --> 01:11:13,359 of that is helped with agents um 2066 01:11:13,359 --> 01:11:15,359 nowadays, right? So just like there's so 2067 01:11:15,359 --> 01:11:16,719 much information that you need to absorb 2068 01:11:16,719 --> 01:11:17,760 and like you know being able to 2069 01:11:17,760 --> 01:11:19,520 understand and reason through it and a 2070 01:11:19,520 --> 01:11:21,359 lot of that is asking good questions 2071 01:11:21,359 --> 01:11:23,039 really about you know how do things work 2072 01:11:23,039 --> 01:11:25,119 and just like going into like the five 2073 01:11:25,119 --> 01:11:26,640 W's which I think you know you can just 2074 01:11:26,640 --> 01:11:28,079 kind of keep digging and digging and 2075 01:11:28,079 --> 01:11:29,920 digging and you know you're learning 2076 01:11:29,920 --> 01:11:31,840 very very fast through that. The other 2077 01:11:31,840 --> 01:11:33,840 thing is being in tune with the 2078 01:11:33,840 --> 01:11:36,000 community or you know the people that 2079 01:11:36,000 --> 01:11:37,840 you're trying to solve a problem for. 2080 01:11:37,840 --> 01:11:39,279 It's like not everything is like solving 2081 01:11:39,279 --> 01:11:40,719 a direct problem. Sometimes you're 2082 01:11:40,719 --> 01:11:42,399 solving a problem that will be useful, 2083 01:11:42,399 --> 01:11:43,920 you know, to like another group of 2084 01:11:43,920 --> 01:11:45,760 people in the pursuit of like solving a 2085 01:11:45,760 --> 01:11:48,000 problem for humans. But just being crisp 2086 01:11:48,000 --> 01:11:50,479 about the taste or the needs or the 2087 01:11:50,479 --> 01:11:52,399 requirements uh and being able to think 2088 01:11:52,399 --> 01:11:53,920 clearly and like you know exercising 2089 01:11:53,920 --> 01:11:56,158 through this clarity of thought feels 2090 01:11:56,158 --> 01:11:57,439 really important to me. Like if you 2091 01:11:57,439 --> 01:11:58,640 can't explain what you're trying to 2092 01:11:58,640 --> 01:12:00,000 achieve, if you can't explain your 2093 01:12:00,000 --> 01:12:02,079 intent, if you don't have a tie to a 2094 01:12:02,079 --> 01:12:04,079 community, if you don't have the taste, 2095 01:12:04,079 --> 01:12:05,920 it's it's um it's going to be much 2096 01:12:05,920 --> 01:12:07,679 harder to do great work. 2097 01:12:07,679 --> 01:12:09,279 >> Awesome, TB. Well, thanks a bunch for 2098 01:12:09,279 --> 01:12:10,800 this conversation. This was awesome. 2099 01:12:10,800 --> 01:12:12,238 >> Thanks for having me. 2100 01:12:12,238 --> 01:12:13,679 >> I've always wanted to get together with 2101 01:12:13,679 --> 01:12:15,439 Tibo, and I'm glad that we finally made 2102 01:12:15,439 --> 01:12:18,079 it happen. I appreciated how Tibo talked 2103 01:12:18,079 --> 01:12:19,600 about not just the upsides of open 2104 01:12:19,600 --> 01:12:21,760 source, but also the downsides. most 2105 01:12:21,760 --> 01:12:23,520 notably how competitors can copy 2106 01:12:23,520 --> 01:12:24,960 features you [music] are just working on 2107 01:12:24,960 --> 01:12:27,359 in the open right now and then ship it 2108 01:12:27,359 --> 01:12:29,250 right before release and just how 2109 01:12:29,250 --> 01:12:30,960 [music] much this stings. Plus, you get 2110 01:12:30,960 --> 01:12:32,479 a lot of low-quality contributions that 2111 01:12:32,479 --> 01:12:34,238 you still need to somehow deal with. 2112 01:12:34,238 --> 01:12:35,760 Another interesting one was Tibo was 2113 01:12:35,760 --> 01:12:37,359 saying how the harness [music] is always 2114 01:12:37,359 --> 01:12:39,279 a step ahead of the model. From the 2115 01:12:39,279 --> 01:12:41,279 inside, the Codex team see their job as 2116 01:12:41,279 --> 01:12:43,600 building clutches for the model with the 2117 01:12:43,600 --> 01:12:45,119 harness, [music] the tools, and the 2118 01:12:45,119 --> 01:12:46,880 setup instruction. And then the next 2119 01:12:46,880 --> 01:12:48,560 version of the model will be trained to 2120 01:12:48,560 --> 01:12:50,960 need fewer of these clutches. I'll be 2121 01:12:50,960 --> 01:12:52,719 honest, as a dev, this sounds a little 2122 01:12:52,719 --> 01:12:55,119 demotivating that the stuff I build in 2123 01:12:55,119 --> 01:12:56,760 the next version of the model, it'll 2124 01:12:56,760 --> 01:12:58,960 [music] just know and we can get rid of 2125 01:12:58,960 --> 01:13:01,760 it. Plus, I do suspect that it's not 2126 01:13:01,760 --> 01:13:04,000 just about building these clutches, but 2127 01:13:04,000 --> 01:13:06,079 also building tools that models will 2128 01:13:06,079 --> 01:13:07,679 use. And it's not like the next version 2129 01:13:07,679 --> 01:13:09,520 of the model will reinvent an MCB 2130 01:13:09,520 --> 01:13:11,920 protocol or scales or plugins. At least 2131 01:13:11,920 --> 01:13:14,479 I hope not. I also enjoyed hearing what 2132 01:13:14,479 --> 01:13:16,800 the merge merging chat GPC and codeex 2133 01:13:16,800 --> 01:13:18,719 look like from the inside. It was 2134 01:13:18,719 --> 01:13:20,800 merging a previously fully local coding 2135 01:13:20,800 --> 01:13:22,640 agent, Codeex, [music] into a managed 2136 01:13:22,640 --> 01:13:24,960 cloud-based stack and doing it efficient 2137 01:13:24,960 --> 01:13:26,479 enough so that it can be included in 2138 01:13:26,479 --> 01:13:29,840 OpenAX $20 per month plan when $20 is 2139 01:13:29,840 --> 01:13:31,760 not all that much in terms of compute 2140 01:13:31,760 --> 01:13:34,000 purchase. It was pretty amusing to hear 2141 01:13:34,000 --> 01:13:36,238 how CODC itself acted as a journalist of 2142 01:13:36,238 --> 01:13:37,520 the whole project [music] as it was 2143 01:13:37,520 --> 01:13:39,198 present in all the Slack conversations 2144 01:13:39,198 --> 01:13:40,880 and all the documents and so it could 2145 01:13:40,880 --> 01:13:42,479 capture all the important debates and 2146 01:13:42,479 --> 01:13:44,399 decisions. I'm not going to lie, [music] 2147 01:13:44,399 --> 01:13:46,719 this part felt a little bit of a big 2148 01:13:46,719 --> 01:13:48,960 brother feel to it where the AI is 2149 01:13:48,960 --> 01:13:50,479 always watching, but it could well 2150 01:13:50,479 --> 01:13:52,319 become the new normal in startups in the 2151 01:13:52,319 --> 01:13:54,000 future. [music] I've not yet decided how 2152 01:13:54,000 --> 01:13:56,000 I feel about this. And finally, I 2153 01:13:56,000 --> 01:13:57,840 appreciated Tibo's advice for engineers 2154 01:13:57,840 --> 01:14:00,000 to succeed. Be curious, [music] 2155 01:14:00,000 --> 01:14:02,399 understand symptoms quickly, and be in 2156 01:14:02,399 --> 01:14:03,679 tune with the group you are building 2157 01:14:03,679 --> 01:14:05,920 for. It's reassuring to hear from Tibo 2158 01:14:05,920 --> 01:14:06,845 as well how much the fundamentals 2159 01:14:06,845 --> 01:14:07,439 [music] 2160 01:14:07,439 --> 01:14:09,039 still matter. Do check out the show 2161 01:14:09,039 --> 01:14:10,399 notes below for deep dives on how 2162 01:14:10,399 --> 01:14:12,158 codecs, clock code, and cursor were 2163 01:14:12,158 --> 01:14:14,560 built and other related topics. If you 2164 01:14:14,560 --> 01:14:16,238 like what you heard, please hit a rating 2165 01:14:16,238 --> 01:14:17,600 on a podcast player that [music] you're 2166 01:14:17,600 --> 01:14:19,600 using. It means a lot to me and to the 2167 01:14:19,600 --> 01:14:21,520 show. Thanks, and I'll see you in the 2168 01:14:21,520 --> 01:14:24,520 next