OpenAI’s Codex Workflows for Knowledge Work

[00:00:00]I think this combination chatgbt codeex and gbg 5.6 is the gold standard to me for knowledge work. It is the place where I spend all my time except for the times when I use fable which we'll get to. I use it for coding. I use it for writing. I use it for a lot of different things. And I think it's uh it has this combination of power, performance, and usability and speed that is just like it just makes it great for collaborative work. And uh we've got two very special people uh joining us from OpenAI. Dom and Roman. Welcome. Please introduce yourselves and tell us what you do at OpenAI.

[00:00:35]>> Yeah. Hi, I'm Dom. I work on developer experience for Codeex and now I guess chat.

[00:00:40]>> Sweet.

[00:00:41]>> Hey everyone, I'm Roman. I lead developer experience working with Dom.

[00:00:43]Uh but yeah, exciting day.

[00:00:45]>> Tell us how things are going. So this has been a bit of a different model launch because it was it's the first model that I remember where uh we knew it was coming before it came out.

[00:00:53]>> Yeah. No, it's a super exciting day. So now GPD 5.6 soul which is our new frontier model is available to to everyone. Uh you know we worked hard to make it uh available to everyone. So it's exciting day for people to to now use it. And yes, we also um noticed with the rise of codeex over the past few weeks like how many like uh people were using uh codeex as an app not just for like software development but also for anything around the code right and and that has like u been true uh at openai for many weeks now. We saw every single team at the company actually leveraging codeex every single day from finance to recruiting to sales. Everyone was like actually trying to use this product to make amazing things with it. And so that's why on one hand we have this model that's like really frontier intelligence for coding. We can talk more about this but also for any kind of knowledge work. And at the same time, we wanted to make sure the surface uh that people use to to access these capabilities was simpler, right? Like if you're working in sales or in finance, you don't really want to see code even if there is code behind the scenes. And so that's why we now have chatg work with this new work agent. Uh that gives so much more power. You can connect to all of your tools, the services that you use every day. and and now you have like as you saw in the live stream kind of like the ability to do any kind of work on your computer uh from Chaz.

[00:02:15]>> Amazing. And and if if I'm a codeex user and I'm trying to think about okay, how do I now use this this one app? How do I think about it and what are the differences? When do I go into the work tab and when do I go into the codeex tab? Yeah.

[00:02:28]>> Yeah. It's a great question. Honestly, for Codex users and developers, it's the same codec that you know and love, right? It's just m we're just making it better. It has its own dedicated space within the chat GPT app now. But if codec works for you, you can just like stay there for all of your work. And in fact, we as engineers, we have many tasks to do every single day that are not coding tasks, right? Like sometimes we write documents, we write documentation or sometimes we have to explain our work, we have to keep up with projects from Slack and different sources. So all of that already works well in codeex. I think what's quite powerful too now with chat GPT and codeex coming together is that we had like feature requests from builders telling us for instance hey I'm ideulating with chat GPT I'm running a deep research for instance about like some ideas for features I should build but then I kind of have to copy and paste that over to codeex which is strange right and now what's nice is with this one app one surface I can start brainstorming with chat GPT even with a deep research and when I'm ready to build I can at@mention that thread and get conversation get going right away. So this interplay is like very interesting because now it's like one surface and all of these tasks and threads they connect to one another. I think also go for >> don't worry too much about the that like you know quoteunquote toggle in the top top corner like you can literally just like stay in codecs or if you're curious like while a task is running just quickly switch between the two and you can see all the sort of like tiny changes that are happening um in the UI if you're switching between the two but you're not losing out if you're like doing knowledge work in codecs for example like quite the opposite like I've been using been doing plenty of knowledge work this week on on codecs.

[00:04:12]>> And by the way, like we've been shipping every week now on Thursday for the past 10 weeks or so. And so today is no exception even for developers using codecs. We have a ton of new features today of course with like soul ultra um to to get like max reasoning budget for your most ambitious tasks. But we also have new things like inline code editing, reviewing PRs directly within the codeex um you know space in the Chad GPT app and and many more things. So, in case you were wondering, we're just getting started for developers as well.

[00:04:44]>> I love that. One of the things that we've been talking about is there's a very hot thing right now called like about loops where if you're a developer, instead of doing the work yourself, you're building the system that does the work and and that's that's the loop the agent is running in more or less. And what one of the things I feel in using this model over the last month or so is that it's the first time where a loop workflow is available for non-developers where you can actually go and delegate tasks and actually like h have it do a lot of your work for you while you're like tending above it. Um it it sounds like there's no work to do. Actually there's there's a lot of work to do.

[00:05:21]Like making the loop work is is is a lot of work, but you have a lot more leverage. So, for example, I have not and I think Raman as we've been emailing you may have seen some of this like I have not done any of my email in the last like two months because like it's just basically 56 in codeex that knows all my preferences and knows how I would respond to stuff and I'm like giving it little tweaks here and there, but mostly it's like doing my email for me. Um, but I'm sort of curious, is that something that you guys have been seeing internally? Is that do you have like examples of of how it has changed how you've worked? Because I really do think that this is uh one of the things we've been saying is and and I'm sorry to say this, but I think Fable is a better overall programmer, but it's less usable. It's it's a less usable model.

[00:06:02]And I think 56 is still very powerful, but it's like way more usable so that someone who's non-technical can use it for delegated delegated work in a way that is uh very new. So I'm curious what your experiences.

[00:06:15]>> Yeah, I think the the interesting thing with 56 is sort of it naturally is more I would almost call like independent. Um especially if you have like computer use and a Chrome extension like it just sort of like gets the job done and it will do what it has to be done, right? So like if you're using like computer use chrome extension all of those aspects even for knowledge work like there's a lot of things where it can just like get it done end to end and verify its own work and like you might feel like it it might take a bit longer on a task but that's because like by the time that you come back it's actually going to be done. Um, like I've noticed this myself where like yesterday I was running five or like six tasks in parallel because you know it was just getting the job done at a level where I'm like I don't have to tend this at all and you don't even necessarily have to like think about it especially as like someone who hasn't been like you know AI pill like we all have like you don't have to think about it as a loop right it's just sort of like the model naturally cross-checks its work and make sure that things are done and it does it in like incredible ways like I'm going to show you a demo >> please.

[00:07:22]>> So as part of this whole merge uh situation, we also revamped the docs. We moved sort of the the codeex docs out of developers.ai.com um into its new learn.gpbc.com and we wanted to have like a lot of like delightful moments. Um and so uh one of the things that we ended up doing here is for example like all of the like a lot of these components are no longer screenshots. Um this is actually like a uh we gave uh codeex a screenshot and it like 56 soul I actually just went ahead cross reference it with a codebase um and rebuilt the whole thing in like actual HTML.

[00:08:04]>> Wow. Um and where this became absolutely wild for me was last night um I added we added a new feature called visualizations in um the app which works in regards of whether you're in chat work or codeex or even on the web and you can ask it to visualize concepts for you and we had let me show it in the blog post actually we have in the blog post here a couple of examples um of what this looks like it's really cool it gives you these sort of like interactive demos um where you can play around with things um to visualize ideas. I gave soul this blog post draft um and I'll show you the task in a second um but it actually was able to like build me the full interactive things from the video. Um, so like it actually inspected the video using the Chrome extension, >> figured out all of the necessary parts, took uh screen caps of like different frames, and then figured out how does like the interactions actually work, what are the things we're trying to show here. And so it was able to completely rebuild this by inspecting like the frames of the video.

[00:09:15]>> Absolutely blew my mind.

[00:09:16]>> And obviously like I had not even seen this. This is awesome. But like we would not take the time to build this if we did not have a model like so at our disposal, right? Because this would be like too much effort or too many turns.

[00:09:28]But the model being so autonomous, this is so cool. If you're looking at this and you're like, I don't know when I would do use a visualization. I think one thing to be really clear on is as the models get more powerful and they're able to do more autonomously, the bottleneck becomes can I understand what it just did? um like do I know like really do I have a real mental model of what's going on? Um and so it's ability to tell stories like that in a visual way that kind of like clicks I think is actually if you're thinking about what is the next frontier I think that kind of thing is a is a huge huge unlock.

[00:10:02]>> Totally. And that's why I think ultimately it comes back to the DNA of Open AAI, right? Like we're a we're a research and deployment company and the two sides are very critical. You want to make the very best frontier intelligence with research, but you also want to make it usable for people. That's why we spend so much time like on the harness uh which is open source, the Codex harness like leveraging all of this um and able to kind of work for a long time very reliably check its own work but also really be delightful in the way it explains like what it did. So I think that's that's all important. And by the way, like one thing that's like magical in GPT 5.6 six if you've not tried it yet is really computer use like computer use is basically solved at this point it's so much faster and the fact that you can like spin up agents to delegate tasks that can be quite like nebulous not even quite precise but the model understands the intent starts navigating your own apps or your Chrome tabs with its own cursors you can continue doing the work and check back when it's done it's really like delightful to see >> I love it I use it all the time I am often in a meeting and my computer like does something on its own and I'm like, "Oh, shoot." Like, Codeex is Codex is doing work. [laughter] Um, so guys, I know that uh I know that we're we're out of time and and that you have a very busy schedule. Thank you so much for coming on. Any final things you want to leave us with before you head off?

[00:11:23]>> I think there's one small feature that I would highly recommend checking out, which is um if you have the Chrome extension installed in the latest version, you actually have a site chat now. Um, so you can actually open the chat inside Chrome directly. And this is incredible because it actually connects to your chat GPT app, which means that like you have local file system access >> and it allows you to do things like looking at like a website or Google doc and it can directly reference files that are on your file system and interact with it too. Um, and like that's just incredible for for things like um, knowledge work.

[00:11:57]>> Totally. And my parting word would be um you know these models are have now reached uh such a level of capability that no idea is impossible. Just like give it your most ambitious projects all of the ideas you've been putting off you've not tried before the hardest bugs and uh we can't wait to see what happens uh because it's a really great model.

[00:12:18]>> I love it. Thanks for joining guys.

[00:12:20]Thanks for the work you're doing. We actually have we have a we have a special guest uh Kyle Cobber from OpenAI. Kyle, welcome. How you doing?

[00:12:29]>> Good. Uh, nice to meet you guys.

[00:12:31]>> Tell us about what you do on on OpenAI and and and uh and your thoughts on the model.

[00:12:36]>> Yeah. So, I would say um very well like very well aware of your thesis where like Codex is becoming an operating system for knowledge work and that's 100% true. I am not a software engineer yet. Now since I'd say February, I've been doing software engineer type work just integrating this into my natural workflows and um you know it's a very exciting day with soul and the merge of codecs and checks. I feel like now everyone is kind of going to get a taste of that codec style workflow where if you've just been using chat and not codeex as the knowledge worker and I jokingly will say don't bots the knowledge worker in it only takes a few months of uh doing it to to where you're kind of able to do almost anything but it's uh it's really exciting that everyone's kind of get a taste of this and it's something that we've been doing internally now for months. Codex is 100% my operating system uh runs basically everything for me today. Can you tell us more about some of those things that it's doing for you that people uh may not be may not think immediately, oh my god, yeah, you can use it for that?

[00:13:35]>> Yeah, it's just way more uh proactive where it's integrated into all your systems where if it's connected to Slack, you know, Gmail, um Outlook, if you're on a, you know, on a Windows computer or basically any of the Microsoft um you know, suites for uh Excel, PowerPoint, Google Slides, it's just unbelievable where it can gather context. So like every morning for me, my chief of staff is basically checking Slack in core channels that I'm very involved in, seeing where I'm potentially behind, giving me context, links from codeex directly to where I need to go, pre-drafting stuff, um, which is extremely powerful. It just allows you to be kind of always connected what's going on. And for me too, I would say there's so much stuff going on in forecast files, closed files, in these kind of G Suite artifacts that we use. and being able to have it kind of run through comments, you know, draft stuff, kind of get my take and be like truly the operating system where you kind of start with codecs and it enables you to get to that like 70 80% and then after a few weeks or months of using it gets you to like 90% because it learns your style. Um, you know, it just kind of is a huge time saver and productivity boost. So, >> take me through that like give me a specific like you're talking about close files and forecasting. like I want the like kind of crazy finance stuff that I might not know about uh that you use it for.

[00:14:54]>> Can't share anything live. So this is kind of like a a Google slide that has a GIF which was kind of neat where Codex made this slide for me from looking at my hosted app or site that I created. So basically like everything Codex is taking a first pass ad even kind of creating this like live demo gif thing, right? So but the real world like messiness I would say is each month compute is extremely complex. um you know we use it for externally for folks like yourself where people are paying users both for consumer or enterprise.

[00:15:24]We also use it internally obviously for research. So there's a very like hierarchy view to it which didn't really exist in any of the existing software applications like a Tableau or other visual things. So we basically went ahead and built our own and what's really neat is this is something now that is a part of our workflow every month where basically we close the books with our computer accounting team. Um, and also what's really neat is this actually does like a whole allocation on the back end where Codex can kind of meet your team where you're at. So let's say you have a very strong data science or data engineering team where everything is in the data lakeink.

[00:15:58]Amazing because Codex can interact with that fast. If not everything's in there and some of it's in a G sheet or an Excel, great. Doesn't matter. Codeex kind of do the work of pairing data across these different systems for you.

[00:16:09]So in this case, this like months ago basically started out where it was basically automating an allocation that maybe took five like two days of accounting's time and then my team we basically spend a few days like auditing all the detail going way deeper into the product usage since we care about like you know what a prouser is doing with codecs or image genen for the month to understand how the business is behaving from the strategic finance lens right.

[00:16:32]So we would basically uh spend a day or two there and then basically do the data lake uh like or Excel to a G sheet outputs into G sheet link those to slides and that was basically the old way of automating it now um and we can still use G sheet and and and slides to you know gut check or check artifacts they almost become extracts now for us but right now we basically go from data lake straight to a custom hosted app which is built from the sites product that we have so um it's basically a way that we compress almost five days of stuff into five hours. And it's not perfect. There's still and it goes a lot of work went into building this. You can't just oneshot uh it doing an entire workflow for you, right? But you can um teach it over the course of, you know, a week or two weeks and then as it runs this monthly process for you, it just gets better and better. You codify the learnings with it. It learns the skills.

[00:17:24]You improve that. And then basically now where we're at is we're like taking what kind of what we did for compute doing it for others areas of the business where people are now able to in Strat Finn use all the skills that went into building this use uh we're kind of creating our own internal plugins and that's something that too will make its way to the enterprise. So it's kind of exciting as a a finance person getting to do software engineering work um and getting to actually improve the product and improve the experience not only for our strap team but eventually too for the you know people like yourself using it.

[00:17:53]Um, so yeah, don't box the knowledge worker in is is is my my joke, but uh yeah, here's a perfect example of what kind of what you're talking about.

[00:18:01]>> That's really interesting. So, help me understand who built this? Like are you did you build it? Did someone in your team build it? Do you have like are you pairing?

[00:18:09]>> I built this with Codex uh basically just working uh outside of my normal work hours probably uh into the evening all of March like for like a month. And so basically the first opportunity we had for this to like you know dual track this with our normal work was in March and now we've had a few months of it. Um and it basically now is like humming where out of the box we're like 95 to 98% of the way there and it's really coming in and just finalizing some of the qualitative side of your monthly close. And I would say also what's kind of neat is whatever like you can basically like imagine like whatever you can imagine is what you can build. So in this demo you can kind of see it's and it's going through you kind of have like an overview tab again we have like the hierarchy view of how we think of compute. You can then ask it or see hey what how does this hit the P&L um and then kind of goes to more of the compute GPU specifics like what were the GPUs that drove this cost or you know what does it take for um you know image gen to run on a plus account for example.

[00:19:06]Um, and then because it has all this information and built the back end of this data set, it can actually take a great first pass at the qualitative side of your closed slides or whatever um, you know, slides you're doing. Like these are virtual slides built into this dashboard. We're not doing we're not using like a like the traditional slides anymore, right? Um, and then what's really neat, you'll see this little mascot bottom right. This is an idea I had kind of inspired by Codex Pets if you guys like your Codex Pets. for people that are maybe on your team that are less familiar with this or like how to navigate a new uh you know piece of software you built. Basically, there's a mascot you can just talk to and it basically like will take you throughout the the dashboard and answer questions.

[00:19:44]And this is all based on like a skill where it basically like runs a Q&A factory and basically gives this like mascot like basically like a way to answer and direct you across the dashboard which is really neat which is something that I mean you couldn't even think of building any of this and now you're just having fun building software that helps your team like understand what's going on with your business.

[00:20:06]>> I love it. I'm going to give people on the team a chance to ask questions. But uh before I before I flip it over, one final question I have for you is um this is a very complicated piece of software and it sounds like you built this over the course of a month or two.

[00:20:22]Um, what did you learn about how to make something like this that if you were going to start over and you could tell yourself a couple things uh from a couple months ago would have helped you avoid some of the pitfalls because I know it's like I know you can make this without coding, but there's a there's a lot to do to like actually make it good.

[00:20:41]>> Yeah, that is a very good question and I would say um it's it's a perpetual for us, right? Like this is not done every month. We're trying to make this better.

[00:20:50]I would say the the big learning is and what some people will try to do is when they use codeex for the first time or you know chat for work they'll basically try to oneshot stuff where it's like I would say take this is like one of the hardest processes we have each month which is why we thought to even try this I would say take it to your hardest like process have it like like do it the old school way but have it gather all the context on all the files the process the like Slack channels and then as you go along the way you basically say like okay this is like a recurring way that we're doing this like let's codify it with a skill. So you're almost like reverse engineering the process over time and then as it kind of learns it you can either build like a larger skill or kind of create like a plug-in for your team or yourself um or or basically enhance your agent MD files along the way. I think one learning too is for the knowledge worker is to don't be scared to try to figure out how all this stuff works where people will say like oh my agents are doing this like the way we think about a or some people think about agents here myself it's like basically your codeex has an agent MD file and then within each of your projects which this compute metrics is kind of one of my projects has its own agent MD file so that kind of my like codeex is able to orchestrate across all these different projects so um stay organized on the back end try how to recursively improve your skills. You can create skills that improve your skills specific to certain projects or just overall like reflect and refine which is kind of a way we think about as a skill most people have or build themselves here which actually integrates really nice with Chronicle if you've tried or used that on the back end of Codeex.

[00:22:25]>> So becomes super powerful. But I would say like don't don't try to like take the time treat it as almost like an extension of yourself. You're like training a version of yourself to take this away. And yeah, the investment is maybe it won't feel like a one shot and maybe it takes like a week or and it could take days. It just depends on how complex the thing is. This is extremely complex data from different resources and different teams working on this. So um if this takes you a month but the nice thing is I never have to make these slides again like each month all the data is magically as it's kind of put together every slide just updates. The drop down for the new month comes online and then we're basically at at that last uh last mile just trying to go through and edit commentary. And now it is collaborative. So it's like once you kind of build this and push it, my our whole Statin team is coming in and editing commentary. So it actually becomes a control and a point where we're have releasing the final stuff before we then kind of have Codex actually export all this to Google Slides since um there's some like you know sharability with like we can't have our board accessing like internal sites.

[00:23:26]But uh but anyway, that's kind of I would say some of the advice and learnings. But a lot of people are just intimidated and I would say just ask it and work with it and treat it as kind of this like an extension of you but also something that can like uplevel yourself and don't be scared just because it's has codeex in the name or you know chat beauty for work. It's like it is limitless in terms of like what you can do.

[00:23:48]>> I love it. Uh so I would say like maybe a a summary is like don't just try to like oneshot this like gigantic system allin-one. Do it just in time. So, as you have something come up in your workday that is like part of a part of a big project, see if you can make it uh make a skill that helps you do a chunk of it and build that up over time. Um, and and sort of it sounds like you parallel track things. So, you were kind of like as you're going through the process this month or a couple months ago, you were like building a system that would help you for the next time, but not for necessarily this time.

[00:24:19]>> Um, so all those things are are awesome.

[00:24:22]I know we only got you for one minute left, so anyone else on the team have a question? Yeah, >> Kyle. Yeah, Kyle, I wanted to ask a question. I work a lot with uh finance teams in hedge funds and and other finan financial firms and uh the big friction point they always have is uh how do I trust it, right? Because they have all this data um and and they they notice two issues and this is with all AI models, but uh either one it hallucinates uh sometimes and that that problem has gone way down, but it still occurs sometimes. Um uh so it makes s uh facts that like weren't in the actual data. Uh and then the other is just like this is way more common but like it misses things. So uh you know when there's like a lot of data it might just like not have looked at that file or it didn't catch that email. Uh so how can you walk me through like how how do you build trust in this piece of software like as you're building it like what what are some steps you take to evaluate? Yeah, that's a great question and it's actually probably a recurring theme we see when I get to join some deployment calls with some of our enterprise customers. But um I would say it's uh it's a process, right? It's the same way if you hire an analyst and they're kind of coming up to speed. Um they're going to get stuff wrong. It won't be perfect. I will say 56 soul with ultra turned on is extremely it's is a big step up and the sub aent use is kind of built in which is just incredible to where it can kind of multitask and do different pieces of the process all at the same time. But um it all comes down to kind of using it periodically where let's say Mike did get something wrong for you. You got to teach it like basically not to do that ever again. And kind of it that's how that's how that process goes where it's like hey I didn't like the way you output this or design this. Maybe if you're doing more of like a building a build or if you're kind of auditing materials. For example, I have a kind of we call it kind of like a Google Drive audit skill which is pretty neat where um before we ship anything, we're able to basically audit a document and look for any inconsistencies between a metric that's maybe mentioned on a memo that's linked to a Google Drive like to a slide that's linked to an Excel or linked to a G sheet. you basically like trace the metric and then basically tell you like what is wrong. The only way it's able to do that though is because I like the first few times it's like hey this is how I think about tracing things. This is how I think about going through these files like watch me and now there's a kind of a watch and replay or replay and record feature now where you kind of use that where it's like watching how you think about doing stuff if it got something wrong so that it kind of gains the context updates your skill and then you that's how you gain the confidence that hey that won't happen again. So, um, again, it's kind of like this like oneshot mindset where you shouldn't expect it necessarily to be like perfect the first time, but you should know as you're bringing it along. I would say there's like a compounding curve where that first time maybe it's like 70% there. You're like catching multiple mistakes. The next time it's 85 and then basically like a month or two from now it's at like 90 like that's it's at 95.

[00:27:16]Like the payoff is eventually there.

[00:27:18]It's not less work initially because you need the time to build it and teach it.

[00:27:21]But the payoff does come and that's how you get more comfortable where like now for me I allowed to take first stabs at work where it is basically trying to incubate work from reading Slack and I'm not having it just fire off the end results but it's uh kind of getting there and it's like oh I get to review figure it out and kind of it's already working on stuff before I even get to read it which is kind of neat. So but yeah >> to get that comfortable you have to be using it for months. So I would say in the Kodak's journey it's something that's like fun and exciting to to get along. But that's the that's how I would answer that. And the surface area there, that's great by the way, but the surface area there is is it like one long running thread for that type of task that you keep putting more information in or is it more that you're like training this skills uh >> create a create a project folder for like thematic stuff that that that's where it's happening where like that project folder is getting better and better skills that can use across all projects or like more specific to that are being used. you can constantly update the like MD like the agents MD file for that project folder so that let's say an inbound comes from Slack your kind of automation of a skill for your chief of staff folder sees it it knows like oh this is for um you know a new piece of data anal like analysis for this project that Mike's team's working on basically like starts a thread in that project and then because there's a MD file and like skills for that project it kind starts a pass and you basically can govern like I never want you to send something back on Slack until I get to read it. Like it it will listen or like it's only as good as the skills you make it. So like how you invest the time to give it the instructions. It'll behave.

[00:29:00]So that's how I get like very confident where it's like it can just be running and I know it's not going to just ship something to our CFO channel. Then Sarah Frier is going to ping me like, "Hey Kyle, what is going on?" So [laughter] >> sweet. Thank Kyle, I know I know you're you're out of time. Thank you so much for joining and for sharing and would love to have you on again.

[00:29:18]>> Yes. Thanks. Love to see you guys.

[00:29:20]>> Thanks.