WEBVTT

1
00:00:02.000 --> 00:00:06.720
Hello everyone and

2
00:00:04.040 --> 00:00:08.960
welcome. Software engineering is

3
00:00:06.720 --> 00:00:11.440
changing and by the end of 2025 it's

4
00:00:08.960 --> 00:00:13.840
going to look fundamentally different.

5
00:00:11.440 --> 00:00:16.080
In 2021 we announced our first model

6
00:00:13.840 --> 00:00:16.880
that we called codeex. This was the

7
00:00:16.080 --> 00:00:18.640
first time that we've really

8
00:00:16.880 --> 00:00:20.960
demonstrated what we would now call Vibe

9
00:00:18.640 --> 00:00:23.119
coding. Just a couple weeks ago, we

10
00:00:20.960 --> 00:00:25.199
released Codex CLI, which is a local

11
00:00:23.119 --> 00:00:27.039
agent that runs on your laptop in your

12
00:00:25.199 --> 00:00:29.320
own terminal that you're able to pair

13
00:00:27.039 --> 00:00:32.480
with and interact with in a synchronous

14
00:00:29.320 --> 00:00:33.840
fashion. Today, we're going to take a

15
00:00:32.480 --> 00:00:35.280
step towards where we think software

16
00:00:33.840 --> 00:00:38.160
engineering is going. And we are

17
00:00:35.280 --> 00:00:40.559
releasing a new system, which is a

18
00:00:38.160 --> 00:00:43.840
remote software agent that can run many

19
00:00:40.559 --> 00:00:46.239
tasks in parallel. We call this system

20
00:00:43.840 --> 00:00:48.399
in the grand tradition of of OpenAI

21
00:00:46.239 --> 00:00:50.239
naming codecs.

22
00:00:48.399 --> 00:00:52.640
The thing that is exciting about Codeex

23
00:00:50.239 --> 00:00:55.440
is that it runs with your repository,

24
00:00:52.640 --> 00:00:56.960
your environment on OpenAI compute. So

25
00:00:55.440 --> 00:00:58.640
you're able to run many, many copies of

26
00:00:56.960 --> 00:01:00.320
it. So you're able to do many tasks in

27
00:00:58.640 --> 00:01:01.600
parallel, fire them off, come back to

28
00:01:00.320 --> 00:01:03.280
them later, which is something that

29
00:01:01.600 --> 00:01:07.600
we'll be showing you today. We're

30
00:01:03.280 --> 00:01:10.320
rolling this out to uh chat GBT for uh

31
00:01:07.600 --> 00:01:12.720
Pro, Enterprise, and Teams users

32
00:01:10.320 --> 00:01:16.960
starting today, and we will be following

33
00:01:12.720 --> 00:01:18.880
up with plus and edu in the future.

34
00:01:16.960 --> 00:01:21.360
Codeex is powered by a new model that we

35
00:01:18.880 --> 00:01:24.479
call Codex one. This is our best coding

36
00:01:21.360 --> 00:01:26.640
model to date. We've taken 03 and we've

37
00:01:24.479 --> 00:01:28.400
optimized it for not just the

38
00:01:26.640 --> 00:01:30.080
benchmarks, but really for the kind of

39
00:01:28.400 --> 00:01:31.439
code that people actually want to merge

40
00:01:30.080 --> 00:01:33.119
into their codebase. So thinking about

41
00:01:31.439 --> 00:01:35.119
comments, thinking about extraneous

42
00:01:33.119 --> 00:01:36.479
changes, thinking about the style so

43
00:01:35.119 --> 00:01:38.880
that it's something that that actually

44
00:01:36.479 --> 00:01:40.880
really accelerates people's work. And so

45
00:01:38.880 --> 00:01:42.799
to show you this system which we're so

46
00:01:40.880 --> 00:01:45.119
excited to to share with you, uh we have

47
00:01:42.799 --> 00:01:46.880
many members of the codeex team. Hey, um

48
00:01:45.119 --> 00:01:49.119
I'm Hansen. I'm one of the members of

49
00:01:46.880 --> 00:01:51.119
the Codex research team. I'm Josh and

50
00:01:49.119 --> 00:01:54.399
I'm an engineer on the Codex team. And

51
00:01:51.119 --> 00:01:56.799
I'm Tibo and I built code agents and

52
00:01:54.399 --> 00:01:58.799
infrastructure to power them. I'm going

53
00:01:56.799 --> 00:02:01.040
to be jumping right into the demo. So

54
00:01:58.799 --> 00:02:02.640
normally you would have to connect your

55
00:02:01.040 --> 00:02:04.640
GitHub account. I've already done that

56
00:02:02.640 --> 00:02:06.799
for myself. And here I'm going to be

57
00:02:04.640 --> 00:02:08.640
selecting my repo. I have selected an

58
00:02:06.799 --> 00:02:10.959
open source repo, the preparedness repo

59
00:02:08.640 --> 00:02:12.800
which contains some of our frontier uh

60
00:02:10.959 --> 00:02:14.640
evals made by the amazing uh

61
00:02:12.800 --> 00:02:16.319
preparedness team that we have here. And

62
00:02:14.640 --> 00:02:18.560
this is relevant like frontier valves

63
00:02:16.319 --> 00:02:20.200
are really about agents. So I'm just

64
00:02:18.560 --> 00:02:23.280
going to be using that

65
00:02:20.200 --> 00:02:25.599
environment. And then I'm greeted with

66
00:02:23.280 --> 00:02:28.000
three tasks that we believe are good to

67
00:02:25.599 --> 00:02:31.920
get started with on any repo. The first

68
00:02:28.000 --> 00:02:34.239
one is an ask task which just ask the

69
00:02:31.920 --> 00:02:36.080
the code agent codeex to explain the

70
00:02:34.239 --> 00:02:37.440
codebase to a newcomer explain the

71
00:02:36.080 --> 00:02:40.080
general structure. The second one is a

72
00:02:37.440 --> 00:02:42.440
code task which asks to find and fix a

73
00:02:40.080 --> 00:02:45.840
bug somewhere in the repo. Most of the

74
00:02:42.440 --> 00:02:48.400
repos as you all know like have bugs. So

75
00:02:45.840 --> 00:02:51.440
let's see let's see if the codeex agent

76
00:02:48.400 --> 00:02:54.160
can find one. Um I did not put any bugs

77
00:02:51.440 --> 00:02:56.000
in there. Uh this is code that we work

78
00:02:54.160 --> 00:02:57.840
with every day. And so we're going to be

79
00:02:56.000 --> 00:02:59.440
showing two repos today. One is the

80
00:02:57.840 --> 00:03:01.920
preparedness repo and the other one is

81
00:02:59.440 --> 00:03:03.519
the codeex cli one. And the final one is

82
00:03:01.920 --> 00:03:05.599
one that I'm really excited about where

83
00:03:03.519 --> 00:03:07.599
I ask it to go through the codebase and

84
00:03:05.599 --> 00:03:09.920
not just explore the codebase but also

85
00:03:07.599 --> 00:03:12.319
come up with proactive task suggestions

86
00:03:09.920 --> 00:03:14.319
that it could be doing on its own. So

87
00:03:12.319 --> 00:03:17.200
let's start those. Now we have three

88
00:03:14.319 --> 00:03:19.120
codeex agents working on these tasks uh

89
00:03:17.200 --> 00:03:21.760
concurrently just in the background. And

90
00:03:19.120 --> 00:03:24.440
what I'm also going to do is um one of

91
00:03:21.760 --> 00:03:26.959
my favorites is find as

92
00:03:24.440 --> 00:03:32.480
many typos

93
00:03:26.959 --> 00:03:33.560
um and grammar mistakes as you can and

94
00:03:32.480 --> 00:03:35.760
fix

95
00:03:33.560 --> 00:03:38.480
them. This I think this didn't matter.

96
00:03:35.760 --> 00:03:41.920
Let's see if it understands also that my

97
00:03:38.480 --> 00:03:44.239
instructions as I make typos. And I have

98
00:03:41.920 --> 00:03:46.000
scheduled a task just before this before

99
00:03:44.239 --> 00:03:48.080
we uh we were going to jump on this

100
00:03:46.000 --> 00:03:49.840
where I asked some something quite

101
00:03:48.080 --> 00:03:53.280
interesting to the codeex agent. I asked

102
00:03:49.840 --> 00:03:54.720
it about one of my goals. I say my

103
00:03:53.280 --> 00:03:56.879
codebase wants I want it to be

104
00:03:54.720 --> 00:03:58.640
maintainable and bug free. Uh read

105
00:03:56.879 --> 00:04:01.040
through the code propose some tasks that

106
00:03:58.640 --> 00:04:03.519
would help with this goal and don't

107
00:04:01.040 --> 00:04:06.159
overemphasize on to-dos. Like to-dos are

108
00:04:03.519 --> 00:04:08.319
everywhere in code bases. Um those I

109
00:04:06.159 --> 00:04:09.760
know about. I want novel things. And so

110
00:04:08.319 --> 00:04:12.720
the codeex agent went through the

111
00:04:09.760 --> 00:04:15.200
codebase um and in its little

112
00:04:12.720 --> 00:04:18.000
environment as we're going to show you

113
00:04:15.200 --> 00:04:20.400
it found out about multiple things.

114
00:04:18.000 --> 00:04:23.600
Let's have a look. One of them here

115
00:04:20.400 --> 00:04:26.400
looks like we are uh having mutable

116
00:04:23.600 --> 00:04:28.639
defaults as arguments. Let's have codeex

117
00:04:26.400 --> 00:04:31.600
fix that. Definitely fix that. So let's

118
00:04:28.639 --> 00:04:33.199
schedule that. Um correct variable

119
00:04:31.600 --> 00:04:34.560
spelling. I believe this will get fixed

120
00:04:33.199 --> 00:04:36.880
as part of the other task, but let's

121
00:04:34.560 --> 00:04:38.880
schedule that anyway. And then here

122
00:04:36.880 --> 00:04:40.680
there was like a little consistency

123
00:04:38.880 --> 00:04:43.759
about how we set

124
00:04:40.680 --> 00:04:47.040
timeouts. One time we set it to 120,

125
00:04:43.759 --> 00:04:49.680
another time we set it to 60. Um,

126
00:04:47.040 --> 00:04:51.680
codeex, this codeex agent here has

127
00:04:49.680 --> 00:04:54.240
proposed a task for itself. So we are

128
00:04:51.680 --> 00:04:57.600
delegating the delegation here which

129
00:04:54.240 --> 00:04:59.840
blows my mind every time. Uh, and I'm

130
00:04:57.600 --> 00:05:01.440
just going to say make it 120. I think

131
00:04:59.840 --> 00:05:04.880
this is like the right time out here.

132
00:05:01.440 --> 00:05:08.000
And let's have it code out for us. And

133
00:05:04.880 --> 00:05:11.840
then let's jump into maybe this task

134
00:05:08.000 --> 00:05:14.560
here and see the codeex agent at work.

135
00:05:11.840 --> 00:05:17.039
So this isn't running on my laptop here.

136
00:05:14.560 --> 00:05:18.800
Um what exactly is going on here and

137
00:05:17.039 --> 00:05:21.280
like how does this work? Yeah, that's a

138
00:05:18.800 --> 00:05:23.120
great question to so um as evidenced

139
00:05:21.280 --> 00:05:26.000
from just seeing all these tasks launch

140
00:05:23.120 --> 00:05:27.840
in parallel. Um we now need new agentic

141
00:05:26.000 --> 00:05:29.520
coding infrastructure for this world

142
00:05:27.840 --> 00:05:32.560
where agents need not only their own

143
00:05:29.520 --> 00:05:34.400
GPUs but also a couple of CPUs. So this

144
00:05:32.560 --> 00:05:35.759
runs on opens compute infrastructure.

145
00:05:34.400 --> 00:05:36.960
It's actually in fact the same

146
00:05:35.759 --> 00:05:38.800
infrastructure we use for our

147
00:05:36.960 --> 00:05:40.240
reinforcement learning and which means

148
00:05:38.800 --> 00:05:42.560
fortunately even ahead of this launch

149
00:05:40.240 --> 00:05:45.520
it's been battle tested on the large

150
00:05:42.560 --> 00:05:48.000
scale uh training runs that we do and uh

151
00:05:45.520 --> 00:05:50.000
crucially it's also consistent between

152
00:05:48.000 --> 00:05:52.160
what the agencies during training and

153
00:05:50.000 --> 00:05:54.080
what the agencies later in production.

154
00:05:52.160 --> 00:05:57.680
So each of these tasks runs it in its

155
00:05:54.080 --> 00:06:01.120
own microVM sandbox, file system, CPU,

156
00:05:57.680 --> 00:06:02.880
memory, network policy and uh it the

157
00:06:01.120 --> 00:06:05.120
agent has free reign within it, right?

158
00:06:02.880 --> 00:06:08.560
So uh the agent has learned how to use

159
00:06:05.120 --> 00:06:10.639
all your pix commands like GP set, knows

160
00:06:08.560 --> 00:06:12.160
how to run linting, formatting and

161
00:06:10.639 --> 00:06:14.720
obviously it likes to write and execute

162
00:06:12.160 --> 00:06:15.840
a whole bunch of code. Yeah, one one

163
00:06:14.720 --> 00:06:17.600
thing that I really love about this

164
00:06:15.840 --> 00:06:18.960
interface is just how lightweight it

165
00:06:17.600 --> 00:06:20.960
makes it to spin up tasks, right? It's

166
00:06:18.960 --> 00:06:22.240
just your task left and right and I

167
00:06:20.960 --> 00:06:24.080
think that really makes a big difference

168
00:06:22.240 --> 00:06:26.000
for how you use it. I think it's it's a

169
00:06:24.080 --> 00:06:27.840
lot of powerful infrastructure and like

170
00:06:26.000 --> 00:06:30.319
the web interface makes it nice to

171
00:06:27.840 --> 00:06:32.400
interface with it but I think it's also

172
00:06:30.319 --> 00:06:35.120
a lot more comfort uh configurable.

173
00:06:32.400 --> 00:06:37.919
Yeah. So each task runs what runs what

174
00:06:35.120 --> 00:06:39.600
we call environment essentially um more

175
00:06:37.919 --> 00:06:41.759
or less a repo but with environment

176
00:06:39.600 --> 00:06:43.360
variables secrets and setup scripts

177
00:06:41.759 --> 00:06:46.400
configured so that you can customize the

178
00:06:43.360 --> 00:06:48.319
runtime to more fully unlock the agent's

179
00:06:46.400 --> 00:06:50.560
capabilities. Um when we worked with

180
00:06:48.319 --> 00:06:52.720
early alpha testers, we found that you

181
00:06:50.560 --> 00:06:54.720
get really big early wins by installing

182
00:06:52.720 --> 00:06:56.639
something as simple as a llinter or a

183
00:06:54.720 --> 00:06:58.319
formatter. Um we actually now install a

184
00:06:56.639 --> 00:06:59.919
lot of those by default. And of course

185
00:06:58.319 --> 00:07:02.240
our power users have all their test

186
00:06:59.919 --> 00:07:04.720
dependencies set up for OpenIs internal

187
00:07:02.240 --> 00:07:07.039
environments, we have uh even pre-commit

188
00:07:04.720 --> 00:07:08.400
hooks set up. So as the agents coding,

189
00:07:07.039 --> 00:07:10.560
it's actually committing seeing what the

190
00:07:08.400 --> 00:07:13.039
commit hooks say. Um really similar to

191
00:07:10.560 --> 00:07:14.639
actually my own dev environment. That

192
00:07:13.039 --> 00:07:17.520
sounds really exciting. Maybe let's

193
00:07:14.639 --> 00:07:19.599
watch it work. Um, and we're going to

194
00:07:17.520 --> 00:07:22.080
jump to an environment that we have

195
00:07:19.599 --> 00:07:24.240
fully configured like the codeex CLI and

196
00:07:22.080 --> 00:07:26.400
let's demo something cool with it. Yeah.

197
00:07:24.240 --> 00:07:28.240
So, I think one of my favorite use cases

198
00:07:26.400 --> 00:07:30.639
for Codeex is its ability to find and

199
00:07:28.240 --> 00:07:32.240
resolve complex issues. So, even leading

200
00:07:30.639 --> 00:07:33.680
up to this launch, I think Codex

201
00:07:32.240 --> 00:07:36.240
actually found and fixed a couple of

202
00:07:33.680 --> 00:07:37.599
critical bugs uh in our code. Um, but

203
00:07:36.240 --> 00:07:39.759
here we're looking at the Codex CLI

204
00:07:37.599 --> 00:07:42.479
which um TBO and I have worked on. Um,

205
00:07:39.759 --> 00:07:44.639
so uh we've got a bug report from a user

206
00:07:42.479 --> 00:07:46.479
here. basically uh when they have file

207
00:07:44.639 --> 00:07:49.280
names with special characters, the the

208
00:07:46.479 --> 00:07:52.400
diff command in the CLI uh shows up with

209
00:07:49.280 --> 00:07:54.000
an error message. Um and so uh one of

210
00:07:52.400 --> 00:07:56.160
the things we've trained our agents to

211
00:07:54.000 --> 00:07:58.080
do is not just like re uh address these

212
00:07:56.160 --> 00:08:00.720
issues, but also um we've introduced

213
00:07:58.080 --> 00:08:02.639
this uh concept of an agent's MD file.

214
00:08:00.720 --> 00:08:04.960
So we know for developers it's extremely

215
00:08:02.639 --> 00:08:07.599
important to provide steerability and

216
00:08:04.960 --> 00:08:09.360
instruction to to the model. And so the

217
00:08:07.599 --> 00:08:11.120
first thing we'll see it do here, um,

218
00:08:09.360 --> 00:08:13.840
it'll actually take a look at the agents

219
00:08:11.120 --> 00:08:15.599
MD file in the repository. Um, and so

220
00:08:13.840 --> 00:08:17.080
here, uh, we've provided some

221
00:08:15.599 --> 00:08:19.840
instructions about the layout of the

222
00:08:17.080 --> 00:08:21.199
repository. Um, you can see one of the

223
00:08:19.840 --> 00:08:23.360
special instructions here. We've asked

224
00:08:21.199 --> 00:08:24.639
it to print, um, some ASKI art to the

225
00:08:23.360 --> 00:08:28.080
terminal. I think does that look like a

226
00:08:24.639 --> 00:08:30.960
cat to you guys? It is. It's a different

227
00:08:28.080 --> 00:08:32.959
cat every time. It passes my cat test.

228
00:08:30.960 --> 00:08:35.279
Pretty good. Um so and then uh I think

229
00:08:32.959 --> 00:08:37.519
we we also saw above that basically um

230
00:08:35.279 --> 00:08:39.680
we've also told it you know here's the

231
00:08:37.519 --> 00:08:42.000
TypeScript part of the repo in in in the

232
00:08:39.680 --> 00:08:43.599
codec cli uh and along with some

233
00:08:42.000 --> 00:08:44.880
instructions about how to run run tests

234
00:08:43.599 --> 00:08:47.200
here which is pretty important for the

235
00:08:44.880 --> 00:08:49.440
agent. Um so one of the cool things

236
00:08:47.200 --> 00:08:52.240
about the way we train these models with

237
00:08:49.440 --> 00:08:54.160
end toend reinforcement learning um they

238
00:08:52.240 --> 00:08:56.640
not just write code but they also know

239
00:08:54.160 --> 00:08:58.240
how to navigate the codebase uh and even

240
00:08:56.640 --> 00:09:00.399
reproduce the issue. So here it's

241
00:08:58.240 --> 00:09:02.240
written a little script for itself. So

242
00:09:00.399 --> 00:09:03.360
um you can see it it's similar to us. I

243
00:09:02.240 --> 00:09:07.200
don't know about you guys but I love to

244
00:09:03.360 --> 00:09:08.959
use print debugging. Um never that's

245
00:09:07.200 --> 00:09:11.040
something we see emergent kind of as

246
00:09:08.959 --> 00:09:12.880
part of the RL training. Um it actually

247
00:09:11.040 --> 00:09:15.040
like write wrote a little script here to

248
00:09:12.880 --> 00:09:17.200
reproduce the issue. It's like written a

249
00:09:15.040 --> 00:09:19.200
file actually like that matches the user

250
00:09:17.200 --> 00:09:23.040
description and it's actually able to

251
00:09:19.200 --> 00:09:25.839
like execute the code um to actually uh

252
00:09:23.040 --> 00:09:26.600
verify that the the issues uh able able

253
00:09:25.839 --> 00:09:29.680
to be

254
00:09:26.600 --> 00:09:32.160
reproduced. is displaying a lot of like

255
00:09:29.680 --> 00:09:35.920
complex behavior that is very similar to

256
00:09:32.160 --> 00:09:38.720
how we all work. Um how do we teach the

257
00:09:35.920 --> 00:09:40.720
agent to do that? Yeah, I think so we

258
00:09:38.720 --> 00:09:42.240
have a lot of these kinds of tasks in

259
00:09:40.720 --> 00:09:43.839
training. We use end to-end

260
00:09:42.240 --> 00:09:45.760
reinforcement learning to not to

261
00:09:43.839 --> 00:09:49.440
actually like verify that it completes

262
00:09:45.760 --> 00:09:51.600
the entire uh cycle from writing code to

263
00:09:49.440 --> 00:09:54.240
running tests and then we verify that

264
00:09:51.600 --> 00:09:56.399
the that that it actually is able to

265
00:09:54.240 --> 00:09:58.240
complete the tasks and satisfy for

266
00:09:56.399 --> 00:10:00.160
example both style checks and

267
00:09:58.240 --> 00:10:02.240
programmatic checks and many other

268
00:10:00.160 --> 00:10:04.480
things. Um and then we can see that you

269
00:10:02.240 --> 00:10:06.440
know on evaluations like Sweetbench uh

270
00:10:04.480 --> 00:10:09.600
this results in state-of-the-art

271
00:10:06.440 --> 00:10:11.760
performance. Um so yeah so it's actually

272
00:10:09.600 --> 00:10:12.959
a it we can see here it's it's actually

273
00:10:11.760 --> 00:10:15.000
found the issue and it's actually

274
00:10:12.959 --> 00:10:18.160
writing a test for itself now um to

275
00:10:15.000 --> 00:10:20.480
actually so we can see that it's it's

276
00:10:18.160 --> 00:10:22.240
verifying its work for us um and it's

277
00:10:20.480 --> 00:10:24.480
almost done here so it's actually you

278
00:10:22.240 --> 00:10:27.120
know now it's even running llinters to

279
00:10:24.480 --> 00:10:30.320
verify that the code is matching our our

280
00:10:27.120 --> 00:10:33.120
style expectations and I I saw it refer

281
00:10:30.320 --> 00:10:35.920
back to its agents.mmd here yeah exactly

282
00:10:33.120 --> 00:10:37.519
so uh as part of the agents MD you can

283
00:10:35.920 --> 00:10:39.279
provide detailed guidelines and for

284
00:10:37.519 --> 00:10:40.640
example like commit messages and PR

285
00:10:39.279 --> 00:10:42.720
messages and how exactly you want your

286
00:10:40.640 --> 00:10:44.800
code to be structured. It looks like

287
00:10:42.720 --> 00:10:46.040
it's preparing files for commit. Yeah, I

288
00:10:44.800 --> 00:10:48.880
think it's almost

289
00:10:46.040 --> 00:10:51.200
done. So just within you know like the

290
00:10:48.880 --> 00:10:52.880
time that we spent talking um this this

291
00:10:51.200 --> 00:10:55.279
change probably would have taken me you

292
00:10:52.880 --> 00:10:59.360
know like at least 30 minutes or even or

293
00:10:55.279 --> 00:11:01.040
even hours to to debug and um and just

294
00:10:59.360 --> 00:11:03.279
in a moment I think we'll see the final

295
00:11:01.040 --> 00:11:06.880
PR. And the other thing is you could do

296
00:11:03.279 --> 00:11:08.160
this from your phone. Exactly. Yeah, it

297
00:11:06.880 --> 00:11:10.079
looks like it's on the P. How does this

298
00:11:08.160 --> 00:11:13.120
look to do?

299
00:11:10.079 --> 00:11:16.320
Let's have a look. Um, it's kind of

300
00:11:13.120 --> 00:11:18.800
tricky to review code so uh so fast

301
00:11:16.320 --> 00:11:21.680
here. But one of the things here is that

302
00:11:18.800 --> 00:11:23.920
it it did add a test. Uh it's a repro

303
00:11:21.680 --> 00:11:26.800
test for uh the issue that we mentioned.

304
00:11:23.920 --> 00:11:29.279
So this seems to match here. And also

305
00:11:26.800 --> 00:11:31.279
what gives me confidence is that it

306
00:11:29.279 --> 00:11:34.000
actually ran this test. And there is

307
00:11:31.279 --> 00:11:36.720
like a little proof here uh which we'll

308
00:11:34.000 --> 00:11:38.640
talk about later um where I gain a lot

309
00:11:36.720 --> 00:11:42.160
of confidence here that this test is

310
00:11:38.640 --> 00:11:45.120
actually passing. So I would just go and

311
00:11:42.160 --> 00:11:47.440
like push and create a PR here. But now

312
00:11:45.120 --> 00:11:49.440
that we have a whole bunch more tasks

313
00:11:47.440 --> 00:11:52.880
that have completed, what do you think,

314
00:11:49.440 --> 00:11:54.320
Greg? I mean I think this is magic. I'm

315
00:11:52.880 --> 00:11:57.800
feel I'm definitely feeling feeling the

316
00:11:54.320 --> 00:12:00.000
AGI here. Thank you. Thank you all. Um,

317
00:11:57.800 --> 00:12:01.760
and you know, I think that what the

318
00:12:00.000 --> 00:12:04.480
system in my mind shows is that we're

319
00:12:01.760 --> 00:12:06.399
moving beyond thinking of our AI systems

320
00:12:04.480 --> 00:12:07.760
as just language models, right? That

321
00:12:06.399 --> 00:12:10.320
we're really building systems around

322
00:12:07.760 --> 00:12:12.639
them. So, it's not just about the core

323
00:12:10.320 --> 00:12:14.639
AI intelligence. It's really about what

324
00:12:12.639 --> 00:12:17.040
tools is access to, the environment that

325
00:12:14.639 --> 00:12:19.200
it is able to operate within, um, the

326
00:12:17.040 --> 00:12:21.200
kinds of of sort of real world uh, you

327
00:12:19.200 --> 00:12:22.639
know, sort of uh, conditions that it's

328
00:12:21.200 --> 00:12:24.079
been trained to be exposed to. It's

329
00:12:22.639 --> 00:12:25.440
starting to feel much more like the

330
00:12:24.079 --> 00:12:27.920
interface that we're going to see for a

331
00:12:25.440 --> 00:12:29.200
real AGI. Uh still still much more to

332
00:12:27.920 --> 00:12:31.200
build, but it's starting to feel like it

333
00:12:29.200 --> 00:12:34.160
has the right form factor. And so here

334
00:12:31.200 --> 00:12:37.920
to tell you more about the how we build

335
00:12:34.160 --> 00:12:40.800
the system are more members of the team.

336
00:12:37.920 --> 00:12:43.440
Uh hi everyone. I'm Andre. I work on uh

337
00:12:40.800 --> 00:12:44.959
research and uh systems for

338
00:12:43.440 --> 00:12:46.880
reinforcement learning and coding

339
00:12:44.959 --> 00:12:49.519
agents. I'm Katie. I'm on the research

340
00:12:46.880 --> 00:12:51.760
team here. I'm Jerry. I also work on on

341
00:12:49.519 --> 00:12:53.680
research here. If you if we look back at

342
00:12:51.760 --> 00:12:55.680
the history, we've been working on large

343
00:12:53.680 --> 00:12:57.920
language models. Code for many years

344
00:12:55.680 --> 00:13:00.480
already. The first codeex models we have

345
00:12:57.920 --> 00:13:02.399
released that powered GitHub copilot and

346
00:13:00.480 --> 00:13:04.160
was little more than a smart

347
00:13:02.399 --> 00:13:06.160
autocomplete and I think they have

348
00:13:04.160 --> 00:13:08.160
started a revolution of AI powered

349
00:13:06.160 --> 00:13:10.240
coding that we really see take off

350
00:13:08.160 --> 00:13:12.079
today. Since then, we've worked on a lot

351
00:13:10.240 --> 00:13:13.600
of things. We've improved every layer of

352
00:13:12.079 --> 00:13:15.680
our pre-training stack to make our

353
00:13:13.600 --> 00:13:18.399
models really, really great at all kinds

354
00:13:15.680 --> 00:13:20.079
of programming tasks. At the moment GPT4

355
00:13:18.399 --> 00:13:21.839
was released, it was by far the best

356
00:13:20.079 --> 00:13:24.240
programming model in the world. We've

357
00:13:21.839 --> 00:13:26.320
also constantly iterated on our on our

358
00:13:24.240 --> 00:13:28.480
post training recipe to make the moles

359
00:13:26.320 --> 00:13:31.200
in chat GPT really great at answering

360
00:13:28.480 --> 00:13:34.320
users programming related questions.

361
00:13:31.200 --> 00:13:36.959
Every day millions of users use CH GPT

362
00:13:34.320 --> 00:13:38.639
to accelerate their work. However,

363
00:13:36.959 --> 00:13:41.120
working with LLMs on a programming

364
00:13:38.639 --> 00:13:43.120
related projects still feels pretty

365
00:13:41.120 --> 00:13:45.920
clanky. It does require a lot of

366
00:13:43.120 --> 00:13:48.160
handholding and context switching. I

367
00:13:45.920 --> 00:13:50.480
believe the current paradigm of scaling

368
00:13:48.160 --> 00:13:52.959
up reinforcement learning finally can

369
00:13:50.480 --> 00:13:55.839
take us to the place where we can

370
00:13:52.959 --> 00:13:58.639
automate larger chunks of work where our

371
00:13:55.839 --> 00:14:01.279
models can work on where we can tell

372
00:13:58.639 --> 00:14:03.120
them what we want to do not necessarily

373
00:14:01.279 --> 00:14:05.760
how and they can work for extended

374
00:14:03.120 --> 00:14:08.959
period of time on that task for a user

375
00:14:05.760 --> 00:14:10.959
on a real production codebase. Even more

376
00:14:08.959 --> 00:14:12.959
importantly the infrastructure that we

377
00:14:10.959 --> 00:14:14.800
are opening up for it is perfectly

378
00:14:12.959 --> 00:14:16.800
scalable. It doesn't run on your laptop.

379
00:14:14.800 --> 00:14:19.040
It runs on the cloud. At a single push

380
00:14:16.800 --> 00:14:22.000
of a button, you can spin up one agent,

381
00:14:19.040 --> 00:14:25.120
10 agents, or 10,000 agents. It's an

382
00:14:22.000 --> 00:14:26.800
ondemand AI power force multiplier. And

383
00:14:25.120 --> 00:14:28.639
you are seeing really great results with

384
00:14:26.800 --> 00:14:30.240
it internally.

385
00:14:28.639 --> 00:14:32.959
Cool. So, I get to do the part of the

386
00:14:30.240 --> 00:14:34.560
demo um that is probably the most

387
00:14:32.959 --> 00:14:36.800
unlucky where I'm not going to kick off

388
00:14:34.560 --> 00:14:39.199
any tasks, but I'm going to take a look

389
00:14:36.800 --> 00:14:41.360
at the tasks that Tibo, Hansen, and Josh

390
00:14:39.199 --> 00:14:42.800
kicked off um and hope that they're

391
00:14:41.360 --> 00:14:46.000
good. And I think this is actually kind

392
00:14:42.800 --> 00:14:48.320
of reflection of where engineering work

393
00:14:46.000 --> 00:14:50.399
has moved over the past few years where

394
00:14:48.320 --> 00:14:52.959
a lot of my time now is spent reviewing

395
00:14:50.399 --> 00:14:54.800
code rather than writing it. And in that

396
00:14:52.959 --> 00:14:56.639
sense, it becomes even more important

397
00:14:54.800 --> 00:14:58.959
for our models to be very aligned with

398
00:14:56.639 --> 00:15:01.040
what we want. So I will talk you through

399
00:14:58.959 --> 00:15:02.880
how we train these models with alignment

400
00:15:01.040 --> 00:15:06.000
in mind um as we look through a couple

401
00:15:02.880 --> 00:15:07.839
of these tasks. So let me let's click

402
00:15:06.000 --> 00:15:09.680
back into the one that Tibo kicked off

403
00:15:07.839 --> 00:15:12.160
here. Um, so I think there's three key

404
00:15:09.680 --> 00:15:13.760
parts here. On the right, we have the

405
00:15:12.160 --> 00:15:16.160
actual code output produced by the

406
00:15:13.760 --> 00:15:18.399
model. When we look at code and when you

407
00:15:16.160 --> 00:15:20.160
review code, you might look for things

408
00:15:18.399 --> 00:15:22.639
like, you know, like having sensible

409
00:15:20.160 --> 00:15:24.560
changes, doing exactly what the PR

410
00:15:22.639 --> 00:15:27.040
description said, and not having extra

411
00:15:24.560 --> 00:15:28.639
changes. Um, you don't want extra

412
00:15:27.040 --> 00:15:30.160
comments littered throughout your code.

413
00:15:28.639 --> 00:15:32.240
This is feedback that we've heard about

414
00:15:30.160 --> 00:15:33.760
our models before. Um, and so something

415
00:15:32.240 --> 00:15:35.760
that we really focused on was like

416
00:15:33.760 --> 00:15:38.560
having good code quality and style so

417
00:15:35.760 --> 00:15:40.000
that code is easier to review. Another

418
00:15:38.560 --> 00:15:42.320
thing that we really focused on here was

419
00:15:40.000 --> 00:15:44.800
interpretability and verifiable u

420
00:15:42.320 --> 00:15:47.040
outputs. So on the left side here we

421
00:15:44.800 --> 00:15:49.600
have a model generated summary of

422
00:15:47.040 --> 00:15:52.399
exactly what it did. Right here it added

423
00:15:49.600 --> 00:15:54.240
an import for this function. It tells

424
00:15:52.399 --> 00:15:56.880
you what it's doing and like why it's

425
00:15:54.240 --> 00:15:58.959
doing it. And it shows you in this

426
00:15:56.880 --> 00:16:00.399
citation view exactly the code that it

427
00:15:58.959 --> 00:16:02.560
was referencing when it's talking about

428
00:16:00.399 --> 00:16:04.639
what it did. And then in the testing

429
00:16:02.560 --> 00:16:06.959
section here, the model actually ran the

430
00:16:04.639 --> 00:16:09.759
test um that it was told to run in the

431
00:16:06.959 --> 00:16:11.360
agents.mmd file. But beyond just

432
00:16:09.759 --> 00:16:12.800
actually running the test, the model

433
00:16:11.360 --> 00:16:15.040
actually reports whether the test passed

434
00:16:12.800 --> 00:16:17.199
or failed. Um and in this case, you can

435
00:16:15.040 --> 00:16:20.079
see exactly where in the work log the

436
00:16:17.199 --> 00:16:22.800
ter um the test was run and you can see

437
00:16:20.079 --> 00:16:27.160
and verify that it succeeded. Let's take

438
00:16:22.800 --> 00:16:27.160
a look at one more task.

439
00:16:27.600 --> 00:16:31.600
Okay, so here there's another task that

440
00:16:29.839 --> 00:16:34.240
was done in the preparedness repo where

441
00:16:31.600 --> 00:16:36.800
I can take a look over this model output

442
00:16:34.240 --> 00:16:39.279
and it might pass like the smell test to

443
00:16:36.800 --> 00:16:40.959
me initially, but I can see from the

444
00:16:39.279 --> 00:16:43.120
model's testing output here that it

445
00:16:40.959 --> 00:16:45.920
actually didn't manage to pass the test.

446
00:16:43.120 --> 00:16:48.320
And so I'm looking here and it looks

447
00:16:45.920 --> 00:16:50.560
like it might be something with like a

448
00:16:48.320 --> 00:16:52.880
missing dependency in the environment.

449
00:16:50.560 --> 00:16:54.399
Um maybe pedantic is not installed. And

450
00:16:52.880 --> 00:16:56.800
so this PR might be something that I

451
00:16:54.399 --> 00:16:59.920
check back out to my computer and rerun

452
00:16:56.800 --> 00:17:01.680
the tests locally and maybe if I see

453
00:16:59.920 --> 00:17:03.279
that this is valuable. This is something

454
00:17:01.680 --> 00:17:06.160
I can go back to my environment

455
00:17:03.279 --> 00:17:09.199
configuration and reinstall. And

456
00:17:06.160 --> 00:17:10.880
ultimately I think we find codecs to be

457
00:17:09.199 --> 00:17:13.360
like as trustworthy if not more

458
00:17:10.880 --> 00:17:15.520
trustworthy than our own co-workers. I

459
00:17:13.360 --> 00:17:17.679
don't have I don't have this kind of

460
00:17:15.520 --> 00:17:19.839
access to like what Andre did on any

461
00:17:17.679 --> 00:17:22.160
given day in terms of like the logs or

462
00:17:19.839 --> 00:17:23.919
the actual test outputs. And I think as

463
00:17:22.160 --> 00:17:26.000
we move towards this world where AI

464
00:17:23.919 --> 00:17:27.280
writes more and more code, this kind of

465
00:17:26.000 --> 00:17:30.200
verifiability is going to be really

466
00:17:27.280 --> 00:17:33.360
important.

467
00:17:30.200 --> 00:17:34.799
So, uh, I'm super excited to share this

468
00:17:33.360 --> 00:17:36.400
with all of you today. I've been working

469
00:17:34.799 --> 00:17:38.160
on this for more than two years.

470
00:17:36.400 --> 00:17:41.200
Actually, a lot of that time with Jerry.

471
00:17:38.160 --> 00:17:44.400
It's been a real blast. Um, and we're

472
00:17:41.200 --> 00:17:46.400
we've been using this internally and

473
00:17:44.400 --> 00:17:48.480
seeing a lot of really magical moments

474
00:17:46.400 --> 00:17:49.840
start to appear. Uh we're sharing this

475
00:17:48.480 --> 00:17:51.600
with you today because we think this

476
00:17:49.840 --> 00:17:54.799
does represent a glimpse into the future

477
00:17:51.600 --> 00:17:56.080
of how software engineering is done. Um

478
00:17:54.799 --> 00:17:58.160
I'm just going to tell you a little bit

479
00:17:56.080 --> 00:18:00.799
about how I use the product. So in the

480
00:17:58.160 --> 00:18:02.240
leadup to this launch, uh I've ended up

481
00:18:00.799 --> 00:18:04.320
doing a lot of coordination work. I

482
00:18:02.240 --> 00:18:06.480
haven't had as much time for coding as I

483
00:18:04.320 --> 00:18:09.600
maybe used to as much as I maybe ideally

484
00:18:06.480 --> 00:18:12.320
would like. Sorry. Maybe Greg Greg can

485
00:18:09.600 --> 00:18:14.000
empathize with that a little bit. Um,

486
00:18:12.320 --> 00:18:16.960
and so if I'm like reviewing someone

487
00:18:14.000 --> 00:18:18.480
else's code or uh reading through Slack

488
00:18:16.960 --> 00:18:20.880
or playing with the product just to make

489
00:18:18.480 --> 00:18:22.880
sure that uh it's developing well,

490
00:18:20.880 --> 00:18:24.320
sometimes I'll have an idea of like a

491
00:18:22.880 --> 00:18:26.160
code change I might like to make, a

492
00:18:24.320 --> 00:18:27.919
change in the product or some refactor

493
00:18:26.160 --> 00:18:29.600
and I'll just kick it off in codeex. It

494
00:18:27.919 --> 00:18:30.960
takes like 30 seconds. It's a very

495
00:18:29.600 --> 00:18:34.000
lightweight thing to do and then I'll go

496
00:18:30.960 --> 00:18:36.559
back to Slack or or wherever I wherever

497
00:18:34.000 --> 00:18:38.240
I am. Uh and then later I'll come back

498
00:18:36.559 --> 00:18:40.880
and the task is sitting there and it's

499
00:18:38.240 --> 00:18:42.240
done. And sometimes it's a small task.

500
00:18:40.880 --> 00:18:44.400
It's like a simple string change.

501
00:18:42.240 --> 00:18:46.400
Sometimes I was ambitious and I put in a

502
00:18:44.400 --> 00:18:48.720
request for like a bigger refactor or

503
00:18:46.400 --> 00:18:50.000
some sort of feature. And this by no

504
00:18:48.720 --> 00:18:51.840
means works all the time. This is a

505
00:18:50.000 --> 00:18:54.679
research preview. Uh we think it's still

506
00:18:51.840 --> 00:18:57.200
early days. Uh but

507
00:18:54.679 --> 00:18:58.559
sometimes it's like a multiund line diff

508
00:18:57.200 --> 00:19:00.320
and I open it and I start reading

509
00:18:58.559 --> 00:19:04.160
through it and I'm like, "Wow, this

510
00:19:00.320 --> 00:19:05.520
actually looks like it's correct." And I

511
00:19:04.160 --> 00:19:07.360
read through what the model did. Maybe

512
00:19:05.520 --> 00:19:09.360
the model ran some tests. Maybe those

513
00:19:07.360 --> 00:19:12.640
tests failed. and the model fixed the

514
00:19:09.360 --> 00:19:14.720
the test failure. I look on the left and

515
00:19:12.640 --> 00:19:16.400
all the little test results are green

516
00:19:14.720 --> 00:19:18.240
and I'm like, "This change actually

517
00:19:16.400 --> 00:19:19.799
looks really good. Let me open a PR.

518
00:19:18.240 --> 00:19:21.919
Maybe I'll send it to a colleague for

519
00:19:19.799 --> 00:19:23.360
review. Maybe they'll approve it and

520
00:19:21.919 --> 00:19:25.280
we'll merge it and it lands in the

521
00:19:23.360 --> 00:19:27.120
codebase." And then I take a step back

522
00:19:25.280 --> 00:19:29.200
and I realize I just landed like a

523
00:19:27.120 --> 00:19:31.280
non-trivial change, like a large change

524
00:19:29.200 --> 00:19:33.840
often in our codebase and that branch

525
00:19:31.280 --> 00:19:35.360
never even hit my laptop. uh it was just

526
00:19:33.840 --> 00:19:36.960
something that happened completely

527
00:19:35.360 --> 00:19:39.679
through this interaction with an async

528
00:19:36.960 --> 00:19:41.600
delegation to to one of these agents and

529
00:19:39.679 --> 00:19:44.320
when that works it's really a magical

530
00:19:41.600 --> 00:19:46.559
moment uh and it's really amazing and

531
00:19:44.320 --> 00:19:47.760
when I talk to other colleagues uh

532
00:19:46.559 --> 00:19:49.200
they're using the product too and

533
00:19:47.760 --> 00:19:50.480
they're using it in different ways and

534
00:19:49.200 --> 00:19:51.840
they're having magical moments and

535
00:19:50.480 --> 00:19:53.760
magical interactions with it of their

536
00:19:51.840 --> 00:19:56.000
own and you can see some of those uh

537
00:19:53.760 --> 00:19:57.559
stories in the videos that we're linking

538
00:19:56.000 --> 00:20:00.960
from the blog associated with this

539
00:19:57.559 --> 00:20:02.880
release. Um, so that's all very exciting

540
00:20:00.960 --> 00:20:04.400
to see, but what I'm most excited about

541
00:20:02.880 --> 00:20:07.120
is to share this with all of you today

542
00:20:04.400 --> 00:20:09.679
and see what you all do with it. One of

543
00:20:07.120 --> 00:20:12.880
the things that I find really exciting

544
00:20:09.679 --> 00:20:15.360
about how Codeex works is it has very

545
00:20:12.880 --> 00:20:17.600
nonhuman strengths and weaknesses. And

546
00:20:15.360 --> 00:20:19.360
so it really means that you get much

547
00:20:17.600 --> 00:20:21.600
more out of it if you start thinking of

548
00:20:19.360 --> 00:20:24.720
of it as not just a static tool that you

549
00:20:21.600 --> 00:20:27.360
just use like you know just without

550
00:20:24.720 --> 00:20:29.360
having to build expertise but if you um

551
00:20:27.360 --> 00:20:31.280
if you really optimize your codebase

552
00:20:29.360 --> 00:20:34.159
around what it can do you start like

553
00:20:31.280 --> 00:20:35.919
honestly most of what codeex benefits

554
00:20:34.159 --> 00:20:38.400
from is just what is good software

555
00:20:35.919 --> 00:20:40.080
engineering practices um in terms of of

556
00:20:38.400 --> 00:20:42.000
modular code bases with good tests and

557
00:20:40.080 --> 00:20:44.000
things like that um you're able to just

558
00:20:42.000 --> 00:20:46.400
move so fast and we've seen that we've

559
00:20:44.000 --> 00:20:49.440
seen that happen uh internally uh with

560
00:20:46.400 --> 00:20:52.000
many people at OpenAI. Um we are

561
00:20:49.440 --> 00:20:55.880
releasing this today to uh chat GBT

562
00:20:52.000 --> 00:20:57.840
enterprise teams and uh

563
00:20:55.880 --> 00:21:00.159
prousers. We're going to have very

564
00:20:57.840 --> 00:21:02.880
generous rate limits. Uh we're going to

565
00:21:00.159 --> 00:21:04.480
start out without any additional pricing

566
00:21:02.880 --> 00:21:05.840
over time as we start to get some

567
00:21:04.480 --> 00:21:07.520
feedback and some sense of how people

568
00:21:05.840 --> 00:21:09.840
are using it, we'll introduce more rate

569
00:21:07.520 --> 00:21:12.320
limits. we'll be rolling out out to plus

570
00:21:09.840 --> 00:21:14.400
and edu users. Um that we'll be thinking

571
00:21:12.320 --> 00:21:17.039
about if there's pricing for how to get

572
00:21:14.400 --> 00:21:18.880
additional queries. Um in general, we're

573
00:21:17.039 --> 00:21:20.320
just so excited to see how people use it

574
00:21:18.880 --> 00:21:23.360
because we've seen every single person

575
00:21:20.320 --> 00:21:26.080
at OpenAI uses codecs differently. Now,

576
00:21:23.360 --> 00:21:27.600
we're not nearly at the end of this this

577
00:21:26.080 --> 00:21:29.120
journey. We're very much at the

578
00:21:27.600 --> 00:21:30.720
beginning uh that we're going to

579
00:21:29.120 --> 00:21:33.200
continue to improve codecs. We're going

580
00:21:30.720 --> 00:21:34.880
to make it be integrated into far more

581
00:21:33.200 --> 00:21:37.200
systems like be there in your issue

582
00:21:34.880 --> 00:21:38.799
tracker release an API so that you can

583
00:21:37.200 --> 00:21:40.799
have it automatically be integrated into

584
00:21:38.799 --> 00:21:42.799
your CI so if there's a CI error you

585
00:21:40.799 --> 00:21:46.320
don't have to fix it anymore that Codex

586
00:21:42.799 --> 00:21:48.880
takes care of it. Um we also are

587
00:21:46.320 --> 00:21:50.720
continuing to develop Codeex CLI which

588
00:21:48.880 --> 00:21:52.480
again is a local agent that runs on your

589
00:21:50.720 --> 00:21:54.720
laptop. We're releasing a mini model

590
00:21:52.480 --> 00:21:56.720
today and we're also going to be

591
00:21:54.720 --> 00:21:59.039
releasing signin with with chatb to make

592
00:21:56.720 --> 00:22:00.159
it easier to get up and running. Now, if

593
00:21:59.039 --> 00:22:01.919
you think about it, there's two

594
00:22:00.159 --> 00:22:04.159
different form factors that we've talked

595
00:22:01.919 --> 00:22:06.799
about. There's the local synchronous on

596
00:22:04.159 --> 00:22:09.679
your computer version, but there's also

597
00:22:06.799 --> 00:22:12.080
what codeex is of an asynchronous in the

598
00:22:09.679 --> 00:22:13.840
cloud runs on its own computer. And we

599
00:22:12.080 --> 00:22:15.679
think that the future is going to be

600
00:22:13.840 --> 00:22:17.520
these two systems coming together,

601
00:22:15.679 --> 00:22:19.760
right? What you really want is you want

602
00:22:17.520 --> 00:22:21.120
a remote co-orker with its own computer,

603
00:22:19.760 --> 00:22:22.640
but who can also look over your

604
00:22:21.120 --> 00:22:24.240
shoulder. And so, you're there typing

605
00:22:22.640 --> 00:22:25.520
away, you're working on some change,

606
00:22:24.240 --> 00:22:27.280
you're like, "Ah, I don't I want to go

607
00:22:25.520 --> 00:22:29.120
to lunch. Codex, can you finish this?"

608
00:22:27.280 --> 00:22:30.720
it just takes it over seamlessly and

609
00:22:29.120 --> 00:22:31.840
runs it in the cloud. Or if you have a

610
00:22:30.720 --> 00:22:33.200
question about something or you want to

611
00:22:31.840 --> 00:22:34.559
pull down a change because your

612
00:22:33.200 --> 00:22:36.240
dependency isn't installed or something

613
00:22:34.559 --> 00:22:38.159
like that, it all just moves around

614
00:22:36.240 --> 00:22:40.320
totally totally seamlessly. So it's a

615
00:22:38.159 --> 00:22:43.520
co-orker, it's a intern that you can

616
00:22:40.320 --> 00:22:45.600
delegate to. It's a mentor, it's a pair

617
00:22:43.520 --> 00:22:47.600
programmer and all of these at once. And

618
00:22:45.600 --> 00:22:50.000
so our goal really is to help accelerate

619
00:22:47.600 --> 00:22:52.080
the useful work that is done to help it

620
00:22:50.000 --> 00:22:53.280
be so there's more software engineers in

621
00:22:52.080 --> 00:22:55.200
the world so that there's more

622
00:22:53.280 --> 00:22:57.280
programming that is done uh that is

623
00:22:55.200 --> 00:22:59.200
useful and and able to move forward move

624
00:22:57.280 --> 00:23:00.080
forward the world. So we're just so

625
00:22:59.200 --> 00:23:02.710
excited to see what you're going to do

626
00:23:00.080 --> 00:23:05.979
with codeex. Thank you all.

627
00:23:02.710 --> 00:23:05.979
[Music]


