Lecture 108: One Layer Deeper competition

[00:00:00]Uh share the share the slides or share the screen.

[00:00:03]>> What I mean yeah like your screen. Oh >> okay. Yes.

[00:00:08]>> All right. Uh okay. Welcome everyone. I see people trickling in. It's it's sort of an awkward time by the way. So I expect most of the viewers will come from like the from regular YouTube. Uh but like thank you everyone for joining on this like last minute broadcast and thank you to to Ben and Sean for uh humoring me for this like last minute talk. This is basically introducing like a new challenge that we've been cooking up and this is a collaboration between like some stuff I've been doing at my my day job at Core and by Ben and Sean at Research and a couple more folks. Um the interesting thing about this this is a new challenge that we we have in the server. It's already open for submissions today. Um it's primarily a model architecture and optimizer competition that we've really made sure is easy for both your humans both for humans and agents to participate in.

[00:00:59]It's also a fairly nasty task and that'll remind you that like inductive biases matter that you know maybe transformers as they are don't really like work for all sorts of tasks. So I'm really honored that Ben and Sean are here to talk to us about it. So they'll go over some of the theory and Sean will give you a demo of how to make your first submission. And this is really meant to be an interactive Q&A type session. So if you have any questions, feel free to drop them in chat and I'll make sure to interrupt the speakers. So yeah, without further ado, Ben, please take it from here.

[00:01:27]>> Yeah, thanks a lot uh Mark and yeah, thank you everyone uh for for coming. Um I don't know that I can see uh the chat uh with how I'm sharing screen. So yeah, feel free to interrupt me. Um yeah so what I want to uh talk about today is this competition that we're doing uh together with the great folks at core automation uh which is kind of like branded one one layer deeper and the the goal of the competition is uh to sort of um see whether or not models can perform deeper computation than they were trained to do. Um and so in order to do this we we needed to devise like a particularly tricky task.

[00:02:02]Um so here's like a quick outline. Uh I just want to go over quickly like why serial computation matters at all. Um some um previous ideas in the space uh that um have kind of like influenced how how we think about things. Um why depth um in its own right can be difficult to evaluate and how they're kind of confounders there. Um the specific task that we settled on which is this like repeated modular scoring task. uh the design of the competition, how you guys can all participate and then uh solicit solicitation of community feedback.

[00:02:35]Okay. So, so why do we even care about uh serial computation at all? Um well, it's really because for for problems that are like inherently sequential um depth is um really what at the end of the day limits difficulty. So harder instances of certain problems um are just going to require longer algorithms.

[00:02:57]Um and so if we just train models to be um uh to only exist at some fixed depth d then if I present you some instantiation of the problem such that uh more than d steps are taken you just uh your model won't be able to solve it.

[00:03:12]Um and we also kind of uh don't want to um spend too much depth or too much computation if if we don't need to. So we we also want to do this adaptively.

[00:03:21]So basically if if the problem is hard we should spend more computation. we should kind of like have a deeper model and then if the the problem ends up being easier then then we don't actually need like a particularly deep model or or deep computation.

[00:03:35]Um so uh what should uh like a model be able to do? Basically it it should be able to to solve things step by step. So we want to actually be able to like chain long uh long dependent steps together. Um it should do this adaptively. So if the if things are harder uh it should spend more computation. Um, and then we we want to be able to solve problems that are harder than the problems that we trained our model on. So, in particular, like if if we have some some training data um uh consisting of a bunch of problems that we actually like know how to solve, that's uh that's not really all we we want our models to be able to do. We also want our models to be able to solve problems that like we humans uh don't necessarily know how to solve. So, um, we want, um, we want the models that we train to be able to use more steps at test time than, uh, they were trained to do during train time. And then finally, because all of this is kind of hard, um, and potentially brittle, uh, we we want this training to actually be stable. Uh, so it should be reliable if if we're doing like long training runs or something.

[00:04:35]Um, so it's not like particularly new that hard reasoning improves with more compute. So there's this like famous graph from the opening 01 paper which is um sort of showing that uh when they allocated more test time compute to to 01 it's uh it's Amy performance rose. Um one of one of the key differences though between this and what we're going to focus on in this uh competition is that this was sort of like adaptive computation in token space. So it would kind of like verbalize more chain of thought tokens before it would solve the problem. Whereas uh our particular focus is actually doing adaptive compute in latent space. So like through through something like looping.

[00:05:12]Um so I'll just briefly go over some some prior work. Um so uh more compute can actually mean a few different things. Um and we're kind of going to focus on the third thing. But one way in which you could actually spend more compute uh is to do sort of like more parallel sampling. So instead of just um generating like a single alternative, you could generate a bunch of alternatives through uh like lots and lots of like parallel sampling. Um you could do sort of like visible serial computation which is what I kind of mentioned in the in the 01 case where you sort of extend the the chain of thought. Um uh or or you could do this kind of in latent space. So you could do like more recurrent uh updates uh more more looping uh and generally like repeated computation within the model.

[00:05:53]Um and this is kind of like a recurring idea. Uh so all the way back in like 2016 um there was like Graves's adaptive computation time paper uh where kind of like a halting probe was was learned. So um the the model could could think for longer um and it would kind of figure out the ideal amount of time to to think subject to some uh sort of like thinking time penalty. Then there's the the universal transformer line of work which is you take some transformer um and it has like some number of layers and uh some number of parameters and you kind of like repeat uh these blocks. Um so you don't actually increase like the number of parameters in your model but you sort of like increase the effective depth by by allowing like different layers to loop more. Then there's kind of these like easy to hard iteration uh extrapolation uh line of work uh especially from like Tom Goldstein's group where you you train for a certain number um of iterations but then at test time you you allow yourself to to go for longer than that. Um and then most recently um work uh for example uh that that Sean has worked on is um yeah something's kind of related where you you you train transformer blocks for a certain amount of train time and then at inference time you allow yourself to go far beyond this.

[00:07:07]Um and I guess just like stressing again the point uh of like where um fixed depth models can fail uh is if you have an easy medium or hard input you're going to um have exactly D updates. So the the specific update will be different whether it's easy, medium or hard. But you're only allowed say like D state updates if you have like D layers in the model.

[00:07:29]Um the whole idea behind adaptive computation time is that you repeat some number of times and then you predict like the probability with which you should halt at that point and then once you've kind of like accumulated a sufficiently large probability then you hold. So it uh the idea being that uh different inputs if they are harder uh will have like a a lower halting probability over time. Um and so it will actually take uh the effective depth of the model will be larger.

[00:07:58]Um and this uh sort of comes at a cost.

[00:08:02]Um if the model could um think for as long as it wanted without any penalty, it might just like think forever. So you also want to discourage unnecessary updates and that's what the sort of like ponder cost penalty is. So more updates might improve the answer but you don't want to do so many that uh your your sort of like aggregated loss is too high.

[00:08:22]And what they found empirically was actually that as the difficulty of the problem increased uh the ponder time also increased um but didn't like really like blow up either precisely because of this sort of like regularization penalty.

[00:08:36]Um then the idea behind universal transformers uh as I mentioned is to sort of just reuse the same block. So the the number of parameters stays fixed but the recurrent depth grows.

[00:08:47]Um and then the the whole idea behind this like recurrence extrapolation is at train time you say like have some number of recurrent iterations K. Um and these are like the easy inputs and then at test time maybe you have harder inputs and and then you'll have like more than K iterations. I think there's um yeah um so it's the same weights um but one iteration is going to like reuse the same the same block basically um and uh let me yeah so uh for example for like um the prefix sum task maybe you train on like 32-bit strings for 10 iterations but then at test time you do like 40 or 44-bit strings and now you can do like between 10 and 16 iterations and you can basically find that at least on this task ask you sort of increase the difficulty, as long as you also increase the number of iterations you're allowed, the accuracy rises.

[00:09:38]Um, and there's this like really nice uh like visual maze example where you train on mazes that are like kind of small, but then the model actually like learns uh the general like algorithm to solve these types of mazes such that if you increase the size of the maze, but the the sort of like type of maze is the same, then actually at test time uh you you do see this like nice generalization.

[00:09:59]Um more recently there's been work uh on sort of serial reasoning in latent space. So instead of like verbalizing i.e like emitting some some new like chain of thought token you actually just uh continue uh continue looping in latent space. Um and uh maybe I should say like the reason why this is so attractive uh from a theoretical perspective is that um there's this idea that actually if you can uh if you can do adaptive computation in latent space then you can sort of simulate chain of thought. So you you you should actually be strictly more expressive if you can do this like latent space style adaptive reasoning rather than token level adaptive reasoning. And this kind of makes sense as well uh intuitively because if you can do this sort of latent level adaptive reasoning you you are like allowing your your model to continually manipulate this like continuous quantity this like vector in in Rn. Um whereas uh you know if if if you're doing the adaptive computation in token level space you're kind of like every forward pass or something like forcing yourself to do this like discretization step uh which um is somewhat artificial.

[00:11:02]Um yeah and uh like similar idea here.

[00:11:04]Um you have like a a transformer core so to speak that at train time you you sample the number of recurrences that you want. Um and then at inference time you you allow yourself to actually like increase uh the this shared core that you loop over and over again. So this prelude block uh in some transformer model is not looped. This like actual prediction block is not looped but then this kind of like shared thinking block or like set of blocks is uh is actually looped some number of times and it's different at train time than test time.

[00:11:33]And what um Sean and and his collaboratories found basically is that as you like increase the number of recurrences, you can uh you can improve accuracy at least on tasks that seem like they model something like serial computation.

[00:11:48]Um okay. Uh but why why can like depth actually be be hard to evaluate? Um so uh there's a few things that uh a few deseterata that you might want for for a depth benchmark. One is that it should actually be serial. So uh we don't want something uh that is actually a shallow task in disguise. Um and in particular like there there's a difference between um a method being serial and a task itself being serial. And I'll give an example of this in a moment. Um then uh some computation uh can sort of be masqueraded as as memorization in disguise. So one way of um uh uh like one example here is you might have some task that like maybe that task itself is is is serial but if you allow uh your model to be large enough then the model can basically just like memorize a lookup table such that it's not actually really doing any serial computation at a certain point it's just uh sort of like retrieving what it believes is the correct answer from this from this lookup table that it's built out. And then this last point uh I think is like really key uh this depth extrapolation idea that some of these uh prior works also touch on is we really want the test instance in instances to require more depth than training. So it it's not it's not enough that like uh the test set actually comes from the exact same distribution as the train set. We want it to we want it to be strictly harder because we really want to sort of like model this idea of like having having models that that that can uh that can discover things or or or do things that like we humans might not know how to solve.

[00:13:19]Um yeah so so as I mentioned a serial procedure does not imply a serial problem. Um so it is not enough uh to to uh um like have a a particular method that is not shallow. We want uh we we want the actual task itself uh to not be shallow. So for example uh if if you're computing the greatest common divisor there is a a very serial way to do this.

[00:13:42]um you start with your uh uh like number a and then at at each step you compute the remainder. So you compute these terms like r1, r2 so on and so forth up to rk and in order to get uh like rk you need to like previously compute like r1 through rk minus one.

[00:13:59]But computing the gcd itself is actually not a serial task. So like one way to just brute force it is I hand you some number, I ask you what the GCD is and you just um sort of brute force compute like all numbers less than or equal to to that number and and determine if that is like a divisor and then you just choose the one that is the largest. So uh the ukidian algorithm is serial but the computation of the gcd does not have to be and yeah uh kind of this idea that I mentioned before we also can have like memorization that sort of like masquerades um in the sense that you might actually have some some function such that if your model is large enough it just learns some lookup table and then at test time it just like interpolates therein.

[00:14:41]Um yeah and and finally uh again we just want training and test depth to be separable. So test should be harder than train and we we should have some like very like nice granular dial such that like we we can actually adjust this and so this task that that we settled on that kind of like satisfies all of these to zeterata is this like repeated modular squaring task. So how does it work? Um we hand you some number x and then we hand you uh some modulus n and some other uh positive integer t and we ask you to compute x to the 2 to the t mod n. So uh concretely you you do x x^2 x 4th x 8 so on and so forth and then uh we just ask you what are the digits of y mod n. Um so why is this nice? Uh there there is no shortcut. Um in fact it is uh sort of um I can't remember if it's conjectured or actually proven uh that sort of like the best you can do is is like log t squar. Um and like at our end we can actually force this depth extrapolation by like at uh in the sort of train set we we hand you guys a certain like n um but then at test time we can actually like increase t for example and then since the the whole point of this competition is is to run everything on one GPU um if we if we on our end choose n and t uh in in a nice enough way it should prevent memorization of the model. Um so concretely how does this work? Um if we had have n= 77, x= 2 and t= 4. So that means x is equal to 2, x1 is uh 2 ^2 which is 4 mod 77. x2 is 4^2 which is 16 mod 77. x cubed is 16^ squar which is 256 but mod 77 that's 25. And then 25^2 is 625 but mod 77 that's 9. So then uh you would be presented basically 2 4 and 77. and the answer would be nine.

[00:16:37]Um, and so this is uh the same computation. Um, but the idea basically is that if you get one part of this wrong, uh you're um uh it's kind of like irreoverable. Uh the answer is just going to be wrong if you get like any of these like um individual scorings incorrect as long as we have chosen n sufficiently well.

[00:16:57]Um and the the idea basically is that you really needed uh the exa exact value of x and k. So uh actually like for for people that like number theory uh uh this is not like 100% true. For example, if we chose uh if we chose n very poorly um and we chose x very poorly um then um then this would kind of be a boring task. So for example, if we chose like n equal to 8 and then x was odd, then um the squares would basically at a certain point just all be one. um and and you wouldn't really need to do anything. Um but uh we're basically we're going to choose n to be like some semi-prime such that uh the the two shortcuts are kind of blocked. So uh the the obvious shortcut is to just like tabulate the entire function. Um but as long as we choose n sufficiently large this is not possible. Um and then the other thing um is I guess like coming from number theory instead of uh computing uh x to the 2 to the t mod n um there's this like trick uh like uh which is basically lrangees theorem which instead of doing x to the 2 to the t mod n what you could do instead is x to the 2 to the t mod of n where five of n is the toion function. So the number of integers co-prime to n. Um but this relies on knowing the factorization of n. Um and as long as we hide this um uh like i.e. like we don't show you the factorization um uh it's it's it's very difficult.

[00:18:28]Um so that's all this idea is is that 5n um is like if n is semi-prime meaning it's just a product of two distinct primes then the number of co-prime integers uh to less than n um is just p - 1 * q - one um but in order to sort of know that you need to know the factoriization of n and so but if you do then basically instead of doing x^ the 2 to the t mod n you can just compute 2 to the t mod 5 of n. Um, so on our end we know this and so we can compute the labels very easily but then on your end your your model kind of like doesn't know this. Um, and so it's actually hard for the participant but easy for us which is kind of nice I guess.

[00:19:09]Um, and then generalization kind of has two independent axes. So one is kind of like new arithmetic. If uh there's like unseen n and x then this kind of prevents uh um memorization. Um and the same depth can be used. But the thing that's kind of like more interesting is that uh if we sort of give you some number of t that and t is like actually not that large but then at test time t is is much larger then this requires kind of like longer execution. It requires uh uh this like adaptive computation behavior.

[00:19:39]Um so how does the competition actually work?

[00:19:42]Um so it's kind of divided into three budgets. Uh the hard budget is the one that actually matters for the sake of the leaderboard but easy and medium exist so that you can kind of like vet your solution beforehand. Um, so ED, uh, easy, you only have 60 seconds to train your model. Um, and there's a few practice data sets and we actually show you what N is. Um, for the medium, um, it's a little bit harder. Um, N is still shown, um, but the results are are private, but now you have 10 minutes to train your model. And then, um, for the for the hard task, uh, N is held out. We don't show you this. We also don't show you T. Um, you don't even like you submit your solution to to some kind of like sandbox in which, um, the only thing you're given is your is your accuracy. So you're not even given any um any information beyond that. Um and now you have an hour. Um but uh again it's it's only going to be on one GPU.

[00:20:29]Um I think yeah um so uh what is fixed uh it's just a single H100. Um we fixed the data. So you're not allowed to modify the data at all. Um, and what you're allowed to control basically is like what your architecture is, what your optimizer is, how you want to allocate this compute, how you want to allocate your depth, etc. You can also tinker with your loss loss function if you want. Um, but what we don't want is just like hacky like data side or like algorithmic hard coding um of solutions.

[00:20:59]And um I actually didn't touch on it that much in the slides, but really like uh the optimizer is likely to play a really important role here. um where uh you know because uh adaptive computation should be harder to train than um like non-recurrence uh it is likely to require like um thoughtful consideration of the optimizer.

[00:21:22]Um yeah [snorts] so how can you participate? Um it's just again like one file a submission.py file. Um you can sort of choose what track you want to submit to. easy medium. Uh you can uh like at this setting uh you can sort of like read all the logs you want. You can adjust the depth, the learning rate, etc. Um and then when you're actually like ready to submit to the hard track, uh you don't get to see anything other than what your score is.

[00:21:49]Um and then what are we actually kind of like looking for here? So again, we're we're not really like looking for like boring like hacky uh ideas. We want uh really like new architecture uh and optimizer ideas that are kind of like too wacky to like maybe work in in in language modeling or at least like too wacky to work in language modeling right now. Um so like if um um if you want to do things like cubic attention or something like that like that is something that you would not really be able to train a language model on to do like long context modeling but here because the the sort of context length is really really uh small then maybe you could do something like that. Um, we want to like incentivize like new optimizers in a way that's like slightly different than maybe something like nanog speedrun, which is also a really great competition. Um, but like where maybe you configure some optimizer that's like particular for for chaining together like large like long serial computation.

[00:22:43]Um, and again uh your submission submission is like one Python file. Um, so just you build the model, you build the optimizer, you build the training loss, and then uh like everything else is kind of done for you.

[00:22:54]Um [snorts] and yeah uh you can register on the website you implement your file uh you can practice on these like easier uh uh suites of tasks the easy and medium task and then uh you can submit a hard version. Um please uh give us feedback.

[00:23:10]Uh so like if there are rules that are unclear let us know. Uh if there constraints we should change let us know. Um what would make the the participation easier? Um I we've we've gotten a lot of really great feedback so far uh from the GPU mode Discord server server in particular like um there there are some modifications we've made to the like specific presentation of the hard task in order to make it like less hackable. Uh so thank you very much for this. This has been super helpful. But yeah, in general like feedback would be great. Um and if you want to try it, here's the QR code and and URL.

[00:23:43]>> Yeah, thanks.

[00:23:44]>> Uh thank you so much. Yeah, like a big thank you to Alex who helped us like find a lot of like reward hacks. So like very grateful for your help, Alex.

[00:23:51]>> Um I guess yeah, I guess maybe we can just like go to Sean's demo and then we can like include questions.

[00:23:57]>> Cool. Yeah, I'm just going to give a quick demo on how to set yourself up to make submissions. Um so yeah, I'll give you a quick walk through of that hopefully.

[00:24:09]All good. Can you guys see my screen?

[00:24:12]>> Yes. Maybe zoom in the one once. Yes, thank you.

[00:24:17]>> Oh, yeah. There we go. Yeah. So, this is the one layer.ai uh website. So, this is where that link that Ben showed you just leads to. So, we're going to run over to the GitHub. So, here links you over to the GitHub that you can see. And then, um the simplest way to do it is just get clone this repo because you need to install the um the CLI that Ben just showed. So you can get clone and hopefully that works like normal.

[00:24:43]Perfect. Uh so there is you don't have to get clone. Um this is just the easiest way I suppose for this demonstration.

[00:24:51]Um and then we'll pull up the readme.

[00:24:56]So if you don't want to use any local GPUs, if you're just going to rely on the GPUs that modal um have kindly provided to us, you can just go ahead and run this single line. You don't even need to run the git clone and it will just install this tool for you. that is called one layer. Uh so when I run it, it's going to uh tell me it's already installed because I did check that this demo works earlier. And you can do things like one layer help and it's going to give you all the commands out like that. I also read the website earlier. So, if we go to one layer deeper, you can when you go over to the uh when you first sign in, at some point you're going to want to uh make your account because we need to make sure that all of the users are actual humans and not just agents being spawned again and again and again. So, at some point, you're going to come over here and click this like continue with GitHub link. I'm not going to do it right now because I've already got my code and I've put it in my bash RC, but it's going to give you the instructions. And the second line of the instructions is going to be this command, which is super simple. And you can download this like example um submission which is literally one transformer block from the submit page where it's here the download sample which is shown here. And as Ben just showed you build the model, you build the optimizer, you build the training loss and then you have a submission.

[00:26:05]There's a few more flags we've added in as the competition's been going and people have requested different features. So feel free to reach out in the Discord if you um if you want new flags added in. We can definitely make that happen. So yeah, it's as simple as downloading that regular submission from the website and then you're going to send your submission up and then you can see here we've cued that submission. So that's like the basic. If you don't want to use your own local GPUs, you can literally just clone uh this tool using UV tool. Uh if you don't have UV installed, uh you can probably ask Chat UT and it'll explain it to you way better than me of how to download that and you'll be able to get going right away and then you can just edit your local submission and keep resubmitting as long as you're not hitting the rate limits. Uh the slightly more difficult version of this is if you want to develop locally, which is also allowed for like the easy and medium tiers and we give you exactly the uh data install scripts to make all that locally. So we've already ran the git clone command.

[00:26:59]So then you're going to CD into one layer deeper, build your uh virtual environment, which should be super simple using UV. So you're just going to copy all these commands out one at a time to build your environment. Uh this one's going to take a little longer as it installs all the packages that the environment needs because it needs PyTorch and all of that to make your neural networks run on your GPU.

[00:27:23]But you can see this other command that we sent off to the server uh using just the CLI is running up there. And that's going to have a 60-second limit. And there you go. It's finished. Uh it's a one layer model which is really really basic. So it's called zero even though this is the easiest one. And you can see here it tells you what n is. It tells you what t is that uh for the problem just explains. This is a super small n and quite small t's for the easiest tier.

[00:27:49]Uh so back to installing the environment. uh the environment should be installed and then you can check that uh all the unit tests run um to make sure the environment's installed properly and that's that takes a little minute.

[00:28:04]Uh in the read me as well you've got the full CLI documented here as well. So you can see that you can validate your submission without sending it to the server as a full one. You can see all the jobs you've run. You can see the status of a job. Obviously for a 60-second easy job, it's uh very quickly to get the response, but for those hour long jobs, it takes a while. You can also pull back some metrics from your job like the loss and the accuracy over training. Uh that's to kind of make it a bit more fair between people who can develop locally and hack the code to print whatever they want and the people who are developing just on the uh remote GPUs. And you can also print the leaderboard um locally even though it is always shown on the uh website here.

[00:28:43]So there we go. The uh environment installed and the test ran. Okay. So then there's a quick local in the bottom of the readme. There's a uh example of testing it on your CPU so that you don't have to have any GPUs. If you just want to be able to look through all the code and test things actually compile, there's a super simple test there called smoke G smoke CPU. And then you can also make the data by running seems that I might not have pulled the latest uh GitHub version. You can make the data by reading the scripts and generate data sets. The data sets are all seeded. So exactly the data you get here is exactly the same data set that will be on modal if you can if you're running locally or on uh remote.

[00:29:26]So to make all the data sets you can run you can either generate all of them uh by running this file or we can just generate the first data set to make it quicker for this demo.

[00:29:40]So there we go. You can see it wrote 850 lines of data into the expected location.

[00:29:46]And if you ever need to edit anything like that, you can see in here all the manifests are here. So this tells the model uh the like code where the data files expected to be, what the batch size is, what the eval batch size is, what you're expecting to shuffle, the data type and compile. All your regular flags are in here that you can kind of play around with uh locally. And if you find any of these useful, we can more than happily add them to the uh the interface as well. So yeah, now we've generated our data. Luckily here I have a few H100s. So I can just take one and train a model on it uh using this command here. So just like the code requires that you only have one GPU visible. And then the benchmark runner, which is the same benchmark runner that's being deployed on modal. So there's like no difference between what you're seeing locally and what you would be seeing on modal. And we also gave you the same uh tiny example with one layer and AdamW as the optimizer to run in the codebase as well as being downloadable from the website. So yeah, you can either train locally, you can train on the cloud very easily using your CLI, so either you or the agent can do it. I'll kill that runoff there. And yeah, um you can also submit through the website.

[00:30:54]Maybe this is a little bit more clunky, but the API key that you'll get when you sign in, uh you can put in there. This is the same easy data set that I just showed running through the CLI. And then you just choose your submission. And also on this page, we have like a few hints. We're still in beta, so we're still refining the rules. You'll see the leader board resets sometimes when we find a loophole. And the hard rules that we've set so far been discussed in Discord as well are all here. So like there's a maximum of 500 million trainable parameters. Uh don't hardcode the weights, like don't hardcode a lookup table into the weights. Uh no hard-coded algorithms in the forward pass. like you can't just code the algorithm that we told you into the forward pass. Uh cuz yeah, we told you the algorithm. Although actually the hard data set as it says here, uh we do have a little trick to make sure it's very very similar to what is uh in the easy and medium data sets, but you can't just hardcode the algorithm and get 100%. So there's a little gap there that we've kept secret to uh try and prevent anybody hard coding the algorithm. So yeah, that's the rough roundup of how to submit to the leaderboard. And yeah, um, if there's ever any questions, feel free to reach out in the Discord and we can solve them all.

[00:32:01]>> Sweet. Uh, thank you so much, Sean.

[00:32:03]Thank you so much, Ben. Uh, I guess like I saw two questions coming up. Like one was from Hara from Slack who works at Modal. He's like asking us like when is this project actually due? Uh, you know, I'm going to sort of make a proposal like how do you guys feel about like the the end of August? August 30th. Sounds good.

[00:32:22]>> Yeah. Yeah. I think maybe [clears throat] it for the entirety of August, >> right? And then um so I see also in chat Rohan has been asking um have you have you guys tried any solutions like basically what would your best attempt at solving this problem like rank roughly on the lead? I think he's challenging you to to give people a strong baseline.

[00:32:42]>> I think Sean's played around.

[00:32:44]>> Yeah, I've been playing around with it a lot because obviously we didn't want to release a leaderboard which everybody can 100% instantly and then we have to completely change the problem underneath it. So, we made sure we found a definitely difficult problem. Yeah, I think the current version that we've got deployed on hard is definitely a trickier version. Um, I can definitely the current score, the best score I think Alex got this morning was 0.4%.

[00:33:06]I think overnight I um yeah, I got a little bit higher than that. So, it's definitely possible to climb higher.

[00:33:11]You've just got to Yeah, the actual like we're we're being very kind. Currently, the hard data set isn't wildly difficult from what we're giving you. Like, uh, yeah, you've just got to really focus on the data sets we're giving you. And when all of those are looking good, and when I say all of those, I mean you've got to make sure all of those are looking good. Uh, you can probably take a decent chunk out of the hard data set, but yeah, the hard data set is meant to be hard. Um, we're very aware of that. And yeah, we'll be discussing in Slack. No, not Slack, sorry, on Discord. um if it becomes a little bit too hard.

[00:33:47]>> Um yeah, I mean folks like this is partly why we're releasing this problem like in the sort of like weird beta until like like for this week at least because we want to see basically how do people feel? Is this too hard? Is it too easy? And we'll sort of make adjustments.

[00:33:59]>> I guess John, I was wondering if you could speak a bit more to some of the reward hacks that you deleted off of the >> off of the leaderboard so far. Like what was like their just >> Yeah. Well, we didn't Yeah, we we didn't think people would instantly go straight for the reward hacks. Um but yeah, it turned out we found all of them. So our first version of our hard problem was literally we just had the same algorithm deployed as hard. So if you read the website, you'd understand what the problem was and like uh as Ben explained and you could hardcode that algorithm in the forward pass. So instead of actually using any neural network weights, you could just use Python to output every answer because you could parse the input and convert it into uh Python. So that was one. And then yeah, there's been some quite smart ones from um Alex has been really pushing um both neural networks. He would yesterday's solution that we deleted from Alex was like a router to different Python functions which the model was uh learning or like at least updating what would be the correct one to use at test time during training by comparing possible different functions that he just assumed we might do be deploying. And yeah, his search base was wide enough to catch one of the ones we were deploying. But yeah, today we've been a little bit less successful.

[00:35:09]We posted in Discord earlier about yeah that um he was searching for reverse digits, but we uh we'd hidden those. So yeah, there was a lot of different things being searched for that we posted in G GitHub ear no Discord earlier which Alex had just slightly missed. So yeah, it's a little bit of cat and mouse at the minute. But yeah, we're keeping the hard data set, that little trick that we're using secret to try and stop people 100% in reward hacking. But yeah, it's proving very >> we encourage you to try and find it.

[00:35:35]>> Yeah. Yeah, >> it's it's like you know these are like famous last words but yeah I mean like first by the folks like this this has been why this like sort of close period has been so so valuable. So >> uh thank you to everyone who like found nasty award hacks we will like acknowledge your work like I think it's as important as winning because it's sort of like one and the same until we nail down the details of the evalu. So thank you for your patience until we figure this out.

[00:36:01]Um, all right. I I think that that might be like a good time to uh close it. So, Ben, Sean, thank you so much. Uh, we will provide this video. This video is actually already on YouTube. Uh, it'll be on the GPU mode uh YouTube. Uh, I'll also sort of briefly edit it to sort of clearly mark which has been which is Sashan's section. Uh, and all of us hang out on Discord all the time. So, if you have any questions, feel free to ping us there.

[00:36:28]All right. Well, thank you folks and see you everyone soon. Bye-bye. Thanks a lot.