Playing Games with Neural Cellular Automata

Added: 2026-05-16

[00:00:00]Hello and welcome back to Data Science CosNet. In today's video I thought I'd share a few little silly experiments I've been playing with um exploring this idea of trying to see what can neural cellular automata um what can they do? What can they learn? Um can we make them play games and things? Um solve mazes. Um so yeah, I've been I've been messing around with this I played pictionary.

[00:00:28]Um and so this video I'll do a little bit of background on neural cellular automata and point you to some resources if you're interested in learning more.

[00:00:35]This was a favorite topic of mine some years ago and it's always been one that I kind of come and play with in the background. Um and then yeah, I'll show you the current experiments of today and suggest how you can play.

[00:00:46]Um so here's an an old article I wrote uh when was this? Back in 2023. Um so it's not that long ago. Um the idea here is that we're going to have this it's it's going to be a learned system but unlike uh say a normal neural network where you have a a big set of weights that that takes in some input and and processes it uh kind of globally instead we have these uh distributed systems where each cell in this grid is going to have um some tiny little brain and some tiny little local neighborhood in which it operates. Um but somehow we're going to try and achieve um some approximation of global coherence or or other interesting behavior um with the coordination of all of these different cells with their neighbors.

[00:01:29]And so this this was uh in 2020 that what the original article that um really just inspired me and is is a a phenomenal piece of work and and the core piece here yeah is this idea that at every cell at every location um we have this model that's just going to look at a very small neighborhood, right? A 3x3 pixel or cell neighborhood.

[00:01:48]That's going to be what it perceives um through some hard-coded filters.

[00:01:51]We have a very small network that produces then an update.

[00:01:55]Um and that's going to be this this piece that's repeating and and giving us this behavior. Um, so fantastic Eddie article was learning to uh grow these these um fixed uh patterns and and do all sorts of fun things with them. Um, then a follow-up that they did that was also incredibly inspiring and interesting one was learning these kind of texture generation tasks um with incredibly small uh cellular automata.

[00:02:17]We're talking, you know, hundreds of parameters or tens even rather than a millions of parameters or billions or trillions of parameters like um modern um neural networks. Anyway, so um yeah, like I said, this is something that I've I've played with over and over over the years. Uh one thing I liked doing back in the day was um using kind of style loss or using uh clip so that you could have this kind of contrastive uh comparison between the pixel view that's generated by the um evolution of the the NCA rule with some text description, right? So we could uh we could do things like um here, this is a cellular automata um that was optimized towards uh I think maybe a reference image and a text description for something like uh glowing dragon scales, right? And so you've got this this beautiful Here, let me zoom in a little bit more. Um yeah, this emergent pattern of behavior. And again, every single pixel in this pattern is just looking at its neighbors. Um and the weights for this are um pretty small, right? I can I can store them all as a as numbers in a shader.

[00:03:17]Um and you can do fun things like um like steer these with um with video or apply them at different scales. Here, there's a a video driving some channels in this um in this grid.

[00:03:28]Um and so yeah, I'll I'll leave you to read more on the papers if you really want to go a little bit more deep, but the core the core picture that you should have in your head, the core piece of this that makes them so interesting is that it's only ever operating this tiny brain on this tiny immediate neighborhood, right?

[00:03:45]Um in this uh original paper here when they you see them growing a lizard or something like that, the grid starts with a single black pixel in the middle and everything else white. Right? And it has to learn from that to build up this image. And if you erase part of the image, the neighboring cells are able to kind of talk to each other to reconstruct it. And they're able to do this despite being an incredibly simple little network.

[00:04:05]Um yeah, okay. So, that's the that's the background. So, now today's uh fun um is applying this to some games. And so, the the sort of main one here you can see is a kind of simplified Pong, right? You're you're trying to bounce the ball back.

[00:04:22]Um and in this case, now obviously um we just said if there's no sort of coherent global network with one single output, right? Um instead, every pixel is just doing its own thing. Um but what I do is for this rightmost column of the um of the state, um I'm going to read out the the max value there and use that to position the paddle.

[00:04:44]Um but you can see these internal states um each each point is looking at its neighbors and then updating. Um and if you if you track the the ball here, let's let's go to the live demo. Um You you can see as the ball moves, you have this propagation of information ahead of the ball as each cell passes on to its uh its neighbor um you know, it guesses that trajectory is or past motion or where the ball currently is.

[00:05:09]That information flows towards the right and you can see the motion of the paddle. Um I find this quite a uh quite a delightful it's it's it's it's it's not not like a big black box convolutional neural network that just opaquely outputs um like a a suggested position, right? You can see these uh cells talking to each other. You can track the evolution of the hidden states. Um so, it's still obviously mysterious, right? We have these simple rules, but they have all sorts of weird emergent behavior. Um but it seems somehow like delightfully visual and uh interpretable in that you really can just like see what's going on within the the shared state and how the messages are being passed around and and how it's uh it's maneuvering and and behaving.

[00:05:47]Um anyway, so so that's one one fun application. Um another trick that I tried here, I think I've got a better video um up was doing um the same thing for mazes. So, you start with the maze um represented in one channel. Um and then within the other channels uh starts initially random or or zeros, and the um each cell has to just look at its neighbors and and update, and you can see it kind of learns to predict uh eventually a like a a value channel. I I I sort of cheated. It It doesn't um it doesn't like uh encode somehow that the direction that your little agent should move to solve the maze. Um but what it does do is learn to produce this like gentle gradient, right? So, that we have a smooth gradient, and and by taking the next step, it's always going to Well, not always.

[00:06:31]>> [laughter] >> It's going to trace through at least some of the time um from start to finish. Uh and this is cool. Like, I trained this I think at um at 15 by 15 or like some s- some smaller size. Um let's uh go back to Chrome there. Um and even at at much larger mazes, like that one there was a a 31 by 31 maze, um it's able to sort of somewhat generalize these patterns of of um it looks it looks like it's doing something like flood fill, right? Like, encoding uh well, I'm this far away, and so my neighbor must be even further, and being able to pass these messages down.

[00:07:01]Anyway, uh I've only just started doodling with all of this, but I thought I'd share um share what's fun and uh and fresh.

[00:07:09]And then the final and and most interesting demo I want to show is um this Pictionary application. Um and so, here again, we're going to have a grid.

[00:07:16]Uh we're going to have no global knowledge or or pooling or anything like that. Each cell is just going to look at its neighbors, um and we're going to have these drawings, which are from a a little demo that I did. It was like an AI plays Pictionary app, which ran for probably more than a year until all the tiny models running it eventually got deprecated, uh and collected like 20,000 games of Pictionary played by random humans on the internet. Um Thousands of different people uh checked it out and tried it out. And you can see this is is not just a single image that we get. We get this kind of sequence as they fill in the strokes.

[00:07:48]Um and in this in this game the person got more and more frustrated because the the models keep guessing like moon or >> [laughter] >> or something like that rather than a ball. Anyway, and so we have this lovely data set of drawings as people try to draw horses and elephants and and so on.

[00:08:04]Um and so I thought well, let's try this as a as a neural cellular automata, right? So as the um as the game evolves, we're going to um we're going to try and solve it. So here, for example, um we we're trying to draw bed, right? So as I start drawing it's going to start um doing its its little neural network thing.

[00:08:25]Um and I'm going to start sketching out a bed. Um not particularly well. You can see already bed is kind of the top vote.

[00:08:31]And also that over time it's getting more and more solid, right? These are communication channels are all the things passing uh passing information to each other.

[00:08:40]Um and the um consensus value that it reaches eventually is is not always correct. In fact, this is pretty bad. Um but I do only have, you know, I think less than 20,000 images.

[00:08:53]Um it's a very small network. It's a very small pixel grid, etc. etc. etc. Um but still it's kind of neat, right?

[00:08:59]>> [laughter] >> Like um you can see pretty much throughout the image apart from around our person here um the the bed channel is is pretty high.

[00:09:07]Um and let's just go let's try something else.

[00:09:10]Uh let's see how good I can be at drawing an elephant. So I'll start with the ear cuz that's a nice big distinctive feature and then I'll do a trunk and a tusk and then we should make sure that there's the rest of the elephant.

[00:09:23]Um and you can see uh there's the elephant uh guess starting to appear.

[00:09:27]And it's going to be laggy, right? Cuz it's going to take a little while, but um it's it's growing. Uh it's being outpaced by nose. I guess an elephant is effectively a walking nose.

[00:09:36]Um So, uh there's that demo failing. Um But, yeah. That's That's uh That's the explorations. So, I don't know. I've been having fun with this. I think it's a it's a completely unexplored niche. Um I think besides uh I don't know. The The authors of these papers and Alexander Mordvintsev, such cool guy. You probably know him. His work on Deep Dream way back. It's one of the earliest generative art uh fun pieces. But, yeah. I think uh you know, there's been folks like David Ha who've had interest in this kind of distributed nature-inspired systems. Um There's been you know, a couple of other friends that I know on Twitter who've toyed with this. But, it really feels like this incredibly niche forgotten branch of of AI. And possibly for good reason cuz it's quite hard to justify exactly what value these things have.

[00:10:20]But, I think this kind of distributed consensus intelligence um it gives you like fault-tolerant and self-repairing properties and uh kind of like test time compute by default, right? Cuz it's current constantly iterating and refining. Um I think they're really cool. I think you could learn to do all sorts of things, generate images, um uh and uh yeah, just have a ton of fun.

[00:10:42]So, I think more people should be um playing with these.

[00:10:45]And so, what I'll do, I'll share the references that I I had. There's my my amazing elephant. Look at that. Um Uh but, I'm not going to necessarily share the code I don't think. I mean, maybe I will. Um but, more relevant to you I think will be this gist um which I'll definitely include in the description where I just had um AI try and summarize the the general approach. Right? So, apologies if anything reads a little bit too much like uh GPT-5. But, um this is all you need is to to have some condensed uh example and um references is even more verbose. I I I'll also share my old notebooks. Um But, yeah. Like a starting point and then I would encourage you like uh hand this over to Codex or Claude your favorite uh your favorite example. I think if you run the the code there, you'll get a somewhat game-playing um minimal um you know, cellular automata.

[00:11:35]but try this out on all sorts of different tasks. Come up with different ways of framing um as long as your objective fits into this idea of oh, we'll we'll be able to encode something in some channels of some grid, and then we'll let the cellular automata evolve, and then we'll be able to read out somehow, right?

[00:11:48]Whether that's reading the whole thing as an image or reading out one part of it as as a read out or one channel. Um and then applying some loss and optimizing from there. Um yeah, I think there's all sorts of fun stuff we could do, and I'd love to see I'd love to see more um more uh evolution of that. Um so anyway, I hope this I hope this is inspiring. Um do let me know if you build anything fun. Um you will have grown the the group of people working on neural cellular automata by probably about 10% if you join. Um and yeah, thanks for watching, and we'll see you in the next one.