Fixing VFX’s Longest Problem using CorridorKey w/ Niko, Corridor Crew | NAB 2026 Puget Systems Booth

Added: 2026-05-04

[00:00:00]This is really cool to see so many people turning out here.

[00:00:02]>> Uh look, I got a replica of the couch here and everything.

[00:00:06]>> Oh yeah, >> I've seen it.

[00:00:09]>> Look at this. We got a a galaxy in splat of the studio behind us here. Uh which is pretty neat, too. Um good to see some familiar faces here. It's uh super cool.

[00:00:17]Uh wow, what a turnout. Well, I'm going to teach you guys about quarter key today. Uh, specifically I'm gonna be talking about uh I'm gonna be talking about how it was made, how it works, how I hope it's going to evolve and change and why it was made, all that kind of stuff. Just going to tell you a story here. So, um, yeah, green screens. Uh, we use a lot of green screens, and I don't like doing green screens at this point. There's always a little bit of something that goes wrong. One part looks perfect, it makes other part look bad. and do you fix another part that looks bad? It makes the part look look perfect look bad and it's a headache and I just I feel like it should be easier by now. So that was, you know, a thing I thought to myself years ago. Um started getting into AI, machine learning, seeing the different tools and I was like you know at some point you know somebody's going to make something that kind of does the job of green screens but you know with machine learning and kind of fix this and a year passes and you know it doesn't get fixed. familiar passes and you get things like uh you know segment anything, magic mask, rotor brush, great alpha tools, great masking tools, but still nothing that quite does what green screening does, which is to say, you know, you have your foreground subject separated not just from the background with the transparency channel, but also with those green colors unmixed from those semi-transparent pixels, which is a key to making all this stuff work. you know, motion blur, blurry edges, uh you know, things like hair, all that stuff relies on this ability to kind of separate the green from the foreground.

[00:01:48]Now, of course, you can do that with professional tools. You know, you can use delta key or IBK like key light. You can get there with professional tools, but the whole idea is that that becomes a kind of an arduous process of like rotoscoping effectively when you want to make it look really good. And you we have this production coming up. We're doing Son of a Dungeon season 3, uh Dungeons and Dragons show. And you know, one of the the key concepts of this this series is that we play the game, we play a Dungeons and Dragons game, but then we eventually go back after it's edited and we refilm scenes cinematically in the world in the game board that we've, you know, played the game on. And that, of course, requires having a lot of stuff on a green screen. And I didn't want to sit there and spend, you know, another couple months struggling with green screen keying in the year 2026 when I can talk to my phone. I can have my car drive itself, but why am I still pulling a green screen key in this day and age?

[00:02:41]Um, and so I had been, you know, waiting for something to come along. And once again, fantastic automatting tools are coming out, but none of them quite took that extra step for that whole foreground unmixing bit. And eventually, I was like, I'm just going to give this a shot. you know, I'm not I'm not a machine learning guy. I, you know, I don't have any sort of degree in like advanced neural network training or anything like that. Um, but I'm just going to give it a shot. I'm going to try and I'm going to sit down with, you know, some large language models and have them hold my hand. I'm going to work through it. I'm going to ask questions to as many people as I can and just go for it, you know. Um, and so that's how I spent my Christmas of 2025 was uh sitting there and talking to Gemini a lot. That was my entire plane ride back to Los Angeles from Minnesota was talking with Gemini on my laptop and kind of just working it out and giving a lot of research papers and kind of trying to hone in on an idea of how I was going to do this. And so it kind of came down to a core idea here, which is I can generate synthetic data pretty easily these days. Um, pretty much we all can, right? It's you fire up Blender or Houdini or, you know, whatever you want, make some sort of procedural system and start rendering out objects on a green screen. And one of the nice things that you can do with that is you can get a perfect alpha channel, you can get a perfect foreground separated from the background, you get a perfect background, kick it all out to EXRs, floating point 16, gives you a whole bunch of options here to then do augmentations and stuff like that down the line. Um, so I took that idea, that concept, shoveled that workload off onto the rest of the crew and had them knock out renders, hundreds of renders over the course of about two to three weeks.

[00:04:12]Um, and so I had this data set now of, you know, basically 200,000 frames, uh, roughly 500 plus shots. And, you know, the thought was, okay, I have all this put together. I have the data set, you know, garbage in, garbage out, good stuff in, good stuff out. You know, let's sit down and start training something. Um, so I fired up uh Microsoft's imageet or yeah, imageet.

[00:04:37]Um, and did a fine tune on that and it kind of kind of worked. kind of look like garbage. Definitely not good enough to be a professional tool, but it was kind of working and it made me feel really encouraged. So, I kept working on it from there. And at this point, I started keeping the results secret from the rest of the gang at Corridor. So, I could get some good reactions for for the video we were making about this. And of course, we have this interesting thing going on at Corridor Corridor where we can uh we can take a passion project or an idea we have and we can turn it into a YouTube video to justify us spending the time working on it. And of course, I was like, well, we have this, you know, I'm working on this video here. We're going to make a video about green screen tools, and you know, maybe it'll get a couple hundred thousand views, and, you know, it'll let me justify the the two months I spent working on this tool to make Son of a Dungeon easier. Um, and the the finale of this video was me doing the big reveals for the guys. And I spent, you know, I spent basically the two days before rendering these shots out. I barely even got to look at them.

[00:05:40]I had no idea if they were even going to work because we went downstairs and we filmed the hardest shots to deal with on a green screen. Jordan Allen drinking from a transparent blue cup and a transparent glass cup with water in them with transparent clown sheets on his head.

[00:05:56]just like the hardest stuff. Um, and I processed it and showed it to them. And one of my one of the gigs I wanted to do here, so those shots that you're looking at right there, those shots right there, those are actually keyed out shots using corridor key. And I kept that secret from the guys. I could do a nice little reveal here because I was very surprised at the the results I was getting. It was very encouraging. Um, and as you can see, they started to have the same reaction once they saw what was happening here.

[00:06:25]Um, and there you have it. And so if we keep going here, you'll see some of the other shots, uh, as well.

[00:06:32]Jordan's losing it. Ren's losing it. Uh, because whatever we seeing, whatever we seeing here is that like, hey, maybe maybe I don't have to spend all that time anymore. Maybe there's some help, some hope here, you know, to, uh, spend less time doing the technical setup and more time doing the compositing and the, you know, the framing and the the artful stuff, the stuff you kind of want to be doing.

[00:06:56]Now, you know, motion blur, motion blur is a hassle. Like, we always shoot high shutter speed on the green screen. We always shoot deep focus. We try to avoid blurry edges as much as possible. But, you know, we're quickly discovering here with some of these tests that that's, you know, maybe something we don't have to worry about quite as much anymore.

[00:07:12]Um, you know, that's uh it's pretty reassuring when you can get a shot like that. So, I was personally really surprised with the fact that this worked. Um cuz to me this wasn't just about green screening. This was also about the idea of inventing your own tools as a filmmaker. Um we're hitting a new era here where you can code. Anybody can code. Anybody can you know put together a script. And you know if we're all working with digital media here like at some point I'm sure you've dealt with the fact like oh I've got a thousand files I need to do some routine thing too. where it's like I'm as an artist, I'm going to go in and just deal with the same wrote setup that I have to do every time. And you've probably thought to yourself, oh, it'd be great if somebody made this plugin or somebody made this tool. And, you know, I've had that, you know, dream myself many, many times. This is kind of me not just trying to solve green screening, but like, can I make tools now as an artist who doesn't who's not a computer programmer? Can I stop having to rely on software companies to have to make the paintbrush that I want to use?

[00:08:16]just going to sit down there and make the paintbrush myself. And the experience was really really positive and really reassuring and gives me a lot of I guess excitement for things that can happen in the future here um around any kind of tool that you might want. So I thought it might be interesting to talk a little bit about the technical details of what's going on inside of core or key and how it works. Um so we're going to get a little technical dense here but you know let's let's go for it. So what I ended up doing to get those results is I ended up training on a model that Meta trained initially called Hyra. And Hyra is the the backhome model that works with segment anything. Um I don't know if you guys ever use segment anything. It's kind of your classic put a few dots on the subject you want to keep and then the AI model will figure out okay what's back on this foreground. It'll give you a rough mask. Um, at its core, what you're basically dealing with is a system that can look at an image and identify what the pixel is. So, you know, you heard the term chroma key. You can always think of this like a concept key where it's going to go and identify, is this a pixel of the background? Is it a pixel of an actor? Is a pixel of a prop? What is it? And little by little, the malt gets trained to be like, okay, well, if this is identified as a background pixel, you're going to spit out an alpha channel pixel that's black. It's identified as a foreground pixel, spit out an alpha channel pixel that's white.

[00:09:38]Um, if it's a fully white pixel, spit out the same original color of that pixel from the input image for the output image. And if it's a semi-transparent pixel, do that unmixing here. And that's where, you know, I relied on the data set to basically train this model how to do physically accurate unmixing. So, I had this hyra model. I had the synthetic data that I made, but you can't just say, here's my can't just say, >> is this working? Oh, this is much better. Now, I have something to do with my hands.

[00:10:09]Okay, so the way this needed to work is I need to teach the computer how to study these images, how to tell if something is working and something is not working. And that was actually the most challenging part of this entire process. Um because you end up with this this system of losses of of weights of saying like hey if this error is big minimize this error but if this error is small you can ignore it etc etc. So, here's an idea of what it took to basically treat teach the computer how to um how to put all this together. So, Christian, if you wouldn't mind opening up one of the uh the debug images here.

[00:10:44]Maybe not the very first one, but like something from halfway through. And these are pretty gigantic images here.

[00:10:49]Um and let's go ahead and zoom in to the top left corner here.

[00:10:54]All right. So, the first thing I had to deal with is the fact that, you know, I had a lot of frames. I had 200,000 frames of data, but that's not enough.

[00:11:01]you need effectively infinite variations.

[00:11:05]So I did everything from uh separating the background from the foreground, doing color adjustments to the background, hue, brightness, contrast, gamma, all that kind of stuff. Same thing with the foreground separately from each other. Um we gave them uh you know adjustments like aspect ratio, squash them, squished them, things like that. Some you know even some mirroring and that was just the input image. So an input image comes into the system but now it needs to do some comparisons. So basically what you give this system is you give it two things. You give it an alpha hint and you give it a uh just your input image. So your input image would be what's whatever's on the green screen. Your alpha hint is just a black and white mask saying hey this is what I want you to keep. This is what I don't want you to keep. So for example if you have tracking markers on your green screen that aren't green. You still don't want to keep those even though they're not green. So you can't just say hey keep things that aren't green.

[00:11:56]That's not intelligent enough. So the alpha hint was kind of like my way of saying, "Hey, if it's not lit up in the alpha channel, right? Even if it's not green, ignore it. And if it is lit up in the alpha channel, even if it is green, keep it." And so it's kind of like that that guide because, you know, you need a guide as an artist to be able to say like, "Hey, this is what I want you to kind of hold on to." So after that, the program does goes and makes a prediction. It predicts the foreground unmixed from that green screen and it predicts the alpha channel. Now, I'm sure you guys are familiar with premultiplied transparency and straight color. Uh, so this is a system that's trained to give you that straight color output, not a premultiplied output. The idea there is that you're basically teaching the computer to fully guess the color of that foreground pixel, even if it's only 1% opaque, because at some point you're going to need to take that thing, you're going to need to have the background color be removed completely, tweak the alpha channels, do all all this kind of stuff to it. And so that that straight color pass is truly the pass that you're going to want for compositing, but that's something that's physically impossible to capture with your camera, but you can use a machine learning model to get it. So all right, so the program spits out the predicted foreground, the straight color foreground, and it spits out the alpha channel. So now if we go to the top left once again, that's fine right there. Um, so you got these two pictures of the guy on the left here, we have the ground truth. So inside the training program, it does some compositing. It takes the input foreground, the ground truth, the rendered shot, and it puts it on a random color background. And it does the same thing for the predicted output.

[00:13:28]Puts on the same random color background. Then it compares the two.

[00:13:32]Lucky for us, color is a pretty easy thing to define on a computer. Red, green, blue, values between zero and one. Very straightforward. Works great for a neural network. Values between zero and one is exactly how a neural network likes to work. So you can do a simple loss comparison where you just subtract one from the other and get the difference. How big is that difference?

[00:13:51]Well, how far apart are the pixels in terms of their RGB values. So that measurement gets done between these two images.

[00:13:58]But what if your hue, your color is off just a little bit on one of those edges?

[00:14:03]Your RGV values might not be very far apart, even though your hue is noticeably broken. So not only do I do an RGB comparison between these two, but I do what's called a Y, you know, a YUV color space comparison. If you're familiar with YUV color space, you basically have your your luma channel, your brightness as Y. And then U and V are your chroma. So you take U and V and you can just do a subtraction between the ground truth UV and the predicted UV to get your true chroma difference. And you can take that and multiply it to amplify any errors there to make them even more extreme. So okay, so I got my comp on a random background. The background color changes here for every single P image that goes through the system. That way it's unbiased. It will compound any color. It'll learn how to fix any color that's going on there.

[00:14:47]It's unbiased. But not only is it doing that, if we go a little further here to the side, the same image gets comped onto a neutral gray background with no color whatsoever. So that way it can't cheat and just make the edge pixels purple and be like, I fixed it because then the purple would show up here on the side. So same exact comparisons happen. RGB, YUV, and of course the UV part gets multiplied. So any errors there get ampl amplified. So now if we go down, we see the alpha channel. Now of course YUV doesn't work for an alpha channel. There's no chroma in it. Um RGB doesn't even quite work for an alpha channel. It's just one channel, right?

[00:15:24]So the other thing with an alpha channel is that the inside is all white. And once you've predicted white, your job is done in terms of the computer and you're not going to get a lot of improvement.

[00:15:33]Even if an edge pixel is messed up, you're getting a bunch of signal from the rest of this image saying, "Hey, things are fine. It's supposed to be white. It's white. Don't worry about it.

[00:15:41]that little pixel on the edge. Don't worry about it. But we want those pixels on the edge to be perfect. So I made something called a spatial waiting map.

[00:15:49]Boom. Right there.

[00:15:52]So basically what I did is I went to the ground truth alpha channel and said any pixel that is semi-transparent, treat that as an edge. And in fact dilate that edge a little bit. You know, give me a little highlighted outline using those semi-transparent pixels as your guide.

[00:16:07]Anything inside of that is considered foreground. Anything outside of that is considered background. And of course, then you have the edge itself. So now I can take that and make a multiplier based on where the pixel is in the in the image. So if the if the pixel is in the edge in that little highlighted white area, you can multiply it that difference. If you're doing like an RGB minus RGB difference, you can multiply that by five, by 10, by 20. Same with the YUV. So, not only can you emphasize just the chroma by doing like a YUV comparison and multiplying the difference between the YUV, you can also do spatial weighting and multiply the difference by where it is in the image.

[00:16:45]And so, I use that spatial weighting to really really really focus the neural network onto the edges of the image to really make sure that any like differences between the ground truth and the predicted result were getting amplified 10fold, 15fold, 20fold if they were on the edge. So that that signal was just as strong as the signal on the center of the image or the outside there. Um if you go a little further here, we can see some other visualizations. You have a difference mat between like a predicted output and the ground truth. In fact, let's go all the way to the top right here. Right there, that guy. Um so this is the ground truth foreground output right there. The the render. And this is the predicted ground truth right here. Or sorry, not the predicted ground truth.

[00:17:26]It's an oxymoron. This is the predicted foreground right there. Now, you might notice the background looks all screwy in that. The thing is it doesn't matter what's going on in this background here because that never makes it into a final image. That's going to get multiplied against a zero for the alpha channel and that's going to give you nothing. So if you try to have a neural network predict what these pixel values should be in the background here, you're effectively you're effectively entering a divide by zero type of situation here where it's going to go crazy trying to predict random noise that doesn't actually affect the output and you're going to break your model which I learned after a few weeks of trying to do this. Um so there's a whole you know once again back to spatial weighting here. I can say, hey, take that background, those background pixels and ignore them completely or, you know, add a little bit of a weight to them if needed to to try to push it down towards black and that keeps it from breaking the model.

[00:18:18]So, it was this combination of all these different compositing pipelines within the training process that let this program actually give me the result I needed because otherwise I ended up with situations where, you know, it would it would give me I mean, the computer effectively learns to cheat, right? If you don't wait it correctly, it might be like, "Oh, well, you're trying to, you know, pull the color out of the background. Well, let's just make all the edge pixels magenta. That fixes it.

[00:18:43]Gets rid of the green, right? Problem solved." Like, well, that's not what I'm trying to do. Or might be like, "Well, let's make all the edges uh black. Let's just put a black pixel around there."

[00:18:50]Because you have a big black background and with that black background, if I make the edge pixels black, I'm rewarded for it. I did my job. You're like, "Well, no, that's not what I'm looking for either." So it was this combination of compositing on a random background color, causing on gray, doing RGB difference measurements, doing YUV difference measurements, breaking out the chroma part of YUV, multiplying that, breaking out the spatial weights, multiplying based on where it was in the whole process there, changing it as it went through the learning pipeline. You might weigh something heavier at the beginning till it starts to kind of figure it out and it can ease off that weight and weigh other things differently. And it was almost like making a soup or a stew, but it took like a month to make.

[00:19:34]Um, so after kind of working this over and over and over and doing process after process after process, honing in little by little on what was working, I ended up getting to a a spot where I was getting results that were pretty good.

[00:19:48]But the challenge is the Hierra backbone, the the model that I was fine-tuning, you know, the concept key, if you will, it's just a 512 x 512 image at the end of the day. Uh, anything more than that just doesn't even fit on really any hardware. Um, this is all being trained on an RTX 6000 Pro. I was lucky enough to have one of those machines put together by a Puget and Nvidia. Um, and that let me, you know, really crank this, put out some high quality stuff. And I wasn't worried about optimizing it yet. I wasn't worried about it running on anyone else's computer. I was just worried about it running on my computer. Um, so I got it working. But that, like I said, that hyra backbone back backbone is only 512 x 512. And if we want this to be professional, you got to be able to see individual hairs, right? It's got to be HD. It's got to be 4K ideally, right?

[00:20:36]Um, I want to come at this with no compromises. I want this to work. I want to be able to use this in my pipeline, not just be a tech demo. So, what I end up what I ended up settling with was I wanted a 2K by 2K image at least. So, I at least had the vertical resolution of a 4K image. And I like to think of it as an anamorphic 4K, right? Just taking the sides and squeezing them in, you got your 2K x 2K image.

[00:20:59]So, in order to get it up to that resolution, there's a second step in this process, which is something called a convolutional neural network, which I actually don't know all that much about.

[00:21:07]I kind of just said, "Hey, Gemini, take my hand." and uh then lead me um which didn't work for weeks and weeks and weeks. So I had to learn and do some research and read up on it online and eventually I got something which I'm still not quite sure exactly that you know the best principles behind it to use but it's working and effectively what that does is it looks at your your original incoming image the full res image it looks at your predicted image which is 512 x 512 and goes okay well in the full res image I take this one single pixel the 512 x 512 and look at the full res image I see little details in there so let me put those details back in it's kind of like a extra smart upres that gets to look at the source footage to cheat and give you the upres version. And once I had that working, once I had the convolutional neural network plus the hyra back on fine-tuned, I ended up with corridor key and it worked fantastically and it blew my mind and it blew all of it blew's mind.

[00:22:07]Um, and I was really proud of it and it was super cool. But I don't want to stop there, right? Like that's about as far as I could take things myself. Uh, I also at a certain point need to make sure I don't just work on tools, but I also take the time to then use them for the things I wanted to use them for, which if I were to just keep working on this and working on this, I wouldn't ever stop and actually work on videos.

[00:22:28]So, this was kind of where I entered the second phase of working on corridor key.

[00:22:34]And that's when I went to release open source, which has been a fascinating experience. And I should be, you know, upfront here, it's technically not true open source. not yet. Uh, for a variety of reasons, which I'll explain. Um, but I'm very much figuring it out as I go and that's been kind of my attitude.

[00:22:52]I've been trying to be open about that and everybody's actually been really supportive about that and okay with me like very publicly learning, which should be a little scary sometimes, you know? It's like, here you go. And it's like, oh, it doesn't work. Screw you, you know. Um, so what I did is after I made this, I released the model and the inference code online. And I didn't release the training data yet. I haven't released the training program that I made yet. Um, but I basically put it out there. I know we have an audience. We have this fantastic story around it that Austin, one of the editors at Corridor, basically helped put together, which was the YouTube video that you guys saw. And we put it up on our Corridor Discord, Corridor Creates. And, you know, I've never really delved into the world of software licenses and things like that.

[00:23:38]Um, but I wanted to try to find a way to be able to release this for free, let people use it however they wanted for free. But I didn't want people to suddenly take it, package it into a plugin, and start selling it. Um, I wanted to kind of keep it for free and keep it communal because the moment people start, you know, grabbing it, and I'm I'm going to put this in a plugin and sell it here, and I'm going to put this in a plugin, sell it here, I'm going to put this in a plugin, sell it here, it it fractures this whole concept of people trying to come together and make it better. Um, so by kind of limiting it to be like, hey, you can all use it for free, but it's not commercialized. Um, and you can commercialize it, that kind of focus people to to work on it together. Um, and kind of by leading by example, throwing it out there, putting the whole story around it, explaining how it worked. I got a lot of really passionate people that jumped into our community, uh, into our Discord in particular, and really started working on it with me.

[00:24:31]And the very first thing that happened was like when I made this it it required you know 24 gigabyte GPU uh so you know 4090 3090 RTX 6000 like that kind of level stuff within a day people had it running at on at six gigs um which is crazy uh you know uh one guy who's a you know graduate student in machine learning came he's like oh you've initialized a whole dimension that you don't need so I'll just snip that and I'll even submit a patch to the official repo there so they can fix them for you and and another person jumps and like, "Oh, and your, you know, your CNN is way too, you know, occupying way too much memory. I can trim that down for you."

[00:25:07]And somebody else comes in like, "Oh, you only take that from FP32 to FP16 and, you know, have the have half the amount of RAM that you need." So, within a day, like people optimize it. It could run on, you know, not anything, but it could run on, you know, normal GPUs. The other thing is I didn't make any sort of graphic interface for this program. It was all, you know, command line stuff.

[00:25:26]That whole idea is that you drag in a folder full of clips and you wait a couple hours and you get all your clips finished. And once again, works great for us in our workflow here. But, you know, for the vast majority of people, they want an interface. And I totally get it. It makes sense. I like interfaces, too. Um, so within two days, another user by the name of Ed came in and made a graphic interface and just knocked it out of the park. And because there's this whole like community around it that's experimenting and trying things, you know, left and right. He was getting feedback. People are trying different things. People were submitting ideas. people are submitting poll requests and it was just like a matter of yeah truly of a couple days until we had like a full-blown graphic interface with tons of optimizations parallelization. Um and it went from taking maybe four to 5 seconds to render a frame to rendering effectively a frame every second. Um which is pretty crazy to be able to knock out, you know, one FPS on something like this. Uh someone else on the discord just took it upon they made their they they made it their life's mission to make this as efficient as possible. Like truly they built a benchmarking system that in in milliseconds defined every single process that this thing went through and they got it running in real time um on a dual 4090 system which is crazy. First off crazy we have a dual 4090 system but but the fact they got it running in real time is bonkers. um you know not quite practical yet. You know, you're not going to fire it up in OBS and have it keying out your green screens yet, but the experimentation is happening and it's happening in the open and people are sharing knowledge around it and that's been really cool. Um where we are at now is I haven't really had the ability to see corridor key in use very much yet. You know, I dropped it basically a month ago. We've started using it a little bit. We haven't actually shot all the green screen stuff for Son of a Dungeon. So, I don't have like a deep personal experience with it yet. And kind of the same thing's happening online. People are using it little by little and I'm getting to see what's working and what's not working.

[00:27:26]And it's definitely scary releasing something into the world. And like truly all I tested it on was the shots that you guys saw there in the video with with the guys drinking water and, you know, dancing around with clown wigs.

[00:27:38]Um, I hadn't tested on anything else. I had no idea this was going to work for people. Um, but it did. and it was working for people except it likes to flicker as you guys probably know if your footage is noisy or you have chroma compression or subsampling you're going to get a little flicker and that would happen with regular keying tools as well unless you den noiseise it and that was one of the things like okay I need to fix that I need to figure out a way to fix that and once again because of this this beauty of the community and this kind of communal effort like true machine learning professionals have joined in and they're like, okay, you just need to add these couple things. It will look at the frames before. It'll make them temporally consistent. And there's experiments going on right now that are just completely removing the flicker altogether. Um, and my hope is that, you know, in the next couple months to a year here, um, we can take this to the point where it becomes a tool that can run on most systems. I don't think it's going to run on any computer, but hopefully it can run on most computers.

[00:28:42]uh it's fast and can handle truly whatever you throw at it. Like that's that's what I want to achieve with this.

[00:28:49]And you know, at this point, the best thing I can do is just encourage other people and give them that energy and give them that support to keep going with it. Um because the my my knowledge of machine learning has been fully tapped out, but my domain knowledge in the world of visual effects has not been tapped out and my ability to tell a story and to like rally people around this idea of, you know, making your own paint brushes, so to speak. You know, I I got plenty of that in me still, too.

[00:29:16]So, it's been a really interesting journey so far. Um, it's been really cool experiencing all this and it makes me really think about what other tools are lying out there waiting for an artist or somebody else to make because nobody else has made them yet or a software company hasn't found the value proposition in making it. Um, the ability to just kind of lay out your design, pass it off to Claude or Gemini or whatever, have it come back and give you a plugin. It's a really brand new territory here and something that I'm really excited about. Like when I think about AI as a tool, like it's not specifically like, oh, here's an LLM that does some crazy things or here's an image diffusion model that does some crazy things. It's, hey, you now as an individual have the tools and the power to sit down and compose a neural network or AI for whatever it is that you need to do. And yeah, it's going to take some time. There's going to be trial and error and all that kind of stuff, but compute is available for everybody. And there's some crazy things you can do with it. Um, and you know, I don't know what people are going to do with it. I don't know if in, you know, 2 years from now we'll have crazy sound tools that can remove the reverb from crappy audio, you know, or tools that will automatically color match every shot, etc., etc. But I'm really excited to kind of see what people do with this.

[00:30:32]And of course, you know, it's not just artists, but the actual professional software designers are accelerating as well and doing interesting things. So yeah, it's been uh it's been a journey and illuminating and really exciting and I'm hoping to kind of just keep it going as best as I can. So yeah, I think we should take a moment here and uh you know, I want to say thank you to Nvidia Studio for helping with this as well. Um but I want to ask you guys some not ask you guys some questions. I want you guys to ask me some questions. Uh give me an idea of some stuff you'd like to know.

[00:31:03]What's next? I just told you quarter key 2.0 with no flickering. Oh, and quarter key for blue screens.

[00:31:10]Well, what I'm really trying to figure out here is where does the need for a green screen end? And I don't mean in the sense like, oh, green screens are bad, because I actually think green screens are a really handy tool. You know, it's great as a filmmaker to be able to have something that you can say, hey, consider this the background. Like, you need some way to control things, right?

[00:31:31]You need some way to direct things. If you just say, "Hey, just guess whatever the foreground is and remove it." Well, there's going to be times when you're filming weird stuff, abstract stuff on a green screen. Like, imagine somebody walking into frame. Sure, when they're in frame, it's like, "Oh, yeah, clearly there's a human being there and AI can pick it up. What about those like couple of frames when it's a blurry blob on the edge of the frame as a person walks in the frame?

[00:31:54]How are you going to dictate, hey, that's a person, not the edge of the green screen, not some shadow that I want to remove. I want to keep that."

[00:32:01]And to me, the easiest way to do something like that is to be like just to fall back in your traditional green screening concepts, which is like, hey, if it's green, it's the background. If it's not green, it's the foreground. So to me, something like a green screen is still a really useful tool, but you have issues like bounce light, you know, color contamination. So is there a world where in the future the ideal screen is like a pastel green screen or a gray screen or a white screen or something like that? you know, something where it's like it's a background color that the computer can at least lean back on if it can't figure out what it's looking at, but something that doesn't, you know, uh, contaminate your image colors anymore or your foreground colors. Um, so there's this whole question of like if you have this kind of keying intelligence, this matting intelligence, how does it affect the physical world of what you're filming and what you're doing? And that's kind of the experiment that we're getting into right now. But before I get into any of that stuff, the very first thing we need to do is finish Son of a Dungeon. Uh because then I can actually see how it works and I can learn and I can tweak things and the code's right there and I can adjust it on the fly. So yeah. All right. Thanks Ram. That was a good good prompt.

[00:33:08]Uh does anybody have any questions? Yes.

[00:33:11]>> So two questions.

[00:33:12]>> Is there going to be an official four-door key open effects plug it? So you can put it into like pitch revolve something like that. The second one for the other part with green screens is transparent objects. So, I know it doesn't do object detections, but could we see potentially a like refraction map or a distortion map for like blasts of water or something like that?

[00:33:36]>> Okay, good question. Uh, all right. So, two questions here. First one was, is there going to be an official cord key plugin, like an official OFX plugin?

[00:33:44]Maybe. I know that's not a very good answer to your question. Um, but there is an OFX plugin already. uh a guy named Al in uh Brazil has been working with me making a runtime C++ tensor RT something uh OFX plugin and it works pretty dang good. Um there's actually a bunch of After Effects plugins that are out as well. Um I don't know where Nuke is at currently. Uh I don't know where other tools like Baselite are at currently. My hope is that they're in progress and that people are working on them and I'd fully encourage it. uh you know, if there's anything I can do to help encourage that, that's, you know, kind of what I'm trying to do here. Um, but the OFX plugin is is great. Uh, it works really well. You can download it right now for free. Uh, it will always be free to download. Um, the uh the second part, what was the second question? I got you invested in the first. Oh, yeah. Transparency. Okay. So, I've saw this question a few times on the Discord as well. What do you deal how do you deal with refraction? Right?

[00:34:44]You see it. You know, somebody has glasses on and in real life, of course, you can see the background distorted through their glasses when they turn their head to the side. But on a green screen, it just becomes full trans fully transparent, right? It's not how glasses work. It's not how refraction works.

[00:34:56]Well, at the same time that this whole keying stuff is happening and map prediction stuff is happening, there are also models out there that are predicting normals, uh, depth, um, albo maps, specular maps, things like that.

[00:35:11]And a lot of what we're doing here is meant for this pipeline. We're going to be using beeble um for the son of a dungeon stuff because that's like a fully built uh almost like material prediction pipeline for your footage where you give it your footage keyed ready to go and it goes okay I'm going to predict all the normals how the surface is you know where surface is facing. I'm going to predict uh you know what's the colors there without any shadows or lighting on them. I'm going to predict how shiny things are. And with that, then you can throw it right into a program like Unreal Engine and you can relight your real footage in Unreal Engine or Blender or you whatever any program, right? Any 3D rendering program. And my thought when it comes to transparency, as long as you predict the transparency correctly and you subtract that background color so you don't have like a green tint in your glasses in order to get refraction, then you have to lean on your normal prediction. Um, you know, what's the curvature of that surface? Uh I did see a really fascinating idea somebody had which was to make another synthetic data set with um UVs uh basically UVs mapped out as another output so you can train the program to predict what the UVs are for the things that you're looking at um which is a crazy concept but once again as this stuff gets out there and my plan is to release the the training data and the training program which once again it's it's a vibecoded you hobby fest like it's probably not that great but at least it can start right.

[00:36:38]Um once that's out there somebody else be like oh well let me just render out these passes instead go you know wait a few days and end up with a model that's trained to predict a different kind of output but running running out the same structure and of course if somebody else comes out with a different backbone model to try or a way to train a model from scratch ideally because this is open in the community people can kind of jump in and do this kind of stuff. So I would encourage you to give it a shot.

[00:37:01]Honestly, you know, it's uh maybe like my plan is to release the uh the training data and the training program once there's a good plugin for every program. So I figured, you know, for it's one thing to have the model, it's one thing to have the inference ability, it's another thing to have the tool. And that's a big thing I realized like cool, I finished it, it works great. And I release it and people like, oh, this is like a command line interface that requires 24 gigabytes of VRAM. This is useless. I was like, yeah, they're right. I need to be able to actually just use it in the program that I use.

[00:37:27]everybody needs to be able to use it in the program that they use. So once you can fire up After Effects, Dinci Resolve, Fusion, Nuke, etc., it's, you know, download for free or it's there or whatever. And you can just use it. Once it's that easy, then I'm like, okay, put the training data out there, put the training program out there, let people go to town. You know, at least it's been centralized, it's been demonstrated. you know, there's a a a focal point for when people train new models or new ideas.

[00:37:53]There's an architecture on how it can then get utilized in software, which is a whole second step from the first step of just train it in the first place. Um, so that's where that's at. All right, I'll take another question. Yes, >> you're testy. Did you consider using apps or flag?

[00:38:16]>> Cool. Thank you. Um, so his question was, "What do you think about doing like a dark gray or black background?"

[00:38:21]Something I thought about. Um, it comes down to being able to queue in the computer as to what you want to keep as foreground and background. Uh, let's say you're filming a miniature exploding and a bunch of tiny little black particles go flying off into the air. How do you know if those aren't tracking markers?

[00:38:35]How do you know that those aren't background? Right? So, I think there is a certain point as a person filming, as a filmmaker, you still need to step up and help the computer a little bit. And that's where I think at the very least a light gray might be the way to go. So you still get your bounce light but it's not contaminated with the color. Or you go with a hue, you know, a touch of magenta or tan or green or something like that so that worst comes to worse, you can still just fall back into a key.

[00:39:00]That's one of the things I like about filming on a green screen is I can still use color keying, right? Like color key tools still exist and they're still not they're not bad. Like they're pretty great. It's just that this is one click uh close to one click and that's the idea. you know, this gets you there quickly and painlessly as much as possible. That's the goal. So, I don't quite know what the end result will be.

[00:39:20]And frankly, my hope is that once again, people experiment, right? It's just a matter of kind of setting up your computer to kick out some renders for a while and then setting up the training program to chunk on those renders for a while and then you'll have a different model. You could train a gray model, you could train a an orange background model, you can train a blue model, etc. So, yeah.

[00:39:39]>> Uh, you mentioned Gemini a couple times.

[00:39:41]Is there a reason Gemini was your go-to?

[00:39:43]>> Good question. All right, so I did mention Gemini a couple times, and I'm sure everybody's like, "Well, why aren't you using Claude?"

[00:39:48]Um, it's because Gemini is cheap.

[00:39:52]Uh, for 20 bucks a month, I can basically get unlimited Gemini, and I don't get unlimited Claude. Uh, the other thing is that the challenge here wasn't so much the programming. Um, it was the design. And frankly, you need to be able to write the entire program in plain English to the exact details of how the compositing formulas work. Once you can do that, you can throw it almost any LLM. Doesn't need to be clawed and it'll get the job done.

[00:40:19]And that's basically what I needed to do here. The other thing I learned is that, you know, LLMs, they're great at taking structured language and turning it into, you know, script code, etc. They're great reasoning logic machines. They don't understand images whatsoever. So when I'm comping an image, it doesn't know what premultiplied is or straight or linear or sRGB sRGB versus 2.2 do like gamma gamut like all these things are you know even though it thinks it knows it doesn't know them and constantly I was dealing with issues where because I had to do basic code a compositing pipeline for this you know for all the testing that the you know that the loss and the weights go through as a pro model is trained and decided to program in like okay let's let's take in this sRGB EXR file let's convert it to linear let's unpremultiply it let's comp it on this background And then, you know, we go through it like, oh, it's using the premultiplied compositing formula, not the straight color compositing formula. So, little by little, you go and you find these little problems and you have to lay it all out there. And, you know, maybe in the future the highest end LLMs will catch that kind of stuff, but my experience has been that you really have to bring your domain knowledge and then the LLM like Gemini will do the coding. And at that point, Gemini was completely capable for what I was, you know, what I was needing. Yeah. All right. Any other questions?

[00:41:39]Nice correlative peppers.

[00:41:54]A lot of the white background.

[00:41:58]It is interesting to see that.

[00:42:03]>> Yeah, good question. Okay, so the question was how do you deal with things like background objects like a light, a stand, the edge of the green screen, edge of the green screen. Um, so that's where something called the alpha hint comes into play. So my thought was rather than having the program completely guess what the foreground and the background is supposed to be, once again you're an artist, you're a filmmaker, you can at least give it a hint, right? Aka an alpha hint. You can pull a green screen key and just be like, "This is the background. This is the foreground. Here's a black and white image that shows that." Or you can do a magic mask or roto brush pass or whatever. Like it doesn't need to be good. Just a way to be like, hey, if it's white pixels, it's a foreground object. If it's black pixels, it's a background object. Even if it's not green and it really comes down to control. Like you need a way to control your shots. Like you can't just be like, AI, take the wheel. Help me. Like you got to have a way to tell it what you want to keep. Um and so that's why I rely on this whole alpha hand concept where, you know, pull a quick key. It doesn't matter. Or use magic mask, but just be like, I want this. I don't want that. And then the model ideally is intelligent enough to divide it from there because the the actual separation of foreground from background isn't happening through a chroma process. The chroma really only comes into play when it's that unmixing on those semi-transparent edges. There's also this concept of like not reinventing the wheel. There are research projects out there that are so good at alpha extraction, like amazing immaculate alpha extraction. There's re research projects out there that do so many different things. I didn't want to have to reinvent that. I just wanted to really focus on the whole unmixing side of things and just making it usable one click or as close to one click as possible for our workflow and then go from there. Yeah.

[00:43:44]>> Weird question. How do you think it would handle retrofosing >> retroreflective compositing? Um, it probably handle it fine, honestly.

[00:43:54]>> Yeah, you know, it's uh I think it probably handle it. I I'm very surprised at how capable this system is at pulling the foreground from the background, whatever the background might be. Um, but just as long as I kind of like a flat wash of color, it could be anything. Um, yeah, you know, but it's also one of those things where I don't know until people try it and it's only going to get better if people try it and kind of give feedback to each other. So, yeah, >> with dealing with things out of focus, I keep on the background. say the shot request to finish this talk to each other. Uh, do you think it'll be helpful to deal with that sort of mouse stream or is that like >> Okay, so the question was how do you like when dealing with really out of focus stuff like an over the shoulder shot, right? Uh, over the shoulder f 2.3 or something like that, you know, it gets really blurry. Um, it handles it pretty well actually currently. Um, I really made sure that the data set that I was feeding it contained extremely blurry shots, extremely motion blurred shots, fine hairs, other crazy kind of things like that. Um, like I basically made sure that there's a render that documented every single problem I've had to deal with over my, you know, 20 plus years of compositing stuff. Um, so that as long as I can handle it in the data, I knew I can handle it in real life. And it generally does. I mean, if you go really extreme and you have a gradient across the entire frame of opacity, yeah, you know, you still got to like try a little bit to like treat your green screen with respect, but it handles it pretty well.

[00:45:32]And I think that's a sign of it working how it should. Um, to me, the the stress test I would do is I'd throw like really transparent smoke at it. Um, you know, wispy smoke elements, things that are already keyed. I'd recomposite them on green so I'd have that ground truth still. I just see how well it could do.

[00:45:46]And it did a pretty dang good job. Yeah.

[00:45:50]Yeah. All right. Any other questions?

[00:45:55]>> All right. Well, guess I answered them all. Okay. Yes.

[00:46:01]>> Oh, I are any looking cors.

[00:46:07]>> Yeah. So, the real time is tricky uh with machine learning stuff. A big thing I've learned with this is that getting a machine learning model, a neural network to run on your computer the correct way is really challenging.

[00:46:22]Um, you know, by default your computer likes to run things as a single thread on the CPU and call it a day, which is fine if you're just going to, you know, change some folder names or do some basic math. But if you're going to run, you know, long matrix multiplication and try to fire it on all thousand of your CUDA CUDA cores or something like that, you need to be able to talk to your hardware in a really specific way.

[00:46:47]And that's been a bit of a challenge.

[00:46:49]Um, I mean, there's definitely tools for it. Like if you're set up correctly, you have the proper, you know, most recent drivers, you have the proper pietorch, etc., etc. Like you can make stuff sing.

[00:46:59]The problem is that a lot of people I mean there's a technical barrier between just installing a program and then making sure that you have all the proper necessary drivers and dependencies and I guess triggers and flags to really make things work correctly on your GPU. And we're also kind of in an era where that hardware has been kind of rapidly evolving over the past couple years.

[00:47:19]It's like you go back to hardware older than four or five years from, you know, ago, like 2020 and earlier, and you start to really lose the abilities to put the stuff through a network fast.

[00:47:30]Um, and at this point, I can just kind of, you know, just put it out there and hope that people will work together to solve those things with that hardware.

[00:47:39]Have any of you guys used quarter keys?

[00:47:41]Has anybody experimented with it?

[00:47:43]>> What was your experience like?

[00:47:45]>> Really, really >> good. Okay, that's good.

[00:47:49]What kind of shot were you keying?

[00:47:50]>> Uh just a basic green screen stuff and I did a I use Blender a lot so I made a lot of like shots with blurry backgrounds smoke and it was very impressive.

[00:47:59]>> Okay, that's I'm glad to hear it worked for you. Um what kind of system are you using?

[00:48:03]>> Um I have a I have a 20 or 30 I have a 350 and a 370.

[00:48:10]>> I ran on both. Uh but I also have a 10 1080 that works on a 1080 too. It was fine. I mean it took a while but yeah it worked.

[00:48:20]>> Yeah. You know like the the next steps for me besides just trying to make the plug-in better and foster that community development is also try to just make sure that like I'm learning a lot about software licensing this process. like right now it's it's under like a creative common license uh that uh protects against non-commercial but I'm learning that creative common doesn't really apply to code and yada yada so just little by little day by day talking to people and they're guiding me through it but at the end of the day as long as people are kind of coming together into a central point and I can like maintain that energy I feel like we can make it happen right we can make green screens easy feel like somebody would have done this six years ago but at least it's happening now.

[00:49:03]Yeah. Um, let me see how much long we got left. That's kind of everything. We got a few more minutes. Um, >> yeah. I guess anything else? Any other questions? Anything that people want to know? We got one in the back there. Yes.

[00:49:15]>> So, what is the advantage of?

[00:49:21]>> Oh, good question.

[00:49:28]>> Gotcha. Okay. Uh you guys familiar with the sodium vapor process? The most mathematically precise way to pull a mat these days. Uh so the question was why didn't why aren't we using sodium vapor?

[00:49:40]Um sodium vapor is very cool but it's very specific for how it can be used. A couple challenges. One because we don't have the magical Disney prism that has the uh split in the middle the splitting wave wavelengths of light. We have to take our light split it first and then filter it. What that means is that every exposure is getting half the light it would otherwise. So you start really having a blaster set with light. The other challenge is it's a rig with two cameras. So the challenges of shooting stereoscopic apply to shooting uh sodium vapor, which is to say you have to have two cameras perfectly lined up, same focal length. If you're going to pull focus, you have to pull focus across both cameras simultaneously. If you're going to zoom, you have to zoom across both cameras simultaneously. Um so there's definitely restrictions of sodium vapor.

[00:50:27]The thing I haven't done that would be interesting to try is to make a data set using sodium vapor. You know, take the time, set it up, put on a tripod, put in the background, you're not going to pull in focus, whatever. Make life easy, right? Take that, record real life footage, get a bunch of different stuff, and then you have your perfect alpha channel mask, you have your perfectly unmixed foreground, etc. That would be a a really interesting way to use sodium vapor, but I think using sodium vapor in real life is really relegated to very specific setups. you're not pulling focus, you're not pulling zooms. Camera could move, but you're going to be dealing with a lot of hardware there.

[00:51:01]You know, some ideas that people had mentioned back when they made the sodium vapor video with Paul here was using sodium vapor for uh the backgrounds of sets like when you're in like a in a skyscraper apartment set or anything else a lot of reflection and glass and that kind of stuff. Things that would be tough green but aren't so bad when you have like a perfectly physically filtered system that's giving you that kind of, you know, perfect wavelength of light. That's a good question. Sodium vapor is really cool and it's like our touch point for like the ground truth.

[00:51:26]Like if we can hit that quality, then we're good. We're not quite there yet.

[00:51:30]Not even with Corey. It's still not quite as good as sodium vapor, but it's way easier to set up. Yeah. All right.

[00:51:37]Any other questions?

[00:51:38]>> Yeah.

[00:51:40]>> I've coded before.

[00:51:42]>> Uhhuh.

[00:51:42]>> I've hit a lot of roadblocks.

[00:51:44]>> Uhhuh.

[00:51:45]>> What were things you learned along the way? I tried doing like a nonlinear editor and just troubleshooting bugs, all that kind of stuff. Yeah, >> very very hard.

[00:51:53]>> So, when it comes to vibe coding, Nico's vibe coding tips. All right, five minutes of Nico's vibe coding tips. Uh, I'm by no means a professional, but I quite enjoy doing it. Um, make sure you're using an intelligent like IDE like cursor or anti-gravity. Uh, I use anti-gravity personally. Um, it makes your life way easier. You're not just deal with copy and pasting and chunks of code and all that kind of stuff. The other thing is to really make sure you take your time to plan and to write out your plan, whether it's by yourself or with an LLM. It could be separate. It doesn't need to be in, you know, your coding environment, but plan everything out. Um, another really important thing to do is to look up previous research.

[00:52:34]Um, find other papers that have been published that deal with the same problem. Read up on them. If there's something that you don't understand, take the time to understand it. Whether it's, you know, Claude, what does this mean? explain it like I'm five, you know, or whatever. It doesn't matter.

[00:52:48]Just try to wrap your head around it.

[00:52:49]Um, I'm a big believer that even the most sophisticated advanced concepts can be understood if they're phrased the right way by almost anybody and you have like the right foundation, the right nuance to get there. Um, I guess I would say the other thing was just uh, you know, building it modular. Uh, don't try to do everything at once. Build it piece by piece. Don't let the the LLM go to town on your entire codebase. It's going to screw things up. it just it doesn't have that awareness. You know, you really have to think of it almost like a translator. Um you're writing the book, the translator will get it into the appropriate, you know, language for you.

[00:53:24]Um but if you're not writing the book, the translator can't sit there and just invent the story points for you. So that's I would say, you know, the pitfalls and things to think about to look out for. Um one really cool thing is that as your program is being made, if the LLM does something that you don't understand, take the time to say, "Hey, explain to me what you did." And walk through it from there as well. So, it's really really important that you steer the ship. And it's really easy to not steer the ship, but that's when things will fall apart.

[00:53:50]>> Yeah.

[00:53:52]Okay. Anybody else?

[00:53:56]That's really Oh, yes.

[00:53:59]>> I don't know too much about that. I said pop up like Steve, but tell us a little bit about that.

[00:54:09]>> Yeah. Yeah. Okay. Uh so the question was to talk a little bit about the Bible integration here and the process there.

[00:54:14]So Bibble it's a funny name um is a AI piece of software that will take your footage um it can do an AI roto on it.

[00:54:23]It will predict normals it'll predict albido specular etc. And then it also offers a very kind of fast pipeline to put that into some 3D software package of your choice. One of the really interesting things about it is the compositing aspect of visual effects kind of gets blended with your 3D layout and rendering side of your pipeline. And it's kind of a a change up and in a really magnificent way. Let's say you're going to be doing a shot in Unreal Engine walking down the dungeon halls, right? You got torches burning and you got dungeon steam, I don't know, in a skeleton in front of you or something.

[00:55:00]Dungeon steam. Um, those are things that you'd have to deal with, uh, compositing in layers, like either going to deal with like, you know, depth compositing or you're going to deal with just, you know, rendering all your assets. You pock them into whatever compositing program you're in, stick things on top of each other, tweak the color levels, get the bloom right, etc., etc. But now if your if your footage of your actor lives in the 3D scene and has normals, so it's lit by that scene and it has the proper specular and highlights and that kind of stuff and it's motiontracked.

[00:55:32]Well, now when you render out the scene, you're rendering out one unified image where all those elements are being comped in that scene as it's rendered, you know, uh, physically based and it's, you know, any lens effects are happening to the entire scene as one full stack.

[00:55:45]And so it it skips over some of this chunkiness that happens when you kind of have to like fake things all being in one composition versus just being in one actual composition. So we're really excited about that because when it comes to this this D&D show, you know, the idea is that we're going to be literally in the game board that we filmed on. Uh you know, the whole game board was built in 3D. It was 3D printed and painted, but we still have the 3D files. So we played this game, but now we can be in the actual game board at the same time.

[00:56:13]And uh you know if you're dealing with Dungeons and Dragons, you're dealing with many many minutes of footage. So you know film your actors in the green screen once, take a two-minute shot, bring it to life in that world, skip the compositing step, just everything's in Unreal Engine or everything's in Blender and just hit render and you're done. So that that's the whole idea here. You can kind of see some examples there of, you know, the the older versions of Son of a Dungeon. Um yeah, so that's kind of where things are at and it's really exciting. Uh, you know, it's cool to see the turnout here, you know, for learning about quarter key and talking about it.

[00:56:47]Um, there's something really fulfilling about making a tool and then people actually finding use from that tool.

[00:56:54]It's uh, it's a very good feeling. Uh, hard to describe. Um, so I'm really really happy with that and it's really energizing and keeps me wanting to go and make more stuff. So yeah, thank you to everybody uh, for coming out and joining us here. It's super cool to see everybody. It's good to be back at NAB.