Ai Creates Ui Inside Unity

追加: 2026-05-08

[00:00:00]Hello everyone. So, I have been using Codex for 1 week and I found it very useful and I have also created a tool that connects Codex to Unity. It is very similar to MCP tool but it is more efficient and has more features. And I wanted to find some ways to automate some processes in Unity editor itself because coding is almost automated nowadays. The AI agents are very smart at writing code and we don't need to write any code at all. We just need to tell the AI what we want. But there are some tasks in editor that AI agents can't do properly as of now and I'm trying to do that with this tool. So, OpenAI launched an image generation tool which is really good and it can create very good images exactly what you want and uh I generated this image for my game for this UI and I wanted to implement this inside Unity but I don't want to like extract all the UI elements and add as sprite manually.

[00:01:16]I wanted the AI to be able to do that.

[00:01:19]So, I built the tool and we will see the demo of it in this video. So, what I have is I have couple of editor scripts in here and a bridge to communicate between the editor and the Codex app and some skills to tell the AI how to use the tool and also how to create better UI in Unity. So, basically understanding of Unity components and stuff. I gave this image and I gave this prompt to create the exact looking UI with modular sprite and stuff in the open scene inside Unity. And I set it to plan mode so that it will ask me more questions about what I want exactly. So, let's see. I think it's misleading to a different scene. I wanted it to create everything in this test scene. So, I will just tell it to not use any script from my folder and create everything in the current open scene which is a test scene. I don't want it to mess up with my current project. So, I created this folder and I told it to put everything in this folder first. So, it gave us the plan. Let's review it. Okay, so it will generate the sprite sheet with image generation tool which is correct. In my earlier experiments, the AI sometimes created the UI using some Python script and not using the image generation tool and that resulted in I mean the UI was still good enough but not exactly matching to the reference image. So, I gave it good instructions and stuff and now I think it is aligned properly. The plan looks good. I will just tell it to implement. Let's see what it does. I think it will show us the image of all the UI elements first. It is using the image gen skill which is a part of Codex and it will also review it itself first and if it's not happy with the image, it will try to regenerate it and once it is happy, it will show us the image and then we can review it and approve or make some more changes. So, it has generated this image and I think it looks good but there are some mistakes.

[00:03:43]It created duplicate hmm the play button look the same in both state. We can ask it to make changes. Hmm looks like it has generated what we want. Okay, I think we can continue with this. So, now what it will do is it will use this image and use a tool in this I mean there is a tool for extracting UI elements from an image inside this tool.

[00:04:09]I mean there are multiple tools small tools like that and it will use that to extract all the sprites inside this folder I think. Let's see. And it will use those sprite in the scene to set it up. We will just watch this game scene for any UI changes. And uh by the way, I'm using the $100 plan. Uh I needed that for all these experimentations and it was worth it honestly. I think it will speed up my game development process as well. The $100 plan will give you a lot of usage. I mean that is the case right now. Maybe it will change in future, who knows because these companies are changing a lot. Earlier Claude was better but now it has reduced the usage a lot and it's not usable at all now. And ChatGPT is better at understanding the user what the user wants and it is much more better in terms of UI experience and a full like a user experience you can say. And you can also view all the things all the files all the terminal commands and stuff. For those who don't know coding or don't even want to understand coding, this is very good tool. And even for developers, I mean I am also using it. Now, I am not coding at all myself. I'm just planning better and like when I started, I didn't even wanted to learn coding but I had to because it was a necessity. I don't enjoy writing the code but I enjoy the problem-solving aspect of it. Mostly physics-based coding I enjoyed and that's what I have been doing till now and I think AI tool is much better for person like me because the thing is AI still makes mistakes a lot of mistakes in creative problem-solving. It doesn't give you creative solutions. It just gives you what it knows. That is the case right now but it might change in future. But AI is much better than a lot of developers now. Okay, it has extracted okay. Okay, it removed the background first. See how well it is doing. It removed the background. I mean it is still not perfect. You can notice some pink lines but I think for prototype it is good. I think I might have to write better tool for like extracting the UI.

[00:06:38]Let's see what it creates inside the scene first. Let's wait for a bit.

[00:06:43]>> [clears throat] >> Okay, so this is what it has created in the first try. I think it will try to recreate it in a bit because it also has an ability to take a screenshot of game view and analyze if everything is good or not. So, it has created this UI inside Unity. It is not just images.

[00:07:04]These are actual buttons and these are all the modular UI elements. The placement is not that good but it's still we can just move it around and when I play when I hover, it has the interactable [snorts] element to it just like what we told it to do. Let me know what you think about this tool and what I can add in this tool.