HTML Is All Agents Need — James Russo, HeyGen

[00:00:01][music] >> Since we're talking about generative media, I thought starting with a video might be good. That video was made completely by an agent in a single shot, all utilizing HTML.

[00:00:46]My name is James Russo. I am the co-creator and tech lead of hyperframes at HeyGen. Today, we're going to be talking about why we think HTML is all your agents need to create great videos.

[00:00:58]Show of hands here who finds creating that launch post or that launch video harder than actually building now with coding agents.

[00:01:06]All right.

[00:01:07]Some hands up there. Um Coding agents have democratized building, made it incredibly easy for anyone to create anything that comes to their minds.

[00:01:15]Um however, we think that launching your product or your feature, getting it out into the world, is still quite hard. And we at HeyGen have been trying to close that gap.

[00:01:27]If you're not familiar with HeyGen, our mission is to solve communication through video. We started by creating the best AI avatar on the market.

[00:01:35]You can think of this as the A-roll, the footage you see here on the left side of the screen. Um it's the narrator, the character, the main subject of the video. However, it's kind of plain, and great videos have a lot more pieces to them. They have B-roll, which is the images and the the other pieces of media and assets. They have animations, they have captions, they have music. All of this is needed to create a great video.

[00:02:00]And for us, it's important that we nail every layer, not just the avatar, to solve communication through video and give agents the right canvas to create great videos.

[00:02:13]So, how do you give agents the ability to generate all of these layers and build up this composition needed to create great video? Our bet is on HTML.

[00:02:23]HTML, CSS, and JavaScript are the native languages of LLMs. Most of their training data, every webpage that gets scraped at the end of day is essentially just HTML, CSS, and JavaScript under the hood.

[00:02:34]Uh when you try to teach a model a new DSL or even your own custom JSON structure, it's forcing it to speak another language. Uh we like to think of this as trying to ask Shakespeare to write a poem in Japanese or Chinese.

[00:02:48]Even if you give them a bunch of examples um and teach them a bunch of things, you will not get the best output from them because it's not their native tongue.

[00:02:57]So, why not let the LLMs and agents talk in their native tongue when creating videos?

[00:03:04]We're not the only ones saying this.

[00:03:06]Over the last few months, Tarik, Andrej Karpathy have been talking a lot about how HTML is a new markdown. It's a great output um for LLMs to give you visual representation. For what's it's worth, I submitted this talk before both of these tweets came out. Uh I can show you proof later if you're curious, but we all kind of came to the same conclusion here, which is that uh let LLMs talk in their native language and you can get better output from them.

[00:03:34]It wasn't necessarily an easy thing.

[00:03:36]We've been working on this for over a year now and tried a bunch of things along the way.

[00:03:41]Uh there's a bunch of different uh frameworks and tools to help create great videos. And we like to think about them uh on this axis of quality of results and agent-friendly.

[00:03:53]Um for us, After Effects, Premiere Pro, these are kind of like the gold standards of creatives. They create great output, but they're not very agent-friendly. And even with the more recent connectors into it, it's more of a a co-pilot or something that can help you facilitate things you already know how to do. It doesn't give it creative output.

[00:04:12]We tried things like Lottie and Rive, which are coding uh languages in JSON or custom XML formats.

[00:04:20]Um and they can get you pretty good output, but agent-friendly isn't necessarily true for those because they're not um their native tongue and they aren't as editable or controllable, which is a big thing for us is that controllability layer.

[00:04:36]We played around with Remotion quite a bit and honestly thought it was a great example of what LLMs and agents could do with coding. Um however, we noticed that we had to teach them the framework. We had to teach them the language and give them a bunch of examples on how to write code properly, which ultimately took away a lot of the creativity of it.

[00:04:57]Um then we have HTML, which around November of last year when Gemini 3 and the latest models came out, we saw a step function improvement in what LLMs could do. When we just gave them examples of what the output we wanted was, they naturally gravitated toward HTML, CSS, and JavaScript and gave us great output. Um and we decided, let's not fight the model, but find a way where we can let them talk in their native tongue.

[00:05:24]So, how we did this was starting with a very small model, Gemini 3 Flash, as our design partner. We knew that if the smaller models could author workable code in a framework, then the larger models and these coding agents could 100% do it as well.

[00:05:42]We tried with a bunch of different wrappers around HTML, CSS, and JavaScript, adding in a lot of context, and making our system prompts bigger, adding in skills. Um but to our surprise, the thinnest wrapper ultimately won, which is essentially just HTML at the end of the day with a few data attributes as metadata to let the agent know, and to let us know uh about timing and things like that.

[00:06:05]Um our Yeah, as I already mentioned, our thinking was very simple. If this smaller model could get it, as the models got better, it would continue to improve. They would naturally understand it more as well as training data came into the picture. Um and this is when we knew the format of what we were building was right.

[00:06:24]Here's a little bit more detailed example of what a video looks like uh in hyperframe. So, here is on the left just HTML um with a few attributes, and on the right, the web page that it renders.

[00:06:37]Um the preview and the render are all done in the browser. The same pixels that the browser sees is ultimately what your video is going to see.

[00:06:45]Um and anything any web page that your LLM or agent knows how to write, it can now write into a video.

[00:06:54]This gave us hyperframes, our open-source framework that turns your agent's HTML into video.

[00:07:03]So, now that we had the HTML and the language part of it, um the next hard part was how do we actually turn this into a deterministic MP4 video that anyone can post or utilize. And this was a lot harder because browsers are async on purpose. Um they have a different set of requirements and concerns to video rendering. They need to work across a bunch of different networks and a bunch of different uh constraints.

[00:07:28]But, that's not true for video. So, on the left you can kind of see here what a a browser render might do. It's okay to loading things asynchronously like fonts. So, as the page loads in, the font and style of the text might change.

[00:07:42]Images and videos and other assets can load in asynchronously as well. Um so you might not see them initially on the page, but they'll come in at some point.

[00:07:50]Whereas for video, we need everything on the page 100% of the time so that we can show you exactly what you expect in the the video.

[00:08:01]How we do this in hyperframes is um essentially freezing the clock and seeking frame by frame. For those who aren't super familiar with video, uh basically a video is a series of frames or images strung together to create motion across an entire video.

[00:08:18]So we took this insight of ours and basically applied it to our rendering of HTML as well.

[00:08:25]We freeze the clock in the browser and then we seek deterministically to every single moment in time or every single frame, uh wait for everything to load on the page, ensure that it's loaded and ready to go, and then we take a screenshot and move on to the next frame, and do that over and over again uh until we get all of the necessary frames to encode that into a video. So the same input that is previewed in the browser is also rendered into the video.

[00:08:52]Hence the name hyperframes.

[00:08:56]The power of this is that anything you can render in a browser essentially can now be in your video.

[00:09:01]Things like 3.js, charts, SVGs, shaders, WebGL, WebGPU, Lottie, all of these are renderable in the browser, and therefore all of them are renderable in hyperframes.

[00:09:13]This is a big part of how we create a lot of our videos is finding inspiration on the internet, taking these as examples, tweaking them to our needs, and then putting them into our videos.

[00:09:25]The next part outside of the actual framework and rendering is how do you get great or good output out of the agents. And a big part of this is the skills that we couple with our framework.

[00:09:37]Our skill is focused on taste and video aspects because the LLMs and agents already know how to write HTML and CSS and JavaScript, we don't have to teach them the language, we just teach them how to create good videos. This is a big difference between other frameworks where if you look at certain skills, it's really just like how do you write something in that framework?

[00:09:57]Um this allows us to focus on the important parts of what makes a great video and a big part of this is constantly evaling and using agents to improve them.

[00:10:07]Uh this allows us to raise the floor of videos and ensure that the base output of a single shot prompt gets you pretty good results.

[00:10:18]Here's an example I'm going to show of a single shot utilizing our website to video skill.

[00:10:24]All you have to do is give it a website a We tell it how to get to that website, retrieve all the assets and information it needs uh to create that design and theming and branding using DesignMD or FrameMD in our case.

[00:10:38]Um and then from there we just teach the LLM basic motion examples and things like that of great videos that you can then go ahead and get good output like this.

[00:10:54]However, great output takes craft.

[00:10:57]Similar to AI coding, you can get decent output by just giving a single prompt and having something that works for your needs, but getting great output from agents requires craft, taste, the same principles of any uh software engineer before AI coding as well.

[00:11:14]Breaking the problem up into individual pieces and working iteratively.

[00:11:18]The same is true for getting great output for HyperFrames videos.

[00:11:22]Um we want to make sure our power users have full control over this and the same way we created videos before HyperFrames is how we create them with HyperFrames.

[00:11:31]We think about the narrative and the vision and the mission of this video. We storyboard it frame by frame and think about what each frame needs to do. We then move that into adding motion frame by frame utilizing HTML, CSS, and JavaScript, merging it into one cohesive video, um and then utilizing our studio that we released with Hyperframes for that last mile editing. Ensuring that humans are always in the loop and have access to do anything that they would do in their normal video editor within the open-source framework's editor so that you can manually drag, tweak, uh etc. And that gets you output that looks like this. Uh we actually just released this today, keyframes in Hyperframes, uh which is a big aspect of what makes great videos for motion designers, and it allows you to basically coordinate all of this different motion uh frame by frame or keyframe by keyframe um in our studio and make sure that you can do anything that a professional motion designer might be able to do in After Effects.

[00:12:35]So, it's not just a demo. We've been uh we released Hyperframes a couple months ago. It's been running at scale in our opinion. Over 1.3 million videos have been rendered by open-source users of Hyperframes in the last 90 days.

[00:12:47]267,000 creators have tried it. We have about 15,000 videos every single day being rendered utilizing the open-source framework and 32,000 GitHub stars. Um but this is only just the start for us.

[00:13:00]As I mentioned already, it's open-source, free forever. Anyone can go ahead and use Hyperframes right now. It works with any coding agent that you have, Cloud Code, Codex, Cursor, any number of other ones up here, any ones I'm not aware of even. If your agent knows how to write HTML, CSS, and JavaScript, it knows how to create a Hyperframes video.

[00:13:19]Now, with this, the same agent that is helping you create your product can also help you make your launch video.

[00:13:28]The one honest thing that we're going to say here is that the models still aren't good at creative work. We spend a lot of time evaling and trying to improve this in our skills and push it even further.

[00:13:38]But we think there's something at a higher level that needs to change here, which is why we started to work on a code to video benchmark where we are trying to work with the LLM labs, any creators who are working on video agents to ensure that we can raise the floor of videos for everyone. If anyone is interested in this space or working on this, we'd be happy to be collaborators, talk more about this. Feel free to find me or any member wearing a Hyperframes t-shirt in the crowd afterwards. Love to chat to you guys more.

[00:14:09]And one more thing, some of you may have seen the AI engineer Warfare showcase video. We collaborated heavily with the team to create this with them utilizing Hyperframes as a little surprise of what it can do.

[00:14:22]Here's a great example of the motion graphics and designing that you can do with HTML, CSS, and JavaScript.

[00:14:31]So yeah, agents are made building incredibly easy. Launching is still quite hard. We think HTML is all your agents need in order to make great videos and launch your product into the world.

[00:14:42]Here is a link to Hyperframes, the project, the open source repo. Feel free to check it out, star it, download the skills. You can reach me on X at Reems_Juso, and thank you for attending. I'll be outside if anyone wants to chat.