A Frame’s Life: Frame Timing, Synchronization, and Latency in UE | Unreal Fest Chicago 2026

[00:00:01]Hello everyone, welcome.

[00:00:05]This is a frames life frame timing synchronization and latency in Unreal Engine. My name is AI. I work in the technical developer relations team here at Epic Games. And uh yeah, I make presentations like these so you can make games better.

[00:00:20]Um I do want to point one thing out for this presentation is that I'm going to be talking about some stuff that are currently in flight. They are. We're currently working on them and I hope this video is going to be out of date very quickly, but I have a text version available on the Epic developer community that I just posted right now.

[00:00:40]So, it's it's just right now it's just copy pasted text on the slides, but like give me give me a couple of weeks and I'll fully fill it in and then soon as any changes happen to any of the systems that I'm going to be talking about in this presentation, uh I'll update the test article. So, the test article will always be up to date. in this video will not.

[00:01:00]So a frame's life basically from input to photon and this journey well it has you know how we measure it has a bunch of names we can call it input latency system latency end to end system latency click to forum latency motion to foreign latency this is all generally talking about the same thing I'm going to be calling it input latency in this presentation but it has a few different meanings I'll elaborate a bit later uh there's a bunch of things that happen from user input to fon and uh these boxes are only proportional to how the text fit in there. It's not actually like how long they take and in this presentation I'm going to be focusing a lot on the latency that we as developers can affect. So when we are making a frame in Unreal Engine uh the engine needs to do a lot of things to draw a frame. We have the gameplay code animation physics and networks that need to happen. We need to prepare the render scene call uh the object and make draw calls. We translate the rendering commands into specific graphics API like um DX 12 Vulcan metal. Even though they all work similarly, they're different enough that we need to have a translation layer between each of them.

[00:02:07]And then this uh graphics API uh we ceue up GPU work by giving them commands basically saying, "Hey, I want this to happen." Basically, a draw call on GPU and then this and then this and then this. And the GPU basically does them at its own leisure at its own cadence. Uh we can of course sync to it with fences.

[00:02:27]Um and then the GPU does it work and then we finally call present on the graphics API to encue a swap chain flip and uh when the GPU finish this work it will do that.

[00:02:39]So swap chain I want to talk a little bit about that just so that we're all clear on the foundation. So the game has a surface otherwise known as technically a buffer that it has on the GPU that it draws to. This like you can basically think of it as a texture or canvas that it paints to and the GPU scans out the buffer to the display and the buffer that is currently being scanned out. We call it the front buffer. The thing is we do not draw to a buffer that is actively being scanned out because then like as it's being scanned out suddenly objects would just pop in. So we lock it and then the game has another back buffer uh sorry another buffer that we call the back buffer that it draws to while the front buffer is being scanned out. So this is basically what we're talking about when we say double buffering. So Unreal games use either double buffering and that's the default for console and mobile or it uses triple buff buffering which is the default on PC. And of course you can change those defaults.

[00:03:33]You can use double buffering on all or triple buffering on all.

[00:03:38]And the game can wait until vertical sync or VSYNC happens to swap them for a smooth transition. And VSync is signaled at the end of a frame scan out. So we have the front buffer is being scanned out like this. And then the GPU goes VSYNC. And during that VSYNC is when we swap the buffers around and then we're ready to render fra scan out frame number two. But we can also swap them out without waiting for VSYNC. Uh but that introduces tearing. So we can see here front buffer number two starts getting scanned out and then we just sweep them in the middle of a scan out and then we get half of frame two and half of frame three and then as an extra step it is up to the operating systems compositor whether the buffer gets scanned out directly to display or composited first.

[00:04:23]Uh I'll get to that compositor but I also want to talk a little bit about VSYNC. So generally when you turn off VSync via R.Vsync Vync equals zero. It does result in lower input latency, which is why esports and competitive players do that a lot, they don't like that. The cost is screen tearing, which while uh if you're moving the mouse fast enough or the controller, it can look like this because you're basically flipping the uh the buffers right while we're scanning out, it's also uh gives us uh less of an ability to absorb hitches by the pipeline. I have a whole section on that, don't worry. So, most console games actually have vertical sync equals one. And also many users, me included, prefer no tearing. It's very like jarring. I don't like how it looks.

[00:05:05]Because of that, I would recommend that if you're making a game uh and you're trying to decide should you have VSync on or off, I would say prepare uh have it on by default and and make it an optional thing that the user can opt out of.

[00:05:20]Or just have it on and just make your frames. Okay, so VSYNC and insights, you can actually turn on R.Vsync information insights to get VSYNC markers and insights and it will show you the flipped frame number and misses as you can see over there. And this is the number that we're using throughout the entire pipeline like the game thread, the render thread, RSI thread, they all have this like frame number. This is that frame number. And if you make your VSync, it will show up there. And if you miss a VSync, it will show up as missed.

[00:05:50]>> It's uh I don't like the bookmarks. So, we actually do have a new feature coming soon. I'm really excited about that. We have a dedicated track for VSync and it looks like this. Super sexy. You see green uh frames there for the VSyncs that you make and then oranges redish for the ones that you miss. I don't know when it's coming, but hopefully soon we have a little bit of refactoring of insights to be able to have arbitrary um game frame missing tracks like that.

[00:06:19]And I mentioned the compositor. So uh it basically does what its name implies. It composits windows together uh into one graphics buffer before sending it to the display. So for example, you can have a game running, you can have a in window mode, you can have a video player and then a browser all in your uh on your computer and then basically what the operating system does is that it paints it all into one texture, one buffer before scanning it out to the monitor.

[00:06:45]And compositing is not free. It's work that needs to happen and it usually takes an extra frame. It's like uh like the VSSync is fired on your game and then you uh make it, but then the operating system waits one extra VSync before sending its own. Uh so it takes at least uh one frame, usually just one frame. The thing is if we're running in full screen game, we don't need any compositing because we're the only thing being drawn out. And uh on Windows, the compos compositor is called the desktop window manager. And consoles do also have a compositor, but you generally don't need to worry about it because most of the time if nothing is overlaid, it will skip it. So you don't have to worry about it. Consoles is just like don't worry about it. So but on Windows I think we really want to skip this compositor.

[00:07:28]Historically only exclusive full screen skipped the Windows compositor. But fun fact, as of Windows 10 and Dex 12 at least, I think it might have even been some part in Windows 8. Anyways, uh with the flip model shop chain, which is what Unreal uses, I set default, but it's just like Unreal just works like that. You can change it. Um it will automatically skip the Windows compositor if you are running in borderless full screen, but not always. It is automatically activated by Windows when it detects that the game covers everything on screen. So, it's quite cool. So, just you're running in borderless windows.

[00:08:04]Guess what? It's performing just as well as exclusive full screen. Also, it is always activated when you have hardware multiplane plane. Yeah, plane pain.

[00:08:14]Multipane. I can't. Oh, Did I get it wrong? Sorry, I was. Anyways, uh supported by the hardware. I think it's multiplane plane. It's planes. Anyways, um even when it's windowed or covered by other windows and I noticed on my work computer, uh I have an Nvidia card there that the overlay support is only on the primary monitor as set in display settings. So, if you're a gamer watching this and you game on your secondary monitor, stop it. You're getting an extra frame of lag of input lag uh on Nvidia because they put all of their overlays in uh on the primary monitor.

[00:08:52]You you can check whether like because I I know it works like that on at least my Nvidia card. I don't know if it works like that on AMD and Intel, but you can just download present mon which is Intel's free tool for uh checking a lot of things. And there's this custom mode that allows you to put in percent mode stand and will tell you whether it's like which window or which compositing mode Windows is currently using for your game. But in either of these cases, if uh the compositor is being skipped because of the full screen thing or because of the multiplane thing, um you can like it's basically performing as well as exclusive full screen and it's more usable because you can still all tap out of it and you can have tempor temporarily uh composited overlays like the volume bar. If you're clicking volume and then suddenly it comes goes over and then uh if you don't have the multiplane overlays then it's going to uh just temporarily composite and then stop it when it's again the only thing on the screen.

[00:09:50]So exclusive full screen actually fun fact Windows apps can't really do exclusive full screen anymore at least the games with the flip model. Uh Windows is lying to you. It just emulates exclusive full screen by still changing the resolution which is maybe what you mostly wanted but it still uses borderless full screen. So yeah, and if it's the only thing again on screen, it will skip it. And usually when you're using nonnative resolution, when you're sending a lower resolution to the monitor than its native resolution is, then you're making the monitor do the upscaling. And the monitor upscaling is it's usually a very cheap and plain and fast spatial upscaler like bilinear. And when I say spatial upscaler, I mean only using data for the current frame as opposed to temporal upscaler which uses many frames.

[00:10:41]Unreal can actually do better quality fast spatial upscaling on the GPU with fancier algorithms. Then we do it with via the secondary upscale secondary upscale. So Unreal does have two upscaling stages.

[00:10:55]This is a graphics from uh the demo presentation. So the uh numbers are actually from it also. I say resolution methods are examples but it's actually what we used on the wizard demo. So primary upscale is for the main temporal upscaling like temporal super resolution Nvidia DLSS and all the other plugins are going to hook into that one. But there's a secondary upscale that I don't think that many people know of. I didn't know of that well when I was researching this. Uh for optional spatial upscaling and you can set the method that it uses, the algorithm that it uses via the art.cale.quality. You can check the SE for help. Just write it plain into the console and it will give you documentation or see the source code. It gives you quite a lot of info for all of the methods it uses. And the cool thing about secondary upscale is that Unreal will draw the UI at native resolution after the secondary upscale. So the UI will look super crisp and nice even though you just upscaled everything else. It's disabled by default. So I think most games aren't using it.

[00:11:54]Although it it does kick in if Windows's DPI scale is more than 100%.

[00:12:00]Uh but in any case, you can set it with the R. Secondary screen percentage game viewport zar.

[00:12:11]I think because of this because the whole native UI better algorithms are fast, I think secondary upscale is a better alternative to exclusive full screen. So, I think that should be like a mode like if you're offering only borderless windowed and like exclusive full screen, which is not real anyways.

[00:12:30]Um, like if you want the player to play, if the player wants to play at a lower resolution to make it faster, just use the secondary upscaler because it's it's going to be just as fast but look much better.

[00:12:41]I say that but don't like maybe not completely replace exclusive full screen because there is the use case scenario where users have a laptop that doesn't have a great GPU but it has 4K display and in that case because you're running on a 4K buffer it's going to be memory bound so there's exception anyways I told you all the facts you can decide yourself how you want to do it so the Unreal pipeline there is a lot of thing we need to do in a single frame. We have the game simulation rendering operations, RII and GPU rendering and they all need to complete before a frame VSYNC. But in Unreal, we don't do them linearly like that. We kind of like parallelize them and they intersect a little bit as they're sending data over.

[00:13:25]When we do things like this, we can do something really cool. We can actually spend like more we can more than double the time we spent on each stage. If we zoom out a little bit, uh even though we're taking like more than two frames to construct our entire frame, because they're all running on threads, we can parallelize them like that. So each is taking more than one frame's time to complete, but they're still running at your target uh refresh rate, like 60 frames per second or 120 or 5,000, whatever. Okay, but what happens if you pause the game and suddenly the gameplay code isn't running and the game thread just like takes almost no time and it just does just start racing ahead like that.

[00:14:10]We need breaking in the pipeline. So each part of the pipeline needs to be able to break to not run too far ahead of the rest because we don't want the game thread to be at frame like 1,000 while we're still rendering frame number one. So let's start with the GPU and work our way backwards to the game thread.

[00:14:26]So the GPU as one of the last thing it last things it does is it writes to the swap chains back buffer. So it needs the back buffer to be available and it actually does it in the last postp process pass. Uh and if it's like saying I I want to write to the back buffer now because I'm going to show it to the user soon. And if all the back buffers are full, it just goes like well I'll wait and it stalls there. We don't really have a marker for that on PC. So if it happens, if you're VSync bound, we don't say we're waiting on VSYNC. It just uh looks like a bubble. It looks like the Yeah, like it will just say nothing.

[00:15:02]Sorry about that. Uh I told the team I want a marker there. We have it actually on some platforms, but NDA stuff. Um like I said the last postp process pass is the one that writes to the back buffer and whichever pass that you see in the GPU bubble uh like for example here I turned on RG events it's just the previous pass it actually finished already and it's just waiting on the back buffer. So here it's lying it's saying I adaptation buffer took 12.72 millconds.

[00:15:31]Uh it's been 12.72 millconds since we started it. We did finish it. But actually over there at the end like that's that's where uh you can see that we were waiting on the tone mapper was the last uh stage in in my case that's what we were waiting on and it's only in the tone in the like once we actually are doing something that we pop the previous thing we were doing. So it's a little bit you just need to know it.

[00:15:54]Sorry. Um for R I thread it waits for the previous GPU frame to finish. Um it's called sync point_we fence in inside. So you can see there um the GPU frame just ended and then it goes and signals the RI interrupt thread which is basically the thread that handles communication between the GPU and the CPU and then uh when that is fully handled it will end the fence. So this is how you see oh why is it waiting in insights. I think it would be cool if we just click it and we show these arrows automatically. I don't know maybe I'll look into it next year if I have time. Um but the RSI thread also waits on the graphics API when the subchain fills up. For example, like uh let's say there's nothing in chain. You can start sending commands to uh to the subchain through the graphics API.

[00:16:41]Sorry. Uh yeah. Uh like hey do this draw call this draw call this draw call then present and then do this draw call this draw call. You can send all of these before it even has started working on the first draw call. So you can start queuing up work. But as soon as you call percent more often than we have back buffers available, the graphics API just says stop. Uh let's not get ahead of ourselves. So direct x for example will block during present when no back buffers are available. And you can see there actually that as soon as vync is done um directx goes like ah we have one available now and it uh resumes render thread. So the render thread waits for the previous TPU frames hardware occlusion queries.

[00:17:26]This is shown as process visibility tasks on the render thread. And it's also shown as sync point weight GPU occlusion on the worker thread. And if you look really close, you can see that it says GPU bound waiting for GPU for occlusion queries underscore CGPU track. Because people were seeing this and they were saying, "Why is occlusion taking 15.5 milliseconds?

[00:17:52]That's a lot of work. Is occlusion that slow?"

[00:17:56]It's it's it's waiting. It's not doing anything. It's just waiting for the GPU to finish it previous frame. So, it's not actual work. Actually, uh these hardware occlusion queries are also only for nonite messages. So if your game is almost completely nanite and you don't want this to show up in your uh traces, you can turn them off with rallow occlusion queries. Again, this is only for uh nonite. Nanite does have occlusion queries, but it's only on the GPU. It doesn't need to make a round trip.

[00:18:22]But profile like um don't turn them off and then suddenly everything is slow.

[00:18:26]Also kind of like take into account like I mentioned, it's not busy. It's just waiting. And if you like remove this weight because you're like I don't want it to be slow. Guess what? It's just going to wait somewhere else because of the frame pacing because all the brakes.

[00:18:36]So uh let's see a closer look of how hardware occlusion queries looks on the profiler. So we have on the GPU uh you can see the begin begin occlusion test and then we get a bubble means the GPU isn't working. We're just sending the data now and it's a lot of data. So we're sending it there to the RI interrupt thread. It uh when it's done it closes the fence. It handles the data on worker threads and then it uh tells the arts a render thread like hey you can stop waiting now and over subscription it's uh a lot of people have been asking about that and I I just want to clarify what is how it works so unreal engine usually when you uh when you make a game start a game it will allocate as many threads as there are hardware threads on the system so like uh basically the C CPU cores and then maybe times two if you you know, hyper threading or equivalent. And sometimes when a worker thread needs to wait for something, it can allow over subscription. And over subscription is basically a really neat way to tell the engine, hey, I am a thread that basically has its own CPU core allocated to myself, but I'm just going to be waiting now for a while. So, we're just wasting one of the CPU cores. So, how about you just wake up a temporary standby thread and do some work there? So, we are utilizing all the core. So this basically keeps all the cores busy even when I work a thread idles while waiting for something. So if you see in your insights that there's this over subscription and you think uh what is busy taking up my CPU time it's not it's just waiting it's just a fence it's a fancy fence okay and uh if you disable them because you're like I don't want this wait this or subscription in my pipeline then uh the render thread only waits for new work. It's basically start for work from the game thread.

[00:20:29]A game thread is the last piece of the puzzle. So the game thread always waits for the start of the previous frames render threat. It also is getting a little complicated now. It also waits for the two frames ago RHI threat. And this fence is to prevent the game and the renter thread from raising ahead together. So it's connecting it to the rest of all the like uh breaking stuff.

[00:20:52]And you usually won't hit this fence.

[00:20:54]You will never see it. uh because by default you have the hard reclusion queries. Um but everything that I just described all of these breaking and stuff this is what Unreal does by default. This is r.gamethread sync type zero.

[00:21:10]Let's visualize it because I don't know about you but this was a lot of complicated words and it's okay if you kind of disconnect it for a little while because I did also. So let's visualize it. This was the last frame.

[00:21:24]We can start the next game threat uh frame already because the previous rendering operations frame has already started. We can start it as soon as the rendering operations start. So we can already start it. We want to start doing the rendering operations but we can't yet. We barely start it but we have to wait for the previous GPU frames hardware occlusion queries. So that is what it's actually waiting for for when you see that over subscription thing.

[00:21:49]And if it's waiting too long and you're like missing uh dropping a frame or something like that, it's because your previous CPU frame was just taking too long, then uh it does the RHI thread and uh that frame actually it it's bas it's basically done with all this work and then it waits until the end of the GPU frame just so that the next RHI doesn't start immediately and then do GPU rendering. Okay, let's do one more frame.

[00:22:15]It cannot start yet because the blue the rendering operations from the previous frame hasn't started yet. So we need to wait for that. And now we can start the game simulation along with the rendering operations. Usually when you uh are running insights and you see game thread takes something and then it says like waiting for tasks. That's it. This is it. Then again it waits for hardware occlusion queries. You can see there's a big weight there because the previous frame took quite a long time. and then it finishes the rest from the start of when the game simulation uh frame starts and to the vync of that frame. That is what we in Unreal uh language call input latency.

[00:22:58]It's basically when we consume the input to when we uh flip the frame. [snorts] So I want to talk about input latency.

[00:23:04]Now we uh we have our own description for it in Unreal. Generally, when we're talking about intellency, it's the time for when the user actually pressed an input or like did some motion with VR to when it was actually when it hits their eyeballs, you know. But in Unreal, uh the input stat over there when you have stat unit, it's actually the time the input was sampled, not when it was pressed, not when the operating system sent it to Unreal, it's at the start of the game frame to when that frames back buffer was flipped to the front.

[00:23:37]For example, here if the user inputed something during the previous frames game simulation that wasn't uh consumed by that frame, we don't count it. Input latency is still from the start of the orange frame.

[00:23:49]So the input latency, you know what it is. So what it isn't is that it isn't the hardware latency. It's not the operating system. It's not the delay if the user presses it before the consuming of the game frame. Uh it's not the compositor delay. It's not the scan through the HDMI cable. It's not the display's processing time or its own scan or the pixels gray to gray.

[00:24:13]I mentioned hitch resiliency. Let me describe what it is. It's basically the engine's ability to absorb hitches and it does cost greater input latency. So, it's a give and take. And also, the game needs to be under frame time budget so we can actually catch up again and fill the pipeline again. So, I'm going to give you an example because that was a lot of words and I'm a visual creature and I hope you are too. Let's say we have a full pipeline. A full pipeline in Unreal looks like this. Six of the frames are in the engine. Two of the frames are basically in the hardware uh like on the way to the monitor and on the monitor. So, let's say frame number one, the first frame that we simulated is currently being drawn on screen. So, display scan out. Frame number two, it's being sent from the GPU scanout engine through the wire. It's being sent now.

[00:25:00]Frames three and four are just waiting in the stock chain's back buffer. Frame number five, it's almost finished on the GPU, but we're waiting for VSYNC because we don't have a back buffer and we're waiting for it. It's the bubble that I told you about. Frame number six is almost finished on the RHI thread, but it's stalled on the graphics uh API's percent call, the VSync that I told you like direct.

[00:25:22]The uh render thread is waiting on the hardware occlusion queries from the previous CPU frame. And frame number eight game thread is done. It is waiting on the previous render thread frame before starting next frame. This is what a full pipeline in Unreal looks like. Uh six of the frames are like the real frames. The two are kind of like it's out of our hands and we cannot start frame number nine yet. Let's say now frame number one is done. We're doing frame number two. Vsync happens and everything moves up a little bit.

[00:25:50]However, frame number nine is taking a little bit of time. We actually go over frame time budget. Actually, let's say we go over two frames frame time budget.

[00:25:58]Actually, you know what? Let's say we go over three frames frame time budget. But in this hypothetical scenario, our frames are fast enough that are able to catch up because like we still there's two full uh frames in the swap chain.

[00:26:12]Like we have like over 100 millconds there to catch up. So, we finally finish game thread number nine. And it's because we can basically skip the back buffers because like we just take the first one that's available. Uh we're able to kind of catch up and notice that the user at no time missed the frame. So we had a hit but it wasn't a real hit because we didn't drop a single frame.

[00:26:34]The engine kind of like like stutter a little bit but we managed to catch up before anyone could [laughter] notice.

[00:26:41]So you can reduce input lenency uh by sacrificing some hits resiliency. So this is great if you're able to keep consistently under frame budget and you know you don't need that hitch resiliency. So now I want to talk about reducing input latency which should be very important to us as developers because you know sometimes we have action games and we want the input latency to be really low. So of course the first we can do is just run at higher frame rate. Super simple um because the max number of frames in the pipeline stays consistent across frame rates. Like I mentioned six frames uh in the engine. So the frame time is not uh that one is variable. So, for example, if you have six frames in the pipeline at 30 frames per second, that's 200 millconds. So, if you're running on like, you know, limited hardware like mobile uh or a handheld and you're running at 30 frames per second with GTS sync type zero, it's 200 millconds is quite a lot of input latency. But if you're running at 60 frames per second, 100 milliseconds of input latency. Let's say you can even run your game at 120 frames per second, 50 milliseconds, still with a full pipeline, still maximum hits resiliency, still maximum parallelism.

[00:27:49]I said it was simple. I didn't say it was easy.

[00:27:53]You can also throttle the game thread.

[00:27:57]So, you can see the impleency there. is quite long and we want to shorten it.

[00:28:03]But wouldn't it be cool if we could just like move when the game simulation happens. So we're not doing like we're not doing less work. We're just moving when it starts to cut the input latency in half. That's kind of cool. It would be even better if we can make it so that just before the VSync we can kind of start the work so that it would finish just before it end. So this would be basically perfect frame pacing because we start the work so that by the time the GPU rendering ends the ving happens.

[00:28:32]This is the this is the least amount of input latency you can get in Unreal and we would be able to achieve this by throttling the game thread. So there's a way to do it in Unreal. We have a fixed way of throttling it. You can use game thread sync type two to kick off the game thread at a fixed time before vync.

[00:28:53]>> [snorts] >> So by default it's always two frames but then you also get the uh sync slack milliseconds which defaults to 10 milliseconds u and it's maximum one frame. So if you're running 60 frames per second you can't put it higher than 16.666 milliseconds meaning that you can have the throttle between two and three frames before the target VSYNC.

[00:29:17]You decide the frame budget you need to keep to it. So this gives you very consistent delta time, minimal input lag, no hit resiliency, but it's okay because your game is not hitting, right?

[00:29:28]Um, you should use this for consoles where the hardware is fixed because you need to decide the frame time. So it's currently available on PS4 and 5, Xbox One and series and Switch one and two.

[00:29:40]And you kind of want to do it only on when you know what hardware it's actually running on because it's harder to keep to a fixed budget on variable hardware like for example on PC and mobile where it's like you don't know what you're getting. Uh so we don't support it also because we just don't have those uh timings. And this is what it looks like. You have two frames that you need to wait at least and then by default sync like milliseconds is 10.

[00:30:04]But so you can see like oh there's a little bit of spare time there because this what is set. have a little bit of buffer, but if you can't keep to your frame time, if it goes like this, just a little bit over, guess what? You dropped the frame. So, you really need to keep your game under budget.

[00:30:20]There's also an adaptive way to throttle.

[00:30:24]We have third party solutions that allow you to implement adaptive or dynamic frame throttling on PC. Those are Nvidia Reflex, AMD Antil 2, and Intel XE low latency, otherwise known as Intel Xeal, or maybe they call it Intel Excel. Ah, probably not. Um, and they dynamically adjust the throttle based on previous frames. It's a bit of a headache having to implement three SDKs just to make sure your game runs like uh with lowest input latency on uh PCs, but your users are going to love it if you if you can do that.

[00:30:56]I was considering if I should put this slide in or even not. You can, if you want, starve the pipeline. So you can set t.m dom maxfps to a value slightly lower than the display's native refresh rate. For example, if the display runs at 60 and you put max fps 25 to 59 and 58, you're basically simulating as if your game was underperforming. Uh so this completely drains the pipeline because your games aren't running fast enough to fill it up. Zero hits resiliency because again, uh lowest dependency, but the bad thing about it is that it drops a frame or two every second because you're basically telling the game thread like, "Ah, wait a little. Okay, now do it. And then the game is like, I don't have enough time.

[00:31:30]Ah, I haven't dropped a frame. So, well, unless the user has a variable refresh rate, then they can kind of like make a little bit of uh delay there. [snorts] So, in the gamers that are watching this presentation, you guys can actually do this already because um there are some games that offer unrounded max FPS settings in their uh settings game. So, they can say like, oh, max FPS is 58, 57, whatever. And this will actually start the pipeline and it'll get the lowest input latency. But us game developers shouldn't really do it because it is a workaround than a solution. Uh very unprofessional. Don't do it. Rather just, you know, get your frame pacing right. Um but yeah, like I've seen esports players or just people that really want like low impedency in their single player games to just download a mod that like injects the C into their game. So like you know uh like I said I wouldn't recommend this but also I'm not too mother. So you know you can do it now. Uh I would actually recommend just shortening the pipeline.

[00:32:31]So if you still want to retain sub hits resiliency but just shorten the pipeline a bit you can use the art one frame threat like cir by default it's one and that's that the render thread can lag behind the game thread by one frame and that's when it can run both at the same time. Really nice maximum parallelism.

[00:32:46]If you turn it off, the game thread needs to wait until the current frames render thread is done before continuing.

[00:32:53]It's less parallelism because now the game thread and render thread can't overlap. I'll show you what it looks like. Um, here we have the situation from before and this is the default R1 frame thread like equals one. Let's turn it off to zero and this is what happens.

[00:33:08]Basically, game simulation needs to now wait for the rendering operations. So, this is how it was before. we just push it. Ah, sorry. Game thread and rendering uh thread cannot run at the same time.

[00:33:18]The bad thing about that is we're basically losing an entire core because two of the threads [snorts] are just they're never running at the same time.

[00:33:25]So like one of the core and the hardware is just going to not do anything. So maybe if if you know this, you can just like there's you can override how many worker threads it spawns and add one more. I don't know. Uh anyways, you can even shorten the pipeline even more uh with game thread sync type equals one.

[00:33:43]So now this waits for the previous RSI RHI threads frame instead of the two from before.

[00:33:51]Uh even less input latency, even less parallelism and even less hit resiliency. This sounds really so I'll show it. I'll show it. Don't worry. Uh so instead of waiting for the rendering operations we just wait a little bit longer for RHI. So the for the previous frames RHI. So it went from this which is like ah we have like basically those two threads can run at the same time and now we have like basically like almost three frames that can't run at the same time but we uh push the game simulation thread out even more.

[00:34:26]I do want to say sorry this is this is not this is not nice it's not proper like shouldn't it just be easy we do have some future plans uh we are reworking frame pissing in Unreal Engine hopefully it should be out soon uh the plan is for a new model with adaptive frame throttling with adjustable target stack so it works a little bit more like those PC methods like the Nvidia uh AMD Intel things what they are doing but just hardware agnostic and like also not on PC logo on everything uh So this would give you still shortest in valency but maximum parallelism and consistent delta time and this is what it would look like will look like once it's ready. You set the target slack over there. That's the only kind of thing you have control over but otherwise everything else is adaptive. So we base it on the previous frame. So you say I want to have like in this case five to seven milliseconds that I want to have as like kind of like a target slack or target buffer because I know that the frames can vary a bit. So I want to not drop a frame if it goes over it. And then if you have enough work that it goes a little bit over that next frame the adapter system will just increase the time a little bit. So just we just want this to be always for everything and the default target Slack to just be like so you don't have to worry about it. Uh but so if you're watching this from the future and we already have this like lucky you good for you. For the rest of you but this also gives us maximum parallelism because we can uh start off the next game threat frame uh like that.

[00:36:00]Yeah. So, I talked a little bit about variable refresh rate, [clears throat] also called adaptive refresh rate, presync, g-sync. Uh, it used to be vendor specific, so that's why we have all these names, but now it's vendor agnostic. It's part of HDMI standard since 2.1. So, if you're buying a monitor and you're wondering, uh, which one should I get? It's all the same. Um well, sorry, don't sue me. Uh vendors because of course, uh they do their own um there's extra stuff now. Okay, lawyers will get off my back now. Um no, no, they do extra what do you call it?

[00:36:39]Like vouching for it like oh this monitor is great. Okay, so variable refresh rate only helps when your game's frames per second can't keep up with the monitor's native refresh rate because it's basically works that so that instead of you just barely dropping a frame like I showed in uh graph before uh the GPU can just delay the start of the next frame. So if it doesn't have a frame yet, it's like I'll wait a little bit longer. Oh no, I got it. Okay. And it's like almost imperceivable to the user because we didn't drop a whole frame. So it's just a little bit extra.

[00:37:08]You can only hold the frame for a limited time though. So like I think nowadays they can like almost triple the like they can if one frame is 16.66 millconds it can wait like two extra uh it varies per um monitor. Don't rely on it anyways. Uh but if if we run out of time the GPU just resends the previous frame. So now as developers do you know what you need to do in Unreal or on a PC to enable VR for your game?

[00:37:34]Trick question. Uh just turn on VSYNC.

[00:37:36]Everything else is handled by the uh user's hardware and also oh have a little bit crappy performing game so that VR kicks in to save your ass. Okay, measuring I want to talk a little bit about the numbers inside of Unreal when you do stat unit and stat FPS.

[00:37:52]The frame time it's a little bit misunderstood. Some people think it's like the how long the entire frame took and stuff like that. No, no, it's literally the wall clock time between the start of game thread frame. So if game thread frame number two uh time and game thread number three time and it just minuses them and that's that's the frame. So it's only on the game thread game the red one underneath that. It's the same time minus idle time. That's the difference between those two. Draw is the same just for the draw thread and also swap chain percent is included in the idle time. the RXIT same um GPU it's just the time the GPU spent doing actual work. So we have a union between all overlapping work basically like the time that the GPU spent being busy.

[00:38:40]So if you have bubbles in your GPU, it actually doesn't count that which can be um confusing because sometimes like it's you the start of the GPU time and the end of the GPU time might be longer than what the actual GPU time says um because the CPU just like wasn't sending the data fast enough. Input uh the yellow one it's uh people think oh no my game is not performing well like don't worry like this is literally just the like when it was consumed to when it was displayed to the user. So it's a bit different from all the other ones.

[00:39:12]Take into account that when you take a screenshot like this of the frame, these are not the numbers that made up the currently rendered frame.

[00:39:20]It is the last number from each phase.

[00:39:24]So if you are rendering frame number one, then it will probably be like the GPU time for that frame. But if uh CPU frame is already on frame number four or five even, it will be like that frame's time. So it's kind of like just the last stage of each pipeline. Also, it's not the real numbers. We do smooth them a bit with an exponential moving average.

[00:39:42]So, we always uh like uh multiply it by 0.1 uh and the 0.9 is the all the previous values uh to reduce the because otherwise they're just jumping all over the place.

[00:39:55]But you noticed maybe because I have both that FPS and stat units. That FPS is those two numbers there on top.

[00:40:00]There's this number there that says 1650 and it's a little bit different than the frame time which says 16.61. Why is that different? I checked it. Uh ah it uses a 0.75 smoothing factor instead of 0.9.

[00:40:14]And the reason ah sorry no reason it's just old code. I noticed and I was like that's stupid. So I fixed it in 5.8. So now in 5.8 when you run stat FPS and stat unit those two numbers are going to be the same number. Cool. Okay. But as developers we see a lot of numbers. We see all these numbers. The gamers they don't see any of these numbers. Not a single one. They see only how many vsyncs your game made per second.

[00:40:42]So, uh the third party tools, that's what they count. Nvidia app overlay, Windows game bar, Steam overlay, they count how many Vsyncs and that's your frame frames per second. Also, present one does, uh have a custom view that has the same FPS display. It does default like if you have the basic view of percent it defaults to counting FPS according to the calls to percent but that's kind of like like first in first out Q to the GPU and it doesn't take into account the swap chain slack so it's it's not exactly the trans being pres present to the user but I guess it makes sense that they would default to that because the app is literally called present monitor um but just know and also digital foundry they count frames using their own in-house house tool called FPS GUI or did they call it guy?

[00:41:28]Anyways, it also works for consoles which is the only one that does and it kind of like uses delta between frames to know when a new one has been rendered. We don't have a vync counter in Unreal. Of course, you can just use one of these solutions. So, you're not really seeing the same FPS number as the users are. We want it, but we need to do some foundational work first. We need to get the flip timing tracking on all platforms. And Direex kind of like I think we just kind of got it there. Anyways, uh we're working on it. It's going to be one of the timers eventually. So if you're watching this video and you're like, is it in yet? Check the text article. Might have been updated. I'll keep it at the top to say what has been changed. Okay.

[00:42:04]Delta time.

[00:42:06]Unreal uses a delta time property. And I want to explain a little bit. Delta time is the amount of time to advance physics sequences, animations, particles, camera moon, basically the entire gameplay for that frame. And the current frames delta time is literally the last frames frame time.

[00:42:24]If we get a hitch, then the next frame delta time increases. And this is good.

[00:42:30]This is fine because it actually helps the game catch up in game time, which basically prevents the game from getting out of sync with timebased system like audio. So if you have like an audio based game and you have a hitch, you don't want like the uh the delta time to be fixed and then suddenly they're like drifting apart. So knowing that we know the delta time is variable is one frame behind and it adds up to match the total game time.

[00:42:57]Now I want to talk about something hotly [snorts] debated not just but in the game street jitter is you might know it as animation error or micro stutter. I prefer jitter. This happens when the simulated delta time that you're simulating every game thread is different from the time actually be uh between displayed frames because the time between the displayed renders frames is static. You know when it's being showed to the user I'm going to see a VR ignore that. So the delta time value for the current frame however is variable especially if it just had a hitch. So this can cause a mismatch between the simulated time and the time between displayed frames which means that the camera movement gets simulated more than it was actually like displayed to the user and then the camera movement and animations feel stuttery. Uh this is jitter. This is what it is. So to understand jitters better I need to talk about the anatomy of a hitch. And a hitch actually have three effects on frame pacing. First we have the dropped frames. Basically, the last frame stays on screen a little longer because we're not sending any new frames and it just looks like the game's frozen to the user. Then we have the recovery.

[00:44:06]I don't think it's the official name, but that's just what I'm calling it where the next frame speeds up simulation by applying a large delta time like I uh mentioned. It's equal to the last frames frame time and it basically looks like the game was running while the screen was frozen.

[00:44:19]Because of this, it's like as if the user just like closed their eyes and open it and it kept running. Catches up to other is like audio. We actually want this. Lastly, I don't see many people talking about this part. Maybe because Unreal uses the whole uh full pipeline thing, but we need to refill the pipeline again because it's emptied. And this only happens when you're not throttling the game thread and not exceeding frame budgets. So like only when you have the full pipeline and relying on hits resiliency. So basically what happens now is that the next frames they run shorter, they're quicker because there's nothing breaking them.

[00:44:50]So they have a breaking as in slowing them down. They're not breaking. Um and that means they have shorter delta time and so this is a hit and a jitter is different because the pipeline basically absorbs the hit. So we don't have any dropped frames which is good user got all the frames but we still get the delta time fluctuating because the recovery and then the refilling of the pipeline and this is what it looks like.

[00:45:17]This is delta time per frame. And we can see when I get a hitch, we get a big delta time. And then we get some shorter frames as we're refilling the pipeline.

[00:45:29]I want to show you a visualization of this. I've seen some presentations that are like uh they're talking about it by showing the camera jittering and they're showing an animation jittering. Like I saw Digital Foundry reviewing a game.

[00:45:40]They're like, "Look, the animation jitters." And I'm like, >> "I can't really see it honestly. Maybe I'm old or maybe the compression of the animations of the video or something like that and also the camera. I was just like I really want to see it but I just can't. So I made a really contrived example still in Unreal. It's a real example that shows this off. There's no more engineering way. I have a spinning wheel. It is split into 60 sections and it runs one circle per second. So it should move 1/60th of a frame. Meaning that all the lines should always be aligned. So if the lines aren't aligned, if they move like this, that's visualizing jitter. I thought that was a quite a clever solution. And like I said, this is actually running in Unreal. I just turned off a lot of systems. This is the most extreme uh I could make it. This is the most extreme version of hits resiliency because I added as much of a hits I could get away with 90 milliseconds.

[00:46:39]Not a single frame dropped because the frames are fast enough to catch up. So, um, it's there in the lower right corner. However, I know that online this is probably going to be 30 frames per second. So, let's slow it down.

[00:46:51][clears throat] You can see there it makes a little jump. And, you know what? Let's slow it down even more. Let's zoom in and let's label it.

[00:47:00]Here we can see we have a bunch of normal frames where it like rotates 1/60th and then we get the absorbed hits where we got a huge delta time uh because we had the previous frame lagging that long and then we have super short frames. This is by the way exaggerated uh well I mean still real uh but this is a very contract example where it's refilling the pipeline. This even though it's the most extreme you'll ever see it. This is what is happening when your game is jittering. The animation speed up, slow down a little bit, and it just looks horrible to the user. So, it's like we want this ability of the engine in the pipeline to be able to absorb by hits, but would it be nice if we could do it cleanly? So, what can we do? Can we fix the delta time?

[00:47:51]Well, because of variable refresh rate, the display's refresh rate is not always fixed. Also, this is something I learned. This is fun. Monitors don't usually have round numbers as the native refresh rate. You think your monitor is 60 frames per second? You might need to guess again. My work monitor is 59.95 frames per second. Um, and I [clears throat] went to a database of a lot of uh models and turns out of like almost 100,000 of them, less than a third of all monitors there have a native round number reference. They're all slightly off. And fun fact about Unreal, if you ask what's my native refresh rate, it says 60 because it only returns as an integer. So my monitor would actually slightly drift the game time half a second every 10 minutes. So you know, be aware of that. And also because of audio video desynchronization, for example, if the game hitches, the audio just keeps playing, your game pauses, and then continues because you didn't uh compensate for the delta time.

[00:48:54]So, what else can we do? Well, I hate pointing out a problem and then just saying there's nothing you can do. Fat you. No, no. Uh, I made a plugin called Delta Time Smoother that is specifically for this. So, it's called Delta Time Smoother plugin. It's smooth Delta Time for games that don't use game thread throw link. You can download it right there. It's available on GitHub right now. Don't worry, I'll leave this QR code up for a little while. So, it has three modes. Don't worry, the cure code is still up there. It has hits absorption mode where for smaller hitches or jitters they can be absorbed by the pipeline. It basically makes it so that the longer delta times have their depth paid by the only the newer shorter delta times. We don't average out immediately because then it would raise a little bit. It keeps them completely flat. And in my example that I showed before the spinning wheel, it completely smooths it out. 90 millisecond hits unnoticeable.

[00:49:42]But then it has hitch mitigation mode.

[00:49:44]This is for mediumsized hitches that can't be absorbed but can't be smoothed out. You set the milliseconds come by the way yourself. I don't know what those values are for your game. And that just immediately averages out all the delta times in the queue, which means that we might have dropped a few frames, but then instead of having uh the recovery like that, we just run a little bit faster, which shouldn't be perceivable by the user. And lastly, we have hits pass through mode because sometimes like if you get a big hit that's like up to 100 millc, 200 millcond, 300, uh, and if we try to like average it out, it's going to like run comically fast. No, we just let that go through and hit. Sorry. Um, but we'll um still smooth out the refilling.

[00:50:26]The reason why I didn't make this as a native engine plug-in is because this is just a temporary band-aid until we get the new frame pacing in Unreal, which I think every studio should be using if they can. So, uh, takeaways, we're uh, running a little bit over time, but I'm just going to like wrap everything up now. So, first, go for lowest input latency.

[00:50:44]Remember, you will not get any hit resiliency. You need to keep all frames under the time budget or you will drop frames. Luckily, we're all professionals here and we don't have any hits in your games.

[00:50:55]>> Use the new frame pacing mode once it's out. Hopefully, very soon. Uh, not yet.

[00:51:01]As of Unreal 5.8. Until then, on consoles, use GTYNC type two for best frame pacing. Use the sync slack per platform to uh adjust it. For PC games, use the Nvidia Reflex, MT Intellect 2, Intel XC, low latency. It's basically doing the same thing that our new frame pacing will do for mobile and other. I'm sorry. Uh there's no good option. You can use DTSync type one, but you lose out on the parallelism. It's just bad.

[00:51:25]Otherwise, if you need the Hitch resiliency, then sure, go ahead. Use DTSync type zero if you absolutely have to. It's the most input lag, but you can just absorb hitches, which is kind of nice. Uh remember, higher frame rate reduces input lag even more. And lastly, optionally, pair it with the Delta Time Smoother plug-in to reduce jitter.

[00:51:44]That's the end of the presentation.

[00:51:45]Remember, you can scan this code and the text article will be there. Before you clap, I want to take a selfie with you all. So, I'm going to say in three, two, one, uh, all of you will yell, "Hi from Chicago." Can we get the house lights a little brighter so I can see this lovely audience?

[00:52:02]>> Okay, so let's practice it. Three, two, one.

[00:52:09]I must say I've had better ones. I' I collect them. I do this at every fest.

[00:52:14]So, uh I want way more. I want to see waving and I want to see your mouth like gaping open like ah. Okay, EVERYONE READY THIS TIME IN THREE, TWO, ONE.

[00:52:27]>> THAT'S WHAT I'M TALKING ABOUT. THANK YOU SO MUCH.

[00:52:31][applause] Have a great annual fest. Thank you.

[00:52:40]>> [applause]