From Scan to Stream: Open Pipelines for Large-Scale 3D in Unreal Engine | Unreal Fest Chicago 2026

[00:00:01]Uh, I'll be chatting about how to bring large-scale 3D data sets into Unreal using a few different open-source pipelines and what to do with them once they're in.

[00:00:12]So, just a little bit of an intro about myself. My name is Arkoon and I'm a lead researcher at the Carleton Immersive Media Studios. Uh, we're a research lab based out of Carleton University in Ottawa, Canada.

[00:00:23]I have a background in architecture and turned into game dev uh, a few years ago and went down this path mainly focusing on immersive arcvis um, and digital twins. So, I I work a lot in Unreal Engine uh, making these twins for architecture, especially for heritage buildings that are undergoing restoration rehabilitation, uh, places where you might want to document and then visualize uh, and also manage large data sets.

[00:00:48]And on top of that, I wanted to share what I've learned. So, um, since a lot of my journey in Unreal has uh, mainly been self-taught. So, I became a content creator under Third Space Interactive where I share tutorials, tips, tricks, devlogs of projects that I'm actively working on. Uh, mainly just to share my learnings uh, with the community um, and then also helping out developers um, showing them the ropes and helping them on their projects.

[00:01:13]So, the role that I'll be mainly speaking about today is my work at SIMS.

[00:01:17]We have five main research streams and work at the intersection between heritage documentation and real-time 3D.

[00:01:24]And the talk today will mainly focus on these three streams in particular.

[00:01:30]So, we got our start in digitizing historic places, buildings, museums, and so on. Um, into point cloud and photogrammetry models. We then use these as reference to create fully fledged heritage BIM models, HBIM, and this includes all the disciplines, so shell, interior, MEP, structural, and then we bring them into real-time 3D software.

[00:01:51]What you can see in the background is a web version of one of our digital twins.

[00:01:56]And we also work with digital twins in game engines, Unreal to be specific, which is primarily my role at the office.

[00:02:03]So, this talk is going to focus on a few different topics, all increasing in complexity as we go.

[00:02:09]First, we'll chat about a problem that we ran into a few years ago, and that's mainly around the sheer amount of data that we were collecting, um and needing to display for these large-scale heritage projects.

[00:02:21]And we'll explore this through through a case study of the university campus where I work. And uh the three file types that we'll mainly be focusing on are point cloud, photogrammetry, and BIM.

[00:02:31]We'll go through three open-source tools that we created to be able to to bring these in for runtime streaming, and talk about just how to prep these models and actually use them in Unreal.

[00:02:42]So, let's start off with the problem.

[00:02:45]A lot of the projects that I work on at Sims are confidential. These are heritage sites undergoing restoration or rehabilitation, um government facilities, places where people are very particular about where their data is stored.

[00:02:58]That means that there are a lot of data and file types produced. So, a moment a tool says, you know, upload your data to our servers, we'll handle the conversion for you, we'll handle the hosting for you, that conversation unfortunately dies pretty quickly.

[00:03:09]The data for us usually isn't allowed to leave the room, let alone the building.

[00:03:13]So, we have to figure out workarounds with local servers and local tools.

[00:03:17]And this quickly became a problem for us since the tools that we needed to actually support, you know, hundreds of gigs to terabyte scale 3D are mostly cloud-first.

[00:03:27]So, we turned towards open source, and the landscape wasn't particularly encouraging for runtime streaming.

[00:03:33]For point clouds, the main tools that we found were Potree and Entwine.

[00:03:38]Both are great, but both are web-first.

[00:03:41]For meshes, the closest thing that we found was obj2.tiles, which converts it to a 3D tile, which handles partitioning, um but there's it has its own uh unique problems, including baking full textures onto each tile, which quickly uh uh uh um blows up file sizes.

[00:03:57]And then for BIM, and I'm mainly focusing on IFC here. I know DataSmith runtime is an option. Um but for IFC at least, the only options were experimental. So, you have the asset importer plugin and the DataSmith plugin as well.

[00:04:09]And the list of libraries that I have shown up here are only the ones that are maybe compatible with Unreal. The rest that we were finding were mainly web first.

[00:04:18]So, what happens if we try to brute force all of these tools to spawn them into Unreal?

[00:04:24]Uh we get something that I like to call nightmare fuel.

[00:04:26]Um the reality is poor optimization. I won't go through this entire list, but files for the datasets that we were working with are huge. So, you know, what we would hope would be a simple runtime import quickly blows up file sizes, memory budgets, and ruins frame rate.

[00:04:43]Uh and for BIM in particular, we were getting the geometry, but we were losing the semantic data. The very thing that makes it a building information model in the first place.

[00:04:52]So, we decided to build our own pipelines, three of them in particular.

[00:04:55]Uh the first one for point clouds, second for photogrammetry, and the third for BIM.

[00:05:00]All three are open source, and I'll note that the first two, point cloud and photogrammetry, are available today in fragments. Uh we're planning to release later this summer.

[00:05:10]So, there's no cloud step anywhere in this chain, uh unless you choose to host the data in your own S3 bucket.

[00:05:15]Everything can be converted and hosted locally.

[00:05:20]Now, before I get into the tools, I want to ground us on one main case study.

[00:05:24]So, when I got accepted to do this talk, I knew I had to find a dataset that was large enough to actually put both push both Unreal and these pipelines to its limits.

[00:05:34]Uh ironically, the topic that I picked was centered around data sovereignty and data security. So, the projects that I've already applied all these pipelines to, I can't actually talk about.

[00:05:44]So, I I one that's instead near and dear to to where I work, and that's the Carleton University campus.

[00:05:51]For the campus, we have 47 BIM models, landscape model of the site, a hundred or so gigabytes of point clouds, and around 25 gigs of photogrammetry. And this scales up as well to the terabyte scale.

[00:06:03]Uh the scale and complexity of these models, however, are good fit for for showcasing what this tech can do, and this will be the baseline that we anchor on.

[00:06:11]And I had some help from my lovely team at the lab to build out an entire scene of the campus, which you can see here is nice and empty, uh just waiting for us to to populate it.

[00:06:21]So, we'll be adding our models to the scene as we go through the talk.

[00:06:25]So, let's begin with point clouds.

[00:06:28]With our web digital twin that I showed earlier, we're using an open-source library called Potree, which streams point clouds uh to directly to the platform. However, porting over that same library into Unreal didn't really work for us with the way that Potree organized its data.

[00:06:43]We wanted something that was compatible with the Cesium for Unreal plugin. Uh Cesium's open-source 3D tile reader for the engine. So, we can always customize this if we decide to do so in the future.

[00:06:54]So, and because this already existed, there was no point in reinventing the wheel there.

[00:06:59]And that's when we came across a library called Entwine.

[00:07:02]It's an open-source cloud uh point cloud indexer that takes your raw scan as a LAS file, and then chops it into a spatial octree, so that a web-based viewer can stream it into its relevant chunks. And it's been around for close to a decade, uh but unfortunately, they deprecated their support for 3D tiles a few years ago.

[00:07:21]But, that's not a big deal for us, since 3D tiles is also an open an open-source spec for streaming massive 3D scenes easily, uh and conveniently, the tiling structure was near identical to Entwine.

[00:07:36]We just needed to do a little bit of file conversion and create something called a tile set manifest JSON, so that our Unreal plugin knows how to serve it into our scene.

[00:07:45]And here are the results of the pipeline. It's essentially a Python wrapper uh that uses these existing libraries to get from point A to point B.

[00:07:52]We start off with an E57 or a LAS file as the main input. We use Entwine to specially partition it into EPT octree format uh to give us a folder full of lovely tiles.

[00:08:04]And then we do a little bit of conversions to get that to a GLB file, which 3D tiles needs.

[00:08:10]And then generate an optimized and more importantly a geolocated uh tile set JSON manifest.

[00:08:16]And after a final compression, we're left with a 3D tile folder with a manifest that we can point towards Unreal.

[00:08:23]And of course, we can parallelize the entire process to batch convert our point cloud models.

[00:08:29]And the script itself is fairly simple to run. You can input a single file, a whole directory of files, and you can set the amount of uh point cloud models that you'd like to batch convert in one go. And here you can also specify geolocation and compression.

[00:08:45]And because we're able to parallelize a lot of this, the only restriction really is in our memory, how much RAM you have, uh and CPU cores.

[00:08:53]For the point clouds of the university campus, we were able to convert the full set in around 4 hours uh with eight parallel workers. My CPU had 16 cores.

[00:09:02]And due to spatial partitioning and compression first to LAS and then to uh our additional gzip compression, the output size was actually 65% smaller. Um and of course, all of these were uh computed locally on my one lonely workstation.

[00:09:18]So, bringing that into Unreal, we can see the details of the hierarchy that is output by the octree pipeline.

[00:09:24]It's a specially aware algorithm, so you can see that the octree doesn't create tiles exponentially uh for each step up in LOD, but rather it's only generating tiles where there's actually detail, where the point count is over a preset threshold. And it adds these tiles by layering detail on top of each other, layering tiles on top of each other, until you get the target amount of detail.

[00:09:45]And on the Unreal side, it's basically a simple step of enabling the Cesium for Unreal plugin, adding an empty 3D tileset actor, and then pointing the actor towards a URL, which in this case is just a file path, meaning that you can either serve it locally or from a remote bucket like S3.

[00:10:04]So, what does it look like at the end?

[00:10:05]Centering back on our Carleton campus scene, in just a few seconds we're able to load around, I think 12 to 14 point clouds. And we can isolate it in the scene to get a better view of what was imported.

[00:10:16]Each of these scans here represent different efforts the SIMs has done over the campus throughout the past decade as the campus has grown and evolved. And as we zoom into different spaces, we can see that the amount of detail increases as per the instructions of our optimized tileset hierarchy.

[00:10:34]So, next up we have photogrammetry.

[00:10:37]Uh this was supposed to be the exact same playbook. You know, try to find the Entwine equivalent that was also open source, wire through that same Python wrapper script, and hopefully we'll be done in couple days, maybe a week if we're optimistic about it.

[00:10:51]The reality was that photogrammetry just does not have the same ecosystem of libraries built up. No one's really spent the time to build an open-source equivalent of Entwine for meshes.

[00:11:02]The few converters that did exist targeted much smaller models, and they were mainly targeted again for web viewers.

[00:11:08]So, often these had their own underlying problems around performance and file size.

[00:11:14]So, we were back to square one and had to build something completely from scratch.

[00:11:19]And interestingly enough, we landed on Blender for this, which might sound strange because I know that Blender is typically a DCC platform, not really a pipeline tool. But, most interesting part for us is that Blender is completely open source as we all know, but it has a very mature library of mesh editing tools built up over the last couple of decades. Decimation, Bmesh operations, UV unwraps, texture baking, all of it's been built in, and most importantly, all of it's available through the Blender Python API.

[00:11:47]Which means that we can run Blender headless and have multiple Blender sessions running in parallel through command line.

[00:11:52]So, this opened the door for batch processing.

[00:11:56]Which now revealed that photogrammetry had its own set of problems that we needed to solve.

[00:12:02]So, firstly, it was the tiling itself.

[00:12:04]If we were to take inspiration from existing tools, we learned that they, you know, do these uniform grid cuts with even tiles everywhere. That's sort of ignorant to the geometry that lies within.

[00:12:16]This doesn't really work for photogrammetry where details in a heritage space aren't exactly even. You know, a a cornice or an ornamental detail in a room would be significantly more complex than the wall or the floor that's beside it. So, it doesn't make sense to treat each of these spaces the exact same.

[00:12:32]And instead, we can learn from how Entwine handles point clouds and use this spatial octree. Basically, we create a box, split it recursively into eighths, and then the important thing here is we need to make sure that that recursion is adaptive. So, every tile that we create, we check it against a triangle threshold, which in our case is around 20,000 triangles.

[00:12:53]If a tile falls under that threshold, we stop, we call it a leaf tile. And if it falls over it, then we continue splitting until we satisfy the LOD depth.

[00:13:03]And this way, we're basically able to follow complexity and minimize the amount of tiles produced, and thus keep the file size down.

[00:13:11]And here are the LODs that we can create with this adaptive spatial partitioning.

[00:13:16]Whereas with the point cloud data, we were constantly adding layers on top of one another as we zoomed in to get detail, with mesh we want to replace the tiles entirely. And that means that we now need to also address texture.

[00:13:30]With point clouds, this wasn't really a problem since it's just a vertex color on a point.

[00:13:35]With mesh, you have these large open-source textures.

[00:13:38]Sorry, we have these large source textures. And uh the way that our open-source libraries currently handle this is uh leads to something called texture redundancy.

[00:13:47]That's where the existing tool takes, you know, a 16K texture, let's say, and then bakes it into each and every tile.

[00:13:54]So, a tile under 20,000 triangles might normally be like a megabyte in size, now balloons up to 50 megs or more depending on the size of your texture. And imagine that at the scale of an entire mesh. So, if we look here, we have 256 tiles * 50 MB each.

[00:14:10]And when we were to actually select the UVs of uh this tile, we can see that it only uses a fraction of the texture.

[00:14:17]So, of course, we add a baking step.

[00:14:20]Blender smart UV project, also exposed via the Python API, gives each tile its own unique 1024 * 1024 texture. And you can see in this case we're able to get the exact same details from a much smaller texture.

[00:14:32]So, same size everywhere for these textures and unique for each tile. And because the tiles vary in physical size as you go through the LODs, this also adapts our texel density for free. So, now we're not only creating mesh LODs, we're also creating texture LODs.

[00:14:49]And then thirdly, we have this thing called an LOD ladder. So, it's a budget-driven rule where we hold this 1024 * 1024 texture for as long as possible until the total texel count for an entire model is achieved. For a 16K texture, that's around LOD 3.

[00:15:06]And once we cross that budget, we drop resolution to a 32 pixel floor. And that way, once we've achieved the target fidelity of the photogrammetry model, um we adapt subsequent LODs and never really invent detail where it didn't exist already in the source texture.

[00:15:23]And this side-by-side compares the results of that adaptive texture technique. For our test model, we were able to achieve the source texture resolution by LOD 5 when you're right up and close to the building. For the open-source alternative, LOD 0 because it didn't handle any decimation or optimization for the texture. And for the cloud converter, it was around LOD 6 with its Draco compression.

[00:15:48]And what comes at the other end?

[00:15:50]Pipeline works in a very similar way. We write a folder of GLB tiles plus that tiles JSON manifest.

[00:15:57]Same 3D tiles format used for point clouds, very similar pipeline. So, Cesium for Unreal reads it the exact same way. It's the same URL pattern, same on-premises conversion process.

[00:16:10]And how does it scale with complexity?

[00:16:12]Typically, we're dealing with models in a 1 to 10 million triangle range. And we can see that the script is able to convert that fairly well.

[00:16:19]But in the name of science, we needed to test if it would work with the source model from our photogrammetry authoring software.

[00:16:25]Now, anyone that's probably ever made a photogrammetry model knows that the output is usually really large and needs to be simplified before we can do anything useful with it.

[00:16:33]In our case, that number was around 90 million.

[00:16:36]And that's around a 10 GB sized input for an OBJ model.

[00:16:42]And a 32K texture.

[00:16:44]So, we ran it through our pipeline and it came out to around 3.8 GB of streamable tiles.

[00:16:51]And which is around a 62% reduction.

[00:16:53]And for the same model, both the open-source and the cloud converter weren't actually able to finish the processing and and they failed at that step.

[00:17:03]And what does the time to convert look like? As you can imagine, it scales with complexity. So, for a 900,000 triangle model, we were around 3 minutes. For a 9 million model, we're around 40 minutes.

[00:17:14]And then for a 90 million model, it's 5 hours. So, it definitely scales, but it scales linearly.

[00:17:20]And for the campus models, because they're all in that 1 to 5 million triangle range, and we were able to batch convert all of them in parallel, we were able to get all of them done in around 2 hours.

[00:17:31]So, coming back to our Carlton scene, we can see that we're able to import it in just a few seconds.

[00:17:36]And with photogrammetry, it takes a little bit more time to slowly filter through and increase the LODs.

[00:17:41]And we can isolate in the scene to see it a little bit better.

[00:17:44]And what we have here is a drone scan of a new rail line that was being installed on the campus, uh plus this lonely building off in the distance that we still need to connect, and hopefully I can show this video to the university as my bid to fly the drone over it again.

[00:17:57]Um but as we zoom in, you can see that more details get added.

[00:18:03]Okay. So, this is the third pipeline.

[00:18:06]It's the one that's taking the most time to produce, and the one that I'm most excited to share.

[00:18:11]And this is Fragments for Unreal, our solution for BIM streaming in Unreal.

[00:18:15]It's the only one of the three that's still under active development, hence why we're planning to open source this later this summer.

[00:18:22]So, how did this all begin?

[00:18:24]Uh about a year ago, my co-developer, Hyro, posted a short video on LinkedIn.

[00:18:29]He had taken that Open Company's web fragment stack, he'd wired it through this thing called the Google's flatbuffers schema, and figured out how to get BIM models spawning in Unreal in around 5 days.

[00:18:41]When I saw this pop up on my feed, I immediately messaged him, and that started a 12-month collaboration, where we've been basically turning this into from its proof of concept into an actual plugin that can hold a campus-scale scene at a reasonable frame rate.

[00:18:56]So, why build an entirely new tool?

[00:18:59]We'd already taken point cloud and photogrammetry, shown that we can convert to 3D tiles, and there is existing support for IFC for 3D tiles.

[00:19:08]Um so, we would already proven that we can build it ourselves.

[00:19:11]However, the gap was something that we saw in the existing tool ecosystem that it's best articulated with this side-by-side comparison. So, looking at these three columns, we have the most um popular uh file types that are used in Unreal and also outside, which is DataSmith, 3D Tiles, and IFC.

[00:19:30]DataSmith is a great offline importer.

[00:19:32]You get Data Prep, which is awesome for creating recipes that are reusable, um a full per element control for materials, merging, LODs, uh you know, baking Nanite, um and all with the hierarchy and the metadata preserved.

[00:19:46]But, it's built for offline use. Runtime streaming is limited, and everything lives inside the application when you package it, which balloons the size of your executable.

[00:19:56]And that last point really matters for us because we typically deploy through Kubernetes and pixel streaming. So, that means that the bigger the app, the slower the cold start.

[00:20:05]What we want is a thin skeleton with the data living on the server, pulled in when needed.

[00:20:12]And 3D Tiles would be the natural second option. Great runtime performance, hosted on a local server or remote.

[00:20:19]And it loads in in a fraction to a second to a couple of seconds, depending on the speed of your read-write to the server.

[00:20:25]Um but in order to get there, it has to merge everything into a spatial hierarchy. You're still able to keep selection through per vertex metadata, but you lose per element control. And then you're generally stuck with whatever the base material uh came from your authoring software like Revit, unless you do a little bit of finagling.

[00:20:44]And then there's IFC itself. It's 30 years of validation in the AEC industry, vendor neutral, and it's rich in metadata that's baked into each and every element.

[00:20:53]The catch, however, is that support in Unreal is mainly experimental. You know, we were getting the geometry coming in, but the semantic data wouldn't always follow.

[00:21:02]So, we have three formats, three strengths, three let's Let's them deal breakers.

[00:21:06]When Fragments came along, Hiero and I saw a way to take the best of all three and put it into one plugin, which we're hoping that can be Fragments Unreal.

[00:21:17]And there's a more personal reason as well. At Sims, everything that we build, campus and national scale, already streams through this schema on the web.

[00:21:26]But the question that we would always get from clients is, "How do you actually make it look realistic?" So, instead of trying to re-author every one of these assets specifically for the web, we began wondering how we could bring Fragments straight into the game engine. You know, give it the fidelity that only Unreal can get with DataSmith per element control and trying to get that streaming performance of 3D Tiles.

[00:21:49]So, how does Fragments handle performance on the web? It's essentially a structure of arrays. It, you know, it's a runtime optimized version of an IFC file where it does all of the pre-processing for figuring out what to instance. So, you have these parallel pools of of instancing. So, how the web Fragments handles it, and I won't go through all of these, but you're able to get, you know, 3 JS instanced meshes, an LOD system with wireframe when you're far away, full geometry when you're up close, frustum culling, picking, transparency, all of that other fun stuff that's already supported with 3 JS. So, we needed to somehow figure out how to port all of this logic into Unreal.

[00:22:31]And before I get into how we did all of that, I want to actually show how to create a Fragment model itself.

[00:22:37]Um, Hiro has created a repo on our GitHub and it's called IFC to Fragment.

[00:22:41]Also, for all the repos that I'm mentioning, I'll have a QR code at the end that you can scan.

[00:22:45]So, with this, you simply just upload your IFC model into a a web viewer.

[00:22:50]Um, and the import process converts it all to a fragment that you can download. And what you can see here is us running it locally. We're just pulling the repo, running it through NPM, and then loading in our files.

[00:23:02]And when you convert a fragment file, it also compresses it down to optimize it for that runtime streaming that I mentioned. So, a 250 MB model like this one gets saved and compressed as a 17 MB model. So, not only is it optimized for streaming, it's also optimized for storage.

[00:23:22]But, where do we actually start with performance?

[00:23:24]Uh one to once I got my hands on the initial plug-in about a year ago, we quickly realized that we had to imagine uh reimagine the entire architecture for how to handle runtime streaming. Since obviously Unreal doesn't work like the web, although there are some similarities in the concepts that we'll use.

[00:23:39]With the original plug-in, we were able to load in a single model in around 10 minutes.

[00:23:45]Um and we were getting a performance of 22 frames per second, which unfortunately is cut off from this. And even when we tried to spawn the mesh with asynchronously, so it wouldn't block the entire game thread, the time didn't really go down.

[00:23:56]So, my initial focus from then onwards became on runtime performance, while Hire has mainly been focusing on functionality and feature use cases that we've been slowly merging together over the last few months.

[00:24:09]And I could spend an entire talk just on the the the the nightmare I spent in Unreal Insights, but let's just focus on the results of our performance.

[00:24:19]The first thing we looked at was instancing in Unreal, which is supported through instance static meshes or hierarchical instance static meshes and their components.

[00:24:26]For our use case, hierarchical components made the most sense since they allow for per instance culling, which gives us both the performance benefits, especially for frustum and occlusion culling, with how the engine handles each of these components.

[00:24:41]And this is the streaming pipeline that came from that since instancing made it a lot easier.

[00:24:45]Uh and this is what it looks like today.

[00:24:47]We basically take our dot frag file, which is the fragment file, we decompress it on a worker thread, use flat buffers to parse it and build a hierarchy, and then we split that in hierarchy into shell and interior.

[00:25:01]With the shell, we're able to create a hierarchical LOD system that then generates an LOD2 proxy. So, just a unified mesh of the exterior envelope, all merged into one shell. And then an LOD1, which are specially partitioned cells. Where we're bringing that idea again from 3D tiles.

[00:25:19]And then once the initial load is complete, we're only populating the building envelope into our scene.

[00:25:24]And then once the user activates the building, and that's either done by clicking on it to activate or through distance proximity, we can begin to lazy load in the interior models and generate their LOD1 meshes at runtime.

[00:25:39]So, what does that actually mean?

[00:25:40]For a model like this, we're actually able to defer a lot of the triangles and therefore a lot of the work that needs to be done to first populate it into the scene.

[00:25:48]By loading the shell first, we're only having to deal with 260,000 triangles that we process initially. So, it only takes a couple of seconds to load into the scene. And then when you activate the interior, we're lazy loading in the remaining triangles, which was around 10.6 million in this case. So, almost 98% of our triangles get deferred.

[00:26:08]And to give you a visual of how our H-LOD system works, LOD2 is this merged cell proxy, one unified mesh.

[00:26:16]Then when you actually activate the building, you get these per-cell merged meshes, which are merged by their spatial region and also by their floor.

[00:26:24]And these also cast shadows when hidden, which is important because in I For runtime, we can't rely on Nanite and virtual shadow maps. So, a lot of our processes are CPU bound, and every millisecond helps. So, now by having these runtime hidden shadow casting for LOD1, we're only having to deal with 134 shadow casting primitives instead of 26,000 shadow casting primitives. So, that also helps with runtime performance. Every little bit helps. And you can see what the individual components look like on the far right.

[00:26:58]And then we have our material system.

[00:27:00]So, with our fragments master material, we're able to leverage per instance custom data to add individual control out of each of our components.

[00:27:09]So, slot zero basically controls base color and highlighting. Slot one controls opacity masks to control visibility on and off and dither fading to control ghosting. Slot two controls position offsets for reveal animations.

[00:27:24]And that really helps us get into the simulation space a little bit more. And in general, a lot of this material animation are happening on the GPU. So, we're able to offload a lot of that to make our performance even more optimized.

[00:27:37]And you might be thinking to yourself, with this H lot system, how do individual instances stay interactable if we're dealing with these merged meshes at different distances? So, for that we created a specific blueprint called select at screen position, which selects the individual instance by being spatially aware. So, wherever you click, it auto promotes those cells and only the cells around where you actually need to select your components. So, you can see everything else stays at that unified mesh merged hierarchy. And that again really helps when we're trying to do everything we can for performance.

[00:28:11]And speaking of blueprint nodes, we've created quite a few.

[00:28:14]Most of the ones on the left will be part of the initial release of our plugin. And again, this will be open source since we really want the community to just, you know, take this up and be able to help us build this into something that is, you know, sustainable and performant.

[00:28:28]And the ones on the right-hand side are the ones that were created specifically for this demo. And if there's interest in them being used for the wider audience, then we can of course include that in the plugin as well.

[00:28:40]So, after all that work, where does it leave us on performance? For a single building, the model loaded in just a few seconds. And in those few seconds it's also creating all of those LODs at runtime.

[00:28:51]So, other than creating the the fragment file itself, there is no pre-process step.

[00:28:57]A single building sits around the 120 to 130 frames per second when you're in the thick of it, and when you're farther away you're getting in that 140 to 150 range, and this is all on a 4080 GPU.

[00:29:09]And you can see as well, just a little bit of a teaser, but we have some PBR materials that have been applied to this model, and I'll get into how we're doing that in just a second, so that we can avoid having uh you know, just the basic materials that you get from a modeling software like Revit.

[00:29:25]But, on the campus scale, how does it work? The 47 fragments take about 5 seconds each. So, right now the full campus comes in around 5 minutes, and that's including again generating all those LODs, building the hierarchy, applying materials, placement overrides, all happening at runtime. And obviously this is sped up for the presentation to around 30 seconds.

[00:29:46]So, with all 47 in the scene, uh you can see that we're holding around 60 to 70 frames per second as you navigate.

[00:29:52]Uh which works pretty well for runtime imported BIM at a campus scale without any baking or cooking step in the editor.

[00:29:59]And once we load everything in, we can see with our LOD uh debugger that clicking on buildings auto promotes them to active, and then clicking on different buildings demotes the previous and promotes the new. And this really helps us minimize the number of complex meshes and geometry that's in the scene.

[00:30:17]And we're also lazy loading in those interiors, which explains the hitches that you can see there as the interior models populate in, and you can see that as well with the antenna on the roof of this building.

[00:30:29]So, where do we go next? Obviously, we want to continue improving runtime performance, but we want to offer some options for offline support as well. So, with some LOD uh editor preparation steps, we were thinking that we can generate LOD 2 proxy in the LOD 1 merge cells with their materials and everything offline, save that to the content browser, which would allow us to populate the LOD 2 shells immediately into the scene when you first import it, and then we can lazy load in your LOD zero or your actual fragment, you know, individual components from the server at runtime.

[00:31:02]And then optionally, uh we can also build these LODs as Nanite to leverage what the engine already has built in with its virtualization system.

[00:31:12]And speaking of editor functionality, our initial release will include a couple of main features. So, you can load in any fragment model into your scene. In this case, I choose to spawn it at the world origin, but you can just easily choose the base coordinates from your authoring software.

[00:31:27]And on the right-hand side, we can just use our regular editor panel to specify location and rotation offsets. And you can see at the bottom left, we have a little button for saving that placement so that it stays persistent through different sessions, but also on different workstations, and I'll explain how in the next slide.

[00:31:44]On this left panel, we also have the ability to manipulate materials, and here you can see me reverting these PBR materials that we've applied to the building back to the original that we got from from Revit. And we're able to use this panel to isolate and highlight these materials in the scene, again using our fragments material to control all of this functionality.

[00:32:03]And then we're able to override these materials as well. So, in the case of the red facade, we can apply uh just red siding. And then for the main beige color that you can see there, we'll apply wood paneling.

[00:32:17]And again, these PBR textures are also saved between sessions and between workstations, and also support runtimes.

[00:32:23]So, nothing is ever really saved in your package build other than the materials.

[00:32:28]So, how do we actually handle this?

[00:32:31]It's through fragments overrides. So, it's a JSON file that's kept at the exact same file location as your fragment model, and it's auto-generated by our plugin as soon as you try to load in your fragment model.

[00:32:41]In our case, we hosted this on a local server, but you can just as easily put it adjacent to your object storage bucket um or wherever you keep your files.

[00:32:49]The plugin then reads this overrides.json file at runtime and imports it into the scene and applies the materials, geolocation, or rather the placement changes.

[00:32:59]And if it doesn't exist, then it reverts back to what the fragment file came with or what the IFC originally came with.

[00:33:07]And here's a snippet of how you can expand that fragment's base material to create a master material with PBR support, animations, weather, and more.

[00:33:16]So, that was a lot of talking and a lot of more more diagrams than demos. So, now that we have these three tools to stream in BIM, photogrammetry, and point cloud, you can see in the background that we're able to flip through the different models of the same building on the campus at runtime. But, how does it look all together?

[00:33:34]So, for that, let's jump into the actual showcase of this tool.

[00:33:38]Here's our scene with all of the models loaded in. You can see we're able to select a model directly in our scene where it begins to lazy load in the interior elements, including some of the MEP stuff on the roof.

[00:33:48]All the elements All of the elements, excuse me, stay interactable, and we can use the material system to override that into our scene and see all of the relevant pieces of metadata. And we can select another building to do the same.

[00:34:00]And you can see here we have a couple of nodes that we've created specifically for controlling floor visibility. And that's also why our merged cells are merged by their floor ID to keep this functionality um performant as well.

[00:34:13]Once a floor is hidden, uh the plugin is able to only to detect what is hidden.

[00:34:18]That way, you can only interact with what is still visible, um allowing you to zoom in, orbit around, select, and see all of the relevant pieces of metadata from that IFC file.

[00:34:31]And then we can have some fun with materials and build off of the existing plugins functionality to add category filtering.

[00:34:38]IFC models are rich in category data, so a lot of the building model elements are assigned one when you first model that at your in your authoring software. And we can use this to isolate the building within the scene, decide to show all or none of the components, and use a slider to control the opacity.

[00:34:54]And then if we disable everything and select let's say some of the structural categories like columns and floor slabs, that allows us to audit the model in some additional and unique ways.

[00:35:06]And then if we have even more fun with material math, we can get into the simulation territory.

[00:35:10]And that's where we use animations using world position offsets and a bit more ghosting, all handled through a material parameter collection in your blueprint graph.

[00:35:19]So, this specifically we're using for a timeline functionality.

[00:35:23]We can use the IFC categories, organize the building into its phasing, so we have foundation, superstructure, envelope, and MEP rough in, and then also the finishes. And then we can use a world offset uh position node uh per custom data to animate these into place as the timeline progresses.

[00:35:40]And we can scrub back and forth as well, see MEP falling into place, if we want we can select it uh or just zoom in to be able to see how everything stacks together as an assembly.

[00:35:51]Where I can really see this being useful is connecting this with an actual construction schedule so that you can animate what your construction process will look like uh in real time. This is just all being read from a JSON file created from a CSV.

[00:36:04]And all simulated in engine as well.

[00:36:08]And then our second example of simulation is energy simulation. So, this is data that we received from the university on energy usage a few years ago. It was a simple CSV that we converted to a JSON so that it could be read by the plugin. And we made a material function just to able just to be able to override uh the colors.

[00:36:26]And what we ended up with was another timeline indicating the energy usage campus-wide over the course of a year.

[00:36:32]And as we go from fall and win- to winter, you can see how the energy usage spikes. And see that both in the top right panel as well as this bottom timeline here.

[00:36:42]And not only can we see it at the campus scale, but also the individual building scale and all of the relevant pieces of of information that we got from it. And the part that really excites me about this is we got this from a static export from the university, but we can just as easily connect this to live data to be able to see what the actual energy usage is at any specific time from the university. So, this could be a really effective tool for simulation.

[00:37:06]And of course, we want to bring all these file types together at the end.

[00:37:09]So, I made this little tool called a comparison slider based off of the split fiction game, uh, where I was really interested in the material math that that game was doing. So, it's a comparison slider that we can use specifically for ArcVis and slide between the three different file types that we've talked about today. On the left, you have your BIM model. In the center, you have photogram- uh, point cloud, excuse me. And on the right side, you have photogrammetry.

[00:37:31]Uh, we usually use this for reality capture model auditing. As I mentioned at the beginning, a lot of our heritage BIM models are created from point cloud.

[00:37:40]So, this really allows us to visualize and audit that information make sure it was actually modeled accurately.

[00:37:47]And with that, I just wanted to thank you all for your time. Here's a QR code of the repos that I mentioned today. And happy to answer questions. We have about 12 minutes left. Thanks, all.

[00:37:58]>> [applause]