AUVSI | MicroVision: How Lidar Gives Multi-Modal Perception the Needed Performance Edge for Defense

[00:00:00]All right. Well, good morning and good afternoon everyone. Thank you so much for joining us today. On behalf of AUVSI, I'm so pleased to welcome you to today's webinar, how LAR gives multimodal perception the needed performance edge for defense. My name is Shannon Walker, the director of programs here at AUVSI and will be your host for today's webinar. Before we get started, if you've joined us before, you know we have a couple of housekeeping items that we always like to go over. So, all of the attendee lines will remain muted throughout today's webinar. We are recording this and a viewing link will be emailed out to everybody within the next few days. Please note, you can view all of our webinars on AUVSI's learning and engagement platform, AEL.

[00:00:43]If you're listening today and you need any technical assistance, please send a note to the webinar leader using the chat feature at the bottom of your screen. And there is definitely going to be time for questions at the end of the presentation. So attendees, we welcome and encourage you to submit your questions at any time throughout the webinar using the Q&A button at the bottom of your screen. Just noting that the questions will be saved to be answered towards the end of the presentation. Please note the Q&A button is a different button than the chat. So, just make sure you're submitting all of your content questions using the Q&A, which can be found along that bottom bar next to the chat, or if you don't see it, hit the ellipse or the dot dot dot where it says more, and it should be right there. All right. Well, let's go ahead and get started. We would really like to welcome back and thank Microvision for their support of today's webinar. And we're excited to welcome our speaker today, Mr. Martin Kyling, the VP of software engineering at Microvision.

[00:01:43]there. He leads the development of perception systems for vehicles, robots, and drones for the commercial and defense applications. With over 15 years of experience leading international engineering teams, Martin is a recognized expert in sensing and autonomous systems, making him a perfect speaker for today's presentation. So, I'm really looking forward to this today, Martin. And with that, I will hand it over to you to kick us off.

[00:02:09]>> Thank you, Shannon, for that introduction. I'm really glad uh that we can be back with AUVSI community for another webinar and today I'm going to speak about lighter's role in multimodal perception with a focus on defense drone applications.

[00:02:25]It's a subject that in my view has reached a really interesting point right now uh because lighter technology has been developing for decades but only in the last few years it has become really practical to be integrated onto smaller drone platforms. So what I want to do today is walk you through um some of the key technical concepts and the technical background in a structured way. Uh we will look at where LAR fits alongside other sensing modalities um that are established in this space. So mainly cameras and radar sensors as well. uh each sensor's capabilities and where it has limitations and then how they are combined in practice and where the combination of heterogeneous um sensors adds value for concrete use case examples.

[00:03:14]Before I walk you through the outline of my presentation today, uh let me start with a brief introduction to Microvision so that you have some context for where our experience and background comes from. Uh Microvision is headquartered in the United States with engineering operations in Europe as well. The company has been working in the lighter domain since the 1990s and over the past several years Microvision has grown significantly through a series of acquisitions.

[00:03:44]We merged with IBO Automotive Systems uh a German company with more than two decades of lighter experience primarily in the automotive sector. At the end of last year, we acquired Scantinel Photonix, a specialist in FMCW.

[00:03:59]And just earlier this year, we completed a merger with Luminina Technologies, which brought in additional longrange lighter sensor capabilities, um, some software depth and also manufacturing capabilities.

[00:04:11]So, today, Microvision is a lighter company with a fairly broad portfolio, uh, both in terms of technology and also in terms of um, product.

[00:04:22]And that covers different uh lighter operating principles, different range classes and also different lighter form factors.

[00:04:30]And that breadth is specifically useful in defense applications because there's not something like a one-sizefits-all solution for the variety of uh drone platforms or air vehicles and not a one-sizefits-all solution for the different mission types. So having multiple sensor architectures available makes it possible to match the right solution to the right application.

[00:04:57]On the software side, uh Microvision develops perception and application software that processes the sensor data in in real time and on the edge. Uh the reasoning for that is fairly simple. So a lighter outputs a point load uh with a very high information density. But what an autonomous system actually needs is a map of the local environment, um, a set of detections or just a motion or localization estimate. And getting from the raw sensor data to those more abstract outputs requires quite a significant lift in software, especially when it comes to combining multiple um, heterogeneous sensors in one system.

[00:05:39]One additional point worth noting on the broader industry context. Um the cost of lighter technology has been falling steadily um and that's driven partly by increased production volumes and partly by design improvements uh and cost improvements on the core technology and this is significant for defense applications because it affects what platforms uh lighter can practically be deployed on.

[00:06:05]uh and I will come back to this point in the swap discussion uh in the swap discussion in the course of this presentation.

[00:06:13]So with this background let me walk you through um how the rest of this presentation will flow. Uh we will start with where lighter actually matters um the use cases in defense drone operations where the difference between having lighter and not having it is um both measurable and operationally significant.

[00:06:35]Uh we will then look at the different drone platforms themselves um the NATO classes, their swap requirements and how these requirements map to the right sensor choice. Uh from there we will look at the full sensor landscape um EOR cameras um the by far most dominant technology in this space at the moment I would say. Um and then we'll also take a look at radar of course and uh LAR.

[00:07:04]Um then we'll go to the core of the argument multimodal perception uh and why a combination of sensors instead of replacement of technologies is the right answer followed by a look at um what sensor fusion actually demands from a system engineering perspective.

[00:07:23]And we will then look at some real performance data. So quantified KPI improvements and concrete use case examples.

[00:07:31]We'll wrap up the session with Q&A in the end. And with that, let's get right into it. I would say the use cases listed [snorts] out on this slide here are scenarios where the lighters properties um namely direct range measurement uh multiple returns per beam and independence from ambient light address real limitations that do make a difference uh in real world operations.

[00:08:03]And the list of examples here or the list of use cases is really not an exhaustive list um but rather a representative uh list of examples.

[00:08:14]So let's take a look at uh real-time terrain mapping. uh photoggramometry the standard camerabased approach to three-dimensional reconstruction requires uh significant computational resources and processing time and it struggles in low texture environments and it cannot really image below a vegetation canopy.

[00:08:35]More recently, uh neural rendering techniques such as gshion splatting um which create photorealistic 3D uh scene representations have attracted a lot of uh interest but they remain very computationally intensive for real-time output um and that constrains them for offline use.

[00:08:55]Lighter in contrast measures range directly uh then generates multiple returns that partially penetrate vegetation and can build a dense three-dimensional terrain model in real time processed on board of the drone itself. And this enables use cases such as extending the perception horizon for autonomous ground vehicles. Uh a topic that we covered in detail in our previous AUVSI webinar which is available online if you're interested.

[00:09:27]Another relevant use case is object detection and tracking. Uh camera based perception systems provide a a very high resolution and very powerful classification uh are very powerful in classifying pre-trained object types in low light with camouflage or um vegetation obscured scenarios.

[00:09:48]Camerabased detection have fundamental limitations.

[00:09:52]lighter uses an active laser source which means that at low ambient lighting condition um the sensor is not really affected um by the environmental conditions and detecting multiple echoes per shot uh allow partial penetration of sparse vegetation and allow to detect objects be below coverage and I'll have a video that uh will show this um later in the presentation fmcwased lighter sensors additionally measure radial velocity per point which is a great support um for target tracking over time and importantly um lighter data allows detection of any object with optical reflectivity independent of whether a classifier has been trained on that object type or not.

[00:10:47]The third use case is um GNSSD9 navigation. Um GPS jamming and spoofing are documented in contested environments and uh physical signal blockage is is also common uh in indoor scenarios, underground scenarios or in urban canyons.

[00:11:06]Lighter based slam is one of the more robust approaches to positioning without requiring any external infrastructure and I will describe this method in a bit more detail later.

[00:11:19]GNSS denite navigation is one of the key enablers for autonomous mission execution in exactly um these environments where GNSS is unavailable.

[00:11:33]The fourth use case is obstacle avoidance uh which is also critical for autonomous mission execution. Uh low altitude flying and also safe landing.

[00:11:43]Reliable obstacle avoidance requires knowing the exact three-dimensional um distance to an obstacle. So not a depth that's estimated or inferred from a camera image, but a but a direct measurement.

[00:11:56]And LADIA provides that measurement at the speed and with the latency and the precision that autonomous flight really requires. And as I mentioned earlier, LADIA captures small or thin objects and also objects of any arbitrary um shape.

[00:12:13]For precision landing, the same direct range measurement allows the system to verify that the landing zone is level um unobscured and also within an acceptable footprint uh for the size of aircraft.

[00:12:31]Um infrastructure inspection is another relevant use case where lighter adds value. Lighter centimeter level range accuracy enables a detailed geometric measurement of of structures like um bridges, power lines, uh pipelines, [snorts] turbine blades and all that just from a flyby without requiring any physical proximity or um direct human access to the infrastructure that's under inspection.

[00:13:00]And the 3D models produced can be analyzed postflight for dimensional changes for um surface deformation or just for structural anomalies.

[00:13:14]And last but not least uh I want to talk about search rescue and ISR.

[00:13:20]Um the feature um that lighters generate multiple returns per lighter beam means that um objects uh intercepted by sparse foilage can still return data from surfaces beneath the canopy. And this allows detection and localization of objects or personnel that are partially concealed from above and invisible to um camerabased perception systems.

[00:13:48]And looking now at the um overview of all these use cases um in each of the cases uh listed here the benefit traces back to the same properties of lighter the direct depth measurement uh operation in darkness and the multi- returnturn capability that allows detection through sparse cover and I will go into more detail on the first three use cases in the later part um of this presentation.

[00:14:21]Before discussing sensors further, it is worth establishing some context on the drone platforms uh because the swap constraints directly determine which sensors are feasible on which aircraft.

[00:14:35]The NATO UAV classification provides a useful framework for that. Um, class one covers platforms under 150 kg broken into micro uh under 2 kg, mini 2 to 20 kg and small over 20 kg.

[00:14:54]Class 2 and uh class 2 covers 150 to 600 kg and class 3 covers large platforms above 600 kg.

[00:15:05]So from a lighter integration standpoint, the interesting boundary has historically been within class one.

[00:15:12]Microclass platforms impose a very tight constraint uh on swap. So a sensor that um a sensor with over 80 to 100 g and drawing more than 5 watts of power significantly impacts the endurance of [snorts] the drone. And until recently, most lighters uh couldn't really meet these requirements.

[00:15:36]And this is changing with the emergence of solid state sensors. So sensors with no moving parts that integrate the entire sensing mechanism into uh highly compact hardware.

[00:15:50]The sensors have been evolving rapidly in both uh the resolution and the performance and their size and power consumption have dropped to levels that make them really compatible with even with microclass platforms.

[00:16:04]The picture at mini and small class is somewhat more permissive. Uh drones in the 2 to 20 kg range can typically accommodate a sensor in the 150 to 400 g range. uh with a power consumption of typically between 10 to 20 watts um and without significantly without significant operational impact.

[00:16:28]For context, the drone motors in this class typically consume um 500 watt or more during the flight. So a 15 watt sensor really just represents a small fraction of the overall power budget on the on the drone.

[00:16:45]Um this weight budget allows uh this weight budget on the larger drones also allows for a more comprehensive multi-ensor setup um that enables more range and more coverage different across different environmental conditions.

[00:17:06]At class two and class 3 lighter has been technically feasible for longer.

[00:17:11]The constraint is less about the absolute swap and more about sensor resolution uh detection range and also the reliability requirements for the targeted mission. Uh the trend um over the past several years has been a consistent reduction in the sensor size uh the weight and also the power consumption and a trend that is particularly particularly pronounced in solid state lighter designs.

[00:17:38]This has progressively opened up smaller platforms um to lighter integration.

[00:17:44]The diagram on the right of this slide illustrates how three lighter architectures uh FMCW time off with a rotating mirror and solid state flash uh compare on size, weight and against detection range.

[00:18:06]Now let me walk you through the sensor types that are commonly used in drone perception systems. Understanding the physics of each sensor and the constraints under which each one degrades is um the the core foundation for understanding why multimodal uh approaches are useful.

[00:18:27]Um I want to be very clear here. Each of the senses is well established and genuinely capable. And the goal here um today is really not to argue that one is superior to others, but to understand where each one's operating envelope ends and where combining different technologies um really adds value for the use case. So let's first take a look at um EO and IR cameras.

[00:18:55]In defense drone applications, cameras are almost always deployed as a EO IR pair, an electrol optical sensor covering the visible spectrum and an infrared sensor covering the thermal band. They are often integrated into the same gimbal unit and used together in one system context. The electrooptical or EO camera works in the visible spectrum essentially the same principle as a conventional RGB camera. The EO sensor provides uh dense imagery and decades of machine learning development have been done against visible spectrum imagery which means that there's a large and mature tool set uh for detection and classification from EO data.

[00:19:44]The key limitation of EO cameras is depth is not directly measured. Uh it must be inferred from stereo geometry uh or from uh optical flow or learned prior uh and all of which introduce uncertainty.

[00:20:01]In low light or darkness, the sensitivity of the senses degrades rapidly and usable imagery at night typically requires either active visible illumination which comes with signature implications or image intensification hardware.

[00:20:20]In rain, dust, uh smoke or fog, the image quality degrades, [snorts] uh which is impacting the performance of any downstream detection or navigation algorithms.

[00:20:32]And camouflage that's specifically designed to match the visual background is directly effective uh against EO systems.

[00:20:43]The infrared or IR camera, more precisely um a thermal imager detects emitted thermal radiation uh rather than reflected light. Uh it makes it effective in complete darkness since it doesn't depend on any ambient light. It is sensitive to temperature differences between the target uh and their backgrounds, which makes it particularly useful for detecting personal and vehicle heat signatures.

[00:21:11]However, um IR cameras share some limitations with EO. Uh depth is still inferred rather than directly measured.

[00:21:21]Um the performance degrades when the thermal contrast between the target and the background is low. Uh a well-known example is the thermal crossover in in early morning uh or evening when the air and the ground temperature um match each other and this reduces the contrast that separates warm uh targets from their surroundings.

[00:21:44]The resolution of IRA is also generally lower um than for EO sensors which can limit the classification capability at range and taken together um EO and IR pairs do cover quite a broad uh operating envelope. EO provides a very high resolution high semantic content imagery in daylight and IR extends the useful imaging into the darkness and both sensors are passive sensors. So neither of them directly measures uh the depth and both have environmental conditions under which their performance degrades.

[00:22:29]Now let's let's take a look at radar sensors. Um radar wavelengths are largely unaffected by rain, fk, um dust and smoke in directly. It directly measures range and radial velocity via the Doppler effect. Um long detection ranges are achievable with modest hardware and the primary limitation is really spial resolution. The angular resolution at millimeter wave frequencies uh means that small or closely spaced objects cannot be separated uh at range.

[00:23:08]Um classification from radar returns alone is challenging uh due to the limited resolution uh which then limits the ability to derive size and shape of objects.

[00:23:22]Finally, let's take a look at LADAR. Um, LADAR uses uh either pulsed or modulated laser light to directly measure range via time of flight or frequency shift in FMCW because it uses its own illumination.

[00:23:41]Uh, it operates largely independently from from ambient light. FMCW variants additionally measure um Doppler velocity per return, a velocity vector attached to each point rather than just a position.

[00:23:59]Lighters limitations um performance degrades in heavy rain and dense fog where water droplets scatter the emitted and reflected laser beams.

[00:24:10]The cost for lighter is um higher than for cameras or radar, though the gap is really narrowing. And the size and power consumption historically exceeded what a small drone um could carry. The constraint that newer um sensor designs are addressing as I already mentioned specifically through the emergence of solid state lighters.

[00:24:35]So let me summarize uh two important observations from this comparison before we move uh to multimodal fusion. The first is that these sensors provide largely complimentary types of information.

[00:24:50]EO and IR cameras give us visual and thermal content. Radar gives us velocity and all weather range and lighter gives us um precise 3D geometry.

[00:25:03]The second important point is that um the degradation conditions are also largely non-over overlapping for the different modalities.

[00:25:13]EO cameras degrade in darkness. IR cameras lose contrast at the thermal crossover.

[00:25:20]Radar has um limited spatial resolution and classification capability and lighter degrades in heavily adverse conditions.

[00:25:30]So a system that combines all four sensors sensor technologies covers a much broader set of operational conditions than any one of them can cover alone.

[00:25:41]These two aspects are basically the core motivation to consider multimodal fusion.

[00:25:52]And I want to pick up um these two driving arguments for multimodal fusion again. Um the first is filling the capability gaps which means uh combining complimentary information that no single sensor provides by itself. One example lighter geometry fused with camera semantics gives you objects that are both classified and precisely located in 3D space.

[00:26:23]The second um driving argument is increasing the confidence where modalities provide overlapping information. So when sensors agree on a detection, the confidence in that detection is higher compared to only a single sensor providing that detection.

[00:26:43]And we also know this from our daily lives. For example, when we have a health issue and we are concerned, um getting a second or third opinion from another doctor is usually um giving us better confidence. [snorts] Sometimes it's not what we what we want to hear, but it's more trustworthy.

[00:27:00]And when sensors disagree, that disagreement by itself is a valuable information for the uh autonomous system. It may indicate either that um one sensor has degraded um that there's inference interference or an unusual target property.

[00:27:19]So a well-designed autonomous system uses both the agreement and the disagreement between the different uh modalities productively.

[00:27:31]The diagram at the bottom of the slide uh shows how multimodal fusion works in practice.

[00:27:38]The individual sensors feed into the fusion pipeline that produces a consistent model of the environment. A model that um also includes existence and confidence estimates and uncertainty measures for each of the output signals.

[00:27:54]To achieve this, it is fundamental that each census degradation can be estimated from current environmental conditions and is provided to the pipeline as an input.

[00:28:05]The fusion layer uh uses this information to weigh each sensor's contribution appropriately. It's downweighing uh a sensor that is performing poorly and relying more heavily on those that are um operating in their favorable conditions.

[00:28:22]And this is what we mean by graceful degradation.

[00:28:28]Let me now walk you through the the key processing steps to get to a robust sensor fusion uh result. The first one is the alignment step.

[00:28:38]Before you can fuse data from different sensors, you need to know precisely where each sensor is relative to the others and precisely when each measurement was taken. And this is called cross calibration and time synchronization.

[00:28:54]If your camera and your lighter are not precisely co-registered um in the three-dimensional space even by a centimeter um uh or even by a fraction of a degree the fusion output will be spartially inconsistent. So objects will appear to be in slightly different uh positions depending on which which sensor you trust more and at a centimeter level navigation uh with a drone um that really matters.

[00:29:25]Time synchronization is equally critical. Um, a cam camera operating at 30 frames per second and a lighter operating at 20 frames per second are capturing the world at different moments. And on a moving drone, even a few milliseconds of temporal offset introduce a significant sparial error.

[00:29:49]And next, I want to talk about the association step. Um so once your data streams are aligned in space and time, you need to determine which signal from which different sensor sensors correspond to the same real world object. And this is called data association. And it is actually a bit harder than it sounds.

[00:30:09]uh an object that appears as a bounding box in a 2D camera image and as a 3D cluster of scan points in a lighter point cloud and as a diffused bunch of radar echoes. Those are three very different representations of the same target in the real world.

[00:30:28]And robust association algorithms bridge these fundamentally different representations even under motion and in clutter and with partial occlusions.

[00:30:41]Let me now show you what these capabilities look like in practice.

[00:30:49]This slide shows uh one concrete example of a real-time sensor fusion at the raw data level specifically the fusion of a 2D camera image and a 3D uh lighter point cloud on a per frame basis. So that means this is not accumulated over time. It's really frame frame wise matching of the camera image and the point cloud. The camera provides a dense 2D image with full color and texture information. The lighter provides a set of range measurements. So each each one a measured distance from the sense origin to the surface in the scene and by projecting the lighter light LA lighter measurement into the camera's image coordinate frame using the the calibrated um the calibrated transform between the two sensors. uh you can associate a measured range from the lighter with a position in the camera image and the result is a combined representation uh that carries both the geometric precision of the lighter with the visual detail of the camera and the result is richer in information than the data that either of the two sensors would be able to produce alone.

[00:32:04]A neural network uh can use the camera's texture and the color to classify objects, identifying what they are while using the associated lighter depth uh to determine precisely where they are in the three-dimensional space.

[00:32:26]Let me now go into uh more detail on three specific use cases. Uh the first one that I want to cover is real-time terrain mapping. The operational context for this use case is ground operations require accurate terrain information to plan routes u identify obstacles and assess traversibility uh for both personnel and vehicles and all that in real time. In environments where the terrain changes rapidly um due to cratering or flooding or deliberate engineering having reliable up-to-date terrain intelligence is really critical uh for safe route planning and for safe mission execution.

[00:33:12]The dominant camerabased technique for 3D uh reconstruction is photoggramometry. I mentioned that um earlier um also called structure from motion and it works by finding matching features across overlapping images from different perspectives and reconstructing the 3D structure that it that is consistent with those observations.

[00:33:37]Um the technique has a few limitations that that matter in this context. It requires significant computational time uh which typically means processing is done after the flight rather than uh in real time or on the edge.

[00:33:54]Uh it performs poorly in areas of low visual texture like over sand or mud uh over snow uh or just in uniform vegetation where the feature matching becomes really unreliable.

[00:34:09]So what benefit does does lighter add in this use case? Lighter directly measures the range to the surface at each beam position. Uh it doesn't require texture or feature matching. Each pulse generates multiple returns. So early returns from the top of the canopy and later returns from the ground surface below it.

[00:34:31]And this multi-return behavior is what allows the lighter to model terrain even under vegetation or artificial coverage.

[00:34:40]Lighter's direct mid-range measurements uh are well suited to computationally efficient processing and this is one of the key enablers for real-time uh reconstruction directly on the drone's uh onboard processor and the generated terrain model uh can be transmitted in real time uh while the drone is still airborne. The resulting model can then support immediate route planning um and obstacle identification rather than waiting um for post-flight processing.

[00:35:14]The accuracy achievable uh around 2 to 5 cm um accuracy um at typical drone serving altitudes and flight speeds is substantially better than what photoggramometry uh produces under the same conditions.

[00:35:31]And um in case you haven't noticed it yet here in the video, uh the generated terrain map that you can see is actually generated from two drones covering an overlapping area uh at different altitudes. So one trajectory is shown in yellow and the second trajectory is shown in purple. So the sensor fusion and generation of a unique environmental model can also be done from data streams that are generated um on different agents that move independently in the scene.

[00:36:09]Um this video uh on this slide shows uh what that mapping process looks like in a low light uh environment.

[00:36:18]Uh the orange line um you can see represents the drone's trajectory over the ground generated by the lighter based um GNSSD light navigation module.

[00:36:29]And in the first stage I'm going to pause the video here for a moment. Um in the first stage um is a three-dimensional mesh uh and in the darkness the texture on the mesh has really low contrast. So it is difficult to get a clear picture and impression of what the environment actually um looks like. Going to play it a bit further. So when we get to the second stage uh in the second stage um uh a height based coloring is added as an overlay to the mesh which means that it's much easier or which makes it much easier to understand the shape and the topology uh of the terrain. So this is still the same data just a different style of visualization.

[00:37:16]And the third visualization um let me play here is a semantic segmentation uh of the 3D terrain model.

[00:37:26]Um here so the algorithm classifies [snorts] every region of that mesh. Um blue marks traversible ground surface. Uh green indicates vegetation. Red identifies buildings and built structures and yellow flags obstacles.

[00:37:48]And that classification runs entirely from lighter data and in complete darkness uh on the drone's onboard processor and in real time. So I'm showing you this to illustrate and highlight uh what information lighter can add in a situation where the where cameras degrade the camera image in this uh environment.

[00:38:10]uh provides little information. The lighter continues to produce a geometrically detailed and semantically uh annotated terrain model.

[00:38:23]The second use case uh is object detection and tracking. The operational context for this use case um a typical IER mission requires reliable detection, classification and characterization of objects of interest and this may be uh vehicles, personnel, um equipment or infrastructure.

[00:38:47]These objects of interest are often partially concealed um in adverse lighting or in weather conditions that uh degrade optical sensor performance.

[00:39:00]RGB cameras or EO cameras require adequate ambient lighting uh for reliable detection.

[00:39:07]Um at low light uh their sensitivity degrades significantly unless active illumination is used uh which has its own operational trade-offs in terms of signature and power consumption.

[00:39:20]And even thermal cameras uh which are more effective in darkness can face challenges in conditions where um thermal contrast between target and background is is low and neither camera type provides direct depth measurement.

[00:39:36]Um which means that while detection may be possible uh precise three-dimensional positioning of the target is really challenging.

[00:39:46]Radar, in contrast, is effective at detecting moving targets in adverse weather, but the um limited spial resolution um also impacts its ability to classify targets or detect stationary objects that are um close together or overlapping.

[00:40:06]So, how does lighter help uh in this use case? Because lighter uses an active laser source, uh its sensitivity to the target doesn't really depend uh on the ambient lighting. Uh the performance at midnight is functionally equivalent to its performance during daytime. Uh because the sensor generates its own illumination and doesn't depend uh on ambient light.

[00:40:30]The narrow divergence of each um emitted laser beam produces a dense and a high resolution 3D point cloud enabling fine discrimination uh between objects [snorts] even very close objects.

[00:40:47]Multiple um returns per beam even allow detection of objects that are partially obscured by vegetation or artificial coverage. And this is actually a very very powerful attribute and you can see it also in the video how the lighter is able um to detect the vehicle that is covered by by a net. In the bottom right corner you can see the bird's eye view of the camera where the vehicle is is not [snorts] visible. And when lighter is combined with the camera data, the camera provides um semantic uh content like classification while radar provides precise uh threedimen dimensional positioning and in FM f FMCW cases also um the direct velocity velocity measure.

[00:41:38]The um the fused output is a [snorts] set of detections with a with a higher certainty than what any of these sensors could um provide individually. And for each target, a reliable classification and a precise geometric estimate is provided which is more useful for any downstream uh mission function than either center provides independently.

[00:42:07]The third use case uh that I want to cover is GNSS denied navigation.

[00:42:14]Um GNSS signals can be unavailable for several reasons. Um in certain physical environments like building interiors or underground environments and urban canyons uh with significant signal blockage um the satellite geometry is poor and signals are disturbed below the usable level.

[00:42:34]And um beyond these physical limitations, jamming and spoofing of GPS signals are documented uh capabilities that affect operational drone use in contested environments.

[00:42:48]A drone that relies on GNSS for position estimates cannot really work reliably under any of these uh conditions. So developing um positioning methods that do not require GNSS is therefore [snorts] an important area of work for autonomous systems and also for autonomous drone systems.

[00:43:09]And one wellestablished camerabased approach to to that is visual adometry which means estimating motion by tracking features across across sequential camera frames.

[00:43:21]Visual adometry works reasonably well in conditions of good lighting and good visual texture.

[00:43:28]Uh it degrades in low light in environments with limited visual features such as uniform walls or featureless terrain and in conditions where motion blur affects the image quality.

[00:43:41]It also accumulates drift uh meaning that um or meaning position error that grows over time because each estimate is made relative to the previous one rather than against a global position reference without periodic uh correction from GNSS or another global reference. This drift becomes significant over long flight distances.

[00:44:07]Lighter based slam uses geometry of the environment as its positioning reference. The lighter continuously scans the drone uh as the drone moves uh building up a 3D map and at each scan the algorithm finds the transformation that best aligns the new scan with the accumulated map. And that transformation directly relates to the estimated motion.

[00:44:35]Um just noticed I forgot to play the video.

[00:44:42]Um because lighter slam uses geometric features rather than uh visual ones uh lighting conditions uh do not really affect the sensor performance.

[00:44:52]So with IMU integration um to provide high rate motion estimates between lighter scans, a position error of typically below 30 cm per 100 m of travel is achievable in typical uh typical outdoor environments.

[00:45:12]Lighter slam is also not drift free over very long flight distances. Um and so it's not drift free over long distances and extremely feature sparse environments remain a challenge. Um but in most operational settings it provides substantially more reliable GNSS independent positioning um than a visualbased approach.

[00:45:43]So this slide summarizes a quantitive performance comparison across the use cases we've discussed.

[00:45:51]I want to be clear about what these numbers represent. Um they are indicative uh figures drawn from published research or field testing and they should be understood as an order of magnitude guidance rather than a precise uh specification.

[00:46:07]So actual results would depend on sensor configuration on the platform dynamics on different environmental conditions and also on algorithm implementation and other factors.

[00:46:20]The intent is to give a sense of the scale of the difference between the different sensor configurations.

[00:46:27]So looking at um real-time terrain mapping adding lighter produces a 40 to 80% improvement in 3D terrain modeling accuracy.

[00:46:38]The mechanism to achieve that is direct range measurement rather than inferred depth.

[00:46:44]Um the second major benefit related to this use case is the vegetation penetration.

[00:46:50]Um, lighter can model the terrain below sparse canopy cover as we've seen um in the in the videos on the previous slides.

[00:47:02]For object detection reliability, um adding lighter to a camera and radar based system produces a 5 to 10% point uh improvement in the detection precision. So the false positive rates [snorts] um detection of objects that are not actually present are reduced by more than 40%.

[00:47:24]And these improvements come from the precise three-dimensional information um that lighter adds which allows better discrimination between real targets and the background clutter as well as additional shape and size information that support the classification of objects.

[00:47:44]related to GNSS SDN navigation. Uh we discussed that camera and IMU based autometry in challenging environments typically produce drift of 2 to 5 m per 100 m of travel. Um lighter slam uh in combination with IMU reduces this to only 10 to 30 cm per 100 meter. So roughly a f factor of um 15 to 20 uh improvement for obstacle avoidance. Camera based systems are very effective at detecting pre-trained uh obstacle classes.

[00:48:19]Lighter extends detection to object types that are thin like for example power lines or cables or irre irregularly shaped obstacles um like tree branches uh with no distinctive visual texture.

[00:48:35]And this extends the operational envelope of uh to complex and cluttered environments [snorts] um that camera or radar only systems cannot safely navigate.

[00:48:47]In infrastructure inspection, uh lighter centimeter level range accuracy enables geometric measurements that support structural assessment from a flyby and without physical access. And this includes uh detecting dimensional changes um surface deformation or structural anomalies in bridges, power lines, uh pipelines and similar infrastructure for search and rescue and ISR. Um, LA's multiple return capability enables terrain reconstruction even below sparse vegetation canopy extending the um, detection and mapping mapping capability into areas that are opaque to camera based systems looking from the above.

[00:49:39]So looking um across across all these six use cases that we've been looking at, the pattern is always the same.

[00:49:50]Adding a lighter to a camera and a radar system produces a measurable improvement in every relevant metric. Here the improvements are really not marginal. In several cases, the difference is an order of magnitude or more.

[00:50:08]With that um let me close with uh three points that I think capture the practical takeaway from uh today's discussion. The first is about the state of the hardware. Next generation lighter sensors now fall within the swap envelope uh of drone class platforms and that was not true a few years ago.

[00:50:32]Um it is um and it changed changes the integration calculus uh significantly.

[00:50:38]The size, the weight, um the power and the cost constraints that made lighter impractical for small class uh UAS have been substantially reduced through advances in sensor design and manufacturing scale and that is a trend that still continues to go on.

[00:50:56]The swap barrier is no longer um the limiting factor.

[00:51:02]um for integration.

[00:51:05]The second point that I want to make is about uh the sensor performance. So across every metric that we looked at today, uh terrain mapping accuracy, object detection reliability, uh GNSS denied navigation drift, um precision landing, the data consistently shows significant and in some cases order of magnitude improvements when lighter is integrated into the system.

[00:51:32]And that pattern holds across different studies uh [snorts] different platforms and different environmental conditions.

[00:51:40]The magnitude varies um but the direction does not.

[00:51:46]The third point is about the engineering. Sensor fusion is a well understood uh problem. The challenges like spartial calibration, temporal synchronization, um the data association and computational efficiency, they all have established solutions.

[00:52:05]um they require careful implementation but they are not research problems anymore which means um the variable that actually determines whether a system gets these capabilities is not the technology readiness it's really the decision to to integrate and with that I want to thank you for your time today and your attention and I'm happy to take questions >> well great gosh Martin thank Thank you so much for that that great presentation. Um, we definitely have had a lot of questions rolling in throughout, so I'm going to go ahead and jump right in. I know we're pushing up on the time, so I want to get in as many as possible.

[00:52:48]>> Okay. So, so starting us off, you know, you you made a strong case, you know, that LAR doesn't replace cameras and radar. It completes them. So for an operator who already has a capable EO or IR system on their platform, what would be the most practical first step towards adding LAR?

[00:53:07]>> Yeah, that's a great question. Thank you. Um, I'd say that any of the use cases that we addressed in the presentation today is really a good starting point to consider adding lighter. Um the good thing is that in all these scenarios the lighter output by itself is useful and adds value before any fusion with other modalities is is really needed. Um the fusion architecture in software can be built incrementally um and over time and the value ads um are there from the beginning and will just increase steadily uh with the implementation of a fusion strategy.

[00:53:46]>> Perfect. Thank you so much. All right.

[00:53:48]Uh, moving right along. The next question that we we have that's come in asks, "What would you recommend for identifying bird nesting or habitats under a canopy or near the top of foliage?" Uh, this attendee is presently planning on thermal IDing and then using photography for specificity, but asking, you know, is lighter a better way to go?

[00:54:11]Yeah, it's another great question and an interesting use case. Um, I do think that camera based systems are a really good choice for the detection and classification uh of specifically birds here in this case. Um, I think the lighter would add really great benefit for the positioning. Um so if um the spial localization of the detections matters or tracking them over time, this is where having the precise 3D measures and the depth measurement from the lighter would really add benefit um to this detection system.

[00:54:51]>> Perfect. Thank you for that clarification and an answer.

[00:54:56]Um we have another question come in that has asked is your multimodal approach already being adopted by class one drone suppliers and deployed in the field or is it still you know in the early days of proving testing and fine-tuning the technology?

[00:55:14]>> Yeah, thank you. Um uh no it's it is in adoption. So um we have these systems um built out based on our um Movia Iris and Halo sensors um together with the software features that were presented over the course of this presentation. Uh and you might have seen a recent announcement that we made with a with a drone partner to deploy these systems in real world use cases. So they are they are available. So anyone who has a use case I would like to encourage to reach out.

[00:55:51]>> Perfect. That's so exciting to hear.

[00:55:53]Thank you for thank you for sharing.

[00:55:56]>> Okay. Another question uh that came in is uh more situational. So so Airbus recently started exploring the use of FMCW LAR on their Optimate smart automation demonstrator.

[00:56:12]>> Yeah. Could you share your thoughts on the use of LAR for various applications of autonomy on larger aircraft such as an Airbus 3 A350 with you know that has a taxi speed of you know approximately 10 meters per second and then a takeoff and landing speeds of up to 70 meters per second.

[00:56:31]>> Absolutely. Yeah, that's an interesting question. I mean the use cases that we covered in today's presentation were mostly focused on smaller UAS. So um aircrafts that have a limited um altitude above above ground. Um but there are different use cases that are applicable for larger aircraft. So collision avoidance during taxiing um for takeoff and landing uh is one relevant use case that we've seen. Um and um I think there's also other use cases.

[00:57:04]For example, um docking or um approaching for uh airborne refueling systems. It's it's just different use cases for large aircrafts, but there are def definitely um use cases where lighter would add uh add value to large aircrafts.

[00:57:24]>> Wonderful. Thank you so much. Um, we are we're pressing up on time, but I'm going to squeeze in a final question before we go to our closing.

[00:57:34]>> So, you know, how does this technology compare with say competitor offerings?

[00:57:39]What makes this unique?

[00:57:42]>> Um, that's a great question. Uh so I would say you know our offering is we do as I mentioned in the beginning we do have a very broad portfolio in terms of technology in terms of sensing sensors and sensor products and also in terms of um vertically integrated with with the software that we provide and that really allows us to fit the right solution uh to the right use case. So we don't have to uh make compromises there. We really have a broad technology and product offering that we that allows us to address uh use cases in a in a proper way. And our sensors are really designed to be integrated uh with drones or or payload modules uh directly. So it's not a bolt-on solution. It's designed for scaled um deployment. Um and having the software stack uh in addition to the sensors um really makes it a very complete offering I would say.

[00:58:45]>> Amazing. All right. Well, thank you. We are we are right up on time. So, unfortunately, we're not going to get to the rest of our questions here. Um but with that, Martin, do you have any kind of closing thoughts that you'd like to share before we close out for the for the day?

[00:59:01]Um I want to thank everyone for their attention and time today. I hope that everyone could uh take take away some useful information and one thing that I would like people to remember is that lighter is ready to be adopted and it really adds value across a broad range of use cases on drones [snorts] uh and different drone types and different mission types. So if you have a relevant use case and you're curious to learn more, then I would like to encourage everyone to reach out to me.

[00:59:34]>> Amazing. Well, thank you. You know, I on behalf of AUVSI just, you know, thank you Martin and thank you to Microvision for their support of today's webinar.

[00:59:44]This was great content, great information. Uh definitely appreciated.

[00:59:49]So attendees, you will receive a link to access this recording within the next few days. And you can access all of the webinar recordings on AUVSI's platform, AEL. And if you have any questions or comments for our team, feel free to reach out to us at education auvsi.org.

[01:00:08]And lastly, today's program is copyrighted by AUVSI with all rights reserved. And on that note, just thank you once again. Thank you attendees for being with us. Thank you, Martin, once again for sharing this excellent content and presentation with us. And we hope everybody has a wonderful rest of their day.

[01:00:26]Thank you.

[01:00:27]>> Thank you.