Google Just Made TensorFlow.js Obsolete (LiteRT.js)

[00:00:00]Google just released a new runtime library that lets you run machine learning and AI models inside the browser at near native speeds. It's called light rt.js and it's super impressive. So, in this video, we'll take a look at light rt.js, see how it works, and we're going to test it out by building a real-time 3D motion capture application running entirely in the browser. It's going to be a lot of fun, so let's dive into it.

[00:00:31]So, what exactly is light rt.js? Well, it's a JavaScript binding for light RT, their on-device inference runtime, that's been running models on Android, iOS, and embedded hardware for years now. But, light rt.js brings the same runtime into the browser via WebAssembly using the light rt.js core NPM package.

[00:00:53]And, here's what makes light rt.js different. So, Google already has a browser machine learning library. It's called tensorflow.js, and it's been around for years. But, tensorflow.js runs on JavaScript-based kernels, and that is a big bottleneck.

[00:01:09]You see, JavaScript kernels can't fully exploit CPU or GPU the way a native runtime can. So, tensorflow.js has always lagged behind native app performance. But, light rt.js uses WebAssembly, and it ships with its own optimized kernel. So, the same native runtime that powers Android and iOS inference is compiled to WASM and then exposed to JS. So, you're not getting a web-flavored abstraction layer anymore, you're actually getting a truly optimized runtime engine. And, this also shows up in their back-end architecture.

[00:01:46]Light rt.js has three different tiers that allows it to perform well on CPUs, GPUs, and even NPUs. So, on the CPU side, it uses XNNPACK, Google's optimized CPU kernel library, which is multi-threaded with relaxed SIMD support. And this is your universal fallback that will run anywhere, no GPU required. But for GPUs, it uses ML Drift over WebGPU, and this is where the real performance lives. It utilizes native GPU kernels, so things like shaders are no longer rendered using some sort of JavaScript orchestration. And for MPUs, it has WebNN, which is still experimental in Chrome and Edge, and it targets dedicated neural processing hardware for power efficient inference.

[00:02:34]And according to Google's own benchmarks, which they conducted on a 2024 MacBook Pro with M4 silicon, it appears to have a three times faster inference than other web runtimes on CPU and GPU across vision and audio models.

[00:02:51]And when running workloads like object tracking, audio transcription, or image manipulation on the GPU or MPU, rather than on the CPU, we get performance boosts that jump from five times all the way to 60 times faster results, depending on the task. And it's worth mentioning that that is the best-case scenario, and Google also acknowledges this. So, your mileage may vary taking into consideration your specific GPU, thermal throttling, and driver quality.

[00:03:21]And there's one more thing that really matters for this runtime. If you already have models that you use to run on PyTorch or TensorFlow, if you've got an existing TFLite file, light rt.js can actually run it directly. And if you're on PyTorch, there's also light rt torch, a conversion path that takes your model straight to.tflite. Plus, there's also an AI Edge Quantizer for shrinking model sizes without a full rewrite. So, legacy workflows you've been using on tensorflow.js can easily be ported to the new light rt.js which is super cool. But, what can you actually do with it? Well, Google has some pretty cool demos out there which give you a sense of the range. So, you can do real-time YOLO object detection, monocular depth estimation with the depth anything library that turns your webcam feed into a live 3D point cloud, and it can also do image upscaling with the real ESRGAN library.

[00:04:20]And all these demos are running completely client-side with no back-ends, no APIs, directly inside your browser. And they've already announced what's coming next, light rt lm.js, and this is going to be for running full language models locally in the browser.

[00:04:36]So, this isn't just for computer vision, soon it will also offer local LLM inference, too. All right, that all sounds great, but I wanted to test it out on my own to see how powerful it actually is. So, for this demo, I decided to create a real motion capture application that uses your webcam for real-time pose estimation. And it also runs entirely on your browser, client-side, and offline with no strings attached. So, I live-coded this app on Cloud Code using the new Fable 5 model, and here's the end result. And by the way, if you want to download this repo and play around with it yourself, I've also added the link to the project in the description below. So, here light rt.js is running through the browser using a blaze pose model with 33 body landmarks driving a 3D character in real-time using 3.js. There's no back-end server, and everything you see is happening on this machine right now, completely offline. So, we can see here that on the CPU, we are averaging around 38 FPS with an inference of 23.3 milliseconds. And we can also see that the pose estimation is lagging behind a bit. But now, if we switch to the WebGPU version, we can see that we are now getting a solid 120 frames per second with an 8.4 ms inference, and the pose estimation is also a bit faster. And that's roughly a three times jump in frame rate and a 2.8 times drop in inference time, which lines up with Google's own numbers, though it's on the modest end of what they published. But we have to take into consideration that Blaze Pose is generally a small model, so there's less raw compute for the GPU to chew through. But even with that, moving the tensor math off the CPU and on the WebGPU cuts the latency enough that the lag is significantly lower. And I also added a feature where you can record your pose capture animation and export it as a JSON or as a Biovision motion capture file, and then open that in something like Blender and retarget that animation to your own custom character. And as you can see here, it's not perfect. The movements are not very detailed or refined, so this is still very much a work in progress, but it's super cool that I was able to get this initial version working in just 10 minutes using the new light rt.js. So there you have it, folks. That is light rt.js in a nutshell. It's honestly amazing to see how far WebAssembly has pushed the capabilities of running complex applications on the browser. And this new light rt.js library truly proves that sometimes JavaScript can be a real bottleneck for getting to those near native performance speeds. So I'm super impressed with this library, and I can't wait to try out light rtlm.js when it finally ships later this year.

[00:07:34]But what do you think about light rt.js?

[00:07:36]Have you tried it? Will you use it? Let us know in the comments down below. And folks, if you like these types of technical breakdowns, please let me know by smashing that like button underneath the video and also don't forget to subscribe to our channel. This has been Andres from Better Stack and I will see you in the next videos.