eBPF Explained: The Complete Deep Dive — BPF VM, Verifier, JIT, Maps, XDP, Cilium & Prod Systems

[00:00:08]Every time a packet arrives at your server, it walks through hundreds of rules one by one linearly. At 50,000 rules, a single packet lookup takes 14 micros secondsonds. Multiply that by millions of packets per second and your CPU is spending more time in firewall rules than serving actual traffic. What if you could inject your own code directly into the Linux kernel at runtime without recompiling, without rebooting, and without risking a kernel panic? What if you could make the kernel do exactly what you need at wire speed with safety guarantees enforced by the kernel itself. That technology exists.

[00:01:00]It's called ebpf, extended Berkeley packet filter, and it has fundamentally changed how we build networking security and observability tools. Meta uses it to load balance billions of connections.

[00:01:14]Netflix uses it to trace every system call in production. Cloudflare uses it to drop 26 million DOS packets per second per server. EBPF programmable kernel from the BPF virtual machine to the verifier from JIT compilation to real world production systems. This is the complete deep dive. Here's what we'll cover. The BPF virtual machine and its registerbased architecture. The verifier, how it mathematically proves your program is safe. JIT compilation from BPF by code to native machine code.

[00:01:52]Maps, the shared data structures between kernel and user space. Program types from probes to XDP to LSM hooks. BTF and Siori. Compile once run everywhere portability. The lib tool and trace developer workflow and real world production systems. Psyllium, Falco, Pixie, and the Hyperscalers. What is EBPF?

[00:02:21]The story begins in 1992.

[00:02:23]Steven McCann and Van Jacobson at Lawrence Berkeley National Laboratory created BPF, the Berkeley packet filter.

[00:02:32]The original goal was simple. let dump filter packets efficiently inside the kernel instead of copying every single packet to user space. Classic BPF was a tiny virtual machine with two 32-bit registers, a 16 element scratch memory, and about 20 instructions. It could only read packet data and return a verdict, accept or drop. In 2014, Alexe Starvo and Daniel Borman revolutionized BPF from within the Linux kernel. They expanded the virtual machine to 11 64-bit registers, added a 512 byt stack, and grew the instruction set to support arithmetic, branching, and function calls. Most critically, they decoupled BPF from networking. The new eBPF could attach to any kernel event, system calls, trace points,uler decisions, file system operations, security hooks. The kernel became programmable. Here's the highle architecture. In user space, you write an ebpf program in restricted C.

[00:03:39]The LLVM and clang compiler chain compiles it into BPF bite code. The BPF system call loads the bite code into the kernel. The verifier checks every possible execution path for safety. If it passes, the JIT compiler translates BPF bite code into native machine instructions x86 ARM or riskv. The program attaches to a hook point and every time that event fires, your code runs at native speed inside the kernel.

[00:04:13]Think of it like airport security. You, the developer, write a set of rules. The verifier is the TSA agent who inspects your rules before letting them through.

[00:04:26]The JIT compiler converts your rules into the language the airport actually speaks. And the hook points are the checkpoints, arrivals, departures, baggage claim, where your rules are enforced. No rule gets past security without proving it's safe. No rule can crash the airport. Today, EBPF has over 30 program types, more than 200 helper functions, and dozens of map types. The kernel ships with thousands of test programs in the self- test/BPF directory. Every major cloud provider, AWS, Google Cloud, Azure runs eBPF programs on every server. The CNCF eBPF ecosystem includes Psyllium for networking, Falco for runtime security, Pixie for observability, and Tetragonon for security enforcement. This is not a niche technology. It's the new kernel programming model. The BPF virtual machine, registers, instructions, and stack. The BPF virtual machine is a registerbased machine with 11 64-bit registers named R0 through R10. R0 is the return value register. It holds the program's exit code and the return value from helper function calls. R1 through R5 are argument registers. When your program calls a kernel helper function, arguments go in R 1 through R5. These are caller saved meaning the k can overwrite them. Our six through our 9 are k saved generalpurpose registers.

[00:06:09]They survive across function calls. And our 10 is the readonly frame pointer. It points to the top of the program stack.

[00:06:18]BPF uses a fixed width instruction encoding. Each instruction is 64 bits.

[00:06:25]Eight bits for the op code, four bits for the destination register, four bits for the source register, 16 bits for assigned offset, and 32 bits for assigned immediate value. For loading 64-bit constants, BPF uses wide instructions, two consecutive 64-bit words, giving you a full 64-bit immediate. The instruction set has eight classes. LU and ALU 64 for 32-bit and 64-bit arithmetic. Add, subtract, multiply, divide, modulo, bitwise, and or XR shift operations. JMP and JMP32 for conditional and unconditional branches. Jump if equal, jump if greater, jump if sign less than, LD and LDX for loading from memory, ST and SDX for storing to memory. Every arithmetic operation works on registers. There are no memory tomemory operations. This is a RAC design. The BPF stack is exactly 512 bytes. That's it. No dynamic allocation, no heap, no maloc or 10 points to the top and you access the stack downward or 10 - 8 or 10 - 16 and so on. The verifier tracks every bite of stack usage and rejects any program that reads uninitialized stack memory. Why so small? Safety. A bounded stack means bounded memory usage, which means the kernel can always guarantee the program won't overflow and corrupt adjacent kernel memory. BPF programs can call kernel helper functions like bpf lookup_m to read from a map or bpf promo kernel to safely read kernel memory.

[00:08:18]Each helper has a well-defined signature and the verifier checks that you pass the correct types. BPF to BPF function calls are also supported. Your program can define and call its own sub routines, each getting its own stack frame on that same 512 byt stack. And there's tail calls. One BPF program can chain into another, resetting the stack.

[00:08:44]This lets you build programs larger than the single program instruction limit. If this sounds familiar, it should. BPF is conceptually similar to the Java virtual machine or web assembly. All three are bite code virtual machines with a verifier or type checker. Java's JVM verifies class files before execution.

[00:09:06]Web assembly validates modules before instantiation. BPF verifies programs before loading into the kernel. The key difference BPF runs in kernel space with direct access to kernel data structures.

[00:09:21]A bug in your JVM code crashes your app.

[00:09:24]A bug in unverified kernel code crashes the entire machine. That's why BPF's verifier is so much stricter. The verifier proving program safety. The verifier is the most critical component in the entire ebpf stack. Every program you load must pass the verifier. No exceptions. Its job is to mathematically prove that your program will terminate, will not access invalid memory, will not leak kernel pointers to user space, and will not cause the kernel to deadlock or crash. Think of it as a compiler that says no a lot. And that's exactly what makes ebpf safe. Verification happens in two passes. First pass control flow graph validation. The verifier builds a directed ascyclic graph of all instructions. It checks that there are no unreachable instructions, no back edges, meaning no loops, and that every path eventually reaches a bpf_exit instruction. If your program has a loop, the verifier rejects it immediately.

[00:10:42]Since kernel 5.3, bounded loops are allowed, but only if the verifier can statically prove the loop will terminate within a fixed number of iterations.

[00:10:53]Second pass, abstract interpretation.

[00:10:57]This is where the real magic happens.

[00:11:00]The verifier simulates every possible execution path through your program, tracking the state of all 11 registers and every stack slot. For each register, the verifier tracks its type. Is it a scalar value, a pointer to a map, a pointer to the stack, a pointer to a packet, or a context pointer? It also tracks the range of possible values.

[00:11:29]This register holds a value between 0 and 255 and it tracks alignment. Is this pointer aligned to four bytes or eight bytes?

[00:11:40]Bounds tracking is how the verifier prevents outofbounds memory access. Say your program reads data at an offset from a packet pointer. The verifier knows the packet length. It's in the context. If your offset could exceed the packet length, the verifier rejects the program. You must add an explicit bounds check. If offset is less than packet length then read. After that check the verifier narrows the range of the offset register to be within bounds. This is called state refinement. Every conditional branch refineses what the verifier knows about register values on each side of the branch. Here's the scaling problem. A program with n conditional branches has up to 2 to the n execution paths. A non-trivial program might have hundreds of branches. Without optimization, verification would take exponential time. The solution is state pruning. At certain prune points, branch targets, function calls. The verifier saves a snapshot of the current register and stack state. when it reaches the same instruction again from a different path, it compares the current state to the saved state. If the current state is a subset of the saved state, meaning every register has equal or narrower value ranges, the verifier can skip that path. It's already been explored. This reduces exponential paths to manageable numbers. The verifier enforces hard limits. 1 million verified instructions maximum. 1 million is a lot. Most production programs verify in under a 100,000. Psyllium's most complex programs verify in about 800,000 instructions. The verifier also limits stack depth to eight nested function calls and total program size to 1 million BPF instructions. If your program is too complex for the verifier, you need to simplify it. Split it into smaller programs chained with tail calls or reduce branching complexity. When the verifier rejects your program, it produces a detailed log. It tells you exactly which instruction failed and why. or one has type equals scalar expected equals map ptr or invalid memory access at offset minus 8 from stack pointer. These error messages are notoriously cryptic for beginners, but they're precise. Learning to read verifier output is the key skill for ebpf development. JIT compilation from bite code to native code. After the verifier approves a program, the JIT compiler translates BPF BTE code into native machine instructions for the host CPU x86-64, ARM 64, RISV, whatever architecture you're running. Without JIT, the kernel would interpret BPF instructions one at a time, roughly 10 times slower than native execution. With JIT, your eBPF program runs at the same speed as handwritten kernel C code. Here's how it works. The JIT walks through each BPF instruction and emits the equivalent native instruction. BPF's register model was deliberately designed to map cleanly onto x86-64 registers. R0 maps to REX, R1 maps to RDI, R2 to RSI, and so on. A BPF add instruction becomes a native ADD instruction. A BPF memory load becomes a native for memory. A BPF function call becomes a native call with arguments in the right registers. The translation is almost one one. The JIT includes security hardening. Constant blinding randomizes immediate values in the generated code to prevent JIT spray attacks where an attacker crafts BPF programs that when jit compiled produce shell code at predictable memory addresses. The JIT also uses repelines for indirect branches protecting against spectre variant 2 attacks. These mitigations add a few nanoseconds of overhead but prevent entire classes of CPU level attacks. The performance difference is dramatic. Consider a packet filtering scenario. With IPALs, 5,000 rules are evaluated linearly, one by one, for every packet. That's about 3 microsconds per packet at best. An equivalent ebpf XDP program uses a hashmap lookup o of one, regardless of how many rules you have. Measured performance. An XDP program can process 24 million packets per second per core on commodity hardware. IPALs with 5,000 rules manages about 3 million packets per second. That's an eight times improvement not from better hardware but from better kernel programming. Maps shared state between kernel and user space. Maps are the data structures that make ebpf program stateful. Without maps, a BPF program runs, returns a value, and that's it. No memory between invocations.

[00:17:26]Maps solve this by providing persistent typed data structures that live in kernel memory. Your BPF program can read and write maps. Your user space application can also read and write the same maps. This is how information flows between the two worlds. Think of maps as shared white bars. The kernel program writes observations and the user space program reads them. The most common map type is the hashmap bpf_map_ype ash. It stores key value pairs with o of one average lookup time. You define the key size, value size, and maximum number of entries. When you create the map, the kernel pre-allocates memory for all entries by default. Use cases counting packets per source IP tracking connection state storing configuration from user space. There's also a per CPU variant bpf_map_p_ash where each CPU core has its own copy.

[00:18:33]This eliminates lock contention at the cost of more memory. Array maps bpf_map underscore type array are even simpler.

[00:18:42]They're fixed size arrays indexed by integer keys from zero to max entries minus one. Lookup is O of one just a memory offset calculation. The JIT can even inline array lookups eliminating function call overhead entirely. Arrays are ideal for global counters per CPU statistics and lookup tables. per CPU arrays. BPF underscoremup_UP underscore array are the fastest possible map type when each CPU tracks its own data. BPF_MUP_TUP ring buff is the modern replacement for PF event arrays. It's a single ring buffer shared across all CPUs. The key advantage events maintain their global ordering with per CPU PF buffers. Events from different CPUs arrive out of order and your user space consumer must sort them. Ringbuff solves this with a lock-free multi-producer single consumer design. The BPF program calls BPF ringbuff_output to submit an event. The user space application pulls the ring buffer with a simple memory mapped read.

[00:19:49]Typical throughput over 5 million events per second with less than one microcond latency. BPF_MAP_MP_ing is specialized for longest prefix match lookups. Exactly what you need for IP address routing and CDR based access control. Given a key like 1 192.168.1.10 the LPM tree finds the longest matching prefix maybe 1 192.168 1.0/24 or 1 192.168.0.0/16.

[00:20:19]This is the same algorithm your network router uses. Psyllium uses LPM tries to implement network policies with CI rules. Lookup time is O of the prefix length at most 32 comparisons for IPv4 128 for IPv6. Beyond these core types, EBPF has maps of maps. A meta map where each value is a file descriptor to another map. This lets you atomically swap entire map contents. There's also sock map for socket level redirection.

[00:20:49]dev map for XDP packet forwarding, stack trace maps for capturing kernel call stacks, and task storage maps that attach data to individual kernel tasks.

[00:21:01]The map ecosystem has over 30 types, each optimized for a specific access pattern. Program types, where ebpf hooks into the kernel. Every eBPF program has a type that determines three things.

[00:21:18]where it can attach in the kernel, what context data it receives, and which helper functions it can call. A program attached to a network hook gets packet data. A program attached to a trace point gets the traced functions arguments. The verifier enforces these type constraints. You cannot call a networking helper from a tracing program. probes bpf_prog_ub probe let you attach ebpf programs to virtually any kernel function. When that function executes, your BPF program runs with access to the functions arguments via registers. Cret probes fire when the function returns, giving you the return value. Trace points. BPF underscore prog trace point attached to stable kernel trace points. The difference probes can break between kernel versions because internal functions change. Trace points are stable ABI. They survive kernel upgrades. For production monitoring, prefer trace points. For debugging, probes give you deeper access. XDP bpf_prog_type XDP is the flagship networking program type. An XDP program runs at the earliest possible point in the network receive path before the kernel even allocates an SK_buff data structure. The program gets raw packet data and returns one of five verdicts. XDP underscore pass to send the packet up the normal stack. XDP drop to discard it immediately. XDP_TX to bounce it back out the same interface. XDP redirect to forward it to another interface or CPU or XDP_ed to drop it and signal an error. Because XDP runs before the kernel networking stack, it can process packets at 24 million packets per second per core. That's four times faster than Iles TC programs bpf_prog_up cls attached to the Linux traffic control layer. They run later than XDP after the kernel has allocated an SK buff, but they can see both ingress and egress traffic. Psyllium uses TC programs extensively for its networking data plane. Socket filter programs BPF_Prog filter are the oldest type descended from classic BPF. They attach to individual sockets and filter incoming packets. Modern tools use raw trace points for socket level tracing instead.

[00:23:56]Group programs BPF_pro_KB and related types let you enforce policies per group. Kubernetes uses this for pod level network policies. any socket operation by a process in that group triggers your BPF program. LSM programs BPF_Prog LSM hook into the Linux security module framework. This means you can write custom security policies as eBPF programs. Want to block a specific system call only when called by a container process with a particular label? Write an LSMBBPF program.

[00:24:34]Tetragonon and other eBPF security tools use this to build kernel level runtime enforcement. BTF and Ciori compile once, run everywhere. Here's the challenge.

[00:24:46]Kernel data structures change between versions. A strruct task_struct in kernel 5.15 might have a field at offset 160, but in kernel 6.1 that same field is at offset 176. If you hard- code offsets, your eBPF program breaks on any kernel version other than the one you compiled for. Before CO, you had two bad options. Ship kernel headers and recompile on every target machine using BCC, which requires LLVM and clang installed in production, or maintain separate binaries for every kernel version. BTF BPF type format solves the type information problem. BTF is a compact metadata format that describes every kernel data structure, strct names, field names, field offsets, sizes, types. Modern kernels ship with BTF embedded at /is/kernel/btf/v Linux. It's typically 1 to 4 megabytes of compressed data describing millions of type definitions. When you compile an eBPF program, clang embeds BTF metadata about the types your program uses. C compile once, run everywhere, is the magic that ties it together. Here's the workflow. You write your program using Linux.h, a giant header generated from BTF using fuel that contains every kernel type.

[00:26:18]You compile once with clang. Applying records co relocations in the ellf object. This program accesses fieldcom at offset x of strruct task_struct.

[00:26:31]At load time, lib reads the target kernel's btf, finds the actual offset of comm in that kernel's task_struct and patches your program's instructions. The compiled by code adapts to whatever kernel it runs on. The impact is massive. Before CO tools like BCC compiled eBPF programs on every startup, taking seconds and requiring hundreds of megabytes of compiler tool chain. With CO, you ship a single 500 kilobyte binary that loads in milliseconds on any kernel from 5.8 onward. Facebook was one of the first to adopt CO at scale. Their BPF-based monitoring agent runs on millions of servers across dozens of kernel versions from a single binary developer tooling lip tool and trace.

[00:27:27]Lymph is the standard C library for loading and managing eBPF programs. It handles the entire life cycle. Opening the compiled ELF object, performing CORE relocations, loading programs into the kernel via the BPF system call, attaching programs to hook points, creating and managing maps, and cleaning up on exit. The typical workflow BPF_ob_open to parse the ELF, BPF_OBM load to verify and JIT, then BPF program_attach to hook it in. Libmph also provides the skeleton API autogenerated C code that gives you type-S safe access to your maps and programs. FTO is the Swiss Army knife for eBPF inspection. Full prog list shows all loaded BPF programs, their ID, type, attached hook, and memory usage.

[00:28:20]Full map dump reads map contents. Full BTF dump generates from Linux.h from the kernel's BTF data. Fulnet list shows all networkattached BPF programs. Tool prog ID42 gives you detailed info about a specific program including its JIT compiled instructions. For debugging production issues, tool is indispensable.

[00:28:45]Trace is the highle tracing language for eBPF. It's off for the kernel. A oneliner like if trace each trace point sys calls SIS read at comm equals count counts red system calls grouped by process name with zero code compilation.

[00:29:03]Under the hood compiles your script to BPF bite code, loads it and streams results. It supports wild cards, histograms, stack traces and printf style output. For quick investigation, which processes are writing to disk right now? Up trace gives you answers in seconds. The modern ebpf development workflow. Write your BPFC program and user space loader. Generate linux.h with compile with clang targeting BPF. Test with trace for quick prototypes. Use for inspection and debugging. Ship a single binary with lift and co. monitor in production. The tooling has matured enormously. What once required deep kernel expertise now has IDE support, documentation, and a growing community of thousands of developers. Realworld production where ebpf runs today.

[00:30:01]Psyllium is the ebpf powered CNI container network interface for Kubernetes. It replaces kuba proxy ipables and traditional network policies with ebpf programs attached at XDP and TC hook points. The result pod networking without any ipables rules service load balancing in the kernel data plane network policy enforcement with zero overhead for allowed traffic in benchmarks. Psyllium with eBPF achieves 8.9 Gbits per second throughput with 100 network policies active while IPAL's based approaches drop to 3.2 Gbits a 64% performance gap. Falco created by cydig and now a CNCF project uses eBPF for runtime security. It attaches BPF programs to system call trace points and monitors every operation in real time. File access, network connections, process execution.

[00:31:00]When a container opens/ etc/ shadow or a process spawns a shell inside a production pod, Falco fires an alert.

[00:31:08]The ebpf approach has near zero overhead. Unlike kernel module or tracebased tools, BPF programs add less than 1% CPU overhead even at high system call rates. Pixie by New Relic and also a CNCF project provides automatic Kubernetes observability using ebpf. It captures HTTP, gRPC, MySQL and postgrql protocol data from the kernel without any application instrumentation. No sidecars, no code changes, no agents inside your containers. Pixie attaches ebpf programs to socket level operations and reconstructs application protocols from raw kernel data. It can show you request latency, error rates, and throughput for every service from the moment you deploy it. The hyperscalers run ebpf at a scale no one else can match. Metaiscatch is an XDPbased layer 4 load balancer handling billions of connections. Netflix uses ebpf for fleetwide performance analysis. Their flame scope tool builds CPU flame graphs from BPF collected stack traces across thousands of servers. Cloudflare's DDoS mitigation processes over 26 million packets per second per server using XDP classifying and dropping attack traffic before it ever reaches user space.

[00:32:32]Google's GKE data plane v2 uses Psyllium's ebpf for all Kubernetes networking. The mental model.

[00:32:42]Here's the complete mental model. User space program compiled with clang loaded via BPF system call. The verifier proves safety DAG walk bounds tracking state pruning. The JIT compiles to native code. The program hooks into the kernel at probes trace points XDP TC group or LSM points. Maps provide shared state.

[00:33:06]BTF and Core provide portability in the ecosystem. Psyllium, Falco, Pixie, Trace turns raw kernel programmability into production grade infrastructure. EBPF didn't just optimize the [music]