What is eBPF, and why does it matter for observability?

Working within the Linux kernel is ideal when implementing security, networking, and observability features. However, it’s not without its challenges. Whether modifying kernel source code or adding modules, developers have traditionally found themselves contending with complicated infrastructure and abstracted layers that are difficult to debug. Extended BPF (eBPF) solves both of these problems.

What is eBPF?

Extended Berkeley Packet Filter (eBPF) is a kernel technology (starting in Linux 4.x) that allows programs to run without having to change the kernel source code or adding additional modules. You can think of it as a lightweight, sandbox virtual machine (VM) inside the Linux kernel, where programmers can run BPF bytecode that takes advantage of specific kernel resources.

Using eBPF eliminates the need to change kernel source code and streamlines the ability of software to leverage existing layers. As a result, it’s a powerful technology with the potential to fundamentally alter how services like networking, observability, and security are delivered.

Here’s a closer look at what it is, how it works, and when to consider implementing it.

eBPF vs BPF

eBPF (Extended Berkeley Packet Filter) and BPF (Berkeley Packet Filter) both serve as technologies for observing and monitoring network traffic, but they are far from identical. 

BPF, the original technology, is relatively limited and was designed mainly for packet filtering. It's like the old-school tool that gets the job done but lacks versatility. eBPF, on the other hand, is a more advanced and flexible version, extending BPF's capabilities beyond just networking. Think of eBPF as BPF's ingenious upgrade—like going from a flip phone to a smartphone. eBPF allows you to run sandboxed programs in the Linux Kernel, giving engineers powerful and customizable insights into system behavior.

It's not just about packets anymore; eBPF can help you explore various system calls, trace points, and more. If BPF is a hammer, eBPF is the entire toolbox. Both are reliable, but eBPF empowers you to achieve a greater level of granularity and control, offering real value in today's complex computing landscapes.

Here’s a closer look at what it is, how it works, and when to consider implementing it.

The basics of eBPF

When diving into the world of eBPF, there are several key terms and concepts you’ll need to familiarize yourself with to fully leverage its empowering capabilities. 

• eBPF bytecode: This is the compiled form of your eBPF program. It's what gets executed by the eBPF virtual machine inside the Linux kernel.
• eBPF virtual machine: A lightweight, in-kernel virtual machine that interprets and runs the eBPF bytecode, offering you real-time data and insights.
• Verifier: Before your eBPF bytecode runs, the verifier checks it for safety and compliance, ensuring that your program won't harm the system.
• Maps: These are key-value stores accessible from both user-space and kernel-space, used for storing and sharing data between eBPF programs and user-space applications.
• Hooks: These are the specific points in the Linux kernel where you can attach your eBPF programs, such as system calls or networking events, to monitor or modify behavior.
• XDP (eXpress data path): A high-performance data path that enables eBPF programs to process packets directly at the driver level, making decisions even before the kernel gets deeply involved.
• BPF type format (BTF): This provides rich type information to make your eBPF programs more understandable and easier to write.
• bpftrace: A high-level tracing language for eBPF, designed to simplify the task of collecting data from the kernel for analysis.

Armed with an understanding of these essential eBPF terminologies, let's delve into how this intricate yet empowering technology actually functions within the Linux kernel.

How eBPF works

eBPF programs are event-driven and attached to a code path. The code path contains hooks which execute any attached eBPF programs when they’re passed. Some examples of hooks include network events, system calls, function entries, and kernel tracepoints.

When triggered, the code is compiled first to the BPF bytecode. In turn, the bytecode is verified before it runs, to ensure it doesn’t create a loop. This step prevents the program from compromising the Linux kernel either accidentally or on purpose.

After a program is triggered at a hook, it then makes helper calls. These helper calls are functions that equip eBPF with many features for accessing memory. Helper calls need to be pre-defined by the kernel, but the list of what functions exist continues to grow.

eBPF was initially used as a way to increase observability and security when filtering network packets. However, over time, it became a way to make the implementation of user-supplied code safer, more convenient, and better-performing.

The advantages of eBPF

eBPF is typically used to trace user-space processes, and its advantages shine here. It’s a safe and useful method to ensure:

• Speed and performance. eBPF can move packet processing from the kernel-space and into the user-space. Likewise, eBPF is a just-in-time (JIT) compiler. After the bytecode is compiled, eBPF is invoked rather than a new interpretation of the bytecode for every method.
• Low intrusiveness. When leveraged as a debugger, eBPF doesn’t need to stop a program to observe its state.
• Security. Programs are effectively sandboxed, meaning kernel source code remains protected and unchanged. The verification step ensures that resources don’t get choked up with programs that run infinite loops.
• Convenience. It’s less work to create code that hooks kernel functions than it is to build and maintain kernel modules.
• Unified tracing. eBPF gives you a single, powerful, and accessible framework for tracing processes. This increases visibility and security.
• Programmability. Using eBPF helps increase the feature-richness of an environment without adding additional layers. Likewise, since code is run directly in the kernel, it’s possible to store data between eBPF events instead of dumping it like other tracers do.
• Expressiveness. eBPF is expressive, capable of performing functions usually only found in high-level languages.
• Real-time, event-driven data capture:  eBPF allows for immediate logging of specific events directly from the kernel, providing timely and accurate data for monitoring and analysis.

eBPF best practices

Since eBPF is such a new technology, many things remain unexplored. Best practices around eBPF are still evolving as the technology gains prominence. While no defined set of best practices exist, there are a few things that you can do to ensure effective, efficient programs.

If you’re using eBPF for your ecosystem, we recommend that you:

• Use LLVM Clang to compile C into bytecode. When eBPF first hit the scene, it was necessary to code and assemble the program by hand. Then, developers used the kernel’s assembler to generate bytecode. Fortunately, it’s no longer necessary to do this. Clang provides infrastructure for frontend and tooling in C languages. 
• Use the BCC toolkit when writing BPF programs. The BPF Compiler Collection (BCC) is a toolkit that can help you create efficient kernel tracing and manipulation programs. It’s especially suited for tasks related to performance analysis and controlling network traffic.
• Leverage eBPF helper functions: The Linux kernel provides a set of helper functions that can be called from within eBPF programs. These functions provide a way to interact with kernel data structures and perform various operations, making your eBPF programs more powerful and flexible.
• Use maps for data sharing: eBPF maps are key-value stores that facilitate data sharing between the kernel and user space, or between different eBPF programs. Utilizing maps effectively can greatly enhance the functionality of your programs.
• Monitor resource limits: Be mindful of the resource limits imposed by the kernel, such as the maximum number of active eBPF programs and maps. Exceeding these limits can lead to program load failure.
• Validate user input: If your eBPF program interacts with user space, ensure that you validate any user input to prevent potential security vulnerabilities.
• Keep abreast of kernel updates: eBPF is a rapidly evolving technology, and new features and improvements are regularly added to the Linux kernel. Keeping your kernel up-to-date will allow you to take advantage of these enhancements.
• Employ eBPF libraries and frameworks: Consider using libraries like libbpf for loading and manipulating eBPF programs and maps, or frameworks like Cilium for network security and observability tasks. These tools can simplify the development process and offer additional features.
• Document your code: Given the complexity and low-level nature of eBPF programs, thorough documentation is essential for maintainability and future debugging.

By following these practices, you can ensure that your eBPF programs are not only effective but also secure, maintainable, and up-to-date with the latest advancements in this technology.

The pitfalls of eBPF

Although it’s powerful, eBPF is not a silver bullet that suits every project or ecosystem. eBPF does have some notable disadvantages, which can make it frustrating to work within certain instances. Some developers might find eBPF inadequate to use because:

• It’s restricted to Linux and a recent kernel. eBPF was developed in the Linux kernel and is completely oriented around it. That makes it less portable than other tracers. Additionally, you need a fairly recent kernel. If you’re running anything older than v 4.13, you won’t be able to use it.
• Sandboxed programs are limited. eBPF derives increased security by limiting what resources programs can access. However, by limiting what parts of the OS a program can access, functionality is also potentially limited.

eBPF use cases

eBPF is rapidly gaining traction in cloud native applications. As a result, eBPF is used most commonly in the following situations:

• Real-time system monitoring: eBPF offers granular insights into various system metrics like CPU utilization, memory allocation, and disk I/O operations, providing a comprehensive view of system health.
• Security: eBPF allows for fine-grained control over system calls and network activities, enabling the identification and mitigation of potential security vulnerabilities.
• Networking: Used for traffic shaping, load balancing, and routing, eBPF can intercept and manipulate network packets directly within the kernel, offering advanced networking capabilities.
• Application performance monitoring (APM): Developers can use eBPF to trace micro-level events in their code, helping to identify bottlenecks and optimize application performance.
• Versatility: Due to its efficiency and flexibility, eBPF is applicable in a wide range of fields including system administration, cybersecurity, networking, and software development.

New Relic and eBPF

Pixie (acquired by New Relic), is an open source, kubernetes-native-in-cluster observability platform that provides instant visibility into Kubernetes workloads with no manual instrumentation. eBPF provides most of the magic behind the Pixie platform. As described earlier, eBPF allows you to run restricted code when an event is triggered. This event could be a function call either in kernel space(kprobes) or userspace(uprobes). Pixie uses both uprobes and kprobes to enable observability across services and applications. 

Pixie automatically harvests telemetry data by leveraging eBPF, and its edge-machine intelligence connects this data with Kubernetes metadata to provide visibility while maintaining data locality. This visibility complements New Relic’s powerful Kubernetes observability solution. And starting in late May, you'll be able to send Pixie-generated telemetry data to New Relic One, gaining scalable retention, powerful visualizations, advanced correlation, and intelligent alerting capabilities.

eBPF observability 

eBPF has been a game-changer in the realm of observability, offering unprecedented insights into system behavior and performance.

By allowing custom code to run directly within the kernel space, eBPF enables real-time data collection and analysis without the overhead typically associated with such tasks. This has revolutionized the way engineers monitor systems, troubleshoot issues, and optimize performance. With eBPF, it's possible to observe system calls, network events, and even hardware-level activities, all while maintaining a minimal performance footprint. This granular level of visibility is not just a boon for debugging and performance tuning, but it also empowers organizations to proactively identify and mitigate security risks. In essence, eBPF enhances observability by providing a more nuanced, efficient, and comprehensive view into system operations.