What is eBPF, and why does it matter for observability?

eBPF is an evolution of BPF (Berkeley Packet Filter). BPF was originally created to allow observing and monitoring of network traffic. It was designed mainly for packet filtering but lacked versatility for other critical capabilities. 

eBPF is far more innovative and extensive, essentially going beyond network packet filtering to observing various activities within the kernel. eBPF allows developers to explore various system calls, trace points, and more, making it an ideal technology for performance monitoring, security, and observability.

eBPF makes the tasks of IT and observability more efficient by running versatile programs in their own environment in the Linux kernel instead of copying data to user space. These programs can give engineers powerful and customizable tools at a lower level of the software stack for deeper insight into system behavior. The technology empowers developers to achieve a greater level of granularity and control, offering real value in today's complex computing landscapes.

Working within the Linux kernel is ideal when implementing security, networking, and observability features. However, it’s not without its challenges. Whether modifying kernel source code or adding modules, developers have traditionally found themselves contending with complicated infrastructure and abstracted layers that are difficult to debug. Extended BPF (eBPF) solves both of these problems.

What is eBPF?

The Extended Berkeley Packet Filter (eBPF) is a groundbreaking kernel technology introduced in Linux 4.x, enabling bytecode to run directly within the Linux kernel. Functioning as a lightweight, sandboxed virtual machine embedded in the kernel, eBPF provides controlled access to specific kernel resources in a secure and efficient manner.

With eBPF, developers can execute custom programs in kernel space without the need to modify kernel source code or add additional modules. This avoids the complexities typically associated with diving into the kernel's internals, making it significantly easier to build software that seamlessly integrates with and utilizes existing kernel functionality.

eBPF revolutionizes modern software development by providing streamlined access to core kernel functionality. This groundbreaking technology has the potential to redefine how essential services like networking, observability, and security are implemented. With its unparalleled power and versatility, eBPF stands as a transformative innovation in the tech landscape.

What is the difference between BPF vs eBPF?

eBPF (Extended Berkeley Packet Filter) and BPF (Berkeley Packet Filter) both serve as technologies for observing and monitoring network traffic, but they are far from identical. 

BPF, the original technology, is relatively limited and was designed mainly for packet filtering. It's like the old-school tool that gets the job done but lacks versatility. eBPF, on the other hand, is a more advanced and flexible version, extending BPF's capabilities beyond just networking. Think of eBPF as BPF's ingenious upgrade—like going from a flip phone to a smartphone. eBPF allows you to run sandboxed programs in the Linux Kernel, giving engineers powerful and customizable insights into system behavior.

It's not just about packets anymore; eBPF can help you explore various system calls, trace points, and more. If BPF is a hammer, eBPF is the entire toolbox. Both are reliable, but eBPF empowers you to achieve a greater level of granularity and control, offering real value in today's complex computing landscapes.

Here’s a closer look at what it is, how it works, and when to consider implementing it.

The basics of eBPF

When diving into the world of eBPF, there are several key terms and concepts you’ll need to familiarize yourself with to fully leverage its empowering capabilities. 

• eBPF bytecode: This is the compiled form of your eBPF program. It's what gets executed by the eBPF virtual machine inside the Linux kernel.
• eBPF virtual machine: A lightweight, in-kernel virtual machine that interprets and runs the eBPF bytecode, offering you real-time data and insights.
• Verifier: Before your eBPF bytecode runs, the verifier checks it for safety and compliance, ensuring that your program won't harm the system.
• Maps: These are key-value stores accessible from both user-space and kernel-space, used for storing and sharing data between eBPF programs and user-space applications.
• Hooks: These are the specific points in the Linux kernel where you can attach your eBPF programs, such as system calls or networking events, to monitor or modify behavior.
• XDP (eXpress data path): A high-performance data path that enables eBPF programs to process packets directly at the driver level, making decisions even before the kernel gets deeply involved.
• BPF type format (BTF): This provides rich type information to make your eBPF programs more understandable and easier to write.
• bpftrace: A high-level tracing language for eBPF, designed to simplify the task of collecting data from the kernel for analysis.

Armed with an understanding of these essential eBPF terminologies, let's delve into how this intricate yet empowering technology actually functions within the Linux kernel.

How eBPF works

eBPF programs are event-driven and attached to a code path. The code path contains hooks which execute any attached eBPF programs when they’re passed. Some examples of hooks include network events, system calls, function entries, and kernel tracepoints.

When triggered, the code is compiled first to the BPF bytecode. In turn, the bytecode is verified before it runs, to ensure it doesn’t create a loop. This step prevents the program from compromising the Linux kernel either accidentally or on purpose.

After a program is triggered at a hook, it then makes helper calls. These helper calls are functions that equip eBPF with many features for accessing memory. Helper calls need to be pre-defined by the kernel, but the list of what functions exist continues to grow.

eBPF was initially used as a way to increase observability and security when filtering network packets. However, over time, it became a way to make the implementation of user-supplied code safer, more convenient, and better-performing.

The advantages of eBPF

eBPF is typically used to trace user-space processes. But it offers even more powerful technology beyond just tracing. Running in kernel space, it’s faster, less intrusive, and more secure, among other benefits. It’s a safe and useful method to ensure:

• Speed and performance. eBPF can move packet processing from the kernel-space and into the user-space. Likewise, eBPF is a just-in-time (JIT) compiler. After the bytecode is compiled, eBPF is invoked rather than a new interpretation of the bytecode for every method.
• Low intrusiveness. When leveraged as a debugger, eBPF doesn’t need to stop a program to observe its state.
• Security. Programs are effectively sandboxed, meaning kernel source code remains protected and unchanged. The verification step ensures that resources don’t get choked up with programs that run infinite loops.
• Convenience. It’s less work to create code that hooks kernel functions than it is to build and maintain kernel modules.
• Unified tracing. eBPF gives you a single, powerful, and accessible framework for tracing processes. This increases visibility and security.
• Programmability. Using eBPF helps increase the feature-richness of an environment without adding additional layers. Likewise, since code is run directly in the kernel, it’s possible to store data between eBPF events instead of dumping it like other tracers do.
• Expressiveness. eBPF is expressive, capable of performing functions usually only found in high-level languages.
• Real-time, event-driven data capture:  eBPF allows for immediate logging of specific events directly from the kernel, providing timely and accurate data for monitoring and analysis.
• eBPF Packet Inspection: Prior to BPF and eBPF, packet inspection had to take place in user space, which added latency to the processes. Since eBPF operates in kernel space, it’s faster and more efficient.

What are the disadvantages of eBPF?

Although it’s compact, and powerful, eBPF may not suit every project or ecosystem. Some developers might find that eBPF is not the right fit for solving particular issues. Here are some of the key challenges using eBPF. 

• Restricted to a newer Linux kernel: eBPF is a Linux technology and runs in kernels 4.13 and above. As a result, t’s less portable than other tracers. Additionally, eBPF has evolved over time and continues to evolve. An eBPF program written for one version of Linux might not run on another version.
• Sandboxed programs are limited: eBPF derives increased security by limiting what resources programs can access. However, by limiting what parts of the OS a program can access, functionality is also potentially limited.
• Resource constraints: eBPF can reach its capacity limit, missing new events and rendering a system blind to potentially malicious activities. Attackers can exploit this and execute actions without detection, compromising your security.
• Complexity: To develop eBPF programs you need a good understanding of kernel-level programming and the intricacies of eBPF.  Also, there’s limited stack space for eBPF, which can make programming more difficult and less intuitive.
• Performance Overhead: Extensive use of eBPF can lead to performance degradation, especially in resource-constrained environments or those with high traffic volume.
• CPU-Intensive Operations: Some operations using eBPF are CPU-intensive and thus too expensive, such as decryption and re-encryption for encrypted flows.
• Security Vulnerabilities: While eBPF programs run in sandboxed environments and must pass a verification process before execution, there is still a potential for security vulnerabilities if not implemented carefully. These include a lack of anti-tamper capabilities, probe unreliability, data truncation, and others.

eBPF use cases

With the benefits offered by eBPF technology, it is rapidly gaining traction for observability use cases, especially in cloud native applications. These use cases include real-time system, network, and security monitoring, as well as application performance monitoring. 

• Real-time system monitoring: eBPF offers granular insights into various system metrics like CPU utilization, memory allocation, and disk I/O operations, providing a comprehensive view of system health.
• Security: eBPF allows for fine-grained control over system calls and network activities, enabling the identification and mitigation of potential security vulnerabilities.
• Networking: Used for traffic shaping, load balancing, and routing, eBPF can intercept and manipulate network packets directly within the kernel, offering advanced networking capabilities.
• Application performance monitoring (APM): Developers can use eBPF to trace micro-level events in their code, helping to identify bottlenecks and optimize application performance.
• Versatility: Due to its efficiency and flexibility, eBPF is applicable in a wide range of fields including system administration, cybersecurity, networking, and software development.

eBPF best practices

Since eBPF is such a new technology, many things remain unexplored. Best practices around eBPF are still evolving as the technology gains prominence. While no defined set of best practices exist, there are a few things that you can do to ensure effective, efficient programs.

If you’re using eBPF for your ecosystem, we recommend that you:

• Use LLVM Clang to compile C into bytecode. When eBPF first hit the scene, it was necessary to code and assemble the program by hand. Then, developers used the kernel’s assembler to generate bytecode. Fortunately, it’s no longer necessary to do this. Clang provides infrastructure for frontend and tooling in C languages.
• Use the BCC toolkit when writing BPF programs. The BPF Compiler Collection (BCC) is a toolkit that can help you create efficient kernel tracing and manipulation programs. It’s especially suited for tasks related to performance analysis and controlling network traffic.
• Leverage eBPF helper functions: The Linux kernel provides a set of helper functions that can be called from within eBPF programs. These functions provide a way to interact with kernel data structures and perform various operations, making your eBPF programs more powerful and flexible.
• Use maps for data sharing: eBPF maps are key-value stores that facilitate data sharing between the kernel and user space, or between different eBPF programs. Utilizing maps effectively can greatly enhance the functionality of your programs.
• Monitor resource limits: Be mindful of the resource limits imposed by the kernel, such as the maximum number of active eBPF programs and maps. Exceeding these limits can lead to program load failure.
• Validate user input: If your eBPF program interacts with user space, ensure that you validate any user input to prevent potential security vulnerabilities.
• Keep abreast of kernel updates: eBPF is a rapidly evolving technology, and new features and improvements are regularly added to the Linux kernel. Keeping your kernel up-to-date will allow you to take advantage of these enhancements.
• Employ eBPF libraries and frameworks: Consider using libraries like libbpf for loading and manipulating eBPF programs and maps, or frameworks like Cilium for network security and observability tasks. These tools can simplify the development process and offer additional features.
• Document your code: Given the complexity and low-level nature of eBPF programs, thorough documentation is essential for maintainability and future debugging.

By following these practices, you can ensure that your eBPF programs are not only effective but also secure, maintainable, and up-to-date with the latest advancements in this technology.

New Relic and eBPF

Pixie (acquired by New Relic), is an open source, kubernetes-native-in-cluster observability platform that provides instant visibility into Kubernetes workloads with no manual instrumentation. eBPF provides most of the magic behind the Pixie platform. As described earlier, eBPF allows you to run restricted code when an event is triggered. This event could be a function call either in kernel space(kprobes) or userspace(uprobes). Pixie uses both uprobes and kprobes to enable observability across services and applications. 

Pixie automatically harvests telemetry data by leveraging eBPF, and its edge-machine intelligence connects this data with Kubernetes metadata to provide visibility while maintaining data locality. This visibility complements New Relic’s powerful Kubernetes observability solution. And starting in late May, you'll be able to send Pixie-generated telemetry data to New Relic One, gaining scalable retention, powerful visualizations, advanced correlation, and intelligent alerting capabilities.

eBPF observability 

eBPF has been a game-changer in the realm of observability, offering unprecedented insights into system behavior and performance.

By allowing custom code to run directly within the kernel space, eBPF enables real-time data collection and analysis without the overhead typically associated with such tasks. This has revolutionized the way engineers monitor systems, troubleshoot issues, and optimize performance. With eBPF, it's possible to observe system calls, network events, and even hardware-level activities, all while maintaining a minimal performance footprint. This granular level of visibility is not just a boon for debugging and performance tuning, but it also empowers organizations to proactively identify and mitigate security risks. In essence, eBPF enhances observability by providing a more nuanced, efficient, and comprehensive view into system operations.