Documentation

• 1: Overview
• 2: Quick Start
• 3: Concepts
• 4: Make Contributions

1 - Overview

What is Merbridge

Merbridge is designed to make traffic interception and forwarding more efficient for service mesh by replacing iptables with eBPF.

eBPF (extended Berkeley Packet Filter) allows users to run programs in the Linux kernel without modifying the kernel code or loading kernel modules. It is widely used in networking, security, monitoring, and other relevant fields. Compared to iptables, Merbridge shortens the data path between sidecars and services, thereby accelerating networking. Additionally, using Merbridge does not change the original architecture of Istio. The original logic remains valid, meaning that if you no longer want to use Merbridge, you can simply delete the DaemonSet and the original iptables will function again without any issues.

What Merbridge can do

Merbridge has following core features:

• Processing outbound traffic

  Merbridge uses eBPF’s connect program to modify the user_ip and user_port, in order to change the destination address of a connection and ensure that traffic can be sent to the new interface. To help Envoy identify the original destination, the application, including Envoy, will call the get_sockopt function to get ORIGINAL_DST when receiving a connection.

• Processing inbound traffic

  Inbound traffic is processed similarly to outbound traffic, but it’s worth noting that eBPF cannot take effect in a specified namespace like iptables, so changes will be global. This means that if eBPF is applied to Pods that are not originally managed by Istio or to an external IP, serious problems may occur, such as not being able to establish a connection.

  To address this issue, a tiny control plane was designed and deployed as a DaemonSet. It can watch and get a list of all pods on the node, similar to kubelet. Then, Pod IPs injected into the sidecar will be written into the local_pod_ips map. For traffic with a destination address not in the map, Merbridge will not intercept it.

• Accelerating your network

  In Istio, Envoy visits the application by the current podIP and port number. Because the podIP exists in the local_pod_ips map, traffic will be redirected to the podIP on port 15006, which creates an infinite loop. A solution to this issue is to use a feedback mechanism where when Envoy tries to establish a connection, it’s redirected to port 15006. When it moves to sockops, the source IP and the destination IP are checked to see if they are the same. If they are, it means that the request is incorrect and it will be discarded in the sockops process. Meanwhile, the current ProcessID and IP will be written into the process_ip map, allowing eBPF to support the corresponding relationship between processes and IPs. When the next request is sent, it will check directly from the process_ip map if the destination is the same as the current IP. Envoy will retry when the request fails. This retry process will only occur once and subsequent connections will go very fast.

Why Merbridge is better

In the service mesh scenario, in order to use sidecars for traffic management without the application being aware of it, the ingress and egress traffic of Pods should be forwarded to the sidecar. The most common solution is using the redirect capability of iptables (netfilter) to forward the original traffic. However, this approach increases network latency because iptables intercept both egress and ingress traffic, which causes duplicated steps to be performed several times. As a result, the data path becomes very long.

eBPF provides a function called bpf_msg_redirect_hash which allows for directly forwarding packets from applications in the inbound socket to the outbound socket. This can greatly accelerate packet processing in the kernel. Therefore, the aim is to replace iptables with eBPF, this is the main idea behind Merbridge.

When to use Merbridge

Merbridge is recommended if you have any of following problems:

1. In scenarios that require high-performance connections, using iptables will increase latency.
   • The performance of iptables control plane and data plane degrades as the number of containers in the cluster increases because it needs to traverse and modify all the rules every time a new rule is added.
   • Systems that use IP addresses for security filtering will come under increasing pressure as the Pod lifecycle gets shorter, sometimes just a few seconds, because it requires more frequent updates of iptables rules.
   • Using iptables to achieve transparent interception requires a conntrack module for connection trace, which causes a lot of consumption when there are many connections.
2. The system cannot use iptables for some reasons.
   • Sometimes it needs to process numerous active connections simultaneously, but using iptables easily causes a full conntrack table.
   • Sometimes numerous connections need to be processed in one second, which will exceed the limit of the conntrack table. For example, if you try to process 1100 connections per second with timeout set as 120 seconds and a table capacity of 128k, it would exceed the conntrack table’s limit (128k/120 seconds = 1092 connections/second).
3. Due to security concerns, some ordinary Pods cannot have too many permissions, but using Istio (without CNI) requires these Pods to gain more permissions.
   • Running the init container may require the NET_ADMIN permissions.
   • Running an iptables command may require the CAP_NET_ADMIN permissions.
   • Mounting a file system may require the CAP_SYS_ADMIN permissions.

In short Merbridge is a better alternative to iptables, as it’s faster, more efficient, and easier to manage in high traffic scenarios.

What Merbridge will change

Using eBPF can greatly simplify the kernel’s processing of traffic and make inter-service communication more efficient.

• Before applying eBPF with Merbridge, the data path between pods is like:

  

  Diagram From: Accelerating Envoy and Istio with Cilium and the Linux Kernel

• After applying Merbridge, the outbound traffic can skip many filter steps to improve performance:

  

  Diagram From: Accelerating Envoy and Istio with Cilium and the Linux Kernel

• If two pods are on the same node, the connection will be even faster:

  

  Diagram From: Accelerating Envoy and Istio with Cilium and the Linux Kernel

Merbridge is a completely independent open source project that is still in its early stages. We would greatly appreciate it if more users and developers could try this new technology to accelerate your mesh and provide feedback. By using Merbridge, you can benefit from faster and more efficient traffic management compared to using iptables, and can help to improve the project by providing feedback and participating in its development.

2 - Quick Start

Prerequisites

1. Use kernel 5.7 or a higher version. Check your version with uname -r.
2. Activate cgroup2 in your system. Check the status with mount | grep cgroup2.

Installation

Merbridge can be installed on Istio and Linkerd2 only.

Install on Istio

Apply the following command to install Merbridge:

kubectl apply -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one.yaml

Install on Linkerd2

Apply the following command to install Merbridge:

kubectl apply -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one-linkerd.yaml

Install on Kuma

Apply the following command to install Merbridge:

kubectl apply -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one-kuma.yaml

Verification

Verify installation

Before you start this verification, make sure all Pods relevant to Merbridge are running well. You can check Pod status in Istio with the following command:

kubectl -n istio-system get pods

If all these Pods are Running, it means Merbridge is successfully installed.

Verify connection

Use the following methods to check the connectivity of Merbridge:

Install sleep and helloworld and wait for a full start

kubectl label ns default istio-injection=enabled
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml

Conduct curl test

kubectl exec $(kubectl get po -l app=sleep -o=jsonpath='{..metadata.name}') -c sleep -- curl -s -v helloworld:5000/hello

If you see words like * Connected to helloworld (127.128.0.1) port 5000 (#0) in the output, it means Merbridge has managed to replace iptables with eBPF for traffic forwarding.

3 - Concepts

eBPF

The full name of eBPF is Extended Berkeley Packet Filter. As the name implies, this is a module used to filter network packets. For example, sockops and redir capabilities of eBPF can efficiently filter and intercept packets.

eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in an operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules.

iptables

iptables is a traffic filter built on netfilter. It implements traffic filtering and interception by registering hook functions on the mount point of netfilter. From the name of iptables, we can guess it may contain some tables. In practice, by mounting rule tables on different chains of netfilter, iptables can filter or modify the traffic packets entering and leaving the kernel protocol stack.

iptables has 4 tables by default:

• Filter
• NAT
• Raw
• Mangle

iptables has 5 chains by default:

• INPUT chain (ingress rules)
• OUTPUT chain (egress rules)
• FORWARD chain (rules of forwarding)
• PREROUTING chain (rules before routing)
• POSTROUTING chain (rules after routing)

Service Mesh

A service mesh is a dedicated infrastructure layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application. A service mesh can guarantee fast, reliable, and secure communication between containerized application infrastructure services. Key capabilities provided by mesh include service discovery, load balancing, secure encryption and authentication, failover, observability, and more.

A service mesh typically injects a sidecar proxy into each service instance. These sidecars handle inter-service communication, monitoring, and security. In this way, developers can focus on the development, support, and maintenance of the application code in the service, while the O&M team is responsible for the maintenance of the service mesh and applications.

Today, the most well-known service mesh is Istio.

Istio

Istio is a service mesh technology originally open sourced by IBM, Google, and Lyft. It can be layered transparently onto distributed applications and provides all the benefits of a service mesh, such as traffic governance, security, and observability.

Istio can adapt to all services hosted with on-premises, cloud, Kubernetes containers, and virtual machines. It is typically used with microservices deployed on a Kubernetes platform.

Fundamentally, Istio works by deploying an extended version of Envoy as a sidecar proxy to each microservice. The proxy network it uses forms a data plane of Istio. The configuration and management of these proxies is done in a control plane, providing discovery, configuration, and certificate management for Envoy proxies in the data plane.

Linkerd

Linkerd is the first service mesh launched on the market, but Istio is more popular today.

Linkerd is an open source, ultra-lightweight service mesh designed by Buoyant for Kubernetes. It is completely rewritten in Rust, which makes it as small, light and safe as possible. It provides runtime debugging, observability, reliability, and safety without code changes in distributed applications.

Linkerd has three basic components: UI, data plane, and control plane. Linkerd works by installing a set of ultra-light, transparent proxies next to each service instance that automatically handle all traffic to and from the service.

4 - Make Contributions

Merbridge is currently hosted on GitHub as an open source project. All code-related issues are managed on GitHub.

If you have any questions or suggestions about Merbridge, please create a New Issue on GitHub. We will review and process it as soon as possible.

If you find a bug in Merbridge and are interested in helping us fix it, you are more than welcome to share your fix code by creating a Pull Request. We will review and process it as soon as possible.