In DDoS Protection, Gcore uses the bundle of XDP and regular expressions (regex). This article will explain why Gcore started using this solution (regex in XDP) and how they bound them via a third-party engine and API development.
Moreover, we’ll also explain their open-source solution for handling regex in XDP and benchmarking results.
The reason behind using the XDP framework
In the early days of their DDoS protection service, they operated on a small number of dedicated servers (nodes) running DPDK (a Linux framework for fast packet processing) and filtered traffic with regular expressions. By using this technology, they have successfully safeguarded their clients’ apps thanks to this bundle. However, as the capacity of DDoS attacks increased from 300 Gbps in 2021 to 700 Gbps in 2022, the underlying infrastructure eventually proved insufficient.
DPDK requires exclusive network adapter access to function effectively. In this way, integrating it with other uses is extremely difficult (and unreasonable) to the point of being impossible. This necessitates the acquisition of new, dedicated nodes for DPDK if the existing infrastructure is to grow without modification. Gcore concluded that this was not a cost-effective option and set out to find one.
In addition to their DDoS protection services, Gcore also offers Content delivery network (CDN) infrastructure consisting of over a thousand CDN nodes. So, it makes sense for them to begin employing it for content distribution and traffic screening since more nodes mean better security against increasingly powerful attackers.
Since DPDK can’t work with CDN nodes (it needs dedicated ones), Gcore has opted to use the XDP framework instead. They claim that the main improvement over the previous version is how well it integrates into the stack with other applications. DDoS Protection was previously only available on dedicated servers with DPDK (Fig. 1), but now it can be integrated into CDN servers (Fig. 2), allowing for greater scalability.
Source: Gcore
Source: Gcore
Why Gcore finds XDP useful:
Cost efficiency. The framework can be installed on servers with any edge-network applications, such as web servers and DNS servers, so it doesn’t require expensive dedicated equipment, and developers don’t need to spend hours integrating other applications with XDP.
Attack detection speed. The framework can be installed on hundreds of CDN servers. That means that DDoS Protection is closer to client applications and malicious traffic sources. As a result, attacks are stopped faster and don’t go deep into the infrastructure.
But it also has several drawbacks:
Lower performance. Separated nodes with XDP have lower performance compared to DPDK, but because of the incorporation of more nodes, the overall efficiency of the solution is ultimately higher.
Inability to handle regex. XDP has no built-in engine to handle regular expressions, so they had to come up with a solution to adapt regular expression processing to it.
Why Gcore kept using regex to filter traffic
There are two approaches to filtering out malicious traffic in DDoS Protection: packet parsers and handling regular expressions (regex).
Packet parsers are manually written filters that are programmed to detect and block suspicious activity in applications using a particular protocol. Writing such packet parsers requires a lot of programming work, especially if the DDoS protection needs to be able to accept new protocols quickly.
Handling regex based on analyses of packet payloads lessens the time it takes to create filters versus the packet parsers approach. Moreover, it’s a more flexible approach that allows packets to be processed more efficiently with a lower kernel load.
Packets that are sent to their customers’ applications are checked in two modes:
Reaction to attacks (manual mode). They analyze malicious traffic generated by a specific payload (pattern). Then create regular expressions that point to this payload and apply it to traffic. All requests that contain a similar payload will be automatically filtered.
Game connection protection mode. Many Gcore clients are online gaming services providers characterized by requests over the UDP protocol and small packet sizes. Packages coming to gaming services have a strict structure that can be described using regular expressions. They create regular expressions for each client’s game service and use it to create an allowlist of packets. All packets that match regular expressions will be allowed. If the packets differ, they’ll be blocked.
Packet processing when using regex is arranged as follows:
Dissector. Analyzes each packet and splits it into headers.
Flow Router. Routes packets to the appropriate protection profile, which is a typical set of rules for traffic protection.
Policy Pipeline. Applies special rules (countermeasures) to packets disassembled into components. One of the countermeasures is the use of regular expressions.
Verdict. Skips or blocks a packet based on countermeasure checks.
How Gcore adapts regex processing in the XDP context: Challenges and solutions
Working with regex is a resource-intensive process, and Gcore claims they’re going to check millions of packets and use regular expressions of different complexity. This led them to conclude that performance is a must-have requirement for a regex engine.
The best engine available is Hyperscan, designed by Intel. It’s open-source with a license compatible with GPL, which is fast because it uses an AVX2/AVX512 vector instruction set and is used as an industry standard for DPI applications.
To adapt the regex processing in XDP, they encountered several challenges, which are described below.
Challenge 1. Limitations of eBPF don’t allow regex filters to be implemented as part of the XDP program.
Solution. Gcore rebuilt the Hyperscan engine as a loadable Linux kernel module providing eBPF helpers. Hyperscan is an engine designed to process regular expressions in DPI (Deep Packet Inspection) systems, checking if a packet’s payload matches any predefined regular expressions.
Challenge 2. eBPF helpers from loadable modules can’t be registered for XDP.
Solution. eBPF helpers in loadable modules were first introduced in Linux 5.16, but registering them for XDP wasn’t possible until Linux 5.18. Because Gcore only had Linux 5.17 during development, they had to provide that possibility. Mainline kernel builds don’t require that kind of patching.
Challenge 3. Vector instructions (FPU) were not supposed to be used during packet processing in the Linux kernel.
Solution. They save and restore the FPU registers state as they enter the module for regex processing. They perform this per packet when required without affecting other packets for which regex processing isn’t required.
What open-source solution does Gcore offer to the community: eBPF API for handling regex in XDP
If your infrastructure needs to handle regular expressions in XDP, you can use a ready-made solution provided by their developers instead of going all the way from scratch.
Their custom eBPF helper ‘bpf_xdp_scan_bytes()’ can now be used in the same way as other eBPF helpers.
struct rex_scan_attr attr = {
.database_id = regex_id,
.handler_flags = REX_SINGLE_SHOT,
.nr_events = 0,
.last_event = {},
};
err = bpf_xdp_scan_bytes(xdp, payload_off, payload_len, &attr);
if (err < 0)
return XDP_ABORTED;
return (attr.nr_events > 0) ? XDP_DROP : XDP_PASS
To evaluate regex against the packet buffer, first add a regex into the loadable module and reference its identifier when calling the eBPF helper:
Create a node using mkdir under /sys/kernel/config/rex
Compile pattern database:
echo ‘101:/foobar/’ > patterns.txt
echo ‘201:/a{3.10}/’ > patterns.txt
build/bin/hscollider -e patterns.txt -ao out/ -nl
Upload compiled regex to the /sys/kernel/config/rex/<node>/database:
dd if=$(echo out/”.db) of=/sys/kernel/config/rex/hello/database
Read or set a new regex identifier at /sys/kernel/config/rex/<node>/id
Transfer regex identifier to eBPF program and use as a helper argument.
The full source code is available in the Gcore GitHub account by link: https://github.com/G-Core/linux-regex-module
What benchmarks do Gcore have on regex usage in XDP
Their DDoS filtering solution is based on 3rd Generation Intel® Xeon® Scalable processors and 100GbE Intel® Ethernet Network Adapter E810. Intel® Hyperscan enabled high-performance pattern matching across data streams.
Intel® provided expert insight, including on the XDP technology that was used for packet filtering. According to them, using the new software on the latest 3rd Generation Intel® Xeon® Scalable processors has increased the filtering capacity from 100 Gbps to up 400 Gbps or 200 million packets per second.
In the charts below, you can see what the results of their own testing.
The linerate (blue line) is the maximum network throughput of 4 × 100 Gbps interfaces.
The base (red line) is a measure of how XDP would have handled it without using regular expressions.
The bottom three lines are packet handling when using regular expressions.
Conclusion. On a packet size bigger than 512 bytes, the system can operate on the line rate speed, effectively filtering traffic. On packets smaller than 512 bytes, the packet rate and the pressure for the system are much higher, and the system can’t maintain linerate performance and works roughly at 40–50% of the line rate speed.
Gcore truly is satisfied with these results. Based on their data, they suit us even though the performance for processing small packets isn’t as high. Because real DDoS attacks tend to use large packets, the efficiency and speed are sufficiently acceptable.
Tests show that regex usage in XDP is suitable for processing heavy processes, yet it has enough speed to handle large traffic.
Sponsored and written by Gcore
