MANRS+: Assessing the feasibility of routing security compliance tests

Routing security through MANRS

Routing on the internet is based on the Border Gateway Protocol (BGP), which allows Autonomous Systems (ASes) to interchange information about which IP addresses can be reached by which routes. However, BGP does not have a mechanism for validating the integrity of routing information and is therefore vulnerable to various types of attack, such as route hijacking. These attacks are common in the wild and can cause internet-based services, such as the DNS infrastructure for .nl, to become unavailable.

To make the internet less vulnerable to these threats, MANRS (Mutually Agreed Norms for Routing Security) recommends routing security measures for AS operators, ranging from filtering BGP announcements to implementing and providing open-source monitoring tools to the community. The MANRS community calls these "Actions," and continuously measures their correct implementation through the MANRS Observatory.

MANRS has become a baseline for good routing security practice since its inception in 2014 and currently involves over 1,200 AS operators globally. At SIDN, we have been a MANRS member since 2018 because we consider a secure routing system to be essential for the .nl ecosystem and the entire internet. Other members in the Netherlands include KPN, VodafoneZiggo, TransIP, BIT and SURF. MANRS is managed by the Global Cyber Alliance (GCA).

MANRS+: taking MANRS to the next level

The MANRS+ working group, also managed by GCA, recently adopted MANRS+. It is an extension to MANRS that (1) defines additional actions and (2) aims to validate MANRS conformance with a high assurance level by means of measurements and audits. MANRS+ was set up to meet the demands of enterprise customers for enhanced routing security, so they can, for example, better protect their communications supply chain and select connectivity providers that actively manage the security of routes (by supporting MANRS+).

MANRS+ focuses on ASes that act as Connectivity Providers (CP ASes) and their role in BGP. A CP AS can have 3 types of BGP relationship: a relationship with its peers, a relationship with its transit providers, and a relationship with its customers. A CP AS provides connectivity to its customers by forwarding their (legitimate) routes to its peers and transit providers and vice-versa.

MANRS+ requires that CP ASes implement different types of so-called “security controls”, for instance for routing security, DDoS attack mitigation and anti-spoofing protection. It also defines guidelines on how to validate that CP ASes have implemented these controls correctly as well as different auditing levels (Self-declared, Audited or Measured). It defines a total of 21 security controls.

Problem: reliably measuring MANRS+ conformance

An important problem is that it is difficult to reliably measure to what extent future MANRS+ members are correctly implementing controls to filter BGP messages, specifically:

• Route Origin Validation (MANRS+ Routing Security control RS-01)

• Prefix Filtering of Customers (RS-02)

• Control a set of customer ASes (RS-03)

• Filtering of bogons (RS-06)

Checking these controls reliably is difficult because it requires that no other ASes are on the path between the measurement infrastructure and a CP AS. Other ASes would influence the test results and make it less clear whether correct filtering was being performed by the CP AS of interest, or by another AS on the path. Therefore, it is necessary to establish a BGP session with the CP AS that is being tested.

GCA contacted us to ask whether we could help them, given our experience with internet measurements. We subsequently carried out our study in collaboration with GCA. Checking the correct implementation of any of the other 17 MANRS+ security controls is outside the scope of our work.

Design of our MANRS+ measurement system

To solve the above problem, we developed a new methodology to measure with a high assurance level whether future MANRS+ participants have implemented the 4 filtering security controls RS-01, RS-02, RS-03 and RS-06. Figure 1 provides a high-level design overview of our measurement system, the source code of which is publicly available on GitHub.

Figure 1: High-level overview of our MANRS+ measurement system: the BGP peerings between the MANRS+ Peer AS, Customer AS and CP AS.

In order to measure the extent to which a CP AS has configured filtering, our system sets up 2 direct peering sessions with the CP AS. One with the MANRS+ Peer AS playing the role of a peer AS (arrow on the left in Figure 1) and one with the MANRS+ Customer AS playing the role of a customer AS (right arrow). In the future, we envisage that the Customer AS and the Peer AS will be operated by a “MANRS+ auditor”, a new business role in the routing ecosystem, which we discuss in more detail later in this blog.

Using the 2 peering sessions, our system enables a variety of tests to be performed to check how the CP AS processes announcements. Specifically, we announce prefixes from the MANRS+ Customer AS and monitor the announcements received by the MANRS+ Peer AS. If the latter receives routes that should be rejected according to the MANRS+ filtering controls, then this is an indication of improper filtering and configuration of the CP AS router. We use prefixes that aren’t used anywhere else on the internet to avoid potential damage if the CP AS were to accidentally leak our test announcements onto the internet.

The tested CP AS does not have to be connected to the measurement infrastructure via a direct physical connection. If there is no such connection, the MANRS+ Customer and Peer ASes can still set up direct BGP sessions with the CP AS, but the BGP messages transparently pass through multiple underlying networks (BGP multihop).

Testing the MANRS+ routing security controls for filtering

Test for RS-01: our system measures the correct configuration of Route Origin Validation (ROV) by sending route announcements with an invalid Route Origin Authorisation (ROA) to the CP AS. If the latter implements ROV, it should discard such BGP announcements, and we will not see them at the Peer AS. ROA objects allow AS operators to cryptographically prove that they have the right to use and announce a range of IP addresses (a prefix). ROA objects are stored in the Resource Public Key Infrastructure (RPKI), a decentralised database which has its trust rooted in the 5 Regional Internet Registries (RIRs).

Test for RS-02 and RS-03: our system injects routing announcements into the CP AS that it should not announce to the Peer AS based on information from the Internet Routing Registries (IRR) and other public sources. For example, the CP AS should drop announcements for a prefix that the Customer AS is not allowed to announce according to the IRR. The IRR consists of several distributed routing policy databases with information on which ASes are allowed to announce which prefixes, contact persons and IP ranges, among other things. In contrast to the RPKI, the integrity of data in IRR databases is not cryptographically secured.

Test for RS-06: we measure the correct filtering of bogon prefix announcements by announcing them to the CP AS. The CP AS should filter those and not announce them to the Peer AS. A bogon announcement is an announcement of a prefix that is not intended to be routed on the public internet. Examples are the private IPv4 and IPv6 ranges.

Our prototype implementation

To assess the feasibility of our approach, we developed a prototype of our measurement system in our lab at SIDN using containerlab, a tool for orchestrating container-based networking labs. In contrast to tools like docker-compose, it allows easy configuration of links between containers and is therefore more suitable for our setup. In our prototype, we use 3 containers to configure a topology that simulates the relationships between the 3 ASes in Figure 1: CP AS, MANRS+ Customer AS and MANRS+ Peer AS.

For the routers in our 3 simulated ASes we use the BIRD software router. It is well maintained and open source. That makes it accessible for everyone who wants to run our prototype. To validate and serve RPKI data to the CP AS router, we run a Routinator instance as a validating cache.

In addition to the routers for our 3 local ASes, our topology includes a local IRR and a local RPKI (not shown in Figure 1 for simplicity). Running a local IRR and RPKI allows us to easily manipulate data to demonstrate the different control test cases. We use Krill and rsync to run a local publication server and IRRd, Postgres and Redis to run a local IRR.

Result: we can measure MANRS+ routing security controls reliably

Our prototype system was able to identify the implementation of the 4 MANRS+ routing security controls at the CP AS reliably. We determined that by running our test scripts for different test cases (SR-01, SR-02, SR-03 and SR-06), which yielded accurate and reproducible results.

We therefore believe our system is a suitable basis for a future “MANRS+ auditor”, a new business role we envisage for auditing the MANRS+ compliance of CP ASes by running the Customer AS and the Peer AS (Figure 1).

Extensions and considerations for a future MANRS+ auditor

Nevertheless, we recommend that such a MANRS+ auditor carefully reviews our design and prototype because our work was a feasibility study. Extensions that we envision include:

• Scheduling tests: our system needs to be extended so that it automatically updates objects in the RPKI and in an IRR database before carrying out tests. For example, when the Customer AS announces a prefix that the CP AS should discard based on an ROV-invalid filter (RS-01), the system first needs to update the corresponding ROA, andthe announced prefix must be part of a valid AS-set in the IRR. That would enable us to be sure that the CP AS is dropping the prefix because of ROV and not because of invalid information in the IRR. Another example is that the system should give CP ASes sufficient time to update their filters. For example, changes to the RPKI can take an hour or longer to propagate, and filters that are based on information in the IRR might propagate even more slowly (e.g. 24 hours in the case of NTT). After that, the MANRS+ auditor can announce the prefix to the CP ASes. On the internet, BGP propagates updates usually in less than 2 minutes, so the operator should stop the test after 2 minutes or less.

• Auditing multiple CP ASes: our system needs to be extended so that the MANRS+ auditor can use it to audit multiple CP ASes in parallel. This means it should be able to announce the same prefixes to multiple CPs at the same time and monitor multiple incoming announcements at the MANRS+ Peer, for instance using the BGP Monitoring Protocol. That will enable the MANRS+ auditor to test the conformance of multiple CPs at the same time without the need for additional resources.

• Limit unintentional route propagation: A failed test means that the CP AS has not implemented the appropriate controls and that invalid announcements may propagate onto the internet. To avoid that, the MANRS+ auditor should implement 3 countermeasures. First, the MANRS+ Customer AS should withdraw the invalid announcement as soon it has been observed at the MANRS+ Peer AS. Second, a CP AS may allow its customers to influence the propagation of their prefixes through BGP communities. In this case, the announcements of the Customer AS should contain BGP communities that instruct the CP AS to propagate the prefixes only towards the MANRS+ Peer AS. Third, if a CP AS propagates test prefixes further, they may attract the attention of other network operators and researchers. Therefore, MANRS+ should publicly document the resources involved in the tests for transparency.

• Results should be validated independently: Tests performed by MANRS+ are based on trust. A CP could treat announcements and peering relationships with the MANRS+ Customer and Peer differently than it treats the rest of its peering sessions. Additional monitoring, similar to the measurements currently used for MANRS, could be deployed to independently monitor for the propagation of invalid routes.

We discuss those and other extensions in more detail in our public report for the MANRS+ working group.

Next steps for MANRS+ and our testbed

The next step is to test our approach on multiple real-world CP ASes on the internet, and to find an organisation that can take on the role of MANRS+ auditor. That is an activity that GCA will lead and that we will support with a presentation about our prototype system at the upcoming MANRS Community meeting on the 11 November.

Happy to get your feedback!

Since this was our first project in the space of routing security auditing, we’d be very keen to hear your opinion. Feel free to drop us an email.