There are currently over 1,100 Internet Exchange Points around the world according to PeeringDB. These Internet infrastructure locations serve as a meeting point for networks all over the planet, allowing for cheap (or free!) exchange of low latency traffic.
The idea is simple: if a lot of networks are in the same datacenter, why have them spend a fortune connecting to each other with direct cables? We can add some switches there, everyone connects to one of them, and then they get a Layer 2 connection to all other participants.
The outcome of this is a more robust Internet, that’s more local, faster, and more secure. The more interconnections these ISPs have between them, the better it is for everyone.
However, when you join an IX, you are plugged into a LAN where tens or hundreds or thousands of others are also there with their equipment. Each device has a direct connection to all others, which is done at a much lower level than IP. So things are bound to happen :)
One of the common problems of that is people forgetting to turn off Layer 2 protocols (or not knowing they are on), and sharing more than you’d like with others. I saw today that BGP Tools added support for detecting such behavior in a small number of participating IXes, so I found the time to blog about my experience.
In order to get an idea of what traffic is flowing in the Peering LANs, I
started a very constrained tcpdump packet capture. I used my own network,
AS210312, and I performed this at some of the
exchanges I am connected to. Keep in mind that anyone can do this, I didn’t
have any elevated privileges in any of these platforms, and this is traffic
going to everyone. Here’s a list of things found:
For those unfamiliar with IPv6, this is the equivalent of DHCP running over there. There are plenty of participants that are sending “RAs” to others, begging them to send all their traffic to their equipment. Thankfully, a properly configured router should not accept these, as they’re only for clients, but if a device does this, it will add a default route pointing to this network’s machine.
A lot of Internet Exchanges have a policy against people using non-automated ways of routing traffic over the Peering LAN. For example, you’re usually not allowed to add static routes to someone’s IP address if they don’t agree to peer with you. I would argue that although RAs are not BGP, this is an automated means of inviting you to send traffic to that network ;) Who doesn’t like free outgoing transit?
The protocol used to route the Internet is BGP. This is a Layer 3 protocol, which means that you need to specify the other routers’ IP addresses so you can establish a (TCP) connection to them. But some IGPs (the routing protocols that are used within an ISP) are Layer 2 based protocols. You don’t specify a peer’s IP address, but you usually specify a network interface and then there’s some kind of automatic discovery of other routers. Examples of such protocols are the RIP family, OSPF, and Babel.
Well… It seems that people accidentally included the IXP interface, or forgot to exclude it, and these protocols are now running in an untrusted environment. If no authentication is configured, by having your own device speak that protocol on the LAN you can learn other people’s internal routing tables, and you can also modify them! Fun!
There are many standard and proprietary protocols for devices to automatically detect others on the network and play nice with them. These tend to be very useful in automating network deployments, but can be very bad when ran in an untrusted environment.
Having all of your switches automatically pick up your network’s VLANs and add them to the trunk ports sounds awesome, unless hundreds of strangers can do that too.
Although I’ve seen plenty of these, and most IXPs ask you to turn them off, for your own good, I’ll focus on one particular such feature today, which is none other than MikroTik’s proprietary MikroTik Neighbor Discovery Protocol (MNDP). There’s nothing special about it, there are other vendors doing it too, but I picked it as it’s less known and turned on by default, without many people knowing it exists at all.
It works by periodically sending a UDP packet to port 5678 at the IPv4
address 255.255.255.255, which is the broadcast one, that all devices end up
receiving the packets for. What’s inside?
Okay, this is interesting. What can we do with that? Let’s create some statistics! I will look at one IXP, where I can find 5 devices with the protocol enabled. What can we learn?
I will not publish the exact names, but I will say that some times people make typos. Also, there’s a clear preference for not using FQDNs, as only one of them is a valid domain name. From the remaining 4, only one has some hierarchy using dots.
This can act as a poor person’s survey of RouterOS 7’s popularity :) I can see the following versions:
Looks like with a tiny sample size of 5, most people are on RouterOS 7 already! And almost at the latest and greatest (7.12) too! Understandably there are people holding back and staying with RouterOS 6, especially for long term stability, but they’re missing out on all the new features of the next major version.
As expected, the uptime is measured in days, and it correlates with the release of the OS’ version. However, I must admit that they are less than I expected for the older software. The smallest uptime was 26 days, and the highest was 177 days, making all of them up for less than half a year.
Since MNDP helpfully provides us with the exact router model, we can look at what people are using these days for a BGP router. We have:
It looks like people are using the company’s virtualized / Bring-Your-Own-Hardware solution, the Cloud Hosted Router (CHR) a lot. It provides an easy and familiar management interface for any x86 hardware at a low one-time fee.
For MikroTik hardware we see most networks are in the older generation Cloud Core Routers (CCR), which is now End of Sale. One of the networks has the company’s former flagship, while the other has one of their smallest ISP-tier routers. Only one ISP has upgraded to the second generation hardware with a mighty 12x10G + 2x25G device (connected with 2x25G to a 4 core ARM CPU ;).
We’ve polled 5 MikroTik operators! What interface do they prefer plugging their IX port to? Do they leave the default names, or do they change them? Find out here!
We have 2 etherN interfaces, one IXNAME interface, one etherN-IXNAME, and
finally one entirely custom. Funnily enough, the custom name contains the word
“test” although this definitely looks like a production setup.
As far as IPs go, all routers had one IPv4 and one IPv6 address configured, no secret uncovered here, with the exception of one of them that only had Legacy IP.
Things aren’t terrible or on fire, this is nothing new. With a few exceptions where the IXP infrastructure will parse the packets and act on them (STP, LACP, others), this doesn’t harm the platform itself, but it could harm other participants (RAs), or it could leak information about the device connected. In some more extreme cases it could also give control of the router or its routing table to a malicious actor.
Network Engineers must pay attention to the features enabled, configuration best practices, and try to minimize their Layer 2 footprint across a shared LAN.
Internet Exchanges can help as well. By utilizing ACLs they can statelessly drop some bad packets, while with more modern solutions they can reduce the BUMs to a minimum. For example, although I am in DE-CIX, one of the largest places by peer count, and although it has plenty of MikroTiks, I haven’t received a single MNDP message there. And I somehow doubt that everyone on it is just super careful.
I don’t think it’s the platforms’ job to do this, but it just seems easier, and a nice-to-have. If a router is misconfigured, it will have the same issues on all of its transit interfaces, private peering interfaces, customer interfaces, and so on. This doesn’t address the actual problem, it just limits the attack surface.