OVH announced today its OverTheBox project, which is basically a link-aggregation solution for Internet access links.

Analysis of the technology

Foreword on link aggregation

First of all, aggregating Internet access links has nothing to do with classical link aggregation (also called bonding or trunking). This is a much harder problem, because the access links typically have very diverse characteristics, in terms of latency, capacity, and packet loss.

Think of aggregating a DSL line, a FTTH line and a satellite connection. If you simply send packets in a round-robin fashion, you will basically get the worst out of each link: packets will be heavily reordered, causing TCP to fall apart. The latency of a flow will basically be the latency of the worst link. Additionally, packet loss on any of the links will heavily impact the whole flow.

Technology used in OverTheBox

For OverTheBox, the main technology used by OVH is Multipath TCP, often abbreviated as MPTCP. Multipath TCP basically allows to split a TCP flow across multiple paths, providing redundancy and increased throughput. It does so in a clever way: each subflow runs TCP independently, providing congestion control and packet loss recovery independently for each path. A scheduler decides on which path to send data, based first on the RTT of each path (lower RTT is preferred) and moving to the next path when the congestion window is filled.

While Multipath TCP was not initially designed for link aggregation, it implements all necessary ingredients to do this efficiently. However, it only works for TCP traffic, and requires that both ends of a TCP connection know how to speak Multipath TCP. This is actually by design: end hosts are in the best position to discover paths and their associated characteristics (the typical use-case being a smartphone with both 4G and Wi-Fi).

OVH used the Linux implementation of Multipath TCP, and based its distribution on OpenWRT, using an existing patch.

Since Multipath TCP is not yet widely deployed in end-hosts, a link-aggregation solution based on Multipath TCP must be transparent for the devices behind the aggregation point. To do this, OVH used a classical solution based on a VPN. The idea is to run a VPN protocol able to tunnel data over TCP, such as OpenVPN. This way, provided both the VPN client and servers and MPTCP-compatible, the VPN will automatically use all available paths, with associated load-balancing and failover benefits.

OVH apparently decided to use vtun, which I had never heard of before. That being said, there are also references to OpenVPN in the code, so I am not sure which one they use.

In addition to that, OVH seems to use a transparent SOCKS proxy, shadowsocks. The goal is to avoid TCP over TCP encapsulation, which is notoriously a bad idea. Thanks to the SOCKS proxy, TCP connections from local clients are terminated locally, and new TCP connections are established from the other end of the tunnel towards the destination. This way, any packet loss on the path towards the destination does not trigger retransmissions inside the VPN.

For UDP traffic, I am not sure whether it also goes through the SOCKS proxy (this is possible with SOCKS5, but would be somewhat useless in this case) or travels directly on the VPN.

Finally, as a last note, OVH decided to shoot IPv6 in the head by completely ignoring AAAA DNS requests in their local DNS resolver. This is a ugly hack, and sounds like a quick and dirty fix for an issue discovered just before the initial release. My guess is that either shadowsocks does not support IPv6, or the IPv6 connectivity provided by some of the access links interferes with the operation of the OverTheBox box. I do hope that this is a temporary fix, because crippling IPv6 like this will certainly not help its deployment. By the way, Multipath TCP of course fully supports IPv6.

By the way, this analysis is based on a rather quick look at the source code, and my own experience. If you think I made a mistake, feel free to send me an email (contact at the domain name of this blog).

Impact of OverThebox

As such, this project from OVH merely assembles existing components. It introduces nothing new, except maybe a nice web interface (which is actually non-negligible in terms of user impact).

And indeed, technically speaking, people have already been doing the exact same thing for a while: Multipath TCP for link aggregation, a VPN such as OpenVPN for encapsulation, and a transparent SOCKS proxy to terminate client TCP connections before entering the tunnel. See for instance this mail on the mptcp-dev mailing list.

But this is, to my knowledge, the first open off-the-shelf solution providing an easy-to-use interface. What's more, OVH released the code, and the solution should work just fine with your own VPN server: it does not force you to use OVH services, which is extremely nice.

This is in huge contrast with existing proprietary solutions for the same problem, such as the products sold by Peplink. Their business model is to sell you the hardware and the service, with associated licensing fees. Since the protocol is proprietary, you are forced to use the Peplink VPN servers (even though they seem to offer to deploy VPN servers in the cloud, that you can manage through their provided interface). OverTheBox is likely to have an effect on this kind of proprietary businesses. On the other hand, providers like Peplink can (and probably should) make a difference by providing custom support for companies, something that OVH probably won't do.

Finally, let us note that there are other solutions to the original problem, such as MLVPN (which is not based on Multipath TCP). But OVH clearly has enough weight to make a huge impact with its nice, integrated solution.