For optimal reading, please switch to desktop mode.
Published: Fri 30 August 2024
Updated: Fri 30 August 2024
By Bertie Thorpe
In Networking .
tags: roce rdma openmpi ucx networking
Bertie is StackHPC's 2024 summer intern, and has just started a permanent position with the engineering team. This blog post
presents a one week investigation he made into configuring RoCE-enabled OpenMPI tasks.
Remote Direct Memory Access (RDMA) is the cornerstone of networking for HPC workloads. RDMA circumvents processor and OS overheads;
it enables more direct data exchange by implementing a transport protocol in the Network Interface Card (NIC). InfiniBand (IB) held sway
for many years as the primary RDMA enabling standard, delivering very high bandwidth and very low latencies for many in the
TOP500 . We have demonstrated previously
that in recent years modern Ethernet stacks have begun to support RDMA, coming close to the ~1 µs latencies of IB.
Though slightly behind IB in latency performance, RDMA over Converged Ethernet (RoCE)
has a number of other significant benefits over IB. The ubiquity of Ethernet is the primary selling point, as existing
infrastructure can be leveraged to enable RDMA at lower cost. The overall system complexity and network management is
simplified because a single data transport unifies RDMA and conventional IP traffic.
While RoCE is certainly very attractive, as always there are hurdles to trip over when actually putting it to work. The
ever-changing networking hardware ecosystem meant OpenMPI has had to adapt. This can lead to some difficult pitfalls to debug
yourself out of if not paying close attention to how OpenMPI is configured; that is the focus of this blog post.
Case Study - RoCE vs TCP
During a task of investigating RoCE performance, we happened across some very peculiar behaviour during a simple OpenMPI
PingPong benchmark. To set the scene we have two Ethernet network devices:
eth0
eth1
These interfaces are used for general network communication using the standard TCP protocol. However, eth1 provides a link
layer to a mellanox device; this interface is configured to support RoCE. We can see these available devices by running ip a
for the ip interfaces, and ibv_devinfo to show host channel adaptor info for the Mellanox NICs,
Initial PingPong results
PingPong Latency (µs)
PingPong Bandwidth (Mb/sec)
1.62
45626.88
The default performance of PingPong, with a latency of ~1.6 µs indicates that Open MPI is utilising RoCE as standard.
This can be further highlighted in Grafana:
As can be seen above, the network traffic predominately goes over the RDMA device with minimal traffic over TCP (note that
our Grafana dashboard refers to this traffic as IB, but in our case it is actually RoCE). At least for TCP interfaces, OpenMPI's
default behaviour is to use all available devices on the machine,
and selects best ones based on performance characteristics (bandwidth, latency, NUMA
locality, etc). In most situations this would be ideal; if you’re investing in RDMA capable infrastructure, you’d want it
fully utilised.
Specifying network devices - Very strange behaviour
However, now suppose we want to have finer-grained control over the network devices used for each task. OpenMPI
recommends using the Unified Communications X (UCX) library directly, a middleware layer that abstracts the complexities of
different communication protocols and hardware. With multiple interfaces,
by default UCX
uses the first two interfaces in proportion to their detected speed ratio. How this interacts with OpenMPI's default behaviour
with multiple interfaces is not clear from documentation. UCX's default behaviour can be overridden by setting UCX_NET_DEVICES
to to specify which device is used. For completeness let’s
export UCX_NET_DEVICES=mlx5_0:1
This is the RoCE device we are aiming to utilise linklayered with eth1, giving us:
PingPong Latency (µs)
PingPong Bandwidth (Mb/sec)
1.59
45612.64
These results seem to be consistent with what we expect; the very low latency approaches 1 µs, and by selecting
the Mellanox NIC directly, it further proves that the previous mpirun above makes full use of the RoCE device as default.
Now lets try using just one of the Ethernet devices over TCP. Setting UCX_NET_DEVICES to eth0 gives us the following results:
PingPong Latency (µs)
PingPong Bandwidth (Mb/sec)
2.14
45649.52
A latency of ~2.1 µs, though higher than the previous 1.6, is far lower than is possible with TCP/IP networking. The
results are plainly not right; we are still getting RDMA latencies even though we're telling UCX to just send over TCP.
To prove this is the case, without a shadow of a doubt, we simply need to look at the prometheus scraped data on Grafana
again:
Yep, just as we suspected! Most traffic, if not all, is going over RoCE. This is the same if we set UCX_NET_DEVICES to eth0, eth1,
foo, or anything else at all. Very strange behaviour indeed. It appears that the devices specified with UCX are being ignored
by OpenMPI. An interesting aside is the fact that the PingPong latencies for these rogue benchmarks are consistently a little
longer than the default or when we specify the Mellanox NIC directly. Whatever mechanism OpenMPI is falling back on is
unoptimal.
A deeper dive into Open MPI
To understand why setting UCX_NET_DEVICES gives us unexpected and seemingly incorrect performance, a deeper dive into OpenMPI’s
architecture might help us. OpenMPI is built from a Modular Component Architecture (MCA), a series of modules, components,
and frameworks pieced together at run-time to create an MPI implementation. Each of these components is responsible for a
specific aspect of MPI functionality. We're interested in the components that supply support for RDMA, and their priorities
within the MCA.
As of OpenMPI 4.x (the version we are running is 4.1.5) four separate components supporting IB, RoCE, and/or iWARP are
potentially available:
openib BTL
ucx PML
hcoll coll
ofi MTL
It's very important to know the option flags our instance of OpenMPI was compiled with. We can find this out with
ompi_info | grep Configure
An effort is being made to streamline the components and as of v4.0.0 the UCX PML is the preferred mechanism for running
over RoCE-based networks. Consequently the --without-verbs
Fixed Configuration
Configuring OpenMPI with
--with-platform=contrib/platform/mellanox/optimized
forces using UCX also with TCP transports when specified. Alternatively we can add the following options to the mpirun command:
-mca pml_ucx_tls any -mca pml_ucx_devices any
This now allows us to correctly set the network device used with UCX_NET_DEVICES as before, with no issues. Finally we
can set the net device and get exactly what we expect.
eth0
PingPong Latency (µs)
PingPong Bandwidth (Mb/sec)
36.12
6533.84
eth1
PingPong Latency (µs)
PingPong Bandwidth (Mb/sec)
9.44
28973.04
And just to make sure we can see on the Grafana dashboard that no traffic is going
via RDMA.
Conclusions
Once a network is RoCE-capable, both the in-kernel and Mellanox drivers make RoCE available to the system with no further
configuration in modern Linux distributions. However the MPI stack is still hugely complicated and despite efforts such as
priotitising the UCX component to make it simpler for users, ensuring it is running as desired is not straightforward. Experimentation is necessary, having
monitoring is crucial, and users need documented "known-good" configurations to base their own workloads on.
Many thanks to the UCX and OpenMPI developers for their help in investigating and understanding the behaviour described here!