For optimal reading, please switch to desktop mode.
Building Mellanox OFED for OpenStack Overcloud Images
A typical HPC system environment is continuously managed: a maintained configuration, updated over time as new software packages become available. The cloud compute model takes a different approach: infrastructure, (such as the OS and run-time environment of an HPC system) is like code, and that infrastructure gets managed through "recompilation", not through being updated in place.
This can have far-reaching consequences for the workflow with which systems are managed. One clear benefit is that any system can be rebuilt according to a formula (likely to be a collection of scripts, Ansible playbooks or Puppet manifests), and that formula can be managed and developed under source control. If a formula is sufficiently precise, it can provide an increased level of repeatability: at some future point we can check out the formula and use it again to rebuild servers to a similar configuration.
One quirky side-effect of this approach is that a cloud-model image contains no 'baggage': no old, superseded packages lying unused after an upgrade. In particular, cloud-model images do not contain the kernel package that originally shipped with that distribution, since updated several times over. This turns out to be a nuisance when installing Mellanox OFED, which assumes this original kernel is present.
When building a cloud-model system image we are typically also building something different from the environment of the build host. One is likely to be a well-stocked sysadmins toolbox. The other should be pared-down and minimally sufficient for the (virtualised) task at hand. We need to prevent the configuration of the build host from affecting the cloud image. We need to avoid this kind of pollution.
How Mellanox OFED is Built
The new kernel should be installed on the build system. Assuming a kernel RPM package is being used, both the kernel and the kernel-devel RPMs should be installed.
Mellanox OFED is downloaded as a tarball, or an equivalent ISO image. Within the image is a yum repo of RPMs and a set of scripts for automating building, installing and uninstalling.
After unpacking the tarball (or mounting the ISO), we use the build automation script, mlnx_add_kernel_support.sh:
KVER=3.10.0-327.22.2.el7.x86_64
./mlnx_add_kernel_support.sh \
--mlnx_ofed $PWD --kernel $KVER --yes --verbose --make-iso
How to Use the Output
If the build succeeds, an updated version of the Mellanox OFED tarball (or ISO image) is generated in /tmp. This output can be used to install in exactly the same way as the Mellanox OFED build first downloaded.