For optimal reading, please switch to desktop mode.

UPDATED: See OpenHPC v2.1 Released below.

The OpenHPC project is a key part of the "HPC" side of StackHPC. It essentially provides a set of packages which make it easy to install the system software for a scheduler-based HPC cluster, including the scheduler itself, compilers, MPI libraries, maths and I/O libraries, performance tools, etc., all integrated into a neat set of hierarchical modules. While OpenHPC’s own documentation has recipes for using Warewulf to create and deploy images for compute nodes, we generally use our stackhpc.openhpc Ansible Galaxy role which can configure all nodes in a cluster from a base image with a single command.

OpenHPC v2.0 was released in October 2020 and we've since deployed it to client systems using our Galaxy role, including the 1000+ node Top 100 and Top 500 systems we recently blogged about. OpenHPC v2.0 is a significant upgrade from v1.3.9 which preceded it (released in November 2019), with new versions of software at all levels of the stack. However it requires Centos 8 so it is not a trivial upgrade, although presumably at least some of these enhancements will eventually show up in the Centos 7-based v1.3.10 which was originally planned for the end of 2020. Let's take a look at what that upgrade gets you.

Starting with the scheduler, OpenHPC v2.0 updates Slurm to v20.02.5. This adds a "configless" mode which enables compute and login nodes to pull configuration information directly from the slurm control daemon rather than having to distribute the config file to every node. As well as simplifying the overall cluster configuration, this approach means that changes to slurm configuration such as added or removed nodes no longer need to be replicated in compute node images or mounted over a network filesystem, simplifying building images for compute nodes.

OpenHPC provides a variety of compilers and MPI libraries, but the GCC + OpenMPI combination will be of interest to many users needing a FOSS toolchain. OpenHPC v2.0 updates the packaged GNU compilers from v8 to v9, and OpenMPI from v3 to v4, but the building of OpenMPI against UCX is possibly the most obvious change to users. UCX is a communications framework which aims to provide optimized performance with a unified interface both “up” to the user and “down” to developers across a range of hardware and platforms. While adding yet another layer into the already-complicated HPC interconnect/networking/fabric stack may feel somewhat unhelpful, UCX does simplify life for users. For example we recently carried out benchmarking of a range of MPI applications to compare performance between InfiniBand and RoCE interconnects. Getting RoCE to work on Mellanox ConnectX4 cards with the "native" OpenMPI Byte Transport Layer required setting:

OMPI_MCA_btl=openib,self,vader
OMPI_MCA_btl_openib_if_include=mlx5_1:1
OMPI_OPENIB_ROCE_QUEUES="--mca btl_openib_receive_queues P,128,64,32,32,32:S,2048,1024,128,32:S,12288,1024,128,32:S,65536,1024,128,32"

with pingpong tests showing some poor performance at some message sizes. Rather than optimising the queues to try to improve this, using UCX provided more consistent RoCE performance by simply setting:

UCX_NET_DEVICES=mlx5_1:1

As a bonus, the MPICH packages in OpenHPC v2.0 also use UCX and Intel MPI supports it from v2019.5, so using multiple MPI libraries gets significantly simpler.

We've just released v0.7.0 of our Ansible Galaxy OpenHPC role. The first major enhancement in this version is support for the new "configless" mode when using OpenHPC v2.0 (on Centos 8 - support for OpenHPC v1.x/Centos 7 is still included but without this mode). The second major enhancement is new options to configure slurmdb and the accounting plugin. This significantly enhances the accounting information available via sacct compared to the default text-file-based storage, and enables us to build job-specific monitoring dashboards for the cluster. Less obviously, a number of internal tweaks have been made to improve using the role in image build pipelines for compute nodes. For a full list of what's new see the v0.7.0 release notes.

To maintain backwards compatibility these features aren't turned on by default, so as an example of what's needed here's the configuration for a Slurm cluster in configless mode, slurmdb/enhanced accounting enabled and 2x partitions:

- name: Setup slurm
  hosts: openhpc
  become: yes
  tags:
    - openhpc
  tasks:
    - import_role:
        name: stackhpc.openhpc
      vars:
        openhpc_enable:
          control: "{{ inventory_hostname in groups['control'] }}"
          batch: "{{ inventory_hostname in groups['compute'] }}"
          database: "{{ inventory_hostname in groups['control'] }}"
          runtime: true
        openhpc_slurm_accounting_storage_type: 'accounting_storage/slurmdbd'
        openhpc_slurmdbd_mysql_password: "{{ secrets_openhpc_mysql_slurm_password }}"
        openhpc_slurm_control_host: "{{ groups['control'] | first }}"
        openhpc_slurm_partitions:
          - name: "hpc"
            default: YES
            maxtime: "3-0" # 3 days 0 hours
          - name: "express"
            default: NO
            maxtime: "1:0:0" # 1 hour 0m 0s
        openhpc_slurm_configless: true
        openhpc_login_only_nodes: login

Using this role for the core Slurm functionality, we're now building a flexible Ansible-based "Slurm appliance" around it which automates deployment and configuration of an entire HPC environment. At present it includes the Slurm-based monitoring as mentioned above, post-deployment performance tests, and additional filesystems, as well as providing production-ready configuration for aspects such as PAM and user limits. Our OpenHPC role makes deploying a Slurm cluster easy, and we're excited that this appliance will provide the same ease of use for a much richer user experience. Watch this space for details ...