Azimuth's Cluster API Engine

The new Cluster API driver reuses components developed to provide Kubernetes support in Azimuth, an open-source portal providing self-service HPC and AI platforms for OpenStack clouds, where they have been battle-tested for a number of years.

Providing a managed Kubernetes service is much more than just provisioning Kubernetes cluster nodes - clusters must have addons installed, such as a CNI, OpenStack integrations and the metrics server.

Whilst Cluster API is excellent at provisioning and operating Kubernetes clusters, it has limited capability for managing cluster addons. To address this, StackHPC developed an addon provider for Cluster API - another Kubernetes operator that provides a declarative API for specifying which Helm charts and additional manifests should be installated on a Cluster API cluster.

For example, this resource specifies that the NGINX ingress controller should be installed onto the Cluster API cluster example using values from an inline template (values from ConfigMaps and Secrets are also supported):

apiVersion: addons.stackhpc.com/v1alpha1
kind: HelmRelease
metadata:
  name: ingress-nginx
spec:
  # The name of the target Cluster API cluster
  clusterName: example
  # The namespace on the cluster to install the Helm release in
  targetNamespace: ingress-nginx
  # The name of the Helm release on the target cluster
  releaseName: ingress-nginx
  # Details of the Helm chart to use
  chart:
    repo: https://kubernetes.github.io/ingress-nginx
    name: ingress-nginx
    version: 4.9.1
  # Values for the Helm release
  # These can be merged from multiple sources
  valuesSources:
    - template: |
        controller:
          metrics:
            enabled: true
            serviceMonitor:
              enabled: true

As well as providing managed Kubernetes clusters for tenants using Cluster API, Azimuth itself also runs on a Kubernetes cluster. Usually this cluster is deployed in a project on the same OpenStack cloud that Azimuth is configured to provision resources in.

In order to share as much code and knowledge between these two use cases as possible, StackHPC have codified the deployment of Cluster API clusters on OpenStack, with addons properly configured, as a set of open-source Helm charts. These charts are used in Azimuth for provisioning both the Azimuth cluster itself and tenant clusters.

Helm's Flexibility Advantage

The biggest benefit of encapsulating good practice using Helm is that the charts can be reused outside of Azimuth, e.g. to operate Kubernetes clusters using GitOps with Flux or Argo CD. For example, deploying a basic cluster with a monitoring stack requires the following Helm values:

# The name of a secret containing an OpenStack appcred for the target project
cloudCredentialsSecretName: my-appcred

# The ID of the image to use for cluster nodes
machineImageId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# The version of kubelet that is in that image
kubernetesVersion: 1.28.6

clusterNetworking:
  # The ID of the external network to use
  externalNetworkId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

controlPlane:
  # The name of the flavor to use for control plane nodes
  machineFlavor: small

# The node groups for the cluster
nodeGroups:
  - name: md-0
    machineFlavor: small
    machineCount: 2

# Enable the monitoring stack
addons:
  monitoring:
    enabled: true

In particular, these charts are used by StackHPC's Magnum driver to produce the Cluster API resources for Magnum clusters. The Magnum driver simply converts Magnum's representation of the cluster template and cluster into Helm values that are passed to the CAPI Helm charts.

This means that although StackHPC's Cluster API Magnum driver is relatively new, it is based on components that have been used in production for several years.

StackHPC will be contributing the Helm charts to the Magnum project to serve as a reference for how to provision Cluster API clusters on OpenStack.

Magnum Cluster API Drivers are Like Buses

The Kolla-Ansible project already does the work to package the driver developed by VEXXHOST and will also be packaging the Cluster API Helm driver once it merges in the Magnum codebase.

In the Bobcat relase of Magnum, it is not possible for both of these drivers to coexist as the current driver selection mechanism does not consider enough information to differentiate them (see the Magnum Docs). Jake Yip, the current Magnum PTL, has been doing some great work to make the driver selection more explicit using image metadata, which will allow the two Cluster API drivers to co-exist from the Caracal release onwards (patch in Gerrit).

There are some technical differences between the VEXXHOST and StackHPC drivers. In particular, as discussed above, StackHPC have chosen to use Helm to template the Cluster API resources for Magnum clusters, whereas the VEXXHOST driver templates the Cluster API resources in Python code.

We chose Helm for a number of reasons:

• The method to deploy Cluster API clusters on OpenStack is reusable outside of Magnum, rather than being encapsulated behind the Magnum APIs.
• The Helm charts, when combined with StackHPC's addon provider, provide a powerful, declarative way for managing addons.
• More flexibility for operators - if an operator needs to modify the way that clusters are deployed, the default charts can be replaced with custom charts as long as the Helm values provided by the Magnum driver are respected.
• The Helm charts can have an independent release cycle, allowing Magnum to support new Kubernetes versions without waiting for an OpenStack release by just creating new cluster templates that reference the new Helm chart version.

A significant difference between the two drivers is use of ClusterClass. Whilst ClusterClass does promise to make certain lifecycle operations easier, particularly upgrades, the Cluster API project still classifies the feature as "experimental (alpha)" and requires a feature gate to be enabled to use it.

The CAPI Helm charts have intelligence baked in to deal with upgrades, backed by extensive use in production and cluster upgrades tested in CI. We are being cautious about switching to an alpha feature that may still be subject to substantial change. It is expected that the Helm charts will move to ClusterClass when it moves into beta or GA.

Despite the technical differences, there are still a number of places where code could be shared between the two drivers. Both drivers derive the Magnum cluster state by looking at the same Cluster API objects, both drivers have a need to make the certificates generated by Magnum available to the Cluster API clusters, and both drivers monitor the cluster health in the same way.

At StackHPC, we are keen to collaborate to see if we can develop a shared library that can be used by both drivers to perform these tasks, reducing duplication and making the most of the available effort to push the Magnum ecosystem forward together.

Get in touch

If you would like to get in touch we would love to hear from you. Reach out to us via Twitter, LinkedIn or directly via our contact page.