For optimal reading, please switch to desktop mode.
Introduction
The familiarity of OpenStack tooling combined with the power of Kubernetes Cluster API.
That was the motivation
behind recent community efforts to develop a new OpenStack Magnum driver combining the Kubernetes
Cluster API (CAPI) and Helm projects.
The first official release
of the Magnum CAPI Helm driver in June 2024 marked a significant milestone in the process
of modernising the Magnum project's Kubernetes offering; however, as with
any new technology, there is a learning curve associated with the new concepts and tooling
introduced by Cluster API.
This blog post is a follow-on from our previous Magnum article
(plus the related presentation at the CERN-hosted OpenInfra Days Europe event)
and will provide a more in-depth look at the anatomy of a Magnum CAPI Helm deployment, with a focus
on introducing and explaining the relevant concepts which may be less familiar to many
OpenStack operators. It is intended as a supplement to the official Magnum CAPI Helm driver documentation.
Prerequisites
The Magnum CAPI Helm driver relies on the core OpenStack components (Keystone, Nova, Neutron, Glance, Cinder) plus the OpenStack Octavia service (for API load balancers) and Barbican service (for cluster certificates) - and, of course, the OpenStack Magnum service. These must be deployed on the target cloud prior to working with the Magnum CAPI Helm driver.
Some level of familiarity with Kubernetes and Helm will also be beneficial.
Core Concepts
Cluster API is a complex and evolving project, and as such, it is not feasible to cover all
features and concepts in a single article. Therefore, the remainder of this piece will be dedicated
to the aspects of Cluster API that are most relevant to the Magnum CAPI Helm driver. For a more
general introduction to Cluster API, see the
official documentation or for an
overview of the core concepts and CRDs, see here.
The general idea behind the Cluster API project is to provide a Kubernetes-native
API for managing the
various aspects of cloud infrastructure which together make up a functional Kubernetes cluster.
However, this presents an immediate problem, since providing such an interface requires a
Kubernetes cluster to exist in the first place... This is where the Cluster API management
cluster comes in.
The CAPI management cluster is a standalone Kubernetes cluster which is responsible for
providing this Kubernetes-native interface in the form of various
Custom Resource Definitions
(CRDs).
However, on their own, these CRDs (i.e. custom API endpoints) are somewhat useless
(they're essentially just an interface for storing and validating JSON blobs).
To make these CRDs useful, we must also add some new
Operators (also referred
to as controllers) to our CAPI management cluster. These controllers are responsible for
watching the relevant Kubernetes Custom Resources for changes and translating the custom
resource state into external actions, such as creating cloud resources via OpenStack APIs.
The key CRDs and controllers within a Magnum CAPI Helm management cluster are summarised in the
following diagram:
The general purpose of each of the CRDs is as follows:
- clusters.cluster.x-k8s.io: The top-level resource representing a single
workload cluster.
- machinedeployment.cluster.x-k8s.io: The Machine equivalent of a Kubernetes pod
Deployment.
- machine.cluster.x-k8s.io: A representation of an individual machine (usually a VM) belonging
to a workload cluster.
- kubeadmconfigtemplate.cluster.x-k8s.io and kubeadmcontrolplane.cluster.x-k8s.io:
Kubeadm is a tool for
bootstrapping standard Linux hosts and converting them into Kubernetes cluster nodes. Combined
with the relevant controllers, these two CRDs provide a declarative interface for node bootstrapping
and configuration.
- openstack{cluster,machinetemplate,machine}.infrastructure.cluster.x-k8s.io: A representation of
the OpenStack cloud resources required by a single workload cluster. The
cluster-api-provider-openstack
(capo-controller-manager in the above diagram) is responsible for reconciling the state of these
resources using OpenStack API calls. The cluster-api-janitor also watches these CRDs to clean up
any OpenStack resources left behind upon deletion of the workload cluster.
- {helmrelease, manifests}.addons.stackhpc.com: A representation of a Helm
release or plain Kubernetes manifest to be installed on a workload cluster. Installation and subsequent
reconciliation are carried out by the cluster-api-addon-provider.
Hint: To view a list of the installed CRDs on a given Kubernetes cluster, run kubectl get crds.
Creating a CAPI Management Cluster
Having established that a CAPI management cluster is simply a standard Kubernetes
cluster with some additional tooling installed, an obvious follow-on question is:
How do we install this extra tooling?
For a basic proof-of-concept setup, we can extract the relevant
section
from the Magnum CAPI Helm driver's development environment setup script. The highlighted
section of this script will do the following:
- Install k3s.
- Install the kubectl, helm, and clusterctl CLI tools.
- Install the various controllers listed in the previous section on the k3s cluster using clusterctl and helm.
Warning: Running the full new-devstack.sh script linked above will also set up an all-in-one
devstack environment on the host. This will involve
making substantial changes to your system during setup; therefore, the full script should only be used
if you intend to do some development work on the driver source code, and if so, should only be run on
a dedicated VM that is not used for any other purpose. For setting up a k3s-backed CAPI management cluster,
the highlighted section of the script should be extracted and run separately (though ideally still in a
dedicated/ephemeral VM).
For advice on setting up a resilient, highly available CAPI management cluster, see later in this
article: A Production-Ready CAPI Management Cluster.
Where's all that Magnum stuff gone?
Having set up a working CAPI management cluster, the next step is to configure Magnum and the CAPI Helm
driver to make use of this cluster. To do so, the Magnum CAPI Helm driver must be (pip) installed in the same
environment (i.e. Python virtual environment or Docker container) as the main Magnum project.
Additionally, the k3s cluster's admin kubeconfig file (found at /etc/rancher/k3s/k3s.yaml in a
default k3s installation) must be copied to a
location which is visible
to the Magnum conductor process. For a Kolla-Ansible (or Kayobe) based OpenStack deployment, the kubeconfig file should
be copied into the Magnum config directory on the Ansible control host and Kolla-Ansible will
handle
the rest. For a more detailed walkthrough of the Magnum CAPI Helm setup process using Kayobe, see this
guide.
The final setup consideration from a Magnum perspective is the network topology. The key routes required for a
functional Magnum CAPI Helm driver are as follows:
- Routing from the OpenStack control plane - specifically, the Magnum conductor(s) - to the CAPI management cluster.
- Routing from the CAPI management cluster to the OpenStack cloud's external network. Each workload cluster will
be created with a load balancer on the external network for the Kubernetes API server; the CAPI management cluster
must be able to reach the public IPs of these load balancers.
- Outbound internet access from both the CAPI management cluster and the workload clusters (for pulling container images).
Now that we understand some of the core concepts behind, and key components of, a CAPI management cluster, and
having configured Magnum to make use of this CAPI management cluster, we next turn to the user-facing aspects
of the setup process.
Reproducible Workload Clusters
To make cluster creation as consistent and reproducible as possible, the Magnum CAPI Helm driver relies
on a set of known machine images that are built and tested as part of an upstream CI pipeline. These
images are then referenced in Magnum cluster templates.
The aforementioned driver development environment script also contains
a section
which downloads the latest stable images, uploads them to a target cloud and then creates corresponding
cluster templates using the OpenStack CLI. Uploading images and creating templates in this way is sufficient
for testing purposes; however, for a production setup, a configuration management solution such as StackHPC's
openstack-config repository is recommended. This repository
includes an Ansible playbook
and some documentation to allow
image and cluster template definitions to be placed under version control and easily updated to the latest versions.
Creating a Workload Cluster
Having uploaded the required images to Glance and defined a suitable (set of) cluster template(s), we're
finally ready to create some useful workload clusters!
To do so, we first need some OpenStack user credentials along with the required OpenStack CLI client plugins:
source /path/to/openrc.sh
python3 -m venv os-env
source os-env/bin/activate
pip install -U pip
pip install python-openstackclient python-octaviaclient python-magnumclient
Note: An openrc
file is required rather than a less-privileged
application credential since the cluster creation
process involves creating additional user trusts and temporary application credentials for use within Magnum.
The OpenStack CLI can then be used to check which cluster templates are available:
(os-env) ubuntu@demo:~$ openstack coe cluster template list
+--------------------------------------+-------------------------+------+
| uuid | name | tags |
+--------------------------------------+-------------------------+------+
| ac27710b-9ddf-4e6b-b18a-aba204a8c47d | kubernetes-1-27-jammy | None |
| 7eed6e06-81e8-4a5c-af02-31392ef51159 | kubernetes-1-28-jammy | None |
| 562c42fc-7e2c-4926-984d-fb341104fa38 | kubernetes-1-29-jammy | None |
+--------------------------------------+-------------------------+------+
A workload cluster can be created based on one of these templates with
(os-env) ubuntu@demo:~$ openstack coe cluster create \
--cluster-template kubernetes-1-28-jammy \
--master-count 1 --node-count 1 \
demo-cluster
which should return
Request to create cluster <cluster-id> accepted.
and the cluster state can be checked using
(os-env) ubuntu@demo:~$ openstack coe cluster list -c name -c status
+-----------------------+--------------------+
| name | status |
+-----------------------+--------------------+
| demo-cluster | CREATE_IN_PROGRESS |
+-----------------------+--------------------+
After a few minutes, the relevant OpenStack resources (e.g. networks, subnets, load balancers, servers etc.)
should start to appear:
(os-env) ubuntu@demo:~$ openstack server list \
--name demo-cluster -c Name -c Status
+-----------------------------------------------+--------+
| Name | Status |
+-----------------------------------------------+--------+
| demo-cluster-ivcsn5ftmi5j-control-plane-24djb | ACTIVE |
+-----------------------------------------------+--------+
and as long as (at least) the first control plane node is active, the kubeconfig for the workload cluster
can be obtained with
(os-env) ubuntu@demo:~$ openstack coe cluster config demo-cluster
This command will write the kubeconfig to a file called config in the current working directory, which
can then be used with the kubectl CLI tool to interact with the workload cluster:
(os-env) ubuntu@demo:~$ kubectl get nodes --kubeconfig ./config
NAME STATUS ROLES AGE VERSION
demo-cluster-ivcsn5ftmi5j-control-plane-24djb NotReady control-plane 3m11s v1.28.8
Once the creation process is complete, the status field will be updated in the OpenStack cluster list
(os-env) ubuntu@demo:~$ openstack coe cluster list -c name -c status
+-----------------------+-----------------+
| name | status |
+-----------------------+-----------------+
| demo-cluster | CREATE_COMPLETE |
+-----------------------+-----------------+
and all nodes will be registered in the workload cluster
(os-env) ubuntu@scott-test-1:~$ openstack server list --name demo-cluster -c Name -c Status
+------------------------------------------------------+--------+
| Name | Status |
+------------------------------------------------------+--------+
| demo-cluster-ivcsn5ftmi5j-default-worker-cwmhs-h92w7 | ACTIVE |
| demo-cluster-ivcsn5ftmi5j-control-plane-24djb | ACTIVE |
+------------------------------------------------------+--------+
(os-env) ubuntu@demo:~$ kubectl get nodes --kubeconfig ./config
NAME STATUS ROLES AGE VERSION
demo-cluster-ivcsn5ftmi5j-control-plane-24djb Ready control-plane 8m22s v1.28.8
demo-cluster-ivcsn5ftmi5j-default-worker-cwmhs-h92w7 Ready <none> 3m37s v1.28.8
At this point, the workload cluster is ready to use.
Future cluster lifecycle operations, such as adding or removing nodes and performing rolling cluster
upgrades, should also be performed via the OpenStack CLI (or tools such as
OpenTofu's Magnum
provider).
See openstack coe -h for a list of available commands and openstack coe <command> -h for detailed information on a specific command.
Wait, what just happened?
The process described above provided a user-centric view of the workload cluster creation process;
however, since a functional workload cluster relies on many different parts of an OpenStack cloud,
it can be useful for administrators to understand a little more about the inner workings of the
Magnum CAPI Helm driver to monitor or troubleshoot the cluster creation process. Non-admin readers
who do not have access to their cloud's CAPI management cluster can feel free to skip this section.
In the examples below, all commands targetting the CAPI management cluster will be denoted with
--kubeconfig capi-mgmt (though in practice the name and location of the management kubeconfig may
differ).
When a user submits a cluster creation request via openstack coe cluster create, Magnum core
performs some validation on the request and then, based on the properties of the cluster template,
hands the cluster creation off to the appropriate Magnum driver by calling its create_cluster function.
When the Magnum CAPI Helm driver receives
the cluster creation request, it installs an instance of the
openstack-cluster
Helm chart onto the CAPI management cluster. The full set of templated Kubernetes manifests created by this
Helm release can be viewed using the helm get manifest command. For example, the list of manifests installed
as part of the above demo cluster is:
$ helm --kubeconfig capi-mgmt get manifest \
-n magnum-af29bdf3eb4d4467a855f54a0441d9ab demo-cluster-ivcsn5ftmi5j \
| grep ^kind: | uniq -c
1 kind: ServiceAccount
12 kind: Secret
1 kind: Role
1 kind: RoleBinding
1 kind: Deployment
1 kind: Cluster
12 kind: HelmRelease
1 kind: KubeadmConfigTemplate
1 kind: KubeadmControlPlane
1 kind: MachineDeployment
2 kind: MachineHealthCheck
8 kind: Manifests
1 kind: OpenStackCluster
2 kind: OpenStackMachineTemplate
where the Cluster, MachineDeployment, Kubeadm*, OpenStack*, HelmRelease and Manifest
resource types are exactly those that were introduced in Core Concepts.
As explained previously, these resource instances alone are not particularly useful. It is the custom
controllers which are responsible for acting on these resources to initiate external actions. An example
of this can be seen by inspecting the logs for the capo-controller-manager pod, which include messages
such as:
$ kubectl --kubeconfig capi-mgmt logs -n capo-system \
capo-controller-manager-544cb69b9d-gjfnf \
| grep demo-cluster | head
I0730 14:27:45.708546 1 openstackcluster_controller.go:262] "Reconciling Cluster" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="magnum-af29bdf3eb4d4467a855f54a0441d9ab/demo-cluster-ivcsn5ftmi5j" namespace="magnum-af29bdf3eb4d4467a855f54a0441d9ab" name="demo-cluster-ivcsn5ftmi5j" reconcileID="2bdca1ff-2fa0-401f-b94e-64ac461adfd4" cluster="demo-cluster-ivcsn5ftmi5j"
I0730 14:27:45.708588 1 openstackcluster_controller.go:432] "Reconciling network components" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="magnum-af29bdf3eb4d4467a855f54a0441d9ab/demo-cluster-ivcsn5ftmi5j" namespace="magnum-af29bdf3eb4d4467a855f54a0441d9ab" name="demo-cluster-ivcsn5ftmi5j" reconcileID="2bdca1ff-2fa0-401f-b94e-64ac461adfd4" cluster="demo-cluster-ivcsn5ftmi5j"
I0730 14:27:45.996274 1 network.go:122] "Reconciling network" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="magnum-af29bdf3eb4d4467a855f54a0441d9ab/demo-cluster-ivcsn5ftmi5j" namespace="magnum-af29bdf3eb4d4467a855f54a0441d9ab" reconcileID="2bdca1ff-2fa0-401f-b94e-64ac461adfd4" cluster="demo-cluster-ivcsn5ftmi5j" name="k8s-clusterapi-cluster-magnum-af29bdf3eb4d4467a855f54a0441d9ab-demo-cluster-ivcsn5ftmi5j"
I0730 14:27:46.986616 1 network.go:202] "Reconciling subnet" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="magnum-af29bdf3eb4d4467a855f54a0441d9ab/demo-cluster-ivcsn5ftmi5j" namespace="magnum-af29bdf3eb4d4467a855f54a0441d9ab" reconcileID="2bdca1ff-2fa0-401f-b94e-64ac461adfd4" cluster="demo-cluster-ivcsn5ftmi5j" name="k8s-clusterapi-cluster-magnum-af29bdf3eb4d4467a855f54a0441d9ab-demo-cluster-ivcsn5ftmi5j"
I0730 14:27:46.986864 1 recorder.go:104] "events: Created network k8s-clusterapi-cluster-magnum-af29bdf3eb4d4467a855f54a0441d9ab-demo-cluster-ivcsn5ftmi5j with id 094e2714-c2c7-4a49-8bbe-1907c7b8dd01" type="Normal" object={"kind":"OpenStackCluster","namespace":"magnum-af29bdf3eb4d4467a855f54a0441d9ab","name":"demo-cluster-ivcsn5ftmi5j","uid":"f78a7f37-01dc-4396-acf1-ba5b0fc0fb5b","apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha7","resourceVersion":"75989071"} reason="Successfulcreatenetwork"
I0730 14:27:47.762945 1 router.go:47] "Reconciling router" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="magnum-af29bdf3eb4d4467a855f54a0441d9ab/demo-cluster-ivcsn5ftmi5j" namespace="magnum-af29bdf3eb4d4467a855f54a0441d9ab" name="demo-cluster-ivcsn5ftmi5j" reconcileID="2bdca1ff-2fa0-401f-b94e-64ac461adfd4" cluster="magnum-af29bdf3eb4d4467a855f54a0441d9ab-demo-cluster-ivcsn5ftmi5j"
which demonstrates the CAPO controller making external API calls to create (some of) the OpenStack
resources required by the workload cluster.
The key milestones in between the Magnum CAPI Helm driver performing a helm install of the
openstack-cluster chart on the CAPI management cluster and a ready-to-use Magnum workload cluster
are:
- The capo-controller-manager creates various OpenStack resources (network, subnet, router, load balancer)
along with the first cluster VM.
- The first cluster VM is bootstrapped into a Kubernetes control plane node by the Kubeadm controllers.
- Core cluster services (e.g. a CNI)
are installed on the 'cluster' (which at this point consists of a single control plane node). These core services
are installed via the cluster-api-addon-provider so their status can be monitored using kubectl get helmrelease -A.
- The cluster creation process will stall until the CNI is ready.
- Once the CNI is up and running, the capo-controller-manager and the kubeadm controllers will create
the remaining VMs, bootstrap them as Kubernetes nodes and then add them to the cluster.
- Finally, the remaining cluster add-ons such as the monitoring stack are installed by the
cluster-api-addon-provider.
There are various other intermediate steps involved in the cluster creation process aside from these key
milestones. The general starting point for understanding the cluster creation process in more detail or
troubleshooting a stalled cluster should be the pod logs for each of the custom controllers listed in the
Core Concepts section.
Having now covered the concepts behind, and practicalities of, the new Magnum CAPI Helm driver using a
simple k3s CAPI management cluster, the next section will provide some guidance on moving towards a more
resilient, production-ready deployment.
A Production-Ready CAPI Management Cluster
At StackHPC we have been running CAPI management clusters in production on various OpenStack clouds for
the past few years as part of our Azimuth platform.
During this time we have learned a thing or two about how to configure and run a reliable management cluster.
This learning has been crystalised in the form of our azimuth-ops
Ansible collection and associated azimuth-config repository; the latter of which
provides an example environment
for deploying a standard CAPI management cluster (as required for Magnum use, rather than the full Azimuth deployment).
When deployed using this Ansible tooling, the Magnum CAPI management cluster is a highly-available, auto-healing
Kubernetes cluster which itself is managed by a lightweight k3s cluster (backed by a Cinder volume, for easy
backup and recovery). The architecture of the management cluster setup in this scenario looks like:
When deployed using this battle-tested Azimuth tooling, a CAPI management cluster benefits from StackHPC's
extensive operational experience, as well as the following useful features:
For now, the canonical source of information for Magnum CAPI management cluster deployments utilising the
Azimuth Ansible tooling is the Azimuth Operator Guide.
Since this documentation is primarily geared toward Azimuth deployments (rather than a simpler Magnum CAPI
management cluster deployment), this information should be supplemented with the Magnum-specific deployment
guide in the
StackHPC Kayobe Config documentation.
In the future, Magnum-specific documentation will be added to the driver's
documentation pages.
Conclusion
To summarise, recent developments in the Magnum community involving the Kubernetes Cluster API
project have generated significant excitement regarding the future of Magnum's Kubernetes support. In
this article, we have introduced the relevant Cluster API concepts and explained how a CAPI management
cluster works in tandem with Magnum and its new CAPI Helm driver to create and manage user workload
clusters. Finally, we provided an overview of the processes StackHPC use to create resilient and
production-ready Magnum CAPI management clusters for our clients' OpenStack clouds.
Get in touch
If you are interested in finding out more about all things OpenStack, Magnum or Cluster API and would like
to get in touch, then we would love to hear from you. Reach out to us via
X, LinkedIn or
directly via our contact page.