Developed by StackHPC and the upstream open source community, Kayobe provides automation for deploying OpenStack
using infrastructure-as-code principles. Beyond the initial software
deployment, so-called day 2 operations are also crucial to automate to
increase productivity and ensure continued operation and knowledge transfer
throughout the lifetime of the infrastructure.
In this short blog post, we are looking at how Kayobe and GitHub Actions can
work together to automate OpenStack database backups.
OpenStack database backups with Kayobe
OpenStack services rely on a relational database to store information.
OpenStack deployments generally use a MariaDB or MySQL database. Keeping
regular backups of database content is critical to be able to recover from a
failure of the control plane.
Thankfully, Kolla Ansible makes it easy to perform database backups
using Mariabackup.
This functionality is also available via a Kayobe command.
Kayobe automation
We work with many different organisations that are running OpenStack. Local
teams using GitOps to manage their infrastructure will already have experience
with a specific CI/CD platform. Imposing a different one for Kayobe could add
much friction and discourage automation. Instead, we strive to make Kayobe easy
to integrate with any CI/CD platform.
To achieve this, we developed kayobe-automation. In addition to running any
Kayobe command, it generates a Kolla configuration diff for change requests,
making it easy to visualise the Kolla configuration changes to be applied on
each OpenStack host.
Initially developed for GitLab, we have since
integrated it with GitHub Actions
using a local self-hosted runner with access to the Kayobe admin network.
Automating backups with GitHub Actions
When integrated with GitHub Actions, each Kayobe command is implemented in a
different workflow, which can be triggered manually through the
workflow_dispatch
event. This allows an operator to perform a database backup at the push of a
button. However, this is still not fully automated. GitHub Actions provides the
schedule event,
which uses a cron syntax to define when to run the workflow:
on:
schedule:
- cron: '0 30 * * *'
And just like that, our OpenStack database is now backed up every night, with
GitHub Actions taking care of the scheduling, providing us with status and logs
of each workflow run through its web interface, and sending us an email
notification in case of failure.
But where is my backup?
Kolla Ansible runs the database backup process on the first host in
mariadb_shard_group, which will usually map to the first controller. The
backup file is stored in the mariadb_backup Docker volume on this host.
This makes it prone to loss in case of hardware failure. To be really effective
against disaster, backup files should be transferred off-site onto a separate
storage system. This can be easily customised for each deployment using Kayobe
hooks. Hooks
allow operators to automatically execute custom Ansible playbooks at
certain points during the execution of a Kayobe command. For example, we can
write a custom playbook called upload-database-backup.yml which will upload
the latest backup file somewhere safe, for example to an off-site object store:
---
# This playbook uploads MariaDB backups to a Swift object store.
- hosts: controllers[0]
vars:
backup_directory: "/var/lib/docker/volumes/mariadb_backup/_data"
swift_venv: "/opt/kayobe/venvs/swift"
tasks:
- name: Ensure swift client is available
pip:
name:
- python-keystoneclient
- python-swiftclient
virtualenv: "{{ swift_venv }}"
- name: Upload backup files
shell: >
cd {{ backup_directory }} && /opt/kayobe/venvs/swift/bin/swift --auth-version 2 \
--os-auth-url {{ backup_swift_url }} --os-username {{ secrets_backup_swift_user }} \
--os-password {{ secrets_backup_swift_key }} --os-project-name {{ backup_swift_project }} \
upload --skip-identical {{ mysql_backup_container }} .
become: True
Creating a hook to run this playbook after each execution of kayobe overcloud
database backup (including when run through a CI/CD platform) is done by
creating a symbolic link from
$KAYOBE_CONFIG_PATH/hooks/overcloud-database-backup/post.d/10-upload-database-backup.yml
to $KAYOBE_CONFIG_PATH/ansible/upload-database-backup.yml:
$ readlink $KAYOBE_CONFIG_PATH/hooks/overcloud-database-backup/post.d/10-upload-database-backup.yml
../../../ansible/upload-database-backup.yml
Similarly, we can run another playbook to delete any old backups from the
controllers, to avoid consuming large amounts of storage over time.
Restoring from backup
Like taking an insurance policy, we perform backups hoping we will never have
to use them. However, it is important to know how to restore from backups and
to check that the process actually works. The Kolla Ansible documentation
covers how to restore backups.
Note that OpenStack will lose track of anything that happened since the backup
was taken, so there may be some resources, such as virtual machines or volumes,
that will require manual cleanup.
Conclusion
Automating OpenStack database backups is only one of the many things possible
with Kayobe and CI/CD platforms. Future blog posts will describe other
features, such as Kolla configuration diffs.
Get in touch
If you would like to get in touch we would love to hear
from you. Reach out to us via Twitter
or directly via our contact page.