OpenStack database backups with Kayobe

OpenStack services rely on a relational database to store information. OpenStack deployments generally use a MariaDB or MySQL database. Keeping regular backups of database content is critical to be able to recover from a failure of the control plane.

Thankfully, Kolla Ansible makes it easy to perform database backups using Mariabackup. This functionality is also available via a Kayobe command.

Kayobe automation

We work with many different organisations that are running OpenStack. Local teams using GitOps to manage their infrastructure will already have experience with a specific CI/CD platform. Imposing a different one for Kayobe could add much friction and discourage automation. Instead, we strive to make Kayobe easy to integrate with any CI/CD platform.

To achieve this, we developed kayobe-automation. In addition to running any Kayobe command, it generates a Kolla configuration diff for change requests, making it easy to visualise the Kolla configuration changes to be applied on each OpenStack host.

Initially developed for GitLab, we have since integrated it with GitHub Actions using a local self-hosted runner with access to the Kayobe admin network.

Automating backups with GitHub Actions

When integrated with GitHub Actions, each Kayobe command is implemented in a different workflow, which can be triggered manually through the workflow_dispatch event. This allows an operator to perform a database backup at the push of a button. However, this is still not fully automated. GitHub Actions provides the schedule event, which uses a cron syntax to define when to run the workflow:

on:
  schedule:
    - cron: '0 30 * * *'

And just like that, our OpenStack database is now backed up every night, with GitHub Actions taking care of the scheduling, providing us with status and logs of each workflow run through its web interface, and sending us an email notification in case of failure.

But where is my backup?

Kolla Ansible runs the database backup process on the first host in mariadb_shard_group, which will usually map to the first controller. The backup file is stored in the mariadb_backup Docker volume on this host. This makes it prone to loss in case of hardware failure. To be really effective against disaster, backup files should be transferred off-site onto a separate storage system. This can be easily customised for each deployment using Kayobe hooks. Hooks allow operators to automatically execute custom Ansible playbooks at certain points during the execution of a Kayobe command. For example, we can write a custom playbook called upload-database-backup.yml which will upload the latest backup file somewhere safe, for example to an off-site object store:

---
# This playbook uploads MariaDB backups to a Swift object store.

- hosts: controllers[0]
  vars:
    backup_directory: "/var/lib/docker/volumes/mariadb_backup/_data"
    swift_venv: "/opt/kayobe/venvs/swift"
  tasks:
    - name: Ensure swift client is available
      pip:
        name:
          - python-keystoneclient
          - python-swiftclient
        virtualenv: "{{ swift_venv }}"

    - name: Upload backup files
      shell: >
        cd {{ backup_directory }} && /opt/kayobe/venvs/swift/bin/swift --auth-version 2 \
        --os-auth-url {{ backup_swift_url }} --os-username {{ secrets_backup_swift_user }} \
        --os-password {{ secrets_backup_swift_key }} --os-project-name {{ backup_swift_project }} \
        upload --skip-identical {{ mysql_backup_container }} .
      become: True

Creating a hook to run this playbook after each execution of kayobe overcloud database backup (including when run through a CI/CD platform) is done by creating a symbolic link from $KAYOBE_CONFIG_PATH/hooks/overcloud-database-backup/post.d/10-upload-database-backup.yml to $KAYOBE_CONFIG_PATH/ansible/upload-database-backup.yml:

$ readlink $KAYOBE_CONFIG_PATH/hooks/overcloud-database-backup/post.d/10-upload-database-backup.yml
../../../ansible/upload-database-backup.yml

Similarly, we can run another playbook to delete any old backups from the controllers, to avoid consuming large amounts of storage over time.

Restoring from backup

Like taking an insurance policy, we perform backups hoping we will never have to use them. However, it is important to know how to restore from backups and to check that the process actually works. The Kolla Ansible documentation covers how to restore backups. Note that OpenStack will lose track of anything that happened since the backup was taken, so there may be some resources, such as virtual machines or volumes, that will require manual cleanup.

Conclusion

Automating OpenStack database backups is only one of the many things possible with Kayobe and CI/CD platforms. Future blog posts will describe other features, such as Kolla configuration diffs.

Get in touch

If you would like to get in touch we would love to hear from you. Reach out to us via Twitter or directly via our contact page.