For optimal reading, please switch to desktop mode.

BeeGFS is a parallel file system suitable for High Performance Computing with a proven track record in scalable storage solution space. In this article, we explore how different components of BeeGFS are pieced together and how we have incorporated them into an Ansible role for a seamless storage cluster deployment experience.

We've previously described ways of integrating OpenStack and High-Performance Data. In this post we'll focus on some practical details for how to dynamically provision BeeGFS filesystems and/or clients running in cloud environments. There are actually no dependencies on OpenStack here - although we do like to draw our Ansible inventory from Cluster-as-a-Service infrastructure.

As described here, BeeGFS has components which may be familiar concepts to those working in parallel file system solution space:

• Management service: for registering and watching all other services
• Storage service: for storing the distributed file contents
• Metadata service: for storing access permissions and striping info
• Client service: for mounting the file system to access stored data
• Admon service (optional): for presenting administration and monitoring options through a graphical user interface.

Introducing our Ansible role for BeeGFS...

We have an Ansible role published on Ansible Galaxy which handles the end-to-end deployment of BeeGFS. It takes care of details all the way from deployment of management, storage and metadata servers to setting up client nodes and mounting the storage point. To install, simply run:

ansible-galaxy install stackhpc.beegfs

There is a README that describes the role parameters and example usage.

An Ansible inventory is organised into groups, each representing a different role within the filesystem (or its clients). An example inventory-beegfs file with two hosts bgfs1 and bgfs2 may look like this:

[leader]
bgfs1 ansible_host=172.16.1.1 ansible_user=centos

[follower]
bgfs2 ansible_host=172.16.1.2 ansible_user=centos

[cluster:children]
leader
follower

[cluster_beegfs_mgmt:children]
leader

[cluster_beegfs_mds:children]
leader

[cluster_beegfs_oss:children]
leader
follower

[cluster_beegfs_client:children]
leader
follower

Through controlling the membership of each inventory group, it is possible to create a variety of use cases and configurations. For example, client-only deployments, server-only deployments, or hyperconverged use cases in which the filesystem servers are also the clients (as above).

A minimal Ansible playbook which we shall refer to as beegfs.yml to configure the cluster may look something like this:

---
- hosts:
  - cluster_beegfs_mgmt
  - cluster_beegfs_mds
  - cluster_beegfs_oss
  - cluster_beegfs_client
  roles:
  - role: stackhpc.beegfs
    beegfs_state: present
    beegfs_enable:
      mgmt: "{{ inventory_hostname in groups['cluster_beegfs_mgmt'] }}"
      oss: "{{ inventory_hostname in groups['cluster_beegfs_oss'] }}"
      meta: "{{ inventory_hostname in groups['cluster_beegfs_mds'] }}"
      client: "{{ inventory_hostname in groups['cluster_beegfs_client'] }}"
      admon: no
    beegfs_mgmt_host: "{{ groups['cluster_beegfs_mgmt'] | first }}"
    beegfs_oss:
    - dev: "/dev/sdb"
      port: 8003
    - dev: "/dev/sdc"
      port: 8103
    - dev: "/dev/sdd"
      port: 8203
    beegfs_client:
      path: "/mnt/beegfs"
      port: 8004
    beegfs_interfaces:
    - "ib0"
    beegfs_fstype: "xfs"
    beegfs_force_format: no
    beegfs_rdma: yes
...

To create a BeeGFS cluster spanning the two nodes as defined in the inventory, run a single Ansible playbook to handle the setup and the teardown of BeeGFS storage cluster components by setting beegfs_state flag to present or absent:

# build cluster
ansible-playbook beegfs.yml -i inventory-beegfs -e beegfs_state=present

# teardown cluster
ansible-playbook beegfs.yml -i inventory-beegfs -e beegfs_state=absent

The playbook is designed to fail if the path specified for BeeGFS storage service under beegfs_oss is already being used for another service. To override this behaviour, pass an extra option as -e beegfs_force_format=yes. Be warned that this will cause data loss as it formats the disk if a block device is specifed and also erase management and metadata server data if there is an existing BeeGFS deployment.

Highlights of the Ansible role for BeeGFS:

• The idempotent role will leave state unchanged if the configuration has not changed compared to the previous deployment.
• The tuning parameters for optimal performance of the storage servers recommended by the BeeGFS maintainers themselves are automatically set.
• The role can be used to deploy both storage-as-a-service and hyperconverged architecture by the nature of how roles are ascribed to hosts in the Ansible inventory. For example, the hyperconverged case would have storage and client services running on the same nodes while in the disaggregated case, the clients are not aware of storage servers.

Other things we learnt along the way:

• BeeGFS is sensitive to hostname. It prefers hostnames to be consistent and permanent. If the hostname changes, services refuse to start. As a result, this is worth being mindful of during the initial setup.
• This is unrelated to BeeGFS specifically but we had to set a -K flag when formatting NVME devices in order to prevent it from discarding blocks under instructions from Dell otherwise the disk would disappear with the following error message:

[ 7926.276759] nvme nvme3: Removing after probe failure status: -19
[ 7926.349051] nvme3n1: detected capacity change from 3200631791616 to 0