The Convergence of HPC, AI and Cloud

For optimal reading, please switch to desktop mode.

Here at StackHPC we've used and experimented with Monasca in a variety of ways, contributing upstream wherever possible.

For the benefit of those using or considering either Monasca or Kayobe we thought we'd share some of our tips for deploying and configuring it.

This tutorial will follow on from the Kayobe a-universe-from-nothing tutorial on Train to demonstrate how to deploy and customise Monasca with Kolla-Ansible.

Assuming you've got a Kayobe environment (see our helpful universe-from-nothing blog post if you haven't already) you're only a few steps away from having a deployed Monasca stack. Here's how.

Before we begin

From your designated Ansible control host, source the Kayobe virtualenv, kayobe-env and admin credentials files. Assuming the virtualenv and kayobe-config locations are the same as in the a-universe-from-nothing tutorial:

$ source ~/kayobe-venv/bin/activate
$ cd ~/kayobe/config/src/kayobe-config/
$ source kayobe-env
$ source etc/kolla/admin-openrc.sh

Any reference to a filesystem path from this point in the guide will be relative to the kayobe-config directory above.

Optional

Optionally, enable Kayobe shell completion:

$ source <(kayobe complete)

Containers

First you'll need Kolla containers, these can either be pulled from Docker Hub or built using Kayobe. Kolla containers are either of the source or binary variety depending on how they were built and this is reflected in their image name. Note that not every component supports both build types and Monasca is only available from source. In practice, this means we'll need to inform Kolla and Kolla-Ansible of the container image to build (unless pulling from Dockerhub) and deploy respectively.

Pulling from Docker Hub

If you've followed a universe-from-nothing build the following script can be used to pull the relevant containers from the Docker Hub Kolla repositories and push them to the seed:

#!/bin/bash
set -e

tag=${1:-train}
images="kolla/centos-binary-zookeeper
kolla/centos-binary-kafka
kolla/centos-binary-storm
kolla/centos-binary-logstash
kolla/centos-binary-kibana
kolla/centos-binary-elasticsearch
kolla/centos-binary-influxdb
kolla/centos-source-monasca-api
kolla/centos-source-monasca-notification
kolla/centos-source-monasca-persister
kolla/centos-source-monasca-agent
kolla/centos-source-monasca-thresh
kolla/centos-source-monasca-grafana"
registry=192.168.33.5:4000

for image in $images; do
    ssh stack@192.168.33.5 sudo docker pull $image:$tag
    ssh stack@192.168.33.5 sudo docker tag $image:$tag $registry/$image:$tag
    ssh stack@192.168.33.5 sudo docker push $registry/$image:$tag
done

Building using Kayobe

Building your own containers is the recommended approach for production OpenStack and is required if customising the Kolla Dockerfiles. The following Kayobe commands can be used to build Monasca and related containers:

$ kayobe overcloud container image build kafka influxdb kibana elasticsearch zookeeper storm logstash --push
$ kayobe overcloud container image build monasca -e kolla_install_type=source --push

The --push argument will push these containers to the Docker registry on the seed node once built.

Configuring Kayobe

StackHPC usually recommends a cluster of 3 separate nodes for monitoring infrastructure but with sufficient available resources it is possible to configure the controllers as monitoring nodes. For separate monitoring nodes see here for an example of adding another node type.

If instead you are running monitoring services on controllers then add the following to etc/kayobe/inventory/groups:

[monitoring:children]
# Add controllers to monitoring group
controllers

Configuring Kolla-Ansible

Add the following to the contents of etc/kayobe/kolla/globals.yml:

# Roles which grant read/write access to Monasca APIs
monasca_default_authorized_roles:
- admin
- monasca-user

# Roles which grant write access to Monasca APIs
monasca_agent_authorized_roles:
- monasca-agent

# Project name to send control plane logs and metrics to
monasca_control_plane_project: monasca_control_plane

This configures Kolla-Ansible with some sane defaults for user and agent roles and finally names the OpenStack project for metrics as monasca_control_plane.

Configuring Monasca

StackHPC makes regular use of the Slack notification plugin for alerts. To demonstrate how this works we'll enable and customise this feature. Customising Monasca requires creating configuration under directories that do not yet exist, so first create both the Kolla config Monasca directory and a subdirectory for alarm notification templates:

$ mkdir -p etc/kayobe/kolla/config/monasca/notification_templates

Monasca-Notification configuration

Populate the monasca-notification container's configuration at etc/kayobe/kolla/config/monasca/notification.conf to enable Slack webhooks and set the notification template:

[notification_types]
enabled = slack,webhook

[slack_notifier]
message_template = "/etc/monasca/slack_template.j2"
timeout = 5
ca_certs = "/etc/ssl/certs/ca-bundle.crt"
insecure = False

[webhook_notifier]
timeout = 5

Slack webhook notification template

Custom Slack notification templates should be placed in etc/kayobe/kolla/config/monasca/notification_templates/slack_template.j2. If you've followed a universe-from-nothing template then the following jinja will work as is:

{% raw %}{% set base_url = "http://{% endraw %}{{ aio_vip_address }}{% raw %}:3001/plugins/monasca-app/page/alarms" -%}
Alarm: `{{ alarm_name }}`
{%- if metrics[0].dimensions.hostname is defined -%}
{% set hosts = metrics|map(attribute='dimensions.hostname')|unique|list %} on host(s): `{{ hosts|join(', ') }}` moved to <{{ base_url }}?dimensions=hostname:{{ hosts|join('|') }}|status>: `{{ state }}`
{%- else %} moved to <{{ base_url }}|status>: `{{ state }}`
{%- endif %}.{% endraw %}

If you've prepared your own deployment then {{ aio_vip_address }} will need to be replaced with the address of an accessible VIP interface as defined in etc/kayobe/networks.yml.

Astute Jinja practitioners may notice that the notification template is wrapped inside {% raw %} tags except for the VIP address: this allows Kayobe to insert a variable not visible at the time Kolla-Ansible templates the file.

Adding Dashboards & Datasources

Monasca-Grafana will need to be configured with the monasca-api address as a metric source. Note elasticsearch can also be configured as a metric source to visualise log metrics. Optionally custom dashboards can also be defined in the same file etc/kayobe/grafana.yml:

# Path to git repo containing Grafana dashboards. Eg.
# https://github.com/stackhpc/grafana-reference-dashboards.git
grafana_monitoring_node_dashboard_repo: "https://github.com/stackhpc/grafana-reference-dashboards.git"

# Dashboard repo version. Optional, defaults to 'HEAD'.
grafana_monitoring_node_dashboard_repo_version: "stable/train"

# The path, relative to the grafana_monitoring_node_dashboard_repo_checkout_path
# containing the dashboards. Eg. /prometheus/control_plane
grafana_monitoring_node_dashboard_repo_path: "/monasca/control_plane"

# A dict of datasources to configure. See the stackhpc.grafana-conf role
# for all supported datasources.
grafana_datasources:
  monasca_api:
    port: 8070
    host: "{{ aio_vip_address }}"
  elasticsearch:
    port: 9200
    host: "{{ aio_vip_address }}"
    project_id: "{{ monasca_control_plane_project_id | default('') }}"

Pulling containers to the overcloud

Once the configuration is in place, it is recommended to prepare for the next step by pulling the new containers from the seed registry to the relevant overcloud nodes:

$ kayobe overcloud container image pull

This also serves to check all required containers are available.

Deploying Monasca

Deploying Monasca and friends using Kayobe can take a considerable length of time due to the number of checks Kayobe and Kolla-Ansible both perform. Being familiar with kolla-ansible we can skip some of these tasks with the --kolla-tags argument:

$ kayobe overcloud service deploy --kolla-tags monasca,elasticsearch,influxdb,mariadb,kafka,kibana,grafana,storm,kafka,zookeeper,haproxy,common

The above command will deploy only Monasca and related services. A word of caution however, limiting tasks in this fashion can have unexpected consequences for inexperienced users and honestly doesn't save much time. If in doubt which tags are required run a full deploy with:

$ kayobe overcloud service deploy

Now would be a good point to grab a cup of tea.

Run in a production environment, this command shouldn't cause any disruption to tenant services (on sufficient hardware) but HAProxy will restart, potentially interrupting connections to the API for a brief period.

Adding Dashboards & Datasources

Assuming the deployment completed successfully, additional tasks are still required to configure Grafana with the datasources and dashboards defined in etc/kayobe/grafana.yml:

$ kayobe overcloud post configure --tags grafana

Testing

You should now be able to navigate to Grafana and Kibana, found by default on ports 3001 & 5601.

To start using the Monasca CLI, install it from PyPI and configure the relevant roles to authenticate in the Monasca project. Create and activate a fresh venv for the purpose:

$ deactivate
$ python3 -m venv ~/monasca-venv
$ source ~/monasca-venv/bin/activate
$ pip install python-openstackclient
$ pip install python-monascaclient
$ source etc/kolla/admin-openrc.sh

Optionally enable shell completion:

$ source <(openstack complete)
$ source <(monasca complete)

Add the admin user to the monasca_control_plane project (and double check):

$ openstack role add --user admin --project monasca_control_plane admin
$ openstack role assignment list --names --project monasca_control_plane

Switch to the monasca_control_plane project and view available metric names:

$ export OS_PROJECT_NAME=monasca_control_plane
$ unset OS_TENANT_NAME
$ monasca metric-name-list

Alarms!

Don't forget to install the Slack Incoming webhooks integration in order to make use of Monasca alerts. Once that is installed and configured in a channel, you'll be provided with a webhook URL - since this is a private URL it should be secured before being added to kayobe-config (for more information see the Kayobe documentation on secrets).

Deploying Alarms from a custom playbook

Alarms and notification definitions can be created using the monasca CLI, but in keeping with the configuration-as-code approach thus far we'd recommend our ansible role for the task - it contains a reasonably sane set of alarms for monitoring both overcloud nodes and OpenStack services.

With some additional configuration, the role can be installed and used by Kayobe.

First create the directory and provide symlinks to Kayobe Ansible as per the documentation:

$ mkdir -p etc/kayobe/ansible
$ cd etc/kayobe/ansible
$ ln -s ../../../../kayobe/ansible/filter_plugins/ filter_plugins
$ ln -s ../../../../kayobe/ansible/group_vars/ group_vars
$ ln -s ../../../../kayobe/ansible/test_plugins/ test_plugins
$ cd -

And then etc/kayobe/ansible/requirements.yml to specify the role:

---
- src: stackhpc.monasca_default_alarms
  version: 1.3.0

An example playbook to deploy only the system level alerts (cpu, disk, memory usage) in etc/kayobe/ansible/monasca_alarms.yml. This assumes you've created a variable for your slack webhook called secrets_monasca_slack_webhook and that the monasca CLI virtualenv is in ~/monasca-venv:

- name: Create Monasca notification method and alarms
  hosts: localhost
  gather_facts: yes
  vars:
    keystone_url: "http://{{ aio_vip_address }}:5000/v3"
    keystone_project: "monasca_control_plane"
    monasca_endpoint_interface: ["internal"]
    notification_address: "{{ secrets_monasca_slack_webhook }}"
    notification_name: "Default Slack Notification"
    notification_type: "SLACK"
    monasca_client_virtualenv_dir: "~/monasca-venv"
    virtualenv_become: "no"
    skip_tasks: ["misc", "openstack", "monasca", "ceph"]
  roles:
    - {role: stackhpc.monasca_default_alarms, tags: [alarms]}

The Ansible galaxy role can be installed using Kayobe with:

$ kayobe control host bootstrap

And the playbook invoking it can be executed with:

$ kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/monasca_alarms.yml