We're Hiring!

Mattermost, Inc.

Mattermost Recipe: Monitoring with Prometheus and Grafana in Under Fifteen Minutes

Problem

You want to set up Prometheus and Grafana monitoring of your Mattermost cluster, but you don’t have more than fifteen minutes.

Solution

1. Configure Mattermost

In the System Console, go to Performance Monitoring and turn it on. Then, install prometheus-node-exporter on your Mattermost server to get hardware performance metrics.

2. Install the Monitoring Docker Compose

Install Docker on a system that can access the Mattermost servers on port 8067, as well as docker-compose and docker swarm. Then, run docker swarm init to initialize the swarm. The current host is added to the swarm automatically.

Then, use git to clone this great docker stack setup

git clone https://github.com/vegasbrianc/prometheus.git

Then edit the file grafana/config.monitoring to set the Grafana administrator password. The default is foobar.

Next, edit prometheus/prometheus.yml to add your Mattermost servers to the node-exporter job and a new job named mattermost Here’s an example for a four node server, plus monitoring for the system running the docker-compose container. I split them into separate jobs for clarity:

  - job_name: 'node-exporter'

    # Install prometheus-node-exporter on your Mattermost app servers as well
    scrape_interval: 5s
    static_configs:
         - targets: ['node-exporter:9100', '172.31.45.9:9100','172.31.45.244:9100','172.31.44.249:9100','172.31.32.51:9100']
  # Mattermost servers have their metrics on port 8067
  - job_name: 'mattermost'
    static_configs:
    - targets: ['172.31.45.9:8067','172.31.45.244:8067','172.31.44.249:8067','172.31.32.51:8067']                                                                                                 

3. Start Monitoring

Run this command to start the containers:

HOSTNAME="<hostname>" docker stack deploy -c ~docker-stack.yml~ prom

Replace <hostname> with the hostname you’d like to use.

Once the services are started, going to http://<hostname>:3000 and log into Grafana with the username admin and the password you set earlier.

Once you’re there, go to Dashboards > Manage. You’ll see there’s already a dashboard that is monitoring the Docker. To import another dashboard, click the Import button, then enter the ID for the dashboard in the text box and click “Import”. On the next screen, select the Prometheus data source. Here are four dashboards to get started:

When viewing the Node Exporter dashboards, make sure to only select the nodes that end with :9100, and for the Mattermost dashboards select the nodes with :8067.

Discussion

Thanks to containerization and orchestartion that was a lot easier than doing it by hand. As for performance, it’s been running all day on a t2.medium (2vCPU, 4GB RAM) for most of today but the load hasn’t gone above 0.01 even though I’ve been watching it all day.

Also, if you know how to show hostnames instead of IP addresses in Grafana leave me a tip in the comments!

3 Likes