You want to set up Prometheus and Grafana monitoring of your Mattermost cluster, but you don’t have more than fifteen minutes.
1. Configure Mattermost
In the System Console, go to Performance Monitoring and turn it on. Then, install
prometheus-node-exporter on your Mattermost server to get hardware performance metrics.
2. Install the Monitoring Docker Compose
Install Docker on a system that can access the Mattermost servers on port 8067, as well as
docker swarm. Then, run
docker swarm init to initialize the swarm. The current host is added to the swarm automatically.
Then, use git to clone this great docker stack setup
git clone https://github.com/vegasbrianc/prometheus.git
Then edit the file
grafana/config.monitoring to set the Grafana administrator password. The default is
prometheus/prometheus.yml to add your Mattermost servers to the
node-exporter job and a new job named
mattermost Here’s an example for a four node server, plus monitoring for the system running the docker-compose container. I split them into separate jobs for clarity:
- job_name: 'node-exporter' # Install prometheus-node-exporter on your Mattermost app servers as well scrape_interval: 5s static_configs: - targets: ['node-exporter:9100', '172.31.45.9:9100','172.31.45.244:9100','172.31.44.249:9100','172.31.32.51:9100'] # Mattermost servers have their metrics on port 8067 - job_name: 'mattermost' static_configs: - targets: ['172.31.45.9:8067','172.31.45.244:8067','172.31.44.249:8067','172.31.32.51:8067']
3. Start Monitoring
Run this command to start the containers:
HOSTNAME="<hostname>" docker stack deploy -c ~docker-stack.yml~ prom
<hostname> with the hostname you’d like to use.
Once the services are started, going to
http://<hostname>:3000 and log into Grafana with the username
admin and the password you set earlier.
Once you’re there, go to
Manage. You’ll see there’s already a dashboard that is monitoring the Docker. To import another dashboard, click the
Import button, then enter the ID for the dashboard in the text box and click “Import”. On the next screen, select the Prometheus data source. Here are four dashboards to get started:
- Mattermost Performance Monitoring - 2542
- Mattermost Performance Monitoring (Bonus Metrics) - 2545
- Node Exporter Full - 1860
- Node Exporter Server Metrics - 405
When viewing the Node Exporter dashboards, make sure to only select the nodes that end with
:9100, and for the Mattermost dashboards select the nodes with
Thanks to containerization and orchestartion that was a lot easier than doing it by hand. As for performance, it’s been running all day on a t2.medium (2vCPU, 4GB RAM) for most of today but the load hasn’t gone above 0.01 even though I’ve been watching it all day.
Also, if you know how to show hostnames instead of IP addresses in Grafana leave me a tip in the comments!