Mattermost, Inc.

Mattermost behind Apache Proxy - wss not working

I’m new to Mattermost but not to server admin/web admin. I’ve got Mattermost up and running and I’m really liking it so far over the jumbled, complicated mess that is Riot/Synapse. Anyway - I’ve got it setup behind an Apache proxy. I’ve tried a few iterations here but none of them seem to want the wss traffic to pass through meaning I get the red banner warning saying “Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.”

The way it’s setup now is that I’ve got my overall apache reverse proxy that has the following config:

<IfModule mod_ssl.c>

<VirtualHost *:80>

        ServerName chat.domain.net

        ProxyPreserveHost On

        ProxyPass / http://10.20.5.30:80
        ProxyPassReverse / http://chat.domain.net

        RewriteEngine on
        RewriteCond %{SERVER_NAME} =chat.domain.net
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

</VirtualHost>

        <VirtualHost *:443>

                ServerName chat.domain.net

                ProxyPreserveHost On
                SSLEngine On
                SSLProxyEngine On

                ProxyPass / https://10.20.5.30:443/
                ProxyPassReverse / https://chat.domain.net:443/

                RewriteEngine On
                RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
                RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
                RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
                RewriteRule .* ws://10.20.5.30%{REQUEST_URI} [P,QSA,L]

                Include /etc/letsencrypt/options-ssl-apache.conf


                SSLCertificateFile /etc/letsencrypt/live/domain/cert.pem
                SSLCertificateKeyFile /etc/letsencrypt/live/domain/privkey.pem
                SSLCertificateChainFile /etc/letsencrypt/live/domain/chain.pem

        </VirtualHost>

</IfModule>

Then on the mattermost server I’ve got nginx running just fine with the following config:

upstream backend {
        server 10.20.5.30:8065;
        keepalive 32;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
        listen 80;
        server_name    chat.domain.net;
        return 301 https://$server_name$request_uri;
}

server {
        listen 443 ssl http2;
        server_name chat.domain.net;

        ssl on;
        ssl_certificate /etc/nginx/ssl/00/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/00/privkey.pem;
        ssl_session_timeout 1d;
        ssl_protocols TLSv1.2;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:50m;

        location ~ /api/v[0-9]+/(users/)?websocket$ {
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                client_max_body_size 50M;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Frame-Options SAMEORIGIN;
                proxy_buffers 256 16k;
                proxy_buffer_size 16k;
                client_body_timeout 60;
                send_timeout 300;
                lingering_timeout 5;
                proxy_connect_timeout 90;
                proxy_send_timeout 300;
                proxy_read_timeout 90s;
                proxy_pass http://backend;
        }

        location / {
                client_max_body_size 50M;
                proxy_set_header Connection "";
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Frame-Options SAMEORIGIN;
                proxy_buffers 256 16k;
                proxy_buffer_size 16k;
                proxy_read_timeout 600s;
                proxy_cache mattermost_cache;
                proxy_cache_revalidate on;
                proxy_cache_min_uses 2;
                proxy_cache_use_stale timeout;
                proxy_cache_lock on;
                proxy_http_version 1.1;
                proxy_pass http://backend;
        }
}

I’m not a big fan of having a proxy feed to a proxy but it’s working for everything but wss. When I turn off nginx on the mattermost box then it stops working completely so I haven’t tried eliminating the nginx proxy and just using the front end proxy only.

Do I need to eliminate the nginx proxy and just use the apache up front or is there a way to get apache to pass the wss on to the mattermost box.

@richardsonje This troubleshooting doc might help: https://docs.mattermost.com/install/troubleshooting.html#please-check-connection-mattermost-unreachable-if-issue-persists-ask-administrator-to-check-websocket-port

Been crazy busy but finally able to look around at this. The documentation you provided is just the troubleshooting steps I’ve already performed. I know it’s a proxy issue … I’m trying to figure out how to resolve the proxy issue.

Not sure if there’s a way to make it happen, however, I have it working now. I had to remove the nginx proxy from the mattermost VM and then I kept only the apache proxy on my primary reverse proxy. Once I got that setup, it works great now.