Nginx Bad Gateway after update


#1

After upgrading from version 3 shipped with Turnkeys Image of mattermost nginx is reporting bad gateway.

I switched out the nginx config with the one from the docs and added the ssl parts to it. It seems something with the proxy_pass is not working right. I use non-standard ports 8440 and 8441 for http and https respectively, but set them inside the server and nginx.

Current nginx config:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m;

server {
   listen 8440 default_server;
   server_name ##URL##;
   return 301 https://$server_name$request_uri;
}

server {
    listen 8441 ssl;
    include /etc/nginx/include/ssl;
    server_name ##URL##;

    location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_pass https://##URL##:8441;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_http_version 1.1;
       proxy_pass https://##URL##:8441;
   }
}

upstream ##URL##:8441 {
   server 127.0.0.1:8065;
   keepalive 32;
}

I have exhausted all information I could find regarding the configuration changes required. The mattermost config itself reads servicesettings as such:

"ServiceSettings": {
        "SiteURL": "https://##URL##:8441",
        "LicenseFileLocation": "",
        "ListenAddress": "127.0.0.1:8065",
        "ConnectionSecurity": "",
        "TLSCertFile": "",
        "TLSKeyFile": "",
        "UseLetsEncrypt": true,
        "LetsEncryptCertificateCacheFile": "./config/letsencrypt.cache",
        "Forward80To443": false,
        "ReadTimeout": 300,
        "WriteTimeout": 300,
        "EnableAPIv3": true,
        "MaximumLoginAttempts": 10,
        "GoogleDeveloperKey": "",
        "EnableOAuthServiceProvider": true,
        "EnableIncomingWebhooks": true,
        "EnableOutgoingWebhooks": true,
        "EnableCommands": true,
        "EnableOnlyAdminIntegrations": true,
        "EnablePostUsernameOverride": true,
        "EnablePostIconOverride": true,
        "EnableLinkPreviews": false,
        "EnableTesting": false,
        "EnableDeveloper": false,
        "EnableSecurityFixAlert": true,
        "EnableInsecureOutgoingConnections": false,
        "EnableMultifactorAuthentication": false,
        "EnforceMultifactorAuthentication": false,
        "AllowCorsFrom": "",
        "SessionLengthWebInDays": 30,
        "SessionLengthMobileInDays": 30,
        "SessionLengthSSOInDays": 30,
        "SessionCacheInMinutes": 10,
        "WebsocketSecurePort": 8441,
        "WebsocketPort": 8440,
        "WebserverMode": "gzip",
        "EnableCustomEmoji": false,
        "RestrictCustomEmojiCreation": "all",
        "RestrictPostDelete": "all",
        "AllowEditPost": "always",
        "PostEditTimeLimit": 300,
        "TimeBetweenUserTypingUpdatesMilliseconds": 5000,
        "EnableUserTypingMessages": true,
        "ClusterLogTimeoutMilliseconds": 2000
    }

#2

Hi @Tampa!

Here are some initial tips/question:

  1. What is your Mattermost server version?

  2. Can you help take a look if these docs might help: https://docs.mattermost.com/install/config-proxy-nginx.html#configuring-nginx-as-a-proxy-for-mattermost-server and https://docs.mattermost.com/install/config-ssl-http2-nginx.html

  3. Can you send us a snippet of your Nginx error logs around the time of the incident?

Let me know if this helps!


#3

It should now be running the latest 4.10.1 on Debian 8.10 Jessie
I followed those instructions as you can see from the nginx config, I only tried around with the configuration to see if I could produce any other errors or get anywhere. Nginx simply reports:

2018/06/06 13:43:41 [error] 3264#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: ##########, server: ##URL## request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8065/", host: "##URL##:8441"

I don’t have any further information available, mattermost log itself is completely empty, but it is running according to systemd.


#4

So I disabled all encryption and now the server works again.

It is listening to 127.0.0.0:8065 which means I cannot enable 80to443 fowarding, but to use letsencrypt I need to do so, but since I am not using 443 for mattermost, instead using 8441 behind a proxy it will tell me to disable.

I am literally deadlocked unless I use the standard ports, which I cannot use because the server is inside a VPN requiring non-standard ports.

What the heck?


#5

Hi @Tampa! Thank you for the additional information.

Can you help check if any of these previous tips might help:

  1. Nginx/1.4.6 (Ubuntu) - 504 Gateway Time-out
  2. [SOLVED] Mattermost Nginx Configuration
  3. No connection to https nginx reverse proxy with lets encrypt cert
  4. This thread: SOLVED - Nginx Config Ubuntu 16.04

If not, I encourage you to join our nightly build server and then join either Developer Toolkit channel or Peer-to-peer help channel to discuss this with our team - I don’t have enough knowledge about this topic, but our engineers can most certainly help and they are more active at the build server.

Let me know if you have any questions!