We're Hiring!

Mattermost, Inc.

Bridge Nginx and Mattermost on the same server

I have Mattermost set up and running using the default web server/configuration. I would like to have it running on Nginx. I’ve configured Nginx following the Nginx Server instructions. I can reach the page in the browser with SSL.

I’m having trouble pointing the Mattermost installation/configuration to the Nginx configuration.

Any help will be greatly appreciated.

When you say “the page” is the page in reference the default SSL page, or is that the mattermost setup/login page?

I meant the intended site page using Nginx, using my site’s sub-domain (mattermost.mydomain.com). What I can’t figure out is how to get Mattermost to load on that site instead of using the default web server.

Below, I’ve attached a copy of my nginx configuration file

cat /etc/nginx/conf.d/mattermost
upstream backend {
   server 0.0.0.0:4431;
   keepalive 32;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
   listen 4431 default_server;
   server_name  mattermost.example.com ;
   return 301 https://$server_name$request_uri;
}

server {
   listen 4431 http2;
   server_name   mattermost.example.com;

   ssl on;
   ssl_certificate /etc/nginx/certs/OCert.pem;
   ssl_certificate_key /etc/nginx/certs/priv_key.pem;
   ssl_client_certificate /etc/nginx/certs/cloudflare.crt;
   ssl_verify_client on;
   ssl_session_timeout 1d;
   ssl_protocols TLSv1.2 TLSv1.3;
   ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:50m;
   # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
   add_header Strict-Transport-Security max-age=15768000;
   # OCSP Stapling ---
   # fetch OCSP records from URL in ssl_certificate and cache them
   ssl_stapling on;
   ssl_stapling_verify on;
   resolver 1.1.1.1;

   location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_pass http://backend;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_http_version 1.1;
       proxy_pass http://backend;
   }
}

Did you remove the default enabled site with sudo rm /etc/nginx/sites-enabled/default and enable your mattermost site configuration by symlinking the configuration file in /etc/nginx/sites-available/my-file to /etc.nginx/sites-enabled/my-file?

Also, can I ask why the server IP address is 0.0.0.0? If your referring to the localhost, it should be 127.0.0.1 or just plain localhost, right?

Yes, I did and set the above config file as the default.

I guess I’m somewhat confused then since It appears in the top portion of the above configuration file you still have the local IP address as 0.0.0.0, and this is incorrect. Here is why:

According to Wikipedia,
In the Internet Protocol Version 4, the address 0.0.0.0 is a non-routable meta-address used to designate an invalid, unknown or non-applicable target. This address is assigned specific meanings in a number of contexts, such as on clients or on servers.
Which directly states that the IP address in question is non-routable.

If you don’t know what a non-routable IP address is, Nexor Cybersecurity provides a nice explanation:
The term non-routable just means exactly that; that IP packets cannot be directed from one network to another.
This states that a non-routable IP address cannot direct any packets to any location on any network, be that internal, public, private, virtualized, etc, and as such, effectively states that 0.0.0.0 is a dead-end or “black hole” of sorts for internet information and network data transfer.

You may be thinking “I should google this, this is speaking in networking terms not related to being on a server.” Don’t worry, the following covers specifically using the IP address 0.0.0.0 on servers!

ThreatBrief provides a great explanation in detail of the use of 0.0.0.0 in server networking, however here I include a small paraphrase of the general information, to get the point across:

0.0.0.0 – you may have noticed this IP address several times during your online sojourns. Many users think this IP address belongs to some hackers while others think nothing of it. Tech geeks know that it’s a ‘no particular address’ placeholder’.
While it is true, 0.0.0.0 actually specifies all IP addresses on all interfaces on the systems. It’s a non-routable meta-address that’s used to define an unknown or invalid target. If we talk about servers, 0.0.0.0 means all IPv4 addresses on the local machine. In the case of a route entry, it means the default route.

Your PC will pop up a 0.0.0.0 address when it is not connected to any TCP/IP network because it has to resolve the IP address. By default, if you see a 0.0.0.0 IP address, your PC/laptop is offline. Similarly, if the host PC is unable to resolve IP failure, it may be automatically assigned by DHCP. Once your DHCP is assigned 0.0.0.0, it can’t communicate with any other device over IP. In simple language, it means no internet or intranet access.

So in other terms, even if somehow this was able to be used as a network-connected address, which would require sophisticated reconfiguration of your entire IP tables setup, and could likely severely impact your DHCP and or even static network connections to the public internet, the standard system processing of the 0.0.0.0 IP address is that it is a placeholder and non-routable address. A further note to add is that in the event that it was being detected and parsed as a wildcard, the full range of IP addresses in the IP block 127.0.0.1/32 would all be being searched for a connection to a webserver at the same time which would also result in errors, if not total failure.

That all being said, your Nginx configuration file is functioning as a reverse proxy, which means that it is taking the IP address found in the upstream backend and listening to that IP address on your local machine and then taking the information and data that it receives from the internal IP address, and porting it to the desired external port. (IE the port you are connecting to from your website, the standard 80, 443, or a custom port. The external port is what is exposed to the world and connected to.)

This is where the issue is.
In the configuration file which you have posted above, the upstream backend IP address is shown as 0.0.0.0 which is effectively a data black hole. (See below for the snippet of code from your configuration file posted above that shows this)

cat /etc/nginx/conf.d/mattermost
upstream backend {
   server 0.0.0.0:4431;
   keepalive 32;
}

Instead of being set to 0.0.0.0 this should be a localhost internal IP address, which is most commonly (and by default) 127.0.0.1 or just plainly localhost. Setting the IP address to either of the two, and ensuring that the Mattermost config.json file reflects the same listening address (Example of the referenced portion of config.json shown below is from my personal Mattermost installation)

{
    "ServiceSettings": {
        "SiteURL": "https://XXXXXXXXXREDACTEDXXXXXXXX",
        "WebsocketURL": "",
        "LicenseFileLocation": "",
        "ListenAddress": ":8065",
        "ConnectionSecurity": "",
        "TLSCertFile": "",
        "TLSKeyFile": "",
        "TLSMinVer": "1.2",
        "TLSStrictTransport": false,
        "TLSStrictTransportMaxAge": 63072000,
        "TLSOverwriteCiphers": [],
        "UseLetsEncrypt": false,
        "LetsEncryptCertificateCacheFile": "./config/letsencrypt.cache",
        "Forward80To443": false,
        "TrustedProxyIPHeader": [
            "X-Forwarded-For",
            "X-Real-IP"
        ],

Seeing that the local port used in your upstream backend is port 4431, I would suggest just verifying that the port declared in "ListenAddress": is set as "ListenAddress": ":4431",
As something to note, you may notice that in the mattermost configuration files, there are no options to set the listening IP address. This is more secure because it forces the user to reverse proxy the server’s instance to the public internet, which in principle is a much safer way to do it, not to even mention that there are many other reasons why it’s just better.

I hope this helps, If your still experiencing troubles or I am somehow misunderstanding the issue, I’ll get back to you as soon as I can!

1 Like

I’m sorry, I’ve not been able to respond promptly.

Back on the issue, we’re trying to solve; I do understand the addressing scheme (I left 0.0.0.0 in the attachment for provisional purposes, pardon my mistake I should have used the generic loopback address). Nonetheless, I appreciate your effortless knowledgeability on the matter and willingness to explain everything in sufficient detail.

While digging deeper into the issue over the weekend, I noticed proxy_pass http://backend; line which I believe should be pointing to the Mattermost installation or address and port, e.g. proxy_pass http://localhost:8065; or the correct port based on the config’.

Now, I’ve not tried this implementation, but I think that could be part of the problem. Also, just to clarify, both my Nginx and Mattermost installations are running on the same server.
I can’t set Nginx to listen on the same port as Mattermost because then one of the two won’t start.

Again, I can get to the Mattermost installation from the browser as well as Nginx, except the fact that Nginx throws a page error 503 Service Temporarily Unavailable :: nginx/1.17.6; see below.

mattermost

At this point, I believe it’s my proxy settings that are failing me.

Yes indeed, you aren’t supposed to proxy your Nginx server and mattermost using the same port, the intended way to do this is to use Nginx as the proxy, to take a port (for example, 8080) and redirect that data to, say, port 80, or for HTTPS, to port 443.

To do this you would need to set your mattermost port as 8080, and proxy it to the port that you want your outside instance to connect to, and take care to make sure you also reverse proxy your WebSockets as well.

I feel like we’re getting closer to the solution, but this is exactly where I’m hazy (proxying). I haven’t really grasped the concept but I’m going to read up on how to get it going especially with Nginx

I do agree, basically a reverse proxy with either Nginx or Apache is along the same concept as if you were to route port 8080 (as I used in my example) to port 80, with iptables or another protocol such as that, except that it is able to be configured to pass headers and allow websockets etc as well

1 Like