We're Hiring!

Mattermost, Inc.

Help on SSL config on Apache

Summary
I cannot manage the SSL configuration with LE certificate on a newly installed MM 5.31.4 server on Ubuntu 20.04 and Apache.

Expected behavior
MM should be accessible via https.
2-3 years ago I successfully installed another 5.3 MM server with a LE certificate. Thus I have a correct Apache conf configuration for comparison.

Observed behavior
Browser always says “timeout” after a longer period.

Apache modules installed

 alias_module (shared)
 auth_basic_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 core_module (static)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 filter_module (shared)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 mime_module (shared)
 mpm_prefork_module (shared)
 negotiation_module (shared)
 php7_module (shared)
 proxy_http_module (shared)
 proxy_module (shared)
 proxy_wstunnel_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 so_module (static)
 status_module (shared)
 unixd_module (static)
 version_module (static)
 watchdog_module (static)

Apaches mm-ssl.conf

<IfModule ssl_module>

<VirtualHost *:80>
        ServerName      gwmm1.vss.xyz.com
        ServerAdmin     hostmaster@mydomain.com

        RewriteEngine	on
        RewriteCond	%{SERVER_NAME} =gwmm1.vss.xyz.com
        RewriteRule	^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>

<VirtualHost *:443>
        ServerName	gwmm1.vss.xyz.com
        ServerAdmin	hostmaster@mydomain.com

        ProxyPreserveHost On
	DocumentRoot	/opt/mattermost

        # Setup the proxy
        <Proxy *>
                Order allow,deny
                Allow from all
        </Proxy>

	# Set web sockets
        RewriteEngine On
        RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
        RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
        RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
        RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
        RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

        <LocationMatch "^/api/v(?<apiversion>[0-9]+)/(?<apiusers>users/)?websocket">
                Require all granted
                ProxyPass ws://127.0.0.1:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
                ProxyPassReverse ws://127.0.0.1:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
                ProxyPassReverseCookieDomain 127.0.0.1 gwmm1.vss.xyz.com
        </LocationMatch>

        <Location />
                Require all granted
                ProxyPass http://127.0.0.1:8065/
                ProxyPassReverse http://127.0.0.1:8065/
                ProxyPassReverseCookieDomain 127.0.0.1 gwmm1.vss.xyz.com
        </Location>

        # Generated by Certbot
        SSLCertificateFile	/etc/letsencrypt/live/gwmm1.vss.xyz.com/fullchain.pem
        SSLCertificateKeyFile	/etc/letsencrypt/live/gwmm1.vss.xyz.com/privkey.pem
	# SSLCertificateFile	/etc/ssl/certs/ssl-cert-snakeoil.pem
	# SSLCertificateKeyFile	/etc/ssl/private/ssl-cert-snakeoil.key

	SSLProtocol	all -SSLv3 -TLSv1 -TLSv1.1
        SSLCipherSuite	ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

        # HSTS (mod_headers is required) (15768000 seconds = 6 months)
        # (Browser never connects in next 6 months to http://, always https://)
	Header always set Strict-Transport-Security "max-age=15768000 ; includeSubDomains"

</VirtualHost>

SSLHonorCipherOrder	on
SSLCompression		off
SSLSessionTickets       off

# OCSP Stapling, only in httpd 2.3.3 and later
SSLUseStapling          on
SSLStaplingResponderTimeout            5
SSLStaplingReturnResponderErrors       off
SSLStaplingCache        shmcb:/var/run/ocsp(128000)

</IfModule>

The example config on Configuring Apache2 with SSL and HTTP/2 (Unofficial) — Mattermost 5.35 documentation also ends with timeout and I would rather have the configs from /etc/letsencrypt/options-ssl-apache.conf (or similar) in my own conf file.

Can anyone see my mistake?

Hello! The issue you are having here has to do with the fact that you have your HTTP configuration set to automatically redirect and or rewrite the HTTP request into an HTTPS request. This means that when you run the LetsEncrypt certificate generator, and the program attempts to make a HTTP request to your domain name, it won’t work due to the request being redirected to HTTPS, when it needs to be HTTP.

1 Like

Hello XxLilBoPeepsxX,
there seems to be a misunderstanding here.
I have not started the LetsEncrypt certificate generator, certbot is not the problem here and my certificate exist since one week.
You can see the LE certificate already embedded in my conf file. What doesn’t work is that Mattermost starts encrypted with https.
http://gwmm1.vss.kapper.net/

Well for that, I believe you need to specify in the Mattermost config.json that you want it to start with HTTPS. Can you provide the first two sections of your config.json?

Current setting is: “SiteURL”: “http://gwmm1.vss.kapper.net”, because the customer want to prepare some chanells before the first MM user will register (under SSL).

I already know this setting. But when I switch to https, mattermost.service no longer starts.

journalctl -xe than says:

May 23 22:16:25 gwmm1 mattermost[10020]: {"level":"error","ts":1621800985.2333126,"caller":"sqlstore/supplier.go:258","msg":"Failed to ping DB","error":"dial tcp [::1]:5432: connect: connection refused","retrying in seconds":10}

So I currently have checked the SqlSettings and it happens to me the second time, that anyone rewrites my MySQL connection to postgres.

    "SqlSettings": {
        "DriverName": "postgres",
        "DataSource": "postgres://mmuser:mostest@localhost/mattermost_test?sslmode=disable\u0026connect_timeout=10",
        "DataSourceReplicas": [],
        "DataSourceSearchReplicas": [],
        "MaxIdleConns": 20,
        "ConnMaxLifetimeMilliseconds": 3600000,
        "MaxOpenConns": 300,
        "Trace": false,
        "AtRestEncryptKey": "64rbhxunrbyjde87ucci59jqt5egbhub",
        "QueryTimeout": 30,
        "DisableDatabaseSearch": false
    },

I will try to reconfigure tomorrow.
Many thanks for your support (y)

Well, after I edited the config.json file 3-4 times because during editing a process always overwrote my SqlSettings and also the SiteURL, mattermost.service is now running (even after a server restart).

● mattermost.service - Mattermost
     Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-05-24 14:51:47 CEST; 1min 23s ago
   Main PID: 404 (mattermost)
      Tasks: 23 (limit: 4915)
     Memory: 432.5M
     CGroup: /system.slice/mattermost.service
             ├─404 /opt/mattermost/bin/mattermost
             ├─453 plugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64
             └─459 plugins/com.mattermost.nps/server/dist/plugin-linux-amd64

May 24 14:51:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860707.3508918,"caller":"app/server.go:693","msg":"Advanced logging metrics enabled"}
May 24 14:51:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860707.35091,"caller":"app/server.go:1391","msg":"License cannot be found."}
May 24 14:51:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860707.3529086,"caller":"app/server.go:1274","msg":"Cannot obtain last advisory run timestamp","error":"getLastWarnMetricTimestamp: Unable to find the system variable., failed to get system property with name=: sql: no rows in result set"}
May 24 14:51:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860707.3783464,"caller":"jobs/schedulers.go:187","msg":"Next run time for scheduler","scheduler_name":"MigrationsScheduler","next_runtime":"2021-05-24 14:52:47.378334782 +0200 CEST m=+68.351821261"}
May 24 14:51:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860707.3811777,"caller":"jobs/schedulers.go:187","msg":"Next run time for scheduler","scheduler_name":"PluginsScheduler","next_runtime":"2021-05-25 14:51:47.381171114 +0200 CEST m=+86408.354657590"}
May 24 14:51:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860707.4036505,"caller":"jobs/schedulers.go:187","msg":"Next run time for scheduler","scheduler_name":"ExpiryNotifyScheduler","next_runtime":"2021-05-24 15:01:47.403642386 +0200 CEST m=+608.377128864"}
May 24 14:51:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860707.407964,"caller":"jobs/schedulers.go:187","msg":"Next run time for scheduler","scheduler_name":"ProductNoticesScheduler","next_runtime":"2021-05-24 15:51:47.407956425 +0200 CEST m=+3608.381442898"}
May 24 14:52:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860767.4100418,"caller":"migrations/scheduler.go:50","msg":"Scheduling Job","scheduler":"MigrationsScheduler"}
May 24 14:52:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860767.4116666,"caller":"migrations/scheduler.go:89","msg":"All migrations are complete.","scheduler":"MigrationsScheduler"}
May 24 14:52:47 gwmm1 mattermost[404]: {"level":"debug","ts":1621860767.412663,"caller":"jobs/schedulers.go:187","msg":"Next run time for scheduler","scheduler_name":"MigrationsScheduler","next_runtime":"<nil>"}

However, Mattermost can still not be reached via https. I guess it is now still due to Apache’s conf file.
https://gwmm1.vss.kapper.net/

My current mm-ssl.conf

that runs in myother MM server since 5-6 years …


<IfModule ssl_module>


<VirtualHost *:80>
        ServerName      gwmm1.vss.kapper.net
        ServerAdmin     hostmaster@mydomain.com

        RewriteEngine	on
        RewriteCond	%{SERVER_NAME} =gwmm1.vss.kapper.net
        RewriteRule	^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>


<VirtualHost *:443>
        ServerName	gwmm1.vss.kapper.net
        ServerAdmin	hostmaster@mydomain.com

        ProxyPreserveHost On
	DocumentRoot	/opt/mattermost

        # Setup the proxy
        <Proxy *>
                Order allow,deny
                Allow from all
        </Proxy>


# begin MM config --------------------------------------------------------
	# Set web sockets
        RewriteEngine On
        RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
        RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
        RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
        RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
        RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

        <LocationMatch "^/api/v(?<apiversion>[0-9]+)/(?<apiusers>users/)?websocket">
                Require all granted
                ProxyPass ws://127.0.0.1:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
                ProxyPassReverse ws://127.0.0.1:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
                ProxyPassReverseCookieDomain 127.0.0.1 gwmm1.vss.kapper.net
        </LocationMatch>

        <Location />
                Require all granted
                ProxyPass http://127.0.0.1:8065/
                ProxyPassReverse http://127.0.0.1:8065/
                ProxyPassReverseCookieDomain 127.0.0.1 gwmm1.vss.kapper.net
        </Location>
# end MM config ----------------------------------------------------------------


        # Generated by Certbot
	SSLEngine on
        SSLCertificateFile	/etc/letsencrypt/live/gwmm1.vss.kapper.net/fullchain.pem
        SSLCertificateKeyFile	/etc/letsencrypt/live/gwmm1.vss.kapper.net/privkey.pem
	# SSLCertificateFile	/etc/ssl/certs/ssl-cert-snakeoil.pem
	# SSLCertificateKeyFile	/etc/ssl/private/ssl-cert-snakeoil.key

	SSLProtocol	all -SSLv3 -TLSv1 -TLSv1.1
        SSLCipherSuite	ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

        # HSTS (mod_headers is required) (15768000 seconds = 6 months)
        # (Browser never connects in next 6 months to http://, always https://)
	Header always set Strict-Transport-Security "max-age=15768000 ; includeSubDomains"

</VirtualHost>

SSLHonorCipherOrder	on
SSLCompression		off
SSLSessionTickets       off

# OCSP Stapling, only in httpd 2.3.3 and later
SSLUseStapling          on
SSLStaplingResponderTimeout            5
SSLStaplingReturnResponderErrors       off
SSLStaplingCache        shmcb:/var/run/ocsp(128000)

</IfModule>

Do you see the cause here?

I believe that you have the IfModule block above the HTTP section, which would essentially force your Apache2 server to parse all HTTP functions as if they were HTTPS. I would suggest trying the following:


<VirtualHost *:80>
        ServerName      gwmm1.vss.kapper.net
        ServerAdmin     hostmaster@mydomain.com

        RewriteEngine	on
        RewriteCond	%{SERVER_NAME} =gwmm1.vss.kapper.net
        RewriteRule	^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>

<IfModule ssl_module>
<VirtualHost *:443>
        ServerName	gwmm1.vss.kapper.net
        ServerAdmin	hostmaster@mydomain.com

        ProxyPreserveHost On
	DocumentRoot	/opt/mattermost

        # Setup the proxy
        <Proxy *>
                Order allow,deny
                Allow from all
        </Proxy>


# begin MM config --------------------------------------------------------
	# Set web sockets
        RewriteEngine On
        RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
        RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
        RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
        RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
        RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

        <LocationMatch "^/api/v(?<apiversion>[0-9]+)/(?<apiusers>users/)?websocket">
                Require all granted
                ProxyPass ws://127.0.0.1:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
                ProxyPassReverse ws://127.0.0.1:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
                ProxyPassReverseCookieDomain 127.0.0.1 gwmm1.vss.kapper.net
        </LocationMatch>

        <Location />
                Require all granted
                ProxyPass http://127.0.0.1:8065/
                ProxyPassReverse http://127.0.0.1:8065/
                ProxyPassReverseCookieDomain 127.0.0.1 gwmm1.vss.kapper.net
        </Location>
# end MM config ----------------------------------------------------------------


        # Generated by Certbot
	SSLEngine on
        SSLCertificateFile	/etc/letsencrypt/live/gwmm1.vss.kapper.net/fullchain.pem
        SSLCertificateKeyFile	/etc/letsencrypt/live/gwmm1.vss.kapper.net/privkey.pem
	# SSLCertificateFile	/etc/ssl/certs/ssl-cert-snakeoil.pem
	# SSLCertificateKeyFile	/etc/ssl/private/ssl-cert-snakeoil.key

	SSLProtocol	all -SSLv3 -TLSv1 -TLSv1.1
        SSLCipherSuite	ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

        # HSTS (mod_headers is required) (15768000 seconds = 6 months)
        # (Browser never connects in next 6 months to http://, always https://)
	Header always set Strict-Transport-Security "max-age=15768000 ; includeSubDomains"

</VirtualHost>

SSLHonorCipherOrder	on
SSLCompression		off
SSLSessionTickets       off

# OCSP Stapling, only in httpd 2.3.3 and later
SSLUseStapling          on
SSLStaplingResponderTimeout            5
SSLStaplingReturnResponderErrors       off
SSLStaplingCache        shmcb:/var/run/ocsp(128000)

</IfModule>

The other thing that I highly recommend doing is running the following command from your command prompt/shell. This command will verify the integrity of your configuration files and let you know if there are any errors in them. The command is: sudo apachectl configtest. If it shows any errors, please do provide them here so I can further assist with troubleshooting the issues your experiencing.

1 Like

Hello XxLilBoPeepsxX,
to move IfModule block to this point is a good idea, but does not change the behavior.

But I’ve found now the solution :slight_smile: :slight_smile: :slight_smile:

A few Apache modules were missing. e.g. ssl :wink: – see my list above.
Now it works like a charme.

Thank’s for your support!
Suzi

1 Like