[SOLVED] Changing from HTTP to HTTPS results in ERR_CONNECTION_REFUSED for both Mattermost & GitLab

Troubleshooting
1

Changing the mattermost_external_url in my gitlab.rb to https instead of http (and then running sudo gitlab-ctl reconfigure) results in an “ERR_CONNECTION_REFUSED” error in Chrome, for both Mattermost and GitLab. No idea why this would be happening, seems odd that it’s afffecting my GitLab instance as well.

I’m using Ubuntu 15.04, and I have Let’s Encrypt set up for my GitLab instance. I haven’t generated any SSL certificates for my Mattermost instance yet.

Here’s a screenshot of the error.

Any idea why this would be happening?

EDIT: I just checked my System Console, and I think this is the error it’s producing when it’s set to HTTPS: Failed to get security update information from Mattermost.

EDIT 2: I just had another thought about this: Could it be causing issues because Mattermost rules on a separate port, and trying to set it to HTTPS is trying to use the same port as GitLab?

2

Hi @revxx14,

Have you tried setting all the following in your gitlab.rb? You’ll need another SSL certificate just for Mattermost.

mattermost_external_url 'https://mattermost.gitlab.example'
mattermost_nginx['redirect_http_to_https'] = true
mattermost_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mattermost-nginx.crt"
mattermost_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mattermost-nginx.key"
mattermost['service_use_ssl'] = true

This is documented here under the section “Running GitLab Mattermost with HTTPS”. Let me know if that helps

3

Ah, no I did not. I think I see how to get this working, I’ll report back if I make any progress.

4

I tried setting up SSL, but I was having issues because I don’t actually have DNS configured yet, I’m just using a HOSTS file locally. I tried reverting back to HTTP, but now I’m getting a 502 Bad Gateway error. It looks like Mattermost won’t start.

Running gitlab-ctl status returns:

run: gitlab-workhorse: (pid 2309) 142s; run: log: (pid 737) 24887s
run: logrotate: (pid 2316) 142s; run: log: (pid 742) 24886s
down: mattermost: 0s, normally up, want up; run: log: (pid 734) 24887s
run: nginx: (pid 2336) 140s; run: log: (pid 736) 24887s
run: postgresql: (pid 2347) 139s; run: log: (pid 738) 24887s
run: redis: (pid 2349) 139s; run: log: (pid 735) 24887s
run: sidekiq: (pid 2364) 139s; run: log: (pid 739) 24887s
run: unicorn: (pid 2369) 138s; run: log: (pid 741) 24887s

As you can see, mattermost is down. If I try to run gitlab-ctl start mattermost, I get:

ok: run: mattermost: (pid 4263) 0s

But then running gitlab-ctl status mattermost, it’s still down:

down: mattermost: 1s, normally up, want up; run: log: (pid 734) 24954s

I’m not sure what I changed that would be breaking this; I thought I reverted everything. The mattermost section of my gitlab.rb is here:

#####################
# GitLab Mattermost #
#####################

## Basic Settings
mattermost_external_url 'http://chat.mydomain.com'
mattermost['gitlab_enable'] = true
mattermost['gitlab_secret'] = "4**************************************************************0"
mattermost['gitlab_id'] = "c**************************************************************f"
mattermost['gitlab_scope'] = ""
mattermost['gitlab_auth_endpoint'] = "https://git.mydomain.com/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://git.mydomain.com/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://git.mydomain.com/api/v3/user"

## SMTP Settings
mattermost['email_feedback_name'] = "GitLab Mattermost",
mattermost['email_feedback_email'] = "git@mydomain.com",
mattermost['email_smtp_username'] = "git@mydomain.com",
mattermost['email_smtp_password'] = "************",
mattermost['email_smtp_server'] = "mydomain.com"
mattermost['email_smtp_port'] = "26"

## SSL Settings
# mattermost_nginx['redirect_http_to_https'] = true
# mattermost_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mattermost-nginx.crt"
# mattermost_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mattermost-nginx.key"
# mattermost['service_use_ssl'] = true

Could it be trying to use the same port as my HTTPS-enabled GitLab still?

5

no, mattermost doesn’t use any port, it just dies in initialization:[quote=“revxx14, post:4, topic:1123”]
down: mattermost: 0s, normally up, want up; run: log: (pid 734) 24887s
[/quote]
look into your mattermost logs and post the stack trace here.

Some of the init code is really wacky. I had the same problem after I put
mattermost['email_connection_security'] = "TLS"
in my gitlab.rb my mattermost got belly up in the same way as yours. I tracked it down to the email initialization panicing out. It works now with:
mattermost['email_smtp_port'] = 587 mattermost['email_connection_security'] = "STARTTLS"

If you don’t find the stack trace, just go back to your last stable config.rb and then change it only line by line so you can see which line is causing trouble.

1 Like
6

Thanks much! It’s definitely something with the email. I commented out my email settings, ran gitlab-ctl reconfigure and chat is now working again. I’ll double check the email settings and see if I can get that working again.

7

Okay, I’ve definitely confirmed its something with the mail settings, but I can’t figure out what. Everything looks perfectly correct to me, and using the exact same settings via system console sends mail just fine. But when I try to configure it in gitlab.rb it crashes mattermost.

Here’s what I’m using in gitlab.rb, this all looks good, right? The names for each mattermost rule are correct?

## SMTP Settings
mattermost['email_feedback_name'] = "GitLab Mattermost",
mattermost['email_feedback_email'] = "git@mydomain.com", # removed for security
mattermost['email_smtp_username'] = "git@mydomain.com",  # removed for security
mattermost['email_smtp_password'] = "************",      # removed for security
mattermost['email_smtp_server'] = "mail.mydomain.com"    # removed for security
mattermost['email_smtp_port'] = "26"
mattermost['email_connection_security'] = ""

Here’s what my cPanel says I need to use for email settings:

# Username:        git@mydomain.com
# Password:	   Use your cPanel password.
# Incoming Server: mail.mydomain.com
# IMAP Port:       143
# POP3 Port:       110
# Outgoing Server: mail.mydomain.com
# SMTP Port:       26
# Authentication is required for IMAP, POP3, and SMTP.

No idea what I’m doing wrong. Password is correct, email is correct, port is correct, domains are correct… Very strange.

8

Not all combinations of these parameters are considered legal by the go smtp library :smiling_imp:
You’ll need a stack trace to go on further. Search in the logs or try to execute the -version command with the CLI, which is incidentally the complete server, as described here

I’ll bet my yesterdays socks that there is some null pointer panic and connectToSMTPServer or newSMTPClient is involved. There is definitely at least one null pointer check missing there. :worried:

9

Thanks. I’m out of the office until Monday, but I’ll try your suggestions then. I do remember seeing something about SMTP errors in the mattermost log.

EDIT: Actually I was able to see the logs via the system console, and this is the error I’m getting:

[2016/03/23 02:30:53 CDT] [EROR] SMTP server settings do not appear to be configured properly err= details=dial tcp: lookup mail.mydomain.com on [::1]:53: read udp [::1]:33043->[::1]:53: read: connection refused

EDIT 2: Okay, so looking through the logs, I’m seeing that I may have been using the wrong domain name for the email address… I’m an idiot :slight_smile: Hopefully fixing that on Monday will get it working again.

10

I finally got another chance to look at this, and I did get the email settings to work correctly. I’m still having trouble setting up SSL, though, and I’m not sure why. I literally make one change:

# Change
mattermost_external_url 'http://chat.mywebsite.com'
# To
mattermost_external_url 'https://chat.mywebsite.com'

And Nginx then refuses to start. No idea why this would be. Any suggestions?

11

Got it working. I think the key was that I had to generate the SSL key before changing the domain, and in order to do that, I had to add the following line:

mattermost_nginx['custom_gitlab_mattermost_server_config'] = "location ^~ /.well-known {\n alias /var/www/letsencrypt/.well-known;\n}\n"
12

I’ve put together a rough tutorial to help anyone else who’s stuck:

Setting up GitLab Mattermost with Let’s Encrypt

This guide assumes you’ve already installed GitLab along with GitLab Mattermost.

The first step is to switch to the root user as you need to run Let’s Encrypt as root:

sudi -i

Clone Let’s Encrypt

cd /root  
git clone https://github.com/letsencrypt/letsencrypt

Create a mattermost.ini

Make sure you’re still in /root, and run the following:

mkdir letsencrypt-config
nano letsencrypt-config/mattermost.ini

Paste the following in to this configuration file:

# Let's Encrypt config for Mattermost

# Use the webroot authenticator
authenticator = webroot
webroot-path = /var/www/letsencrypt

# Use the standalone authenticator on port 443
# authenticator = standalone
# standalone-supported-challenges = tls-sni-01

# Generate certificates for the specified domain
domains = chat.yourdomain.com

# Register with the specified email address
email = youremail@yourdomain.com

# use a 4096 bit RSA key instead of 2048
rsa-key-size = 4096

Create the folder where authentication files will reside

mkdir -p /var/www/letsencrypt

Update gitlab.rb to alias chat.yourdomain.com/.well-known/ to /var/www/letsencrypt/

Edit your gitlab.rb:

nano /etc/gitlab/gitlab.rb

And add the following line:

mattermost_nginx['custom_gitlab_mattermost_server_config'] = "location ^~ /.well-known {\n alias /var/www/letsencrypt/.well-known;\n}\n"

Reconfigure GitLab to initialize the new setting:

gitlab-ctl reconfigure

Generate the Certificates

/root/letsencrypt/letsencrypt-auto certonly -c /root/letsencrypt-config/mattermost.ini

Update gitlab.rb to reflect live HTTPS settings

nano /etc/gitlab/gitlab.rb

Make the following changes:

mattermost_external_url 'https://chat.yourdomain.com'
mattermost['service_use_ssl'] = true
mattermost_nginx['redirect_http_to_https'] = true
mattermost_nginx['ssl_certificate'] = "/etc/letsencrypt/live/chat.yourdomain.com/fullchain.pem"
mattermost_nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/chat.yourdomain.com/privkey.pem"

Then run:

gitlab-ctl reconfigure

Set up a cron job to auto update the certificates

nano /etc/cron.monthly/renew-ssl-certificates

Add the following:

#!/bin/bash

/root/.local/share/letsencrypt/bin/letsencrypt certonly -c /root/letsencrypt-config/mattermost.ini --renew-by-default

gitlab-ctl restart

And you’re done!

1 Like
13

I received the same msg ERR_CONNECTION_REFUSED and was blocked from doing any searches.
(But I could listen to and watch Youtube videos!)
I don’t know what your computer said, but I am running Google Chrome on windows 7 and above the message it said to:
1) Click on the Chrome Menu (the 2 bars in upper right hand corner of the Google Home page.
2) go to Settings > then "show advanced setting"
3) click Change proxy settings

  1. click LAN Settings
  2. UNclick "use a proxy server for your LAN"
    Was still getting ERR_CONNECTION_REFUSED
    Then closed all windows and programs
    REBOOTED PC
    Now, everything is fine! (REBOOTING (RESET) IS A NUISANCE, BUT IT HELPS)
    I hope this info helps you.
    http://www.deskdecode.com/err_connection_refused/
14

@revxx14 and @john, thanks for posting the tutorial and your solution! I’ll mark this resolved for now.

15

I found a webpage from which i get rid out of this type of error issue, now you can also check this once for

getting rid of it completely from here: http://www.techinpost.com/fix-error-err_connection_refused-code-problem

16

Thanks for posting this information for others @Techinpost!

17

You can try with clear cookies method.

  • For example copy chrome://settings/clearbrowserData
  • Now hit enter, You will see pop up of clear browsing data.
  • After then just clear browser data and restart your browser.

You can try any form this link and also this link if need more ways.
Thanks