Error "Failed to upgrade websocket" behind some firewalls without using nginx or Apache


#1

I have read everything I could find regarding this topic (on this forum, on github and even on stackoverflow) and nothing helped me so far. Most of the problems are caused by reverse proxy configuration, but in my case there is no such thing. Mattermost listen directory to port 80 and 443.

Summary

Some clients can not establish websocket connection behind corporation proxy and they don’t see new messages (fallback to AJAX doesn’t work). Users have to refresh the page in order to see new messages.

Steps to reproduce

Just use the web interface with the latest Chrome version behind that proxy. Unfortunately I also don’t have access to it.

Expected behavior

Web interface should be able to use websocket (it works for other websites) or at least fallback to AJAX polling.

Observed behavior

Logs are filled with errors like this:

[2017/12/04 15:21:03 UTC] [EROR] websocket connect err: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
[2017/12/04 15:21:03 UTC] [EROR] /api/v4/websocket:connect code=500 rid=zg3o4w4iyjdqfbbsnwozj57epc uid=bq55bpuup3nmpd658tqaij4agy ip=111.111.111.111 Failed to upgrade websocket connection [details: ]

screenshot: chrome console errors
screenshot: chrome network errors

Setup

Running Mattermost version 4.3.1 on Ubuntu with mysql (no nginx and no apache in front of it). I have set “Site URL” to https://example.com and Enable cross-origin requests from to * (temporary). The mobile apps and even the web interface work as expected outside that firewall.

It happens for months now, and initially I gave up thinking that proxy doesn’t allow it. It might have started after an upgrade, but can’t tell which one. Today I realized that this wss test website works behind the same firewall and I think this is a Mattermost configuration/server issue.

Questions:

  • Do you think moving Mattermost behind nginx might solve the issue?
  • How can I debug this further?

#2

Question, does your proxy forwards all the request headers? looks to me that some headers might not be forwarded by the proxy is there a way you can verify that?


#3

Hi @elias,

Thanks for you reply. To be honest, I don’t know where to check that.

Behind the firewall

Since I have limited access I can only ask someone to do debugging for me - like the screenshots linked above.

Based on the experience with wss test website, the headers required for wss are forwarded. Is there any header in particular that might cause this but it’s not required for basic wss but required by Mattermost?

At the server level

I have no idea where can I see this in Mattermost, since there is nothing in between.

Should I change the log level in Mattermost?


#4

Before digging can you confirm that if the server is in a DMZ or without that firewall in front all works well?

I just want to identify where the problem is originated


#5

and you are using http://… or https:// on your client to connect to mattermost?

(because a week ago i had an interesting error regarding firewalls+upgrade of http connections (but not mattermost).
but that error was only occuring when connecting on http/port 80, as the firewall only then is able to interpret/misinterpret/discard the upgrade within the non-encrypted header…)

so it could work better when using https as a first workaround!


#6

The server is on Digital Ocean. Plain Debian installation and only Mattermost hosted on it (following the official tutorial of installing it without nginx). No firewall was set up.

The server is already using https (with Let’s Encrypt). It redirects http to https automatically.


#7

@Luxian does this happen with every client you connect or just some of them?

do you have the logs from the server? logs from the clients?


#8

Just some of them (see first post)

Nothing usefull, a lot of repeating errors. Pasted them in first post:

[2017/12/04 15:21:03 UTC] [EROR] websocket connect err: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
[2017/12/04 15:21:03 UTC] [EROR] /api/v4/websocket:connect code=500 rid=zg3o4w4iyjdqfbbsnwozj57epc uid=bq55bpuup3nmpd658tqaij4agy ip=111.111.111.111 Failed to upgrade websocket connection [details: ]

The client is using only the browser (Chrome, latest edition). Unfortunately, I can find the printscreens at the monement, and the links in the first post are not working. I will reply again when I have them.


#9

Upgraded Mattermost to latest version 4.6.0 and the error still persists.

Here’s the screenshot with the errors: Image Pasted at 2018-1-17 09-00|573x499 (and a Google Photos link in case the previous stops working).

Any ideas?


#10

Hi @Luxian,

Some feedback from one of our engineers,

It is weird that it’s hitting the Mattermost server and then failing to upgrade. I’m quite sure we’re not doing anything special, unless it’s an issue with gorilla/websocket, we’re following the example pretty closely