I have read everything I could find regarding this topic (on this forum, on github and even on stackoverflow) and nothing helped me so far. Most of the problems are caused by reverse proxy configuration, but in my case there is no such thing. Mattermost listen directory to port 80 and 443.
Summary
Some clients can not establish websocket connection behind corporation proxy and they don’t see new messages (fallback to AJAX doesn’t work). Users have to refresh the page in order to see new messages.
Steps to reproduce
Just use the web interface with the latest Chrome version behind that proxy. Unfortunately I also don’t have access to it.
Expected behavior
Web interface should be able to use websocket (it works for other websites) or at least fallback to AJAX polling.
Observed behavior
Logs are filled with errors like this:
[2017/12/04 15:21:03 UTC] [EROR] websocket connect err: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
[2017/12/04 15:21:03 UTC] [EROR] /api/v4/websocket:connect code=500 rid=zg3o4w4iyjdqfbbsnwozj57epc uid=bq55bpuup3nmpd658tqaij4agy ip=111.111.111.111 Failed to upgrade websocket connection [details: ]
Running Mattermost version 4.3.1 on Ubuntu with mysql (no nginx and no apache in front of it). I have set “Site URL” to https://example.com and Enable cross-origin requests from to * (temporary). The mobile apps and even the web interface work as expected outside that firewall.
It happens for months now, and initially I gave up thinking that proxy doesn’t allow it. It might have started after an upgrade, but can’t tell which one. Today I realized that this wss test website works behind the same firewall and I think this is a Mattermost configuration/server issue.
Questions:
Do you think moving Mattermost behind nginx might solve the issue?
Question, does your proxy forwards all the request headers? looks to me that some headers might not be forwarded by the proxy is there a way you can verify that?
Thanks for you reply. To be honest, I don’t know where to check that.
Behind the firewall
Since I have limited access I can only ask someone to do debugging for me - like the screenshots linked above.
Based on the experience with wss test website, the headers required for wss are forwarded. Is there any header in particular that might cause this but it’s not required for basic wss but required by Mattermost?
At the server level
I have no idea where can I see this in Mattermost, since there is nothing in between.
and you are using http://… or https:// on your client to connect to mattermost?
(because a week ago i had an interesting error regarding firewalls+upgrade of http connections (but not mattermost).
but that error was only occuring when connecting on http/port 80, as the firewall only then is able to interpret/misinterpret/discard the upgrade within the non-encrypted header…)
so it could work better when using https as a first workaround!
The server is on Digital Ocean. Plain Debian installation and only Mattermost hosted on it (following the official tutorial of installing it without nginx). No firewall was set up.
The server is already using https (with Let’s Encrypt). It redirects http to https automatically.
Nothing usefull, a lot of repeating errors. Pasted them in first post:
[2017/12/04 15:21:03 UTC] [EROR] websocket connect err: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
[2017/12/04 15:21:03 UTC] [EROR] /api/v4/websocket:connect code=500 rid=zg3o4w4iyjdqfbbsnwozj57epc uid=bq55bpuup3nmpd658tqaij4agy ip=111.111.111.111 Failed to upgrade websocket connection [details: ]
The client is using only the browser (Chrome, latest edition). Unfortunately, I can find the printscreens at the monement, and the links in the first post are not working. I will reply again when I have them.
It is weird that it’s hitting the Mattermost server and then failing to upgrade. I’m quite sure we’re not doing anything special, unless it’s an issue with gorilla/websocket, we’re following the example pretty closely