Mattermost, Inc.

WebSocket connection pending forever

Summary

Connection via WebSocket stays in pending state forever

Steps to reproduce

Using Mattermost 5.16.2 from GitLab Omnibus.

  • Log into Mattermost
  • Error message is displayed: “Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.”

Error message only shows for some users using Firefox 71.0-1. Other Chrome and Firefox users don’t see this message, but the WebSocket fails to connect anyway. We checked that using the browser’s developer consoles. The WebSocket connect stays in “pending” state forever. The last status code shown is “101 Switching Protocols”.

Expected behavior

WebSocket connection should be fine.

Observed behavior

On the server side, we see the following error in the logs:

{"level":"error","ts":1576066796.4229283,"caller":"api4/websocket.go:27","msg":"websocket connect err.","error":"websocket: not a websocket handshake: `Sec-WebSocket-Key' header is missing or blank"}
{"level":"error","ts":1576066796.450573,"caller":"mlog/log.go:174","msg":"Failed to upgrade websocket connection","path":"/api/v4/websocket","request_id":"bab5nd5hxfym8mcjfwau1oqwje","ip_addr":"172.21.36.13","user_id":"xn8optx113ywpm4s4f35s9tg1c","method":"GET","err_where":"connect","http_code":500,"err_details":""}

I already checked the forum and the troubleshooting guide with no success. I also tested in the same network where the server is located to make sure no proxies or firewalls are in the way, but no success. This is a corporate network.

We are using GitLab Omnibus 12.5.3.

indent preformatted text by 4 spaces

Here is the auto-generated gitlab-mattermost-http.conf (replaced the real domain with “mattermost.domain”):

# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

## GitLab Mattermost

upstream gitlab_mattermost {
  server 127.0.0.1:8065;
}

server {
  listen *:80;

  server_name mattermost.domain;
  server_tokens off;


  location / {
    return 301 https://mattermost.domain:443$request_uri;
  }

  access_log  /var/log/gitlab/nginx/gitlab_mattermost_access.log gitlab_mattermost_access;
  error_log   /var/log/gitlab/nginx/gitlab_mattermost_error.log;
}

server {
  listen *:443 ssl http2;
  server_name mattermost.domain;
  server_tokens off;     # don't show the version number, a security best practice
  client_max_body_size 0;

  ssl_certificate /etc/gitlab/ssl/mattermost.domain.crt;
  ssl_certificate_key /etc/gitlab/ssl/mattermost.domain.key;

  ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
  ssl_prefer_server_ciphers on;
  ssl_protocols  TLSv1.2 TLSv1.3;
  ssl_session_cache  builtin:1000  shared:SSL:10m;
  ssl_session_timeout  5m;

  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html

  access_log  /var/log/gitlab/nginx/gitlab_mattermost_access.log gitlab_mattermost_access;
  error_log   /var/log/gitlab/nginx/gitlab_mattermost_error.log;

  location / {
    ## If you use HTTPS make sure you disable gzip compression
    ## to be safe against BREACH attack.
    gzip off;

    proxy_read_timeout      3600;
    proxy_connect_timeout   300;
    proxy_redirect          off;

    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Frame-Options SAMEORIGIN;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header X-Forwarded-Ssl on;

    proxy_pass http://gitlab_mattermost;
  } 
}

Turned out it indeed was a client (proxy) issue. A user was using a local squid proxy to avoid authenticating every process to the corporate proxy and whitelisting Mattermost did not work. Configuring Firefox to use the corporate proxy.pac, which whitelists Mattermost together with other internal services, works fine.