We're Hiring!

Mattermost, Inc.

Nginx and plugin issue

Plugin issue due to a bad nginx configuration

Summary
{"level":"warn","ts":1616599242.1224701,"caller":"api4/websocket.go:38","msg":"The HTTP version field was detected as 1.0 during WebSocket handshake. This is most probably due to an incorrect proxy configuration. Please upgrade your proxy config to set the header version to a minimum of 1.1."}

This happens with the grumble plugin, when I’m trying to connect to the mumble server. It is certainly a problem with my nginx config:

 upstream backend {
   server 10.10.10.2:8065;
   keepalive 32;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
  listen 80 default_server;
  server_name   mattermost.lecrtm.tk;
  return 301 https://$server_name$request_uri;
}

server {
   listen 443 ssl http2;
   server_name    mattermost.lecrtm.tk;

   http2_push_preload on; # Enable HTTP/2 Server Push

   ssl on;
   ssl_certificate /etc/letsencrypt/live/mattermost.lecrtm.tk/fullchain.pem;
   ssl_certificate_key /etc/letsencrypt/live/mattermost.lecrtm.tk/privkey.pem;
   ssl_session_timeout 1d;

   # Enable TLS versions (TLSv1.3 is required upcoming HTTP/3 QUIC).
   ssl_protocols TLSv1.2 TLSv1.3;

   # Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
   # prevent replay attacks.
   #
   # @see: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
   ssl_early_data on;

   ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE>   
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:500m;
   # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
   add_header Strict-Transport-Security max-age=15768000;
   # OCSP Stapling ---
   # fetch OCSP records from URL in ssl_certificate and cache them

   ssl_stapling on;
   ssl_stapling_verify on;

   add_header X-Early-Data $tls1_3_early_data;

   location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 500M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_http_version 1.1;
       proxy_pass http://localhost:8065;
   }


   location / {
       client_max_body_size 500M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_http_version 1.1;
       proxy_pass http://localhost:8065;
   }
}

# This block is useful for debugging TLS v1.3. Please feel free to remove this
# and use the `$ssl_early_data` variable exposed by NGINX directly should you
# wish to do so.
map $ssl_early_data $tls1_3_early_data {
  "~." $ssl_early_data;
  default "";
} 

Steps to reproduce
Latest version of MM
Ubuntu 20.04

You just have to install the grumble plugin

Expected behavior
I can’t access to the channels

Observed behavior
There is a similar topic, but the solution was not working for me.

Small edit : this was before updating my configuration and mattermost.

Here is what I have now:

{"level":"info","ts":1618646597.9716687,"caller":"mlog/sugar.go:21","msg":"Stopping grumble server","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"info","ts":1618646597.9718294,"caller":"mlog/sugar.go:21","msg":"Saving server state","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"info","ts":1618646597.985461,"caller":"mlog/sugar.go:21","msg":"Saved server state, users=1 channels=3","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"error","ts":1618646597.9868252,"caller":"mlog/log.go:232","msg":"RPC call OnConfigurationChange to plugin failed.","plugin_id":"com.mattermost.mattermost-grumble-plugin","error":"unexpected EOF"}
{"level":"info","ts":1618646598.0145843,"caller":"go-plugin@v1.4.0/stream.go:15","msg":"2021/04/17 08:03:18 LoadPluginConfiguration API failed to unmarshal: json: cannot unmarshal string into Go struct field CustomAttribute.CustomAttributes.GroupIDs of type []string","plugin_id":"com.mattermost.custom-attributes","source":"plugin_stderr"}
{"level":"warn","ts":1618646627.7493033,"caller":"plugin/health_check.go:59","msg":"Health check failed for plugin","id":"com.mattermost.mattermost-grumble-plugin","error":"plugin RPC connection is not responding"}
{"level":"error","ts":1618646627.7493713,"caller":"mlog/log.go:232","msg":"RPC call OnDeactivate to plugin failed.","plugin_id":"com.mattermost.mattermost-grumble-plugin","error":"connection is shut down"}
{"level":"warn","ts":1618646627.7494023,"caller":"plugin/hclog_adapter.go:70","msg":"error closing client during Kill","plugin_id":"com.mattermost.mattermost-grumble-plugin","wrapped_extras":"errconnection is shut down"}
{"level":"warn","ts":1618646627.7494106,"caller":"plugin/hclog_adapter.go:72","msg":"plugin failed to exit gracefully","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"info","ts":1618646627.9288735,"caller":"mlog/sugar.go:21","msg":"Activating plugin","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"info","ts":1618646627.9307673,"caller":"mlog/sugar.go:21","msg":"Recovered server state, users=1 channels=3","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"info","ts":1618646627.93096,"caller":"mlog/sugar.go:21","msg":"Unfreezing server state","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"info","ts":1618646627.931159,"caller":"mlog/sugar.go:21","msg":"Applying configuration","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"info","ts":1618646627.931946,"caller":"mlog/sugar.go:21","msg":"Starting grumble server","plugin_id":"com.mattermost.mattermost-grumble-plugin"}
{"level":"info","ts":1618646627.9326193,"caller":"go-plugin@v1.4.0/stream.go:15","msg":"[1] 2021/04/17 08:03:47.932407 Started: listening on [::]:64738 and 0.0.0.0:8090","plugin_id":"com.mattermost.mattermost-grumble-plugin","source":"plugin_stdout"}
1 Like

@BCPG Thanks for sharing the solution of this. :star_struck: :heart_eyes: :heart_eyes:

Not actually the solution, but the problem :sweat_smile:

Check changelog for version 5.33: Mattermost Changelog — Mattermost 5.34 documentation

WebSocket handshakes done with an HTTP version lower than 1.1 will result in a warning, and the server will transparently upgrade the version to 1.1 to comply with the WebSocket RFC. This is done to work around incorrect Nginx (and other proxy) configs that do not set the proxy_http_version directive to 1.1. This facility will be removed in a future Mattermost version. It is strongly recommended to fix the proxy configuration to correctly use the WebSocket protocol.

This might help in nginx:

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $host;

Sorry I didn’t sent you my nginx configuration !

Here it is :


 upstream backend {
   server 10.10.10.2:8065;
   keepalive 32;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
  listen 80 default_server;
  server_name   mattermost.lecrtm.tk;
  return 301 https://$server_name$request_uri;
}

server {
   listen 443 ssl http2;
   server_name    mattermost.lecrtm.tk;

   http2_push_preload on; # Enable HTTP/2 Server Push

   ssl on;
   ssl_certificate /etc/letsencrypt/live/mattermost.lecrtm.tk/fullchain.pem;
   ssl_certificate_key /etc/letsencrypt/live/mattermost.lecrtm.tk/privkey.pem;
   ssl_session_timeout 1d;

   # Enable TLS versions (TLSv1.3 is required upcoming HTTP/3 QUIC).
   ssl_protocols TLSv1.2 TLSv1.3;

   # Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
   # prevent replay attacks.
   #
   # @see: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
   ssl_early_data on;

   ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE>
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:500m;
   # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
   add_header Strict-Transport-Security max-age=15768000;
   # OCSP Stapling ---
   # fetch OCSP records from URL in ssl_certificate and cache them
   ssl_stapling on;
   ssl_stapling_verify on;

   add_header X-Early-Data $tls1_3_early_data;

   location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 500M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_http_version 1.1;
       proxy_pass http://localhost:8065;
   }

   location / {
       client_max_body_size 500M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_http_version 1.1;
       proxy_pass http://localhost:8065;
   }
}

# This block is useful for debugging TLS v1.3. Please feel free to remove this
# and use the `$ssl_early_data` variable exposed by NGINX directly should you
# wish to do so.
map $ssl_early_data $tls1_3_early_data {
  "~." $ssl_early_data;
  default "";
}

Hi, @BCPG

While I understand that we have been taking a peek at your NGINX config, would you mind sharing the output of the following command?

cat /opt/mattermost/config/config.json | grep "TLSMinVer"

For example, this is how it looks like on my end:

ahmaddanial@mattermost:~$ cat /opt/mattermost/config/config.json | grep "TLSMinVer"
        "TLSMinVer": "1.2",

I also noticed that you have your proxy HTTP version set to 1.1:

proxy_http_version 1.1;

Would you mind checking on @vinci 's suggestion and have it set to 1.2 instead?

I got the same output as yours for the command.

But when I try to change the proxy_http_version to 1.2, it says :

nginx: [emerg] invalid value "1.2" in /etc/nginx/sites-enabled/mattermost:63

Just to be clear, I never suggested setting the http version to 1.2 (which as far as I know does not exist), I suggested 1.1 before reading the nginx config.
At this point I’m not sure what the issue is.

Sorry, misunderstand here :sweat_smile: