Mattermost Peer-to-Peer Forum

Upgraded from 3.7 to 3.8 broke websockets! help!


#1

Hello everything was working fine… I upgraded like I always do… log in… got a msg to add a URL in the console/configuration field… I added my url… and after that I get:
websocket connect err: websocket: origin not allowed
/api/v3/users/websocket:connect code=500 rid=4che7w3383r88qxswssi53nkwh uid= ip=107.182.x.x Failed to upgrade websocket connection [details: ]

I changed the URL to all my interfaces and restarted with same issue…

MY setup?

nginx port 443 port 80 redirect to 443
config for nginx:

server {
      listen 443 ssl;
      server_name chat.mydomain.info o5imyonion.onion chat.mydomain;

      ssl on;
      ssl_certificate  /etc/letsencrypt/live/chat.mydomain.info/fullchain.pem;
      ssl_certificate_key /etc/letsencrypt/live/chat.mydomain.info/privkey.pem;
      ssl_dhparam /home/mattermost/cert/dhparam.pem;
      ssl_session_timeout 5m;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
      ssl_prefer_server_ciphers on;
      ssl_session_cache shared:SSL:10m;


      location / {
        gzip off;
        proxy_set_header X-Forwarded-Ssl on;
        client_max_body_size 50M;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Frame-Options SAMEORIGIN;
        proxy_pass http://127.0.0.1:8065;
     }
   }

on mattermost I have added this two changes…
on the URL “enforced” field now…(I dont know why you guys force this… when it SHOULD be an option to admins… not you…
I have http://127.0.0.1 (had my external ip, my external domains all of them, this is just the last one I tried)
for port I have:
127.0.0.1:8065

why? nginx is forwarding the connection to localhost:8065 as you can see in the nginx config above…

Please help all my users are complaining … :confused:


#2

I notice that your NGINX config doesn’t have a location for /api/v3/users/websocket… maybe that’s the issue?

The install docs say to have the following config for /api/v3/users/websocket:

location /api/v3/users/websocket {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_pass http://backend;
   }

#3

Interesting, I got my config from one of the first releases so this maybe have changed so I must to miss it, in any case thanks for noticing, I will test tonight and report back in case someone have same issue.
BTW do I need to add that or substitute what I have with “/”?

Thanks


#4

You add it. The instructions are here: https://docs.mattermost.com/install/install-ubuntu-1604.html#configuring-nginx-as-a-proxy-for-mattermost-server. Although they are in the Ubuntu instructions, the config is the same for every OS. Also, the TLS config might be of interest: https://docs.mattermost.com/install/install-ubuntu-1604.html#configuring-nginx-with-ssl-and-http-2

I’m very interested to see what the solution turns out to be so that I can update the documentation.


#5

I have same problem,

there is my mattermost.log

[2017/04/21 14:46:36 CST] [EROR] websocket connect err: websocket: origin not allowed
[2017/04/21 14:46:36 CST] [EROR] /api/v3/users/websocket:connect code=500 rid=gj88f7qtu3yd3m5kdebw11ot9w uid=rhynu9jrebr1zcgpo4t1gwag8w ip=172.16.2.254 Failed to upgrade websocket connection [details: ]

#6

https://github.com/mattermost/platform/issues/6183

I tried building it locally but not sure if I built it wrong or what, but it didn’t seem to fix my issue.


#7

Just tested 3.8.2. Still seeing the same issue.

I have a nginx reverse proxy. Was working on 3.7.3

[2017/04/21 15:08:58 PDT] [EROR] websocket connect err: websocket: origin not allowed
[2017/04/21 15:09:44 PDT] [EROR] /api/v3/users/websocket:connect code=500

My nginx config

upstream mattermost {
  server 127.0.0.1:8065;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
  listen *:80;
  server_name chat.mycomp.com;
  server_tokens off;
  return 301 https://chat.mycomp.com:443$request_uri;
  access_log  /var/log/gitlab/nginx/mattermost_access.log;
  error_log   /var/log/gitlab/nginx/mattermost_error.log;
}

server {
  listen *:443 ssl http2;
  server_name chat.mycomp.com;
  server_tokens off;     # don't show the version number, a security best practice
  client_max_body_size 250m;

  ssl on;
  ssl_certificate /etc/gitlab/ssl/mycomp-chained.crt;
  ssl_certificate_key /etc/gitlab/ssl/mycomp.key;
  ssl_session_timeout 5m;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;


  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html

  access_log  /var/log/gitlab/nginx/mattermost_access.log;
  error_log   /var/log/gitlab/nginx/mattermost_error.log;

  location /api/v3/users/websocket {
 proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_pass http://mattermost;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_pass http://mattermost;
   }

}

#8

@ChrisFernandez and @congalong: Mattermost 3.8.2 is available for download here: http://about.mattermost.com/download/

The dot release should fix the WebSocket connection issues you’ve described.

Please let us know if you hit any further issues, and thank you for the report!

PS: @congalong You may have grabbed an invalid build for 3.8.2: the first release build automation didn’t finish successfully, so we re-launched it afterwards. Can you try again, taking the build from http://about.mattermost.com/download/?


#9

Thanks for your help.

I update our Mattermost to 3.8.2 and use browser client
but got same error message.

[2017/04/22 15:20:32 CST] [EROR] websocket connect err: websocket: origin not allowed
[2017/04/22 15:20:32 CST] [EROR] /api/v3/users/websocket:connect code=500 rid=88ybn74wqib9mj5c6joixicjze uid=k84pboggc7f5fd944ry35cugky ip=36.225.139.27 Failed to upgrade websocket connection [details: ]

#10

Same here (using community edition 3.8.2)


#11

BTW does the Config for nginx have to be changed to also accommodate the new /api/v4/… location?


#12

@mouson @yoshtec

Do you have AllowCorsFrom setting set for cross-origin requests: https://docs.mattermost.com/administration/config-settings.html#enable-cross-origin-requests-from

If you’re using multiple domains, you should add all the alternative domains to AllowCorsFrom setting after which the websockets would work. Otherwise you’ll get websocket connect err: websocket: origin not allowed errors.

Let us know if the issue keeps persisting


#13

@jasonblais

Thank you for your help.

When i type "AllowCorsFrom": "*", websockets is work,
but i type "AllowCorsFrom": "*.mydomain", "AllowCorsFrom": "same site url domain" or "AllowCorsFrom": "" websockets does not work.


#14

@mouson what’s your Site URL set to?


#15

Our config.json like:

"SiteURL": "https://our.domain.com"
"AllowCorsFrom": "*",

that is work


"SiteURL": "https://our.domain.com"
"AllowCorsFrom": "*.domain.com"

or

"SiteURL": "https://our.domain.com"
"AllowCorsFrom": "our.domain.com"

that is not work


#16

Thanks @mouson, can you try setting “AllowCorsFrom”: “https://our.domain.com” ?


#17

Thanks @jasonblais,
I am try to set "AllowCorsFrom": "https://our.domain.com"
websocket not work


#18

Hmm, okay…that should have worked

  1. Which OS did you use to install Mattermost? Is it running on Ubuntu?

  2. Which DB are you running Mattermost on (MySQL, Postgres, or something else)?

  3. Are these the error messages you’re seeing in the logs? (you posted them earlier, just checking to see if they have changed)

[2017/04/22 15:20:32 CST] [EROR] websocket connect err: websocket: origin not allowed
[2017/04/22 15:20:32 CST] [EROR] /api/v3/users/websocket:connect code=500 rid=88ybn74wqib9mj5c6joixicjze uid=k84pboggc7f5fd944ry35cugky ip=36.225.139.27 Failed to upgrade websocket connection [details: ]

  1. Can you confirm the Mattermost server version number from Mattermost Main Menu > About Mattermost dialog?

Just want to confirm the upgrade to 3.8.2 was successful.


#19

Thank for @jasonblais help.

We have two Server, 1. Mattermost service, 2. Nginx Reverse Proxy Service

  1. Mattermost Service Service:

ENV:

  • OS: CentOS release 6.9 - 64bit
  • DB: postgresql-9.5
  • Local IP: 172.16.7.22
  • Mattermost
[mattermost]# ./bin/platform version
Version: 3.8.0
Build Number: 3.8.2
Build Date: Fri Apr 21 21:30:35 UTC 2017
Build Hash: dd1db760e378d2df7a52976e812f77f54f85c2c0
Build Enterprise Ready: true
DB Version: 3.8.0
  • error log on mattermost is same the previous

  1. Nginx Reverse Proxy
  • OS: CentOS release 6.8 (Final) / 64bit
  • WEB SERVER: nginx/1.11.12
  • local ip: 172.16.7.20
  • nginx config:
upstream mattermost {
   server 172.16.7.22:8065;
}
server {
    listen 80;
    server_name    ourdomain.com;
    return 301 https://$server_name$request_uri;
}
server {
    listen 443 ssl http2;
    server_name ourdomain.com;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/ourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ourdomain.com/privkey.pem;
   ssl_session_timeout 5m;
   ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
   ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:10m;

   access_log  /var/log/nginx/ourdomain_access.log;
   error_log   /var/log/nginx/ourdomain_error.log;

   location /api/v3/users/websocket {
       access_log  /var/log/nginx/ourdomain_websocket_access.log;
       error_log  /var/log/nginx/ourdomain_websocket_error.log;

       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Origin $remote_addr;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_pass http://mattermost;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_pass http://mattermost;
   }
}


#20

@mouson

Thanks!

  1. Just to confirm - are you using only one domain for your Mattermost instance?
  2. Can you try "AllowCorsFrom": "", which disables cross-origin requests?
  3. Can you also try "AllowCorsFrom": "domain.com", which should work (I touched base with one of the core developers)