Mattermost, Inc.

Chat messages are lost via WSS

Hello all.

I have a stand with two servers running. They’ve got two Mattermosts running on them. Both servers have the same configuration for both Nginx and Mattermost and other services.

When I send a chat message in a web application to a mobile application or vice versa, I only receive half of the messages - the other half just does not come via WSS but comes via HTTPS - I see the missing messages after refreshing the page. And I didn’t see any errors in the logs about it.

With all this, the problem is eliminated when I disable MM on the second server.

What are the reasons for WSS malfunctioning and how can I fix them? (Disabling MM on the second server doesn’t suit me).

Thanks.

@xayctoff What Mattermost server version are you on? What OS and version are you using?

CentOS 7.6.1810
Mattermost 5.16.0

Would you be open to upgrading your server to a more recent version? Our supported versions include v5.19 and later versions.

I’m not sure that in my case it’s possible to do that.
I’d appreciate it if the problem could be solved for the current version.

@xayctoff, can you confirm you have an E20 enterprise licensed installed and High Availability enabled? Clustering more than one server together requires the servers to coordinate and is not something we support in the Team Edition.

I can’t confirm it because:

{“level”:“error”,“ts”:1589337627.4322038,“caller”:“cluster/cluster.go:228”,“msg”:“This server is not licensed to run in High Availability mode.”}

Hi there, @xayctoff

Based on the error that you are getting on the mattermost.log, the license that you are using is not meant for multi server deployment on Mattermost, which is only meant for E20 license. May I know how did you setup the instances to run on HA and which documentation you referred to?

At the same time, would you be open to consider upgrading your license to E20 by reaching out to the team?

Hi!

HA was only used as a possible solution to this problem - High Availability is not used on the project in MM instances. We’re not looking at the possibility of updating.

There are WSS location parameters in nginx.conf:

location ~ /chat/(?<chatapi>.+)/websocket$ {
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    client_max_body_size 50M;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Frame-Options SAMEORIGIN;
    proxy_set_header Origin '<site name>';
    proxy_buffers 256 16k;
    proxy_buffer_size 16k;
    client_body_timeout 60;
    send_timeout 300;
    lingering_timeout 5;
    proxy_connect_timeout 90;
    proxy_send_timeout 300;
    proxy_read_timeout 90s;
    proxy_pass http://mattermosts/$chatapi/websocket;
}

location /api/v4/websocket {
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    client_max_body_size 50M;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Frame-Options SAMEORIGIN;
    proxy_buffers 256 16k;
    proxy_buffer_size 16k;
    client_body_timeout 60;
    send_timeout 300;
    lingering_timeout 5;
    proxy_connect_timeout 90;
    proxy_send_timeout 300;
    proxy_read_timeout 90s;
    proxy_pass http://mattermosts;
}

MM config.json:

“ServiceSettings”: {
“SiteURL”: “”,
“WebsocketURL”: “”,
“LicenseFileLocation”: “”,
“ListenAddress”: “:8065”,
“ConnectionSecurity”: “”,
“TLSCertFile”: “”,
“TLSKeyFile”: “”,
“TLSMinVer”: “1.2”,
“TLSStrictTransport”: false,
“TLSStrictTransportMaxAge”: 63072000,
“TLSOverwriteCiphers”: ,
“UseLetsEncrypt”: false,
“LetsEncryptCertificateCacheFile”: “./config/letsencrypt.cache”,
“Forward80To443”: false,
“TrustedProxyIPHeader”: [
“X-Forwarded-For”,
“X-Real-IP”
],
“ReadTimeout”: 300,
“WriteTimeout”: 300,
“MaximumLoginAttempts”: 10,
“GoroutineHealthThreshold”: -1,
“GoogleDeveloperKey”: “”,
“EnableOAuthServiceProvider”: false,
“EnableIncomingWebhooks”: true,
“EnableOutgoingWebhooks”: true,
“EnableCommands”: true,
“EnableOnlyAdminIntegrations”: true,
“EnablePostUsernameOverride”: false,
“EnablePostIconOverride”: false,
“EnableLinkPreviews”: false,
“EnableTesting”: false,
“EnableDeveloper”: false,
“EnableSecurityFixAlert”: true,
“EnableInsecureOutgoingConnections”: false,
“AllowedUntrustedInternalConnections”: “”,
“EnableMultifactorAuthentication”: false,
“EnforceMultifactorAuthentication”: false,
“EnableUserAccessTokens”: true,
“AllowCorsFrom”: “”,
“CorsExposedHeaders”: “GET, POST, OPTIONS, PUT, HEAD, DELETE, PATCH”,
“CorsAllowCredentials”: true,
“CorsDebug”: false,
“AllowCookiesForSubdomains”: false,
“SessionLengthWebInDays”: 180,
“SessionLengthMobileInDays”: 180,
“SessionLengthSSOInDays”: 30,
“SessionCacheInMinutes”: 10,
“SessionIdleTimeoutInMinutes”: 43200,
“WebsocketSecurePort”: 443,
“WebsocketPort”: 80,
“WebserverMode”: “gzip”,
“EnableCustomEmoji”: false,
“EnableEmojiPicker”: true,
“EnableGifPicker”: false,
“GfycatApiKey”: “”,
“GfycatApiSecret”: “”,
“RestrictCustomEmojiCreation”: “all”,
“RestrictPostDelete”: “all”,
“AllowEditPost”: “always”,
“PostEditTimeLimit”: -1,
“TimeBetweenUserTypingUpdatesMilliseconds”: 5000,
“EnablePostSearch”: true,
“MinimumHashtagLength”: 3,
“EnableUserTypingMessages”: true,
“EnableChannelViewedMessages”: true,
“EnableUserStatuses”: true,
“ExperimentalEnableAuthenticationTransfer”: true,
“ClusterLogTimeoutMilliseconds”: 2000,
“CloseUnusedDirectMessages”: false,
“EnablePreviewFeatures”: true,
“EnableTutorial”: true,
“ExperimentalEnableDefaultChannelLeaveJoinMessages”: true,
“ExperimentalGroupUnreadChannels”: “disabled”,
“ExperimentalChannelOrganization”: false,
“ImageProxyType”: “”,
“ImageProxyURL”: “”,
“ImageProxyOptions”: “”,
“EnableAPITeamDeletion”: false,
“ExperimentalEnableHardenedMode”: false,
“DisableLegacyMFA”: true,
“ExperimentalStrictCSRFEnforcement”: false,
“EnableEmailInvitations”: false,
“ExperimentalLdapGroupSync”: false,
“DisableBotsWhenOwnerIsDeactivated”: true,
“EnableBotAccountCreation”: false,
“EnableSVGs”: false
},

This problem with WSS is pretty weird for me because it’s never happened before - WSS didn’t lose messages.

@xayctoff, the behaviour you are observing is effectively by design. We don’t support horizontal scaling on team edition, since it requires more than just a shared database. Each peer in a Mattermost server cluster must coordinate their shared caches and proxy websocket messages between each other to ensure all connected users (spread across multiple Mattermost servers) are notified of changes.

As @ahmaddanial noted, if you’re interested in an E20 license (even if on trial), I’d encourage you to reach out.