Mattermost is unstable after update to 4.7.3 (smtp TLS connection isn’t working, but this might be a coincidence) - keeps on restarting every 15-30 minutes.
Steps to reproduce
Debian 9, Nginx, Mattermost Enterprise Version 4.7.2 (upgraded to 4.7.3)
follow standard procedures to update as indicated in the guide
Expected behavior
Same steps worked flawless to upgrade to 4.7.2
Observed behavior
systemctl status mattermost shows that the server is running, but is stuck on ‘pinging sql master database’ (edit). Nginx shows 502 Bad Gateway, then after aun unusally long time mattermost starts running, but keeps on shutting down 1-2 times an hour.
We’ve upgraded to 4.8.0 today, which also attempted (and succeeded) in upgrading the database server. Seems to be stable again now. Will report back if problems persist.
the server runs stable most of the time, but weirdly some random actions cause a restart. Most recent example is uploading a certain image (8.4 MB), which causes a restart with the logs not showing any details on why it restarted. It just shows this:
[2018/03/28 10:15:39 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’
[2018/03/28 10:15:39 +07] [INFO] Server is initializing…
[2018/03/28 10:15:39 +07] [INFO] Pinging SQL master database
[2018/03/28 10:15:40 +07] [INFO] Current version is 4.8.0 (4.8.0/Thu Mar 15 23:11:12 UTC 2018/d1ed79a66c7930020b7fda046a11050f5890c84e/6f563c1fa497d123134fd499eb37674eb2511325)
[2018/03/28 10:15:40 +07] [INFO] Enterprise Enabled: true
[2018/03/28 10:15:40 +07] [INFO] Current working directory is /opt/mattermost
[2018/03/28 10:15:40 +07] [INFO] Loaded config file from /opt/mattermost/config/config.json
[2018/03/28 10:15:40 +07] [INFO] Connection to S3 or minio is good. Bucket exists.
[2018/03/28 10:15:40 +07] [INFO] License key from https://mattermost.com required to unlock enterprise features.
[2018/03/28 10:15:40 +07] [INFO] Starting up plugins
[2018/03/28 10:15:40 +07] [WARN] unable to create user namespace: fork/exec /proc/self/exe: permission denied
[2018/03/28 10:15:40 +07] [WARN] plugin sandboxing is not supported. plugins will run with the same access level as the server
[2018/03/28 10:15:41 +07] [INFO] Starting Server…
[2018/03/28 10:15:41 +07] [INFO] Server is listening on XXX.XXX.XXX.XXX:8065
[2018/03/28 10:15:41 +07] [INFO] API version 3 is scheduled for deprecation. Please see https://api.mattermost.com for details.
[2018/03/28 10:15:41 +07] [INFO] Starting 2 websocket hubs
[2018/03/28 10:15:41 +07] [INFO] Starting workers
[2018/03/28 10:15:41 +07] [INFO] Starting schedulers.
Adding new users with the platform command also seems to cause a restart (although the users are successfully created, while the file upload is stuck indefinitely)
Another thing I noticed, not sure if this is supposed to happen:
Whenever I create user accounts through the platform commands it re-initiates the server
For example these two commands:
sudo ./platform user create --email xxxxx@xxxx.com --username xxxxx --password xxxx
sudo ./platform user verify xxxxx@xxxx.com
Show this in the logs for each user created this way:
[2018/04/09 09:04:00 +07] [INFO] Server is initializing…
[2018/04/09 09:04:00 +07] [INFO] Pinging SQL master database
[2018/04/09 09:04:00 +07] [INFO] License key from https://mattermost.com required to unlock enterprise features.
And then a bit later:
[2018/04/09 09:09:07 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’
[2018/04/09 09:09:07 +07] [INFO] Server is initializing…
[2018/04/09 09:09:07 +07] [INFO] Pinging SQL master database
[2018/04/09 09:09:09 +07] [INFO] Current version is 4.8.0 (4.8.0/Thu Mar 15 23:11:12 UTC 2018/d1ed79a66c7930020b7fda046a11050f5890c84e/6f563c1fa497d123134fd499eb37674eb2511325)
[2018/04/09 09:09:09 +07] [INFO] Enterprise Enabled: true
[2018/04/09 09:09:09 +07] [INFO] Current working directory is /opt/mattermost
[2018/04/09 09:09:09 +07] [INFO] Loaded config file from /opt/mattermost/config/config.json
[2018/04/09 09:09:09 +07] [INFO] Connection to S3 or minio is good. Bucket exists.
[2018/04/09 09:09:09 +07] [INFO] License key from https://mattermost.com required to unlock enterprise features.
[2018/04/09 09:09:09 +07] [INFO] Starting up plugins
[2018/04/09 09:09:09 +07] [WARN] unable to create user namespace: fork/exec /proc/self/exe: permission denied
[2018/04/09 09:09:09 +07] [WARN] plugin sandboxing is not supported. plugins will run with the same access level as the server
[2018/04/09 09:09:10 +07] [INFO] Starting Server…
[2018/04/09 09:09:10 +07] [INFO] Server is listening on XXX.XXX.XXX.XXX:8065
[2018/04/09 09:09:10 +07] [INFO] API version 3 is scheduled for deprecation. Please see https://api.mattermost.com for details.
[2018/04/09 09:09:10 +07] [INFO] Starting 2 websocket hubs
[2018/04/09 09:09:10 +07] [INFO] Starting workers
[2018/04/09 09:09:10 +07] [INFO] Starting schedulers.
The users are created and work, but should the server initialize again each time?
[2018/04/09 09:04:00 +07] [INFO] Server is initializing…
[2018/04/09 09:04:00 +07] [INFO] Pinging SQL master database
[2018/04/09 09:04:00 +07] [INFO] License key from https://mattermost.com required to unlock enterprise features.
after running a platform command is normal but it shouldn’t cause an entire server restart a few minutes later. One thing you might be able to try is manually running the server attached at the command line with ./bin/platform. If you can catch a server restart that way it should also print out the stderr if it’s being caused a by a crash
It seems like updating to 4.9 has resolved the issue. I just tried to upload files when starting the server from the platform command and from the normal systemctl start command. In both cases uploading the files worked without a server crash/restart.
One thing I saw when starting the server from the platform command was this (over and over again) though:
[2018/04/20 15:41:09 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=m5i7o7nhbfguzenwg1bcrnuyyo uid=8oyg7kc89tfjbrbpmb7hos173y ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details: ]
[2018/04/20 15:41:47 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=n1ezh7bciirztpxuhntp5eit6w uid=ej6n4ptsc7no5jkthdhaxqeufr ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details: ]
[2018/04/20 15:41:47 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=zk55okuehtrydcdqtex3oe9fdr uid=8ugq1t7363fhfq5hkijumfrp3a ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details: ]
[2018/04/20 15:42:42 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=ra3p1o4ccjyf8nygd3d4hiueze uid=4xa5eyw3pt83zch8hw6ba5dhzo ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details: ]
[2018/04/20 15:42:55 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=8ww3e1anijdo9dw77qdequtj1w uid=zywk4d55wtgzmkmjrm4ny1mcao ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details: ]
[2018/04/20 15:43:02 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=bdcgehanb7rs3jefnrhxthjauh uid=uq1dehr39tbkig51dd38jzqe8r ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details: ]