Summary
mattermost log file has reached 115,470 lines, and still has not been rotated.
Steps to reproduce
I’m using mattermost v5.8.0, hosting is on aws ec2, in an amazon linux ami.
Expected behavior
mattermost log file is rotated after either 10k lines, or 50k lines. This bug talks about the issue, and quotes old documentation that talked about automatic log rotation, but I can’t seem to find any mention of it in the latest docs.
Observed behavior
I first realised there was a problem when I wanted to check the mattermost version from the linux console:
(tldr is that after every log line output, there is a warning message saying that the log file could not be renamed/rotated)
ec2-user $ /opt/mattermost/bin/mattermost version
{"level":"info","ts":1554364338.4573255,"caller":"utils/i18n.go:79","msg":"Loaded system translations for 'en' from '/opt/mattermost/i18n/en.json'"}
2019-04-04 07:52:18.457732853 +0000 UTC m=+0.243608902 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-18.457.log: permission denied
{"level":"info","ts":1554364338.4578655,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
2019-04-04 07:52:18.457957197 +0000 UTC m=+0.243833226 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-18.457.log: permission denied
{"level":"info","ts":1554364338.4646955,"caller":"sqlstore/supplier.go:215","msg":"Pinging SQL master database"}
2019-04-04 07:52:18.464804317 +0000 UTC m=+0.250680347 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-18.464.log: permission denied
{"level":"info","ts":1554364338.90649,"caller":"filesstore/s3store.go:82","msg":"Connection to S3 or minio is good. Bucket exists."}
2019-04-04 07:52:18.906661758 +0000 UTC m=+0.692537875 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-18.906.log: permission denied
{"level":"info","ts":1554364338.9103534,"caller":"app/license.go:42","msg":"License key valid unlocking enterprise features."}
2019-04-04 07:52:18.910471057 +0000 UTC m=+0.696347115 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-18.910.log: permission denied
{"level":"info","ts":1554364338.9135077,"caller":"app/plugin.go:130","msg":"Starting up plugins"}
2019-04-04 07:52:18.9136039 +0000 UTC m=+0.699479944 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-18.913.log: permission denied
{"level":"error","ts":1554364339.5401752,"caller":"app/plugin.go:159","msg":"Failed to unpack prepackaged plugin","error":"installPlugin: Unable to install plugin., lstat ./plugins/plugins~: permission denied","path":"/opt/mattermost/prepackaged_plugins/jira-1.1.1.tar.gz"}
2019-04-04 07:52:19.540338592 +0000 UTC m=+1.326214628 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-19.540.log: permission denied
{"level":"error","ts":1554364340.1098795,"caller":"app/plugin.go:159","msg":"Failed to unpack prepackaged plugin","error":"installPlugin: Unable to install plugin., lstat ./plugins/plugins~: permission denied","path":"/opt/mattermost/prepackaged_plugins/zoom-1.0.4.tar.gz"}
2019-04-04 07:52:20.110033301 +0000 UTC m=+1.895909377 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.110.log: permission denied
{"level":"error","ts":1554364340.1101313,"caller":"app/plugin.go:55","msg":"Unable to get available plugins","error":"lstat ./plugins/plugins~: permission denied"}
2019-04-04 07:52:20.110221695 +0000 UTC m=+1.896097777 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.110.log: permission denied
{"level":"info","ts":1554364340.110705,"caller":"app/server.go:194","msg":"Current version is 5.8.0 (5.8.0/Fri Feb 15 23:00:08 UTC 2019/24f11a12758fe75ce06269be7f9fb7bb790aec43/2af5b29934977d279f119b8237e9f775ff4ca692)"}
2019-04-04 07:52:20.110791798 +0000 UTC m=+1.896667879 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.110.log: permission denied
{"level":"info","ts":1554364340.1108313,"caller":"app/server.go:195","msg":"Enterprise Enabled: true"}
2019-04-04 07:52:20.110905644 +0000 UTC m=+1.896781683 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.110.log: permission denied
{"level":"info","ts":1554364340.1109612,"caller":"app/server.go:197","msg":"Current working directory is /opt/mattermost"}
2019-04-04 07:52:20.111037295 +0000 UTC m=+1.896913393 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.111.log: permission denied
{"level":"info","ts":1554364340.1111264,"caller":"app/server.go:198","msg":"Loaded config file from /opt/mattermost/config/config.json"}
2019-04-04 07:52:20.111198862 +0000 UTC m=+1.897074947 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.111.log: permission denied
{"level":"info","ts":1554364340.1611774,"caller":"sqlstore/post_store.go:1287","msg":"Post.Message supports at most 16383 characters (65535 bytes)"}
2019-04-04 07:52:20.161294415 +0000 UTC m=+1.947170443 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.161.log: permission denied
{"level":"error","ts":1554364340.162959,"caller":"app/plugin.go:55","msg":"Unable to get available plugins","error":"lstat ./plugins/plugins~: permission denied"}
2019-04-04 07:52:20.163067928 +0000 UTC m=+1.948943905 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.163.log: permission denied
{"level":"error","ts":1554364340.1631424,"caller":"app/plugin.go:55","msg":"Unable to get available plugins","error":"lstat ./plugins/plugins~: permission denied"}
2019-04-04 07:52:20.163241854 +0000 UTC m=+1.949117822 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.163.log: permission denied
{"level":"info","ts":1554364340.1696565,"caller":"app/license.go:42","msg":"License key valid unlocking enterprise features."}
2019-04-04 07:52:20.169704857 +0000 UTC m=+1.955580911 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.169.log: permission denied
{"level":"error","ts":1554364340.169738,"caller":"app/plugin.go:55","msg":"Unable to get available plugins","error":"lstat ./plugins/plugins~: permission denied"}
2019-04-04 07:52:20.169768402 +0000 UTC m=+1.955644415 write error: can't rename log file: rename /opt/mattermost/logs/mattermost.log /opt/mattermost/logs/mattermost-2019-04-04T07-52-20.169.log: permission denied
This command was run by the ec2-user user. If I run it using the mattermost user, then no errors are shown. However, the log file is still not rotated.
Afterthought: The server runs under the user ‘mattermost’, and all files and directories are owned and read/writable to the mattermost user.