Multiple file uploads can still crash Mattermost (and entire server). Any tipps?

Summary

Although file upload improved with Mattermost 5.0, multiple big upload files can still crash Mattermost and I would like to know, if there’s anything to do about it

Steps to reproduce

  • Upload two or three file attachments (50 - 100 MB) simultanously
  • Best case scenario: Server is very slow for a couple of minutes, but uploads work
  • Worst case scenario: Server crashes, uploads fail, client sometimes get an 502-error (as mattermost service is down and has to be restarted).

Expected behavior

As I cannot coordinate or control our team user’s upload behaviour, I’d like to see, that Mattermost server would be able to “pipeline” bigger and simultanous file uploads more save and efficient. I also “played” around with Nginx proxy settings, but nothing worked so far. If there’s any other configuration tipp, please let me know.

Observed behavior

Our Mattermost installation (5.4.0) is hosted on a

  • AWS t2.micro instance
  • with 1 GB RAM,
  • 1 vCPU,
  • Nginx as proxy, SSL enabled
  • 1,7 GB additional SWAP are allocated.
  • Current max file upload size: 100 MB.
  • Files are saved on Amazon S3.

For our Team (a podcast project) it’s crucial to be able to handle 50 - 100 MB Audio-Files from time to time in Mattermost. But bigger uploads crashed Mattermost in the past. With ver 5.0 we hoped for an improvement (Release note: * Improved file upload API to stream files instead of loading them entirely into memory.) And usually big file uploads work. But: When multiple bigger files are uploaded simultanously, the Mattermost app still crashes.

Hi @kraehenpost1,

What log error messages are you seeing when the server crashes?

We are also still working on at least ticket for file upload improvements: https://github.com/mattermost/mattermost-server/issues/8641.

1 Like

hey @amy.blais, thanks for checking back. I just ‘ran’ a new crash with logging in DEBUG mode, but I’m afraid this wouldn’t help a lot. (I only copied the lines from just around the crash. See the time gab between 15:03:27 and 15:11:16. This is when the server crashed):

2018-11-12T15:02:59.655Z debug web/handlers.go:50 GET - /api/v4/users/me/teams/zobpun3ro7rh3k4fxbcrarr4nr/channels/members
2018-11-12T15:03:01.598Z debug web/handlers.go:50 POST - /api/v4/users/status/ids
2018-11-12T15:03:02.797Z debug web/handlers.go:50 POST - /api/v4/users/status/ids
2018-11-12T15:03:10.696Z debug web/handlers.go:50 POST - /api/v4/files
2018-11-12T15:03:11.010Z debug web/handlers.go:50 POST - /api/v4/files
2018-11-12T15:03:11.873Z debug web/handlers.go:50 GET - /api/v4/channels/ckuwqgqo1i8etewxmw9t8jbbjr/posts
2018-11-12T15:03:11.877Z debug web/handlers.go:50 POST - /api/v4/channels/members/me/view
2018-11-12T15:03:11.877Z debug web/handlers.go:50 GET - /api/v4/channels/ckuwqgqo1i8etewxmw9t8jbbjr/posts
2018-11-12T15:03:13.959Z debug app/email_batching.go:110 Email batching job ran. 0 user(s) still have notifications pending.
2018-11-12T15:03:25.073Z debug web/handlers.go:50 POST - /api/v4/users/status/ids
2018-11-12T15:03:27.361Z debug web/handlers.go:50 POST - /api/v4/files

2018-11-12T15:11:16.222Z info utils/i18n.go:78 Loaded system translations for ‘de’ from ‘/opt/bitnami/apps/mattermost/server/i18n/de.json’
2018-11-12T15:11:16.225Z info app/app.go:204 Server is initializing…
2018-11-12T15:11:16.225Z debug utils/html.go:30 Parsing server templates at /opt/bitnami/apps/mattermost/server/templates
2018-11-12T15:11:16.262Z info sqlstore/supplier.go:209 Pinging SQL master database
2018-11-12T15:11:16.269Z info sqlstore/channel_store_experimental.go:47 Enabling experimental public channels materialization
2018-11-12T15:11:16.331Z debug sqlstore/preference_store.go:41 Deleting any unused pre-release features
2018-11-12T15:11:16.345Z debug jobs/schedulers.go:29 Initialising schedulers.
2018-11-12T15:11:16.707Z info commands/server.go:82 Current version is 5.4.0 (5.4.0/Tue Oct 16 16:48:28 UTC 2018/d5b613cb1bbdaa856fa126e7102d94783b9e80b6/none)
2018-11-12T15:11:16.707Z info commands/server.go:83 Enterprise Enabled: false
2018-11-12T15:11:16.707Z info commands/server.go:84 Current working directory is /opt/bitnami/apps/mattermost/server
2018-11-12T15:11:16.708Z info commands/server.go:85 Loaded config file from /opt/bitnami/apps/mattermost/server/config/config.json
2018-11-12T15:11:16.750Z info filesstore/s3store.go:82 Connection to S3 or minio is good. Bucket exists.
etc…

Hi @kraehenpost1,

We still have a few tickets open to address issues with file uploads, so it is possible that you’re hitting issues that these tickets would improve: