We're Hiring!

Mattermost Discussion Forums

After an unknown hickup, Mattermost instance delivers only a blank page

Summary
After an unknown hickup, Mattermost instance delivers no (HTML) data via http.

Steps to reproduce
Unknown. Version is

Version: 5.33.2
Build Number: 5.33.2
Build Date: Wed Mar 24 14:49:44 UTC 2021
Build Hash: 06e9db54def615f8b087728638cac21231b0a00d
Build Enterprise Ready: true
DB Version: 5.33.0

Expected behavior
When going to the the url of the instance, a front end should be delivered to the client, be it a browser or the official mattermost desktop client.

Observed behavior
An established instance that had been running without issues for over a year experienced some sort of hickup yesterday morning. Some people’s messages did not come through, while other people’s messages appeared multiple times. This was followed up by the instance not being reachable for a few minutes, partly returning a 502 http error. This was caused by the mattermost server restarting. No logs are available from this incident.
Since then, the behavior is as follows:

  • Users that had the website open before the hickup in their browser can use the service just fine
  • Users that use the mobile clients on Android or iPhone can use the service just fine
  • Any other users or users that log out or refresh their browser tab or dektop client tab receive an empty webpage. Literally no content, but with the http code 200 or 304.
  • Users that are still connected in some way or another to the service see occasional websocket errors in the webdev console (e.g. “Die Verbindung zu wss://..social/api/v4/websocket wurde unterbrochen, während die Seite geladen wurde.”). The connection so far has been always be reestablised.

What I have tried

  • The instance is hosted at https://uberspace.de. I informed the admins and asked if they had changed something or saw any issues in their network. That is not the case.
  • I restarted the mattermost server several times with no changes.
  • I started loggign, but can see no errors whatsoever in them. People that receive the blank page are simply getting a 200 (or 304) entry in the log:
2021-11-27T14:46:26.190+0100    debug   web/handlers.go:104     Received HTTP request   {"method": "GET", "url": "/", "request_id": "i577p7cwh78e7rua56akzsxuoh", "host": "<sub>.<domain>.social", "scheme": "https", "status_code": "200"}
2021-11-27T14:46:26.407+0100    debug   web/handlers.go:104     Received HTTP request   {"method": "GET", "url": "/favicon.ico", "request_id": "pm3zinazhb897qrs4mc8exowfh", "host": "<sub>.<domain>.social", "scheme": "https", "status_code": "200"}
  • I disabled all caches in my browser. No change.
  • I ran the integrity check:
Found 0 records in relation CommandWebhooks orphans of relation Channels
Found 0 records in relation ChannelMemberHistory orphans of relation Channels
Found 0 records in relation ChannelMembers orphans of relation Channels
Found 0 records in relation IncomingWebhooks orphans of relation Channels
Found 0 records in relation OutgoingWebhooks orphans of relation Channels
Found 0 records in relation Posts orphans of relation Channels
Found 0 records in relation CommandWebhooks orphans of relation Commands
Found 81 records in relation FileInfo orphans of relation Posts
Found 0 records in relation Posts orphans of relation Posts
Found 0 records in relation Posts orphans of relation Posts
Found 0 records in relation Reactions orphans of relation Posts
Found 0 records in relation Channels orphans of relation Schemes
Found 0 records in relation Teams orphans of relation Schemes
Found 2644 records in relation Audits orphans of relation Sessions
Found 0 records in relation Channels orphans of relation Teams
Found 0 records in relation Commands orphans of relation Teams
Found 0 records in relation IncomingWebhooks orphans of relation Teams
Found 0 records in relation OutgoingWebhooks orphans of relation Teams
Found 0 records in relation TeamMembers orphans of relation Teams
Found 0 records in relation Audits orphans of relation Users
Found 0 records in relation CommandWebhooks orphans of relation Users
Found 0 records in relation ChannelMemberHistory orphans of relation Users
Found 0 records in relation ChannelMembers orphans of relation Users
Found 0 records in relation Channels orphans of relation Users
Found 0 records in relation Commands orphans of relation Users
Found 0 records in relation Compliances orphans of relation Users
Found 0 records in relation Emoji orphans of relation Users
Found 0 records in relation FileInfo orphans of relation Users
Found 0 records in relation IncomingWebhooks orphans of relation Users
Found 0 records in relation OAuthAccessData orphans of relation Users
Found 0 records in relation OAuthApps orphans of relation Users
Found 0 records in relation OAuthAuthData orphans of relation Users
Found 0 records in relation OutgoingWebhooks orphans of relation Users
Found 0 records in relation Posts orphans of relation Users
Found 0 records in relation Preferences orphans of relation Users
Found 0 records in relation Reactions orphans of relation Users
Found 0 records in relation Sessions orphans of relation Users
Found 0 records in relation Status orphans of relation Users
Found 0 records in relation TeamMembers orphans of relation Users
Found 0 records in relation UserAccessTokens orphans of relation Users

I see the following errors when running for example the integrity check:

{"level":"error","ts":1638019485.1146317,"caller":"app/plugin.go:843","msg":"Failed to install prepackaged plugin","path":"/home/<user>/mattermost/prepackaged_plugins/mattermost-plugin-incident-collaboration-v1.4.0-linux-amd64.tar.gz","error":"Failed to install extracted prepackaged plugin /home/<user>/mattermost/prepackaged_plugins/mattermost-plugin-incident-collaboration-v1.4.0-linux-amd64.tar.gz: installExtractedPlugin: Unable to restart plugin on upgrade., a valid Mattermost Enterprise E20 license is required to use this plugin","errorVerbose":"installExtractedPlugin: Unable to restart plugin on upgrade., a valid Mattermost Enterprise E20 license is required to use this plugin\nFailed to install extracted prepackaged plugin /home/<user>/mattermost/prepackaged_plugins/mattermost-plugin-incident-collaboration-v1.4.0-linux-amd64.tar.gz\ngithub.com/mattermost/mattermost-server/v5/app.(*App).processPrepackagedPlugin\n\tgithub.com/mattermost/mattermost-server/v5/app/plugin.go:895\ngithub.com/mattermost/mattermost-server/v5/app.(*App).processPrepackagedPlugins.func2\n\tgithub.com/mattermost/mattermost-server/v5/app/plugin.go:841\nruntime.goexit\n\truntime/asm_amd64.s:1374"}
{"level":"error","ts":1638019485.3816772,"caller":"mlog/log.go:232","msg":"Unable to activate plugin","plugin_id":"com.mattermost.plugin-incident-management","error":"a valid Mattermost Enterprise E20 license is required to use this plugin"}
{"level":"error","ts":1638019485.464327,"caller":"mlog/log.go:232","msg":"Unable to activate plugin","plugin_id":"com.mattermost.plugin-incident-management","error":"a valid Mattermost Enterprise E20 license is required to use this plugin"}

These plugins have been disabled. I can also uninstall them if you want. As they are prepackaged, I was not sure if that was a wise idea.

I am happy to give you the full url in DM if you want. Also happy to answer any questions you might have. But I am not sure what sure what do.

The mattermost instance was setup according to this manual:

The general layout of an uberspace can be found here:

Would you be open to trying out a more recent version of Mattermost? We support versions 5.37 and later.

I did the following:

  • I downloaded the current released version of the mattermost 5 and unpacked it
  • copied the data folders from the old to the new ( client/plugins , config , data , logs and plugins)
  • copied the database into a fresh one
  • connected the new database with the new mattermost
  • started it

This worked. However, I would not mark this as solved as I do not know what caused the issue or if it was solved by the new version or if it was something else like a bad cached somewhere.

If you you encounter the issue, please let me know. I am curious what happened still. You can try updating and it might solve your problem.