Hubot mattermost missing messages

Troubleshooting
1

Hi!

I’ve installed Mattermost on a GKE cluster (Google Cloud Kubernetes).
I set up the bot with hubot-matteruser, in the same cluster.

The bot receives some messages, but not all from Mattermost.
And sometimes the bot responds with a message (showing the following logs) that never appears on Mattermost.

[Mon Jan 11 2021 00:59:11 GMT+0000 (Coordinated Universal Time)] DEBUG POST /posts
[Mon Jan 11 2021 00:59:11 GMT+0000 (Coordinated Universal Time)] INFO api url:http://kaggle-mattermost-nodeport/api/v4/posts
[Mon Jan 11 2021 00:59:11 GMT+0000 (Coordinated Universal Time)] DEBUG Posted message.

I first though the problem was linked to the websocket being terminated by the load balancer, but I confirmed it’s set up properly, and the websocket seems to be long lived on the app.

Screen Shot 2021-01-10 at 3.42.20 PM
Screen Shot 2021-01-10 at 3.42.20 PM1042×379 32.6 KB

I tried running the very same bot against a managed instance of Mattermost ( (philmod.cloud.mattermost.com) and it works perfectly in that case (i.e. not missed messages).

Any idea or suggestion is more than welcomed :slight_smile:

Cheers,
Mod

2

Hello, @philmod! I’m sorry to hear you’re having difficulties, I’m confident we can get this resolved!

If you enable your Mattermost Instance’s logging level to DEBUG and attempt to reproduce the undesired behavior, do you see anything out of the ordinary in the logs?

In addition, can I ask if you are using a reverse proxy, and if so, which one you are using?

3

Hey!

Thanks for your quick answer.

I actually realized this morning there seems to be 2 (related?) problems:

  1. The bot doesn’t receive all the messages sent to it
  2. Some messages sent from the bot (or any user) don’t appear on the Mattermost UI (until the page is refreshed)

The following logs on Mattermost are for the first problem (no logs on the bot side):

{"level":"debug","ts":1610381956.750951,"caller":"web/handlers.go:58","msg":"POST - /api/v4/posts"}
{"level":"warn","ts":1610381956.7685251,"caller":"app/notification.go:333","msg":"Notification not sent","logSource":"notifications","ackId":"","type":"message","userId":"nayb48qybbbw9jg3e864fj8sih","postId":"uaqa5qtzhby8fx5onfmcqdzcco","status":"Not Sent due to preferences"}

For the second problem, I do see a log "msg":"POST - /api/v4/posts" proving a message was sent to Mattermost, but the UI doesn’t show it. Websocket issue?

I followed this documentation for installation: docs.mattermost.com/install/install-kubernetes.html

But I modified slightly by adding:

  • a nodeport (with extended timeout for websocket)
  • a https ingress (which is needed to use Google Cloud IAP)

Here is the config: gist:d136f9f41b69c98d4cb8fd2d5bb448a9 · GitHub

The bot talks directly to the mattermost-nodeport.

Thanks a ton!
Mod

4

I think the second problem happens when the user is connected (websocket) to one pod, and the message is sent to the API on a different pod. Is that a known issue?

Maybe this feature is not offered on the open sourced version? Scaling mattermost to multiple servers · Issue #9080 · mattermost/mattermost-server · GitHub

UPDATE: confirmed this is a feature for E20 version.

So we are left with the first issue of the bot not receiving all messages.

1 Like
5

I reduced the deployment to a single pod in the meantime, and it fixes both problems :smiley:

1 Like
6

That’s good to hear that you got it to work! At this point in time, do we still have any issues or is it all figured out?

7

All good, thanks.

The only Action Item may be to update the documentation[0] to make sure people understand it won’t work as expected with >1 pod (which is the default configuration).

[0]Installing Mattermost on Kubernetes — Mattermost 5.30 documentation

8

Alright, I’ll ask about that!

@amy.blais do you think this is a change that would be able to be pushed through?

9

Would you be open to creating an issue in our docs repo Issues · mattermost/docs · GitHub?

1 Like
10

You got it: Request for documentation: kubernetes installation doesn't work well with Team version · Issue #4234 · mattermost/docs · GitHub

3 Likes