Mattermost Peer-to-Peer Forum

Messages sometimes be inserted in wrong order


#1

Summary

Messages sometimes appear to be sent much earlier (time) than they actually are so the order of the messages is wrong.

Steps to reproduce

Hard to reproduce.
Docker Container 5.5.0-rc1 on Linux
Windows Client 4.1.2

Expected behavior

We expect the message to be appended at the end of the message history.

Observed behavior

The message sometimes will be inserted just somewhere in the past history.


Too frequent Retry/Cancel message
#2

My team had the same issue. The server manager had to sync all the servers in the proper time to each can talk to each other. You may want to look at it in that approach. Once our server manager completed the troubleshoot, messages were in order.

I hope this can help.


#3

from @amy.blais “We haven’t been able to reproduce this internally, but there was another report on this issue. It seems to be it could be an issue with network latency between the app and DB servers. One thing would be to check to make sure you are not hitting a TCP socket connection limit on the DB?”


#4

Hi Ovim,

thanks for your suggestions. We cant reproduce it either. But this is what I ve seen:

  • The message shows up instantly, but the Timestamp is in the past (1-3 minutes). This is the confusing part, because answers to questions show up before the questions. :confused:
  • The server has plenty of free resources. There is a hardware firewall in between, but from what I know we dont have other problems with it.
  • Timestamps are correct most of the time (99,9%). Client is the x64-Windows Version: 4.1.2
  • Server Version is 5.5.0-rc1.
  • We have 8 Users in this test-Environment and I dont think there is much to do in the database. We have 50 messages per day.

I have increased the specs for the VM and will report back here as soon as I know more.

Best Regards


#5

Ok. I will ask my team and see if we have more information about this. Good luck.


#6

the problem just came back several times and we’ve tracked more information on this issue:

  • Windows Client is now x64 4.2
  • some messages are in order and just the next message from the same client is not
  • all Timezones are the same, all Server, VM, Docker, Client Times are in sync
  • timestamps are in the past, when error shows up, but the messages showing up instantly (without delay)
  • it is not possible to reproduce till this point

We are trying out Mattermost, but it is unlikely that we go for productive use with a show stopper like that. Are there logs or debug infos on the server we can use for further investigation?


#7

Hey @x0nn,

Mattermost messages are stored with the CreateAt field set to the current unix timestamp + milliseconds of the application server it is running on. So if one application server’s internal clock is more than ~1/10th of a second ahead of the others in the cluster its messages will appear to show up “before” the others. This is especially noticeable if you have users on different servers messaging each other.

If you’ve already verified the host machines’ clocks are showing the same time, then we can investigate the request itself to see if there’s any latency issues between the Mattermost servers and database server. This would cause a similar problem, where one application server’s requests are being delayed when attempting to write to the database. To do that we’ll need the CreateAt value for the post in question, and an excerpt from the Mattermost logs for both application servers showing that time, as well as whether the DB server was running under high load at the time.


#8

@x0nn, Also feedback from one our community members: Any particular reason you are using an old release candidate ( v5.5.0-rc1 )? Also, you can increase the verbosity of the Mattermost logging to debug (via system console).


#9

Hi,

thanks for your help. There are 2 things I want to add:

1.) We are using this version because it is a preview-container and we want to evaluate the software for enterprise use, but on premise. I can upgrade, but I already did to this version, which was the latest at time of the update.

2.) We tested and the messages arent sent at the time shown in the chat. We tested with sending the current timestamps of the computer clocks as message text. A collegue is sitting next to me in the same room and his message is timestamped several minutes in the past, at a time he didnt wrote, yet. The next message, send seconds after, is in correct order and message text (time) is in sync with Timestamp. Sometimes from 3 short messages send in 10 seconds apart, 2 are correct, one is not (all from the same client). And for the record, the messages are showing up immediately.

Despite this problem, we have freezes and white screens on several Windows 10 Clients with Client 4.2 and problems with Empty Search Window and links to already deleted messages crashing the client. I think the software isnt mature enough for our needs. I will try to update the Server to newest software and check the logs.

Best Regards