Mattermost, Inc.

Channels jumping back in time

Summary

Channels jump back in time after implementing infinite scroll

Steps to reproduce

Run Mattermost server latest versions through an Nginx reverse proxy. Access the chat with a fresh desktop Chrome browser and switch channels. Randomly the channel is jumped back in time.

Expected behavior

The latest posts should always be shown when switching to a channel.

Observed behavior

We ran into a strange behaviour after installing the version in which the infinite scroll feature was implemented. When the users switch channels the scrolling randomly jumps back in time, showing a day or week old posts. It seems to occur totally randomly and also it seems to be random how much backwards the scrolling jumps.

Multiple users experience the same problem on multiple devices and all the usual troubleshooting steps (clearing browser cache, observing browser network requests etc) have been taken. I strongly believe the problem is related to our nginx reverse proxy setup in front of Mattermost server, but cannot prove that for now.

So far I haven’t seen or heard the problem occuring on mobile devices, so it seems to be limited to desktop browsers and possibly only Chrome and other Chrome-based browsers. Chrome versions have been up to date all the time, so the problem has been observed on multiple fresh versions on both MacOS and Windows.

Vast majority of our users are using Chrome though, so I cannot confirm that the problem does not occur on other browsers. One user reported he has never seen the problem and he’s running Firefox. Another one said he has the problem with the standalone Mattermost Windows client.

We are running Team Edition 5.16.1 at the moment, but the problem has existed in all the versions between the infinite scroll release (I really cannot remember which one it was :slight_smile: ) and the current one. Nginx version is 1.10.3 on Debian 9.5.

Mattemost configuration is fairly standard and the Nginx website is configured like below, in case someone can spot a clear problem in the configuration:

Nginx website config

----- clip ----
upstream backend {
server 127.0.0.1:8065;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
listen 443;
server_name censored.censored.com;
error_log /var/log/nginx/mattermost.access.log;
ssl on;
ssl_certificate /etc/ssl/certs/censored.com_bundle.crt;
ssl_certificate_key /etc/ssl/private/censored.com.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_ciphers ‘ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA’;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_stapling on;
ssl_stapling_verify on;
add_header Strict-Transport-Security max-age=15768000;

location /localimages {
root /var/www/html;

}

location ~ /api/v[0-9]+/(users/)?websocket$ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
client_max_body_size 100M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_pass http://backend;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-NginX-Proxy true;
proxy_pass_header Set-Cookie;
}

location / {
client_max_body_size 100M;
proxy_set_header Connection “”;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_pass http://backend;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-NginX-Proxy true;
proxy_pass_header Set-Cookie;

}

location ~ /.well-known {
allow all;
}
}

server {
listen 80;
server_name censored.censored.com;
return 301 https://$host$request_uri;
}
----- clip ----

@stnz Hey, We coudn’t figure out a repo for this. Is there any information you have that can help us narrow the repo down? Here are few things to look out for to help us narrow the issue down.

Does this happen the first time you visit a channel(i.e you see a loading… and then the posts)? or does it happen going back to a channel on a recurring visit?
Does the repo has any instances of waking the system up from sleep and using the tab from previous instance.
How many new messages were there in channel when this happened(Ballpark number should be good enough)?

Thanks for the reply. Some answers:

  • No, it happens when going back to a channel. For example earlier today I kept having the same happening for multiple times when switching back to one particular channel. Usually it’s more random though. I had the same browser tab open all the time and there was just minutes (or seconds) between the visits to that channel.
  • No, I think that doesn’t affect. I can try to kill Chrome after waking up from sleep though, and see if it continues to occur.
  • Anything from zero to some dozens. Doesn’t seem to affect.

It’s definitely something with this particular deployment and possibly the Nginx proxy though, as we see it happening for all of our Chrome users. The users have invidual browser settings and add-ins etc, so it’s not related to any centralized settings either.