We're Hiring!

Mattermost, Inc.

Specific users becoming inactive daily

Summary

Certain users are marked as inactive every morning.

Steps to reproduce

  • Login through GitLab (CE 13.5.3)
    
  • Mattermost Version: 5.25.4
    
  • Database Schema Version: 5.25.0
    
  • Database: postgres
    
  • Session lengths are the default 30 days
    

Existing discussion in github: https://github.com/mattermost/mattermost-server/issues/16275

Expected behavior

Users should remain logged in as long as the session length is configured. Direct messages should not become archived just because the users has not logged-in recently.

Observed behavior

A small subset of users (~10) become marked as inactive every morning and all direct messages with them become archived. They can send messages but cannot receive any until they logout and login again. Once they re-login, their status changes to member in the system console, but their direct messages are archived until they send a message.

Note - the users do not have issues with their GitLab session length.
Note - this affects only people with names starting with A , B , C or D , so it looks like some user filtering going wrong.

Solutions Attempted:

  • Revoke all sessions for the users and have them login fresh.
  • I have provided some logs in the GitHub issue, and will gather other logs as needed.

Hello @itwasonlyabug! I read the GitHub issue, and I have to say, this is a very interesting situation that you are dealing with, I hope I’ll be able to contribute to helping this get resolved.

If I can ask, what are you using as your reverse proxy? (Ex. Apache, Nginx, etc)
In addition, can you confirm that you have reverse proxied the web socket as well? This is important, and something that I personally had issues with at first as well, because the web socket must have a separate proxy entry in the configuration files, otherwise it won’t be reverse proxied, because it is not a HTTP/HTTPS request.

Hi! We have NGINX, which is configured as per this page: https://docs.mattermost.com/install/install-ubuntu-1604.html#installing-nginx-server

We do not have customizations on top of those settings.

Alright, and are you using a secured HTTPS connection, and if so, what SSL protocol are you currently using?

Okay, so I just verified and we are not using HTTP/2 like shown in the docs, but the websocket and general location settings are the same.

For SSL we are using TLSv1.2.

After some testing, I think maybe all users with A and B letters are deactivated ( we don’t have that many) . I’ll verify this and reply.

1 Like

Alright, sounds good, I’ll be watching for the reply :slight_smile:

Confirmed, all users with names starting with A and B get deactivated each morning. Users with C & D names have stopped experiencing this issue.

1 Like

That is very intriguing to me, and certainly quite unique! @ahmaddanial I’m wondering, do you think this is more likely a server-side issue, or an in-transport/network issue? Just thinking of where would be best to start the debug and assessment process.

Hello, @itwasonlyabug

Users can get deactivated automatically based on the following circumstances in your case:

  • Gitlab authentication is disabled from System Console.
  • Through user or System Administrator interaction (for example, users are allowed to deactivate their account)
  • Some sort of automation/provisioning from outside.

Since you were able to narrow down that the issue is now only happening for A&B users versus C&D, let us dive deeper to understand the differences between these two sets of user groups. Do you observe anything obvious between them?

Additionally, can the deactivated users still login to Gitlab based on the article I came across here?

Deactivating a user is functionally identical to blocking a user, with the following differences:

  • It does not prohibit the user from logging back in via the UI.
  • Once a deactivated user logs back into the GitLab UI, their account is set to active.
1 Like

Hi @ahmaddanial, here’s some information on the setup:

  • The users are not deactivated/blocked in GitLab.
  • The users can access GitLab just fine and their sessions are also preserved so they don’t have to login each morning.
  • The issue appears only in Mattermost.
  • The users also do not have this issue with pure AD authentication.
  • We are only 2 users with System Admin permission in Mattermost and only I am actively administering it (I’ve checked :)) so nobody is deactivating the users manually.
  • There is no custom automation added at any step between GitLab and Mattermost.
  • We have started experiencing this issue in the past 6 months, there were no such problems before that. We haven’t really changed anything related to Mattermost in that time.
  • I upgraded the Mattermost server 2 months ago from 5.5.0 to 5.25.4 (following the docs strictly) but I can verify that the issue started before that.

@XxLilBoPeepsxX For it to be happening only on the A-B users and not All users, I’m thinking something like session house-keeping or database-related. If it was transport / proxy I would expect it to be affecting more people with random names.

Alright, could you send your database schema of the Users table here, please?
It will look something like this, if you use the same commands that I did, which are also included below.

root@ubuntu:~# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 8.0.22-0ubuntu0.20.04.3 (Ubuntu)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mattermost
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show columns from Users;
+--------------------+--------------+------+-----+---------+-------+
| Field              | Type         | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+-------+
| Id                 | varchar(26)  | NO   | PRI | NULL    |       |
| CreateAt           | bigint       | YES  | MUL | NULL    |       |
| UpdateAt           | bigint       | YES  | MUL | NULL    |       |
| DeleteAt           | bigint       | YES  | MUL | NULL    |       |
| Username           | varchar(64)  | YES  | UNI | NULL    |       |
| Password           | varchar(128) | YES  |     | NULL    |       |
| AuthData           | varchar(128) | YES  | UNI | NULL    |       |
| AuthService        | varchar(32)  | YES  |     | NULL    |       |
| Email              | varchar(128) | YES  | UNI | NULL    |       |
| EmailVerified      | tinyint(1)   | YES  |     | NULL    |       |
| Nickname           | varchar(64)  | YES  |     | NULL    |       |
| FirstName          | varchar(64)  | YES  |     | NULL    |       |
| LastName           | varchar(64)  | YES  |     | NULL    |       |
| Position           | varchar(128) | YES  |     | NULL    |       |
| Roles              | text         | YES  |     | NULL    |       |
| AllowMarketing     | tinyint(1)   | YES  |     | NULL    |       |
| Props              | text         | YES  |     | NULL    |       |
| NotifyProps        | text         | YES  |     | NULL    |       |
| LastPasswordUpdate | bigint       | YES  |     | NULL    |       |
| LastPictureUpdate  | bigint       | YES  |     | NULL    |       |
| FailedAttempts     | int          | YES  |     | NULL    |       |
| Locale             | varchar(5)   | YES  |     | NULL    |       |
| Timezone           | text         | YES  |     | NULL    |       |
| MfaActive          | tinyint(1)   | YES  |     | NULL    |       |
| MfaSecret          | varchar(128) | YES  |     | NULL    |       |
+--------------------+--------------+------+-----+---------+-------+
25 rows in set (0.01 sec)

MySQL>
1 Like

@XxLilBoPeepsxX sorry for the late response, here it is (keep in mind that it’s from PSQL not MySQL):

Column          |          Type           |                                                Modifiers                                                | Storage  | Stats target | Description 
--------------------------+-------------------------+---------------------------------------------------------------------------------------------------------+----------+--------------+-------------
 id                       | character varying(26)   | not null                                                                                                | extended |              | 
 createat                 | bigint                  |                                                                                                         | plain    |              | 
 updateat                 | bigint                  |                                                                                                         | plain    |              | 
 deleteat                 | bigint                  |                                                                                                         | plain    |              | 
 username                 | character varying(64)   |                                                                                                         | extended |              | 
 password                 | character varying(128)  |                                                                                                         | extended |              | 
 authdata                 | character varying(128)  |                                                                                                         | extended |              | 
 authservice              | character varying(32)   |                                                                                                         | extended |              | 
 email                    | character varying(128)  |                                                                                                         | extended |              | 
 emailverified            | boolean                 |                                                                                                         | plain    |              | 
 nickname                 | character varying(64)   |                                                                                                         | extended |              | 
 firstname                | character varying(64)   |                                                                                                         | extended |              | 
 lastname                 | character varying(64)   |                                                                                                         | extended |              | 
 position                 | character varying(128)  |                                                                                                         | extended |              | 
 roles                    | character varying(256)  |                                                                                                         | extended |              | 
 allowmarketing           | boolean                 |                                                                                                         | plain    |              | 
 props                    | character varying(4000) |                                                                                                         | extended |              | 
 notifyprops              | character varying(2000) |                                                                                                         | extended |              | 
 lastpasswordupdate       | bigint                  |                                                                                                         | plain    |              | 
 lastpictureupdate        | bigint                  |                                                                                                         | plain    |              | 
 failedattempts           | integer                 |                                                                                                         | plain    |              | 
 locale                   | character varying(5)    |                                                                                                         | extended |              | 
 mfaactive                | boolean                 |                                                                                                         | plain    |              | 
 mfasecret                | character varying(128)  |                                                                                                         | extended |              | 
 timezone                 | character varying(256)  | default '{"automaticTimezone":"","manualTimezone":"","useAutomaticTimezone":"true"}'::character varying | extended |              | 
 acceptedtermsofserviceid | character varying(64)   | default ''::character varying                                                                           | extended |              |

Thanks for providing the schema, and letting me know it was PSQL, I took a look, it seems to all be correct, which leads me to narrow the issue down to either a network or API/WebSocket issue at this point. I’m going to look into my own installation to see if I can discover for myself the protocols and workflow of this function to see if I can tamper with it and figure a way to reproduce it out. Hopefully, it might work, I have no idea :laughing:

1 Like

Do you have an idea what kind / level of logging I can get that might shed some clarity? e.g. NGINX access logs, mattermost Trace logging?

I would suggest following the steps found here to capture logs, specifically the first step, which will write to the Nginx error log file, which will be easier/easiest to send here.

https://serverfault.com/a/435575

1 Like