We're Hiring!

Mattermost, Inc.

Workaround: AD/LDAP Group Sync Incorrectly Removed Users

Issue:

You use LDAP Sync to manage your teams and channels. You’ve configured access to specific teams through Group Sync within the System Console. An LDAP sync occurs within Mattermost and now you see a large portion of users getting deactivated. These users are not automatically being restored to their teams on any following LDAP syncs.

OR

You tested out LDAP Groups by adding/removing a group a few times and now the users will not automatically add to channels.

Reason:

When Mattermost queries your LDAP database it uses only the most recent payload of users.

With LDAP Sync users are removed from Mattermost for one of two reasons. The user is not returned when Mattermost queries the LDAP database. This is because the user was deleted from your LDAP database. Your LDAP database had an error and failed to return a full payload of users. The user is not a part of the filter you have configured. This means a filter configured in the System Console > LDAP does not include this user anymore. A value on their LDAP profile was changed.

Removing users from Mattermost is usually an intentional change made by you or a system admin within your Mattermost database. However, if your LDAP database fails to send all the users but instead sends a portion of users back to Mattermost without also returning an error this will be treated as the proper set of users within Mattermost.

Solutions:

Depending on your number of users there are a few solutions. First, Take note that when a user is deactivated from Mattermost they are also removed from all channels they were a part of. All of these solutions require your LDAP database to have successfully returned the proper set of users after this issue occurred.

Solution 1:

If you have a small number of users you can manually invite the users back by using /invite @username or the Invite People option within Mattermost.

Use the ‘Invite People’ option within Mattermost to invite the user back to the team.
Then use /invite @username to invite the user to that specific channel.

You will have to do this for each channel & team you want the user to be a part of again.

Solution 2:

If Solution 1 returns an error when attempting to invite users who were deactivated you can instead invite the user by using the CLI. This has the benefit of being able to add multiple users to a single channel, but you will still have to do this for each channel/team.

  1. Log into your Mattermost Server
  2. Navigate to /opt/mattermost
  3. Run sudo ./bin/mattermost add {channel} {users}.
    • {channel} - This must be a channelID or channel name.
    • {users} - this can be an email address, user ID, or username.

For more information view our documentation on this here.

Solution 3:

If you have a large set of users who were deactivated then the above two solutions may not be ideal. The final option is to programmatically add users back to the channels they were a part of based on their ChannelMemberHistory table within Mattermost.

  1. You need to identify the time this started. You can do this by looking through the logs to find a sync that did not return the same number of lines as prior syncs. You could also use the system message where it removed the users as a reference point.

  2. Once you locate the time identify a time before that stamp you want to revert back to and convert it to UNIX time by using any number of tools. My suggestion is Current Millis.

  3. Run the below command within your SQL database to see if this returns the users who were removed. This will list all your users who have been updated since your timestamp, showing your username and the channels they were a part of.

    • Make sure you run USE Mattermost; first to switch to the Mattermost database.

    This command will return a list of users we are going to modify.

SELECT DISTINCT u.Username, u.UpdateAt, u.DeleteAt FROM ChannelMemberHistory AS cmh
JOIN Users AS u ON cmh.UserId = u.Id
JOIN Channels AS c ON c.Id = cmh.ChannelId
WHERE u.UpdateAt >= ‘yourTimestampHere’ AND c.Type != "D";

To check the specific channels you’ll be adding the users back to run the below:

SELECT DISTINCT c.DisplayName, u.Username, u.UpdateAt, u.DeleteAt FROM ChannelMemberHistory AS cmh
JOIN Users AS u ON cmh.UserId = u.Id
JOIN Channels AS c ON c.Id = cmh.ChannelId
WHERE u.UpdateAt >= ‘yourTimestampHere’ AND c.Type != "D";
  1. Once you have confirmed the users in the above query look correct, we’ll need an exported CSV of these users. You can choose to run this directly in a SQL GUI like MySQL Workbench or directly on your Mattermost server.

    • The query when running on a GUI. You need to save this CSV as ‘ChannelMembers.csv’.
SELECT DISTINCT cmh.ChannelId, cmh.UserId FROM ChannelMemberHistory AS cmh
JOIN Users AS u ON cmh.UserId = u.Id
JOIN Channels AS c ON c.Id = cmh.ChannelId
WHERE u.UpdateAt >= ‘yourtimestamphere’ AND c.Type != 'D';
  • When running this directly on the server
sudo mysql mattermost -e "select distinct cmh.ChannelId, cmh.UserId from ChannelMemberHistory as cmh
JOIN Users as u on cmh.UserId = u.Id
JOIN Channels as c on c.Id = cmh.ChannelId
WHERE u.UpdateAt >= '1603897632286' and c.Type != 'D';" | tr '\t' ',' > ChannelMembers.csv
  1. Now that you have the ChannelMembers.csv file, download the script here and place it in your /opt/mattermost directory on your server. Also, move your ChannelMember.csv file to this same directory.
  2. Navigate to /opt/mattermost and run python3 add_members.py. This script should output a file quickly called add_members.sh.
  3. Now run sudo add_members.sh. This will take some time to completely run depending on the total number of users you are attempting to add.
  4. Once the script above has completed running, go to your Mattermost UI > System Console > Web Server and ‘Purge all Caches’. Now when you navigate back to your team you should see these users automatically added back.