Mattermost, Inc.

Bulk import with existing users

Hi all,

The latest posts on this issue are a few years old now, so I create a new one.

Can the bulk import tool deal with users that already exist in mattermost? For me it fails with
“app/user.go:317”,“Couldn’t save the user”,“error”:“SqlUserStore.Save: An account with that email already exists.”

I have also tried the command line import slack that can merge the users, but it only imports private messages to users that have not yet been messaged on Mattermost. Existing chats are skipped.

I’m on Mattermost 5.23.0 Team edition with postgresql 9.4 as database.

Thanks in advance

Hello, @bartels

Based on a discussion with one of the engineering team members, the import tool should be able to handle existing users by updating them with whatever changed.

However, I would like to further understand this statement:

  • Were you originally referring to the bulk loading tool? If yes, what was the outcome of running the following command prior to running the bulk load command?
sudo -u mattermost bin/mattermost import bulk data.jsonl --validate
  • Can you please confirm what do you mean by command line import slack that can merge the users? Which documentation / article that you were referring to when you tried this?

  • Were you trying to upload the .jsonl from another communication tool into Mattermost (Slack)? If yes, which method did you try based on the Migrating from Slack documentation?

Hello @ahmaddanial,

Thanks for your reply. Sorry if things were unclear, I’ll provide more detail.

We are running a workspace that was migrated from Slack with the Standard export that only contains users and public channels. We have been using that one for a while but now got our Slack Corporate export and also want to import our Slack direct message history. That means that all the users already exist and most of us already have some direct messages between us in Mattermost. Now we want to merge the messages from Slack.

I was following the instructions under
https://docs.mattermost.com/administration/migrating.html#migrating-from-slack

I have tried 2 approaches:

  1. Bulk import tool

I ran the slack-advanced-exporter to include the attachments into the exportet .zip file (Only the command to add attachments, not mail addresses. These are already present in the export and the linked necessary API Tokens seem to be deprecated.). I then used the mmetl tool to turn the .zip file with attachments into a .jsonl which finishes successfully.

Running
sudo -u mattermost bin/mattermost import bulk workspace.jsonl --validate
shows that everything is OK:
Validation complete. You can now perform the import by rerunning this command with the --apply flag.

But if I run with --apply it fails:

Running Bulk Import. This may take a long time.

{"level":"error","ts":1590402795.3863447,"caller":"app/user.go:279","msg":"Couldn't save the user","error":"SqlUserStore.Save: An account with that email already exists., user_id=fr4w5m1ijb8g5gomdijyep7p5e, pq: duplicate key value violates unique constraint \"users_email_key\""}
SqlUserStore.Save: An account with that email already exists., user_id=fr4w5m1ijb8g5gomdijyep7p5e, pq: duplicate key value violates unique constraint "users_email_key"
Error occurred on data file line 14
{"level":"info","ts":1590402795.5344722,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1590402795.534502,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"warn","ts":1590402795.5345318,"caller":"app/web_hub.go:135","msg":"We appear to have already sent the stop checking for deadlocks command"}
{"level":"info","ts":1590402795.534698,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1590402795.535528,"caller":"app/server.go:453","msg":"Server stopped"}
Error: SqlUserStore.Save: An account with that email already exists., user_id=fr4w5m1ijb8g5gomdijyep7p5e, pq: duplicate key value violates unique constraint "users_email_key"

Everytime I run the command a different user_id shows the violation.

  1. Migrating from Slack using the Mattermost CLI

This one does not seem to avoid duplicates, so I removed the public channels from the exported zip which have already been migrated to Mattermost.

I then ran
sudo -u mattermost bin/mattermost import slack myTeam slackExportWithAttachments.zip

That produces outputs like:

Slack user merged with an existing Mattermost user with matching email
Unable to import Slack channel .
Unable to import Slack channel mpdm-XXX(direct messages with multiple participants)
Notes:
=======

- Some messages may not have been imported because they were not supported by this importer.
- Slack bot messages are currently not supported.
- Additional errors may be found in the server logs.

Finished Slack Import.

This manages to import direct messages even with attachments, but only in those chats where no messages have been sent on Mattermost yet. If messages between the users already exist, no messages from Slack are merged into the chat.

I hope, this provides sufficient detail.

Thank you in advance!