HipChat Importation

Hi,
now we are under way on migration from HipChat to local Mattermost instance.
HipChat data converted by Hipmost utility.

We are successfully imported direct messages using this method but have issue with importing room’s data.
CLI command finished without errors but new room I’ve found only from CLI command line. I able to see new room in Mattermost only after manipulation with CLI command line:
- manually add himself to new team
- manually add himself to channel
But after this I’ve found that new room is empty, without any message. However source JSOL file contains many messages.

Please help me resolve this situation.
Sergiy

Thanks @Sergiy for following up! Pinging @ylluminarious who may have insight on above.

Thank you for advice!

Hi,
now we are under way on migration from HipChat to local Mattermost instance.
HipChat data converted by Hipmost utility.

We are successfully imported direct messages using this method but have issue with importing room’s data.
CLI command finished without errors but new room I’ve found only from CLI command line. I able to see new room in Mattermost only after manipulation with CLI command line:

  • manually add himself to new team
  • manually add himself to channel
    But after this I’ve found that new room is empty, without any message. However source JSOL file contains many messages.

Please help me resolve this situation.
Sergiy

I think I may have ran into a similar issue that you are having while testing a MatterMost install for our team. The issue with us is that in the .jsonl file, HipMost was interpreting the exported HipChat channel as a “direct_post”, which to MatterMost, is a direct message. This seems to happen with private rooms converted using HipMost from HipChat exports.

The fix was to change the .jsonl file and substitute the “direct_post” to just “post” and remove the “channel_members” object. This seems to be working our development setup. I implemented the fix outside of HipMost using a Python script, which I unfortunately am probably not able to share due to some legalese that I had signed prior to employment at my company. However, I suspect that such a substitution can be made into HipMost, by someone with the appropriate ruby programming experience, or just a straight bash script using sed.

1 Like

Hi,

thank a lot for your help!
I did as you wrote (repalce “direct_post” -> “post” and remove “channel_members” object) and received a validation error as a result:

[tsaruk@mattermost-1 bin]$ sudo ./mattermost import bulk data.jsonl --validate
{"level":"info","ts":1546432032.3000233,"caller":"utils/i18n.go:78","msg":"Loaded system translations for 'ru' from '/opt/mattermost/i18n/ru.json'"}
{"level":"info","ts":1546432032.300167,"caller":"app/app.go:143","msg":"Server is initializing..."}
{"level":"info","ts":1546432032.3065329,"caller":"sqlstore/supplier.go:207","msg":"Pinging SQL master database"}
{"level":"info","ts":1546432032.4146852,"caller":"app/plugin.go:130","msg":"Starting up plugins"}
{"level":"error","ts":1546432032.414776,"caller":"app/plugin.go:138","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
Running Bulk Import Data Validation.
** This checks the validity of the entities in the data file, but does not persist any changes **
Use the --apply flag to perform the actual data import.

{"level":"info","ts":1546432032.4170878,"caller":"sqlstore/post_store.go:1286","msg":"Post.Message supports at most 16383 characters (65535 bytes)"}
BulkImport: Missing required Post property: Team., 
Error occurred on data file line 12
{"level":"info","ts":1546432032.4171906,"caller":"app/app.go:216","msg":"Stopping Server..."}
{"level":"info","ts":1546432032.4172096,"caller":"app/web_hub.go:120","msg":"stopping websocket hub connections"}
{"level":"warn","ts":1546432032.4172215,"caller":"app/web_hub.go:125","msg":"We appear to have already sent the stop checking for deadlocks command"}
{"level":"info","ts":1546432032.4183903,"caller":"sqlstore/supplier.go:850","msg":"Closing SqlStore"}
{"level":"info","ts":1546432032.4188824,"caller":"app/app.go:237","msg":"Server stopped"}
Error: BulkImport: Missing required Post property: Team., 

Are you faced with this error?
I can send to you examples of my file (before and after replacement) If it helps.

Cincerely
Sergiy

Hello,

You’ll want to make sure that the “post” objects follow the documentation at https://docs.mattermost.com/deployment/bulk-loading.html#post-object . The error message seems to suggest that “post” object at around line 12 is missing the “team” value.

Each line should look something like the following:
{ "type": "post", "post": { "team": "hipchat-import-example", "channel": "example_channel_name", "user": "username_here", "message": "some_message_imported_from_hipchat", "create_at": <date value of message here> } }

Hi again,
really good job from you - I’d changed file as you describe and validate step passed.
But I’d got a new error message when running import with “–apply” key:

[tsaruk@mattermost-1 bin]$ sudo ./mattermost import bulk data.jsonl --apply
{"level":"info","ts":1546516929.9373665,"caller":"utils/i18n.go:78","msg":"Loaded system translations for 'ru' from '/opt/mattermost/i18n/ru.json'"}
{"level":"info","ts":1546516929.9377325,"caller":"app/app.go:143","msg":"Server is initializing..."}
{"level":"info","ts":1546516929.9443572,"caller":"sqlstore/supplier.go:207","msg":"Pinging SQL master database"}
{"level":"info","ts":1546516930.043418,"caller":"app/plugin.go:130","msg":"Starting up plugins"}
{"level":"error","ts":1546516930.0435271,"caller":"app/plugin.go:138","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
Running Bulk Import. This may take a long time.

User.IsValid: Your password must contain at least 8 characters made up of at least one lowercase letter, at least one uppercase letter, and at least one number., 
Error occurred on data file line 4

I’m looking in my data file but see nothing extraordinary in line 4:

    { "type": "version", "version": 1 }
    { "type": "team", "team": { "display_name": "ua", "name": "ua", "type": "I", "description": "ua", "allow_open_invite": false } }
    { "type": "channel", "channel": { "team": "ua", "name": "offline-sale", "display_name": "Offline Sale", "type": "P", "header": "Offline office's issues" } }
    { "type": "user", "user": { "username": "adoroshenko", "email": "a.doroshenko@busfor.com", "teams": [] } }
    { "type": "user", "user": { "username": "barylskiy", "email": "a.barylskiy@busfor.com", "teams": [] } }
    { "type": "user", "user": { "username": "offlinekv", "email": "e.medynska@busfor.com", "teams": [] } }
    { "type": "user", "user": { "username": "oksiuta", "email": "a.oksiuta@busfor.com", "teams": [] } }
    { "type": "user", "user": { "username": "darnopyh", "email": "d.darnopyh@busfor.com", "teams": [] } }
    { "type": "user", "user": { "username": "pasichenko", "email": "r.pasichenko@busfor.com", "teams": [] } }
    { "type": "user", "user": { "username": "tsaruk", "email": "s.tsaruk@busfor.com", "teams": [] } }
    { "type": "user", "user": { "username": "vishnevskaya", "email": "o.vishnevskaya@busfor.com", "teams": [] } }
    { "type": "post", "post": { "team": "ua", "channel": "offline-sale", "user": "offlinekv", "message": "Я так понимаю они пренадлежат охранной фирме, а у нас есть характеристики камер или лучше обратится в охранную службу ? ", "create_at": 1544610447000 } }
    { "type": "post", "post": { "team": "ua", "channel": "offline-sale", "user": "offlinekv", "message": "Коллеги, добрый день! Можете подсказать, камеры в офисе пишут? ", "create_at": 1544610391000 } }
    { "type": "post", "post": { "team": "ua", "channel": "offline-sale", "user": "offlinekv", "message": "все в порядке. спасибо ", "create_at": 1544438837000 } }

If you have any idea about this, please tell me.
Once more many thanks to you for your advices and participation in my issue.
Segiy

The error message may not be in the file. I’ve noticed that during the import, MatterMost will compare the users listed in the .jsonl file with the users listing in the database. The issue may actually be the password of the user stored in the database itself.

Hello,
all import worked fine when I did a fresh ire-install Mattermost (without ttracks of my previous experiments) - private rooms data was loaded successfully.
But notice: to display data in the UI I had to consistently add users to team and channel both from CLI.

Thank you very much for your help!

Sincerely,
Sergiy.

I have a similar problem to @Sergiy (the answers already cleared it up mostly) that on import, users who have sent messages in the imported team/channel are not members. I also cant figure out how to edit the import of the team or channel to automatically add the users to that channel.

I had a few problems with this as when I added a new team called “all” and tried to enter it after adding myself to that team, mattermost was trying to redirect me to “all”'s “townsquare” constantly but this resulted in looping rejections as I was not a member of “all”'s “townsquare” channel yet. I ended up having to use the command line tools to add an additional public team called “Aa” (I had a theory I was being redirected to “all” townsquare as it was alphabetically the first team I was in) and add myself to it’s townsquare, as even when I added myself to “all” I was still rejected til I was able to get Mattermost to redirect me somewhere else. After that I was able to enter “all” fine. All the same, it would be nice to be able to add users to channels and teams automatically on import (especially adding them to the townsquare automatically!) to avoid this problem, so if anyone can direct me as to what to edit in the json files or add to them, itd be a great help.

I haven’t found a way to automatically add users into teams or channels other than via scripting that goes through the “user” object in the jsonl and then add that user to the team and channel listed in the jsonl file by sending commands to the MatterMost CLI.