HipChat Importation

We found that the HipChat migration import documentation was faulty… The Mattermost-ETL framework just simply does not import HipChat. We discovered this after chatting with the ETL developers.

We went ahead and implemented our own HipChat importation tool so that we can import things in a piecemeal fashion: hipmost -v --path . "HipChat Room Name" "Mattermost Team":"Channel Name"

This will handle both the importation of HipChat Rooms & People (1-to-1) chats. It does NOT perform the importation itself - it just creates a data file that one can then import via the mattermost CLI.

Any interest in this?

Given the recent announcement that Hipchat (and Stride) are being killed off in favor of Slack, and the anger I’ve seen on Atlassian’s forums from their on-prem hipchat licensees, I’d imagine that anything that facilitates a hipchat-to-mattermost migration would be of interest.

We have several teams affected by this issue, and I’d certainly like to see them move to Mattermost.

Great to hear. I’m thinking we might open source this. I’m not sure yet, I’d like to recoup our development costs if possible, but this is going to be super important to everyone in this same situation. I think we’ll just put up a donation badge for anyone willing to drop something in.

We’ve got it working right now pretty good. Some bugs to probably work through and refinements, but it does actually work and we’re getting rooms converted to channels presently. Nice thing is that it also imports the rooms into existing rooms with data so it’s NOT destructive. Important since we had to work with the new chat system while bringing old conversations in.

I’ll say this: the Mattermost search function for history is SO much better than HipChat and overall our experience with Mattermost is much better than it has been with HipChat. We don’t feel neglected any longer! :slight_smile:

First thing you need to do is to request your data from Atlassian so if you haven’t definitely get that in the pipeline as you have to fill out a form and the have to collect, archive and encrypt it for you which seems to take about a day depending on the size of your history and files.

1 Like

@thejimnicholson are you ready to test the import system? I think we’re (about) ready to release what we’ve got for folks to start using / testing.

I’m vacationing until the last week of this month, and my access is limited. But I’d be glad to try it once I get back.

I would also be interested in testing this. I do not have our data from Atlassian yet, but am working on that. How can I get involved with testing the import system?

@thejimnicholson @apophisnow Sorry for the delay. We just released our import system today, called hipmost. Please test it out and let us know what you think! Thanks!

EDIT: For some reason, people are flagging my posts as spam so I have to edit them now or else my posts will be hidden. I’m really confused what’s going on. If someone has a problem, please try to explain it before just flagging my posts as spam. I’m trying to help members of the community.

1 Like

Hi,

now we use HipHat Cloud but plan to switch to the Matterost local server.
To do this we created file with message history, user logins and so on (with the tool from the administrative console).

To the import this data in Mattermost we use Hiptost.
Actually, there are several questions about it using:

  1. this utility (Hipmost) must be run on same server with the Mattermost isntallation?
  2. decrypt and unzip the data file (received from HipChat admin tools) first need before import process begining?
  3. may you have typical scenario of data transfer from HipChat Cloud to Matterost local server? Please, share it.

With best regards,
Sergiy

In an attempt to migrate our HipChat installation to MatterMost, I came across this post and Hipmost. Unfortunately I can’t get it to work. Except for the “hipmost public list” the other commands seem to fail.

Running hipmost private list will produce:

/var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:24:in []': no implicit conversion of String into Integer (TypeError) from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:24:in block (2 levels) in direct_channels’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:23:in each' from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:23:in map’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:23:in block in direct_channels' from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:21:in each’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:21:in flat_map' from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:21:in direct_channels’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/private.rb:24:in list' from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/private.rb:19:in run’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost.rb:14:in run' from /var/lib/gems/2.3.0/gems/hipmost-1.0/exe/hipmost:48:in <top (required)>’
from /usr/local/bin/hipmost:23:in load' from /usr/local/bin/hipmost:23:in

Attempting a conversion with hipmost room import produces:

/var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:62:in block (3 levels) in save': undefined method to_jsonl’ for nil:NilClass (NoMethodError)
Did you mean? to_json
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:61:in each' from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:61:in block (2 levels) in save’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:60:in each' from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:60:in block in save’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:50:in open' from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:50:in save’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:43:in import' from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/room.rb:19:in run’
from /var/lib/gems/2.3.0/gems/hipmost-1.0/lib/hipmost.rb:12:in run' from /var/lib/gems/2.3.0/gems/hipmost-1.0/exe/hipmost:48:in <top (required)>’
from /usr/local/bin/hipmost:23:in load' from /usr/local/bin/hipmost:23:in

Any help is greatly appreciated

Hi @ylluminarious, FYI on the above feedback ^

Hi @Sergiy and @Niels, please also take a look at our import guide for further details: https://docs.mattermost.com/administration/hipchat-migration-guidelines.html#step-3-import-your-data-into-mattermost.

Hi everyone. I sincerely apologize for the delay that it’s taken to respond here. My team and I have been awfully busy with work for a client over the past several weeks so it’s been hard to find time to devote to this. Also, I didn’t see @Sergiy’s post until I saw @Niels’s post, so I apologize especially for that.

In response to @Sergiy:

  1. No, Hipmost can be run on any system which has the Hipchat data on it. It operates on the data retrieved from Hipchat, converting it to JSONL files which can be imported into Mattermost.

  2. Yes, decrypt and unzip the file before trying to use Hipmost. We’ve added new documentation on how to do this.

  3. I’m not quite sure what you mean by this, but I’ll try to answer as best I can. The new documentation covers all of this in more detail, but in short: the typical scenario of data transfer from Hipchat (Cloud or otherwise) to Mattermost, would consist of setting up a Mattermost server, exporting Hipchat data, converting the data using Hipmost, and importing the data into Mattermost one room at a time.

Please feel free to ask anymore questions. I will make sure to reply in a much timelier fashion this time.

In reply to @Niels:

I have tried to fix the first error which you posted. Unfortunately, I cannot link to the commit right now since it seems that someone is flagging my posts as spam. I’m really unsure why. Anyway, please tell me whether it fixes your first error. You can update the gem to the latest commit via:

$ gem install specific_install
$ gem specific_install -l https://github.com/orbitalimpact/hipmost.git

I have tried to find out what could be causing your 2nd error, but it is very perplexing to me. That variable should not be nil and I cannot reproduce it with our own data. Would you be willing to share a small sample file which can help to reproduce this error?

@amy.blais Umm, why are my posts being flagged as spam? Two of my three messages on this thread have been hidden because apparently multiple community members have flagged them as “spam.” I am very confused and I see no way to contact moderators about this except here. I don’t think I’ve violated any rules at all, so please unhide my posts or at least explain what’s going on. Thanks.

Hi,
many thanks for your help. Don’t worry about delay with answer, I understand.
I will closely read the new version of documentation and write to you if I have any questions.

Many thanks again.

S

Hi @ylluminarious,

Those messages were flagged by the Forum’s moderator’s because you had added multiple links to the same domain (github.com) (it happens to many people). I review any flagged posts daily and unhide the ones not actually spam, no need to worry!

1 Like

Hey @ylluminarious many thanks for your reply and fix to the code. Unfortunately I’m presented with a different error when running the hipmost private list command.

/Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:30:in block (2 levels) in direct_channels': undefined method ’ for nil:NilClass (NoMethodError)
from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:23:in map' from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:23:in block in direct_channels’
from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:21:in each' from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:21:in flat_map’
from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost/hipchat.rb:21:in direct_channels' from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/private.rb:24:in list’
from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost/cmds/private.rb:19:in run' from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/lib/hipmost.rb:14:in run’
from /Library/Ruby/Gems/2.3.0/gems/hipmost-1.0/exe/hipmost:48:in &lt;top (required)&gt;' from /usr/local/bin/hipmost:22:in load’
from /usr/local/bin/hipmost:22:in `<main>’

What kind of sample set would you need? Depending, I’ll could try and wade through some files and check the content.

@ylluminarious Also getting the same error when using hipmost public import room team:channel as @Niels
I do get a successful .jsonl file created when using hipmost private import

However, that jsonl file fails when using Mattermost’s bulk import procedure.
image

Thanks so much for clearing that up and explaining what’s going on!

@Sergiy Thanks! Feel free to ask any questions you may have, or file an issue at our repository.

Niels, Thanks for posting that new error message. Unfortunately, now it’s even harder for me to determine what’s causing the problem without some sample data.

What kind of sample set would you need?

Anything which produces both errors you’re seeing would be helpful. The more, the better. Sadly, these errors are hard for me to decipher from afar esp. since I’m doing this in my spare time. If you want to contact me privately, please send me a PM or email me at georgedp at orbitalimpact dot com.


@Klee I suggest also that you provide some sample data. Errors with the Mattermost validator are especially hard for me to analyze without any context. I realize this may be inconvenient so I apologize. Unfortunately, I have reached my reply limit for this thread since I am a new user. As such, I am starting a new thread here for Hipmost support issues. Please post questions there or on our issue tracker on our repository.