Mattermost, Inc.

"Processing" during image upload

I have a mattermost instance running on a Pi3 and so far am really impressed. There is only one issue that would prevent adoption. Bear with me - I know that ARM isn’t strictly supported and a 1Gb box is small, but the same box runs Matrix, SuiteCRM and Nextcloud without breaking a sweat.

My issue is this - I share a jpg image and as soon as the size exceeds (around) 500k, the upload looks complete (in terms of progress bar) but I then get a popup which says “Processing”. It will say that for up to 3-5 minutes depending on image size while all 4 cores show pegged at 100% CPU.

My question is - what is it doing while “Processing” - if I look at Nextcloud and Matrix which are doing basically the same tasks of copying the file to local storage and generating a small thumbnail plus some DB indexing, I just don’t understand what it being done in Mattermost that doesn’t seem to be required in the other apps?

I have trawled issues and posts and the most common responses were to the two caveats I mentioned at the top. I accept those but would dearly like to understand what is driving this behaviour.

Thanks

Mike

1 Like

Also - clearly not related to images per se which was my original assumption. I just uploaded a 10mb SQL dump - that reached 89% then stopped and also stopped my server…

I’ve seen comments about loading the entire file into ram but even if that’s correct - 10mb is neither here nor there.

I’m going to try and spin up a 1G instance on an x86-64 architecture at some point and see what happens (perfectly prepared to accept it’s related to ARM arch) but would still appreciate a dev or someone who knows what “processing” means.

Thanks

Mike

Ok… I set up a 64bit Virtualbox instance with 1gig and the system now works as expected.

It’s a shame it won’t run on the Pi but it seems not.

I’m still curious what happens during “processing” because I see it on the VM, it just doesn’t hang the box…

1 Like

There are two things I am aware that could cause it to hang for awhile in “Processing”.

  1. We do some image processing.
  2. Plugins can do their own processing on file uploads eg. https://github.com/mattermost/mattermost-plugin-antivirus

Is the Pi3 hanging because of memory pressure? If it’s using swap space that could cause the CPU to max out. It’s possible that we use more memory then the file size.

Hi

It’s possible to be memory pressure although the x64 VM I am now running is the same 1gb and that never gets over 4-500 mb real… I have encountered occasional swap usage but tested by clearing that (swapoff -a / swapon -a) and then uploading an image. Still stuck and even with small images, the wait seems excessive (10-15 secs).

I guess I’m just curious regarding what processing is required other than thumbnail(s)? I don’t have any plugins enabled on the Pi.

Rgds

I don’t think there is any further processing beyond images and plugins.

Hello, please see here for the details I’ve collected as well, as I’ve run into this as well on a Raspberry Pi 4. My 4GB RAM is far from being full. No swap at all, I have it disabled.

This problem seems to be purely CPU-bound. Not RAM, and not disk-IO-bound. I’m watching what’s happening with both htop, and iotop. I tried migrating Mattermost’s data folder to a good-quality USB 3 stick, to split up the disk IO. That didn’t help at all.

I’m not running any additional Plugins beyond a basic install. No Clam-AV plugin.

I would sure appreciate a radio button somewhere in the System Console to make that image processing very lightweight, or not at all (as in, just attach the file, no matter what the file extension). Then let the web browser itself, or smartphone OS itself (with some Gallery-like app) show an image, using it’s own image-rendering ability (and the user must press the back button to go back to Mattermost).

I did my Mattermost install, using these (docker-avoiding) instructions. I’m on Raspbian Debian 10, with Mattermost 5.21, and MariaDB.

Here’s what htop looks like, during the image processing (and you can see the RAM is just fine, and the swap is off):

It seems the 398% is a summation of the next 4 lines below it…

BTW: Uploading a .jpg just smaller than 1MB required 46 seconds of maxed-out CPU time (just after upload to server, but just before user taps Enter to post), and nothing else running on the Raspberry Pi 4 was competing with Mattermost!

I strongly suspect that this crippling performance hit on a Raspberry Pi 4 has to do with how the NPM library “image-webpack-loader” gets used. It in turn calls “imagemin”. Imagemin performs the “wonderful” service of “Minify images seamlessly”.

It seems Mattermost is transparently compressing the images, before storing them! But formats like .jpg, .gif, and .png are already (internally) compressed. How does that make any sense? Spending all those CPU cycles, just to save like 1% off the file size? Please, say it ain’t so!

Please, can anyone explain how this transparent compression of images can be disabled?

Mattermost devs, could there please be a radio button for this, in the Mattermost System Console, or even a variable in config.json (which I don’t mind hand-editing)?

In image-webpack-loader (see link above), there is an option to “disable” it, but how to gracefully do this disabling, without hand-compiling all of Mattermost from source?

> In your webpack.config.js, add the image-loader, chained after the file-loader:
> 
> rules: [{
>   test: /\.(gif|png|jpe?g|svg)$/i,
>   use: [
>     'file-loader',
>     {
>       loader: 'image-webpack-loader',
>       options: {
>         bypassOnDebug: true, // webpack@1.x
>         disable: true, // webpack@2.x and newer
>       },
>     },
>   ],
> }]

If you don’t believe me (that re-compressing .gifs, .jpgs, and .pngs is a bad idea), then please try compressing a few .gifs, .jpgs and .pngs laying around on your computer for yourself, and you’ll see that I’m not kidding.

.svg, OTOH, is not compressed by default. It makes sense to compress pretty much this image format only, as it just contains a bunch of XML text (which is very compressible).

The test above could intelligently call image-webpack-loader just for the .svgs, but then also disable image-webpack-loader, for the .jpgs, .pngs, and .gifs

Edit:
Yes indeed, in the source tree (v5.21.0), look at mattermost-webapp/webpack.config.js. Compression is turned on for those already-compressed file formats, see starting from line 196:

            {
            test: /\.(png|eot|tiff|svg|woff2|woff|ttf|gif|mp3|jpg)$/,
            use: [
                {
                    loader: 'file-loader',
                    options: {
                        name: 'files/[hash].[ext]',
                    },
                },
                {
                    loader: 'image-webpack-loader',
                    options: {},
                },
            ],
        },

That all makes perfect sense @esbeeb - thank you for doing this and identifying a strong candidate!

Over to Devs to please, please provide what would appear to be a simple patch assuming this pans out…

1 Like

I took a look at each of those formats.

Some are certainly compressed (png gif mp3 jpg eot woff2 woff), some are certainly not compressed (svg ttf), and one is annoyingly sometimes compressed (tiff).

I’m not sure whether it would be wise to one-sidedly compress all tiffs or not. My sensibility is that disk is cheaper than CPU (and people’s time waiting for compression unnecessarily, is the most expensive thing of all).

So I would personally incline towards just storing all tiffs, with the assumption that they are already internally compressed (and not compress them with image-webpack-loader). If a few uncompressed tiffs get through, then oh well, the disk is cheap enough. tiff is a pretty uncommon case these days, so I say it’s no biggie either way.

1 Like

@justinegeffen, what do you think of this attachment-compressing issue, as stated above? I would argue that making instructions to install Mattermost on a Raspberry Pi are well and good, but sort of futile because the end users are likely to get seriously frustrated once the Raspberry Pi gets paralyzed on, say, a 2.8MB image upload.

@esbeeb if you still have your instance available, could you confirm that it is only impacting images for you? I had a similar issue uploading a 10mb SQL backup file. I would have expected no issue but the system locked solid.

That may have been an outlier and I didn’t re-attempt because I aborted the experiment and re-homed on an x64 VM. My preference is still to host on the Pi because that’s extremely efficient to run.

Thanks again

Mike

Hi @esbeeb,

Thanks for raising this, it’s really helpful feedback! @crspeller, do you have any suggestions around this? I found some some discussions around it on our Community server but nothing definitive.

Thanks!

Thanks for your comments, @justinegeffen and @b3lt3r.

I’ve created a user account in the Contributors Community channel myself, saying more there:
https://pre-release.mattermost.com/core/pl/h3sqfoaeriy4tmo5yohuocq7yo

image-webpack-loader and imagemin are only used for static images bundled with the web app (icons, emojis, logos, etc), and they’re only run at compile time. They aren’t used for file uploads, so they wouldn’t have any effect on them.

The processing that’s done on image uploads is to resize them so that they can be displayed in thumbnails in the apps without taking as much bandwidth. For anything other than small images, this is a significant bandwidth savings. That only applies to images though, so if you’re seeing performance issues with other file uploads, there’s something else causing issues here.

1 Like

It turns out I was wrong as to which code is making the Raspberry Pi slow down so much. So my PR is certainly no fix. (removed the link above) Plus, it seems I was wrong to assume that Mattermost is compressing already-compressed image files when a user uploads an image.

The CPU-intensive code is shown here:

I’m grateful to jupenur and @hmhealey for tracking down the relevant image-processing code. What that code does, according to jupenur, is:

"… the explicit image processing we do in file.go on the server [not the webapp]. That would also still line up with the testing you did using .xcf files, because HandleImages is only invoked on files with an image/* MIME type – whereas the MIME type for .xcf is (on many systems) application/x-xcf .

Edit: In fact even more likely the root cause is in postprocessImage in file.go ."

Wild and crazy idea. Could the video hardware, which the RPi4 has, be used to do hardware-accelerated rendering of the images? Is there some easy way to say to the video hardware “please resize this image down to the following thumbnail sizes”?

Are there any video nerds out there?

There are go bindings for SDL.

There is a golang binding for FFMPEG.

The Raspberry Pi foundation is apparently also working on a hardware-accelerated FFMPEG library, for the Raspberry Pi.

That’d be interesting to investigate. I’ve always liked the idea of integrating some sort of video transcoding into the server as well, and that would almost certainly involve FFMPEG.

I’ve asked for a clue, here.

There is a tutorial on how to use the hardware-accelerated SDL library on the Raspberry Pi.