We're Hiring!

Mattermost, Inc.

Failed to install marketplace plugin

Summary

Hi everyone, I have trouble installing plugins on my Mattermost instance. I’ve noticed this during the migration of my system-installed Mattermost instance to a dockerized installation, but the problem also appear with my existing system-installed Mattermost instance. I’ve also tried to boot up a new instance from scratch (new database, no data, fresh config.json) but I have the same problem.
The instance is exposed through my system-installed nginx service. Since this Mattermost instance is for on-premises purpose only, we don’t serve over https, so the nginx config is quite simple.

Steps to reproduce

I am using the Team version 5.33 (mattermost/mattermost-team-edition:5.33.0 to be precise).
Once I’m logged in the web UI, I navigate to the side menu, Plugin Marketplace, and try to install any plugins.
This is my nginx configuration:

upstream backendmm {
server 127.0.0.1:9065;
keepalive 32;
}

proxy_cache_path /var/cache/nginxmm2 levels=1:2 keys_zone=mattermost_cache2:10m max_size=3g inactive=120m use_temp_path=off;

server {
listen 80;
server_name mm2.tahiti-web-management.com;

location ~ /api/v[0-9]+/(users/)?websocket$ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
client_max_body_size 50M;
proxy_set_header Host $http_host;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
client_body_timeout 60;
send_timeout 300;
lingering_timeout 5;
proxy_connect_timeout 90;
proxy_send_timeout 300;
proxy_read_timeout 90s;
proxy_pass http://backendmm;
}

location / {
client_max_body_size 50M;
proxy_set_header Connection “”;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache2;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_http_version 1.1;
proxy_pass http://backendmm;
}
}

Expected behavior

I expect the plugin to be installed.

Observed behavior

Instead, I have the following error:

In the logs, I also have this:

mattermost | {“level”:“debug”,“ts”:1617239523.6523678,“caller”:“web/handlers.go:104”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:"/api/v4/system/ping",“request_id”:“tjzcygjdw7gb9jijkt578t3wwc”,“host”:“localhost:8065”,“scheme”:"",“status_code”:“200”}
mattermost | {“level”:“debug”,“ts”:1617239526.270383,“caller”:“web/handlers.go:104”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:"/api/v4/users/status/ids",“request_id”:“o3atygfj3jr8fnmwxzn66yfm9r”,“host”:“mm2.tahiti-web-management.com”,“scheme”:“http”,“status_code”:“200”}
mattermost | {“level”:“error”,“ts”:1617239546.510492,“caller”:“mlog/log.go:232”,“msg”:“Failed to install marketplace plugin.”,“path”:"/api/v4/plugins/marketplace",“request_id”:“i8hwq4h9fbrj9y77j9ug7sjkdw”,“ip_addr”:“192.168.128.1”,“user_id”:“pyxfzhyke38bdxr9rrgjq7xyza”,“method”:“POST”,“err_where”:“InstallMarketplacePlugin”,“http_code”:500,“err_details”:“download failed after multiple retries.: failed to fetch from https://plugins-store.test.mattermost.com/release/mattermost-plugin-dice-roller-v3.1.1-linux-amd64.tar.gz: Get “https://plugins-store.test.mattermost.com/release/mattermost-plugin-dice-roller-v3.1.1-linux-amd64.tar.gz”: dial tcp: i/o timeout”}
mattermost | {“level”:“debug”,“ts”:1617239546.5117588,“caller”:“web/handlers.go:104”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:"/api/v4/plugins/marketplace",“request_id”:“i8hwq4h9fbrj9y77j9ug7sjkdw”,“host”:“mm2.tahiti-web-management.com”,“scheme”:“http”,“status_code”:“500”}
mattermost | {“level”:“debug”,“ts”:1617239546.9635653,“caller”:“web/handlers.go:104”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:"/api/v4/channels/members/me/view",“request_id”:“78inqfebt3y95g8yzdiy7j8yie”,“host”:“mm2.tahiti-web-management.com”,“scheme”:“http”,“status_code”:“200”}

Note that I can fetch the referenced plugin (https://plugins-store.test.mattermost.com/release/mattermost-plugin-dice-roller-v3.1.1-linux-amd64.tar.gz) by execing into the container.

Your Mattermost server can connect to the marketplace server: TCP I/O timeout.
You can try to curl or wget the url on the server to break down the issue, but I suggest: Firewall/Security.

Best
JP

2 Likes

I have the same problem, but firewall disabled, running without container technology. Tested manually with curl and wget to the target url and it always worked, could use some help to dig deeper.

I enabled the DEBUG log but I didn’t get any related log about the plugin update, it only show up with “error” level log, no other context. (Mattermost Version: 5.33.2)

{"level":"error","ts":1617806043.7547944,"caller":"mlog/log.go:232","msg":"Failed to install marketplace plugin.","path":"/api/v4/plugins/marketplace","request_id":"48andmat9id19poa9k9pp5nd8r","ip_addr":"IP_ADDR","user_id":"xrtamaq6mbyh8xoi978cd36xio","method":"POST","err_where":"InstallMarketplacePlugin","http_code":500,"err_details":"download failed after multiple retries.: failed to fetch from https://plugins-store.test.mattermost.com/release/mattermost-plugin-dice-roller-v3.1.1-linux-amd64.tar.gz: Get \"https://plugins-store.test.mattermost.com/release/mattermost-plugin-dice-roller-v3.1.1-linux-amd64.tar.gz\": dial tcp: i/o timeout"}

I can curl the plugins from a shell session inside the container without problems. I’ve noticed plugins do get installed when I try to install multiple plugins at once, and usually the first once fail and they others manage to install successfully.
It might be a problem with our infrastructure network policies but I can’t verify.
There’s at least 2 workarounds:

  • curl from inside the containers to the plugins folder
  • install multiple plugins all at once

I would also suggest checking your individual web server error.log files, in which the information related to the 500 HTTP error code responses should be logged. This information should be extremely helpful.

Thanks for your answer, I did try to look at my nginx server (it’s actually acting as a reverse proxy) and there wasn’t any errors. Actually error 500 would mean something wrong happened on the server side, and that would be (based on my previous log trace):

https://plugins-store.test.mattermost.com

Well, somewhat. An HTTP response with a status code between 400 and 500 is throwing an error, and since there is a reverse proxy involved, it is entirely possible that the server we’re running in this case could be having an issue.

The other reason I say this is because simply copying and pasting this URL into a browser and navigating to it will download the file, which tells me that it is not a issue with the file server.

Also:

actually I fixed my problem … it’s just caused by a slow dns resolver, and mattermost(or the framework, library behind it) can’t tolerate the latency of it, after change to use a faster resolver, everything works pretty well now