Plugin marketplace does not load

Troubleshooting
1

Summary

Loading the plugin marketplace fails (has never worked)

Steps to reproduce

Load the plugin marketplace

Expected behavior

Plugin marketplace to load successfully.

Observed behavior

What did you see happen? Please include relevant error messages and/or screenshots.
Receive the following error
{"level":"error","ts":1588488299.8669755,"caller":"mlog/log.go:175","msg":"Error saving plugin state in config.","path":"/api/v4/plugins/marketplace","request_id":"redacted","ip_addr":"redacted","user_id":"redacted","method":"GET","err_where":"getRemotePlugins","http_code":500,"err_details":""}

Environment:
Ubuntu 16.04.4
Postgresql 9.5
nginx (reverse proxy for mattermost)
No special customization here, just followed directions to set this up four years ago and keep it patched (5.22.1)

Tried to upload a screenshot, but can’t for some reason. Error displayed in the webgui is

Error connecting to the marketplace server. Please check your settings in the System Console.

2

What are your System Console settings set to for Plugins?

3

Hi, @bediaa

As mentioned by @amy.blais earlier, the first step is to understand the current configuration of the Plugins page.

image
image2870×1638 258 KB

  • Can you provide the screen shot of the page either via System Console or from the config.json?

image
image1378×1628 227 KB

For config.json, this is the example command:

ahmaddanial@mattermost:~$ cat /opt/mattermost/config/config.json | grep -A19 "PluginSettings"
    "PluginSettings": {
        "Enable": true,
        "EnableUploads": true,
        "AllowInsecureDownloadUrl": false,
        "EnableHealthCheck": true,
        "Directory": "./plugins",
        "ClientDirectory": "./client/plugins",
        "Plugins": {},
        "PluginStates": {
            "com.mattermost.nps": {
                "Enable": true
            }
        },
        "EnableMarketplace": true,
        "EnableRemoteMarketplace": true,
        "AutomaticPrepackagedPlugins": true,
        "RequirePluginSignature": false,
        "MarketplaceUrl": "https://api.integrations.mattermost.com",
        "SignaturePublicKeyFiles": []
    },
  • What is the error that you are getting when you tried to click on the Plugin Marketplace from the list?

image
image618×1388 87.2 KB

  • Can you also confirm the permission that you have set on the plugins and prepackaged_plugins are set to the mattermost user? For example:
root@mattermost:/home# ls -lAh /opt/mattermost/plugins/ /opt/mattermost/prepackaged_plugins/
/opt/mattermost/plugins/:
total 4.0K
drwxr--r-- 5 mattermost mattermost 4.0K May  5 00:43 com.mattermost.nps

/opt/mattermost/prepackaged_plugins/:
total 99M
-rw-rw-r-- 1 mattermost mattermost 8.7M Apr 23 14:58 mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 8.7M Apr 23 14:58 mattermost-plugin-autolink-v1.1.2-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-autolink-v1.1.2-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 8.8M Apr 23 14:58 mattermost-plugin-aws-SNS-v1.0.2-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-aws-SNS-v1.0.2-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 8.8M Apr 23 14:58 mattermost-plugin-custom-attributes-v1.0.2-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-custom-attributes-v1.0.2-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 9.7M Apr 23 14:58 mattermost-plugin-github-v0.11.0-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-github-v0.11.0-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 9.3M Apr 23 14:58 mattermost-plugin-gitlab-v1.0.1-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-gitlab-v1.0.1-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 9.0M Apr 23 14:58 mattermost-plugin-jenkins-v1.0.0-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-jenkins-v1.0.0-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost  10M Apr 23 14:58 mattermost-plugin-jira-v2.3.2-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-jira-v2.3.2-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 8.7M Apr 23 14:58 mattermost-plugin-nps-v1.0.3-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-nps-v1.0.3-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 8.6M Apr 23 14:58 mattermost-plugin-welcomebot-v1.1.1-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-welcomebot-v1.1.1-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 mattermost mattermost 8.9M Apr 23 14:58 mattermost-plugin-zoom-v1.3.0-linux-amd64.tar.gz
-rw-rw-r-- 1 mattermost mattermost  566 Apr 23 14:58 mattermost-plugin-zoom-v1.3.0-linux-amd64.tar.gz.sig
4

I think these may answer your questions

1189×1057 

$ grep -A19 "PluginSettings" config.json
    "PluginSettings": {
        "Enable": true,
        "EnableUploads": false,
        "AllowInsecureDownloadUrl": false,
        "EnableHealthCheck": true,
        "Directory": "./plugins",
        "ClientDirectory": "./client/plugins",
        "Plugins": {},
        "PluginStates": {
            "com.mattermost.nps": {
                "Enable": true
            }
        },
        "EnableMarketplace": true,
        "EnableRemoteMarketplace": true,
        "AutomaticPrepackagedPlugins": true,
        "RequirePluginSignature": false,
        "MarketplaceUrl": "https://api.integrations.mattermost.com",
        "SignaturePublicKeyFiles": []
    },

No plugins folder, but do have prepackaged. I execute mattermost under user ‘bediaa’

bediaa@toracon:~/mattermost$ ls -lAh prepackaged_plugins/ plugins/
ls: cannot access 'plugins/': No such file or directory
prepackaged_plugins/:
total 498M
-rw-rw-r-- 1 bediaa bediaa  25M Dec 18  2018 jira-1.0.3.tar.gz
-rw-rw-r-- 1 bediaa bediaa  25M Jan  6  2019 jira-1.0.4.tar.gz
-rw-rw-r-- 1 bediaa bediaa  25M Jan 23  2019 jira-1.1.0.tar.gz
-rw-r--r-- 1 bediaa bediaa  25M Feb  9  2019 jira-1.1.1.tar.gz
-rw-rw-r-- 1 bediaa bediaa 5.2M Jul 24  2018 mattermost-jira-plugin-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.6M Jan 21 18:32 mattermost-plugin-antivirus-v0.1.1.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.7M Apr 23 10:58 mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 8.5M Sep 16  2019 mattermost-plugin-autolink-v1.0.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.6M Jan 21 18:32 mattermost-plugin-autolink-v1.1.1.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.7M Apr 23 10:58 mattermost-plugin-autolink-v1.1.2-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-autolink-v1.1.2-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 8.8M Jul 19  2019 mattermost-plugin-aws-SNS-v1.0.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.8M Apr 23 10:58 mattermost-plugin-aws-SNS-v1.0.2-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-aws-SNS-v1.0.2-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 8.8M Jan 21 18:32 mattermost-plugin-aws-SNS-v1.0.2.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.9M Jul 19  2019 mattermost-plugin-custom-attributes-v1.0.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.9M Sep 16  2019 mattermost-plugin-custom-attributes-v1.0.1.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.8M Apr 23 10:58 mattermost-plugin-custom-attributes-v1.0.2-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-custom-attributes-v1.0.2-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 8.9M Jan 21 18:32 mattermost-plugin-custom-attributes-v1.0.2.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.4M Sep 16  2019 mattermost-plugin-github-v0.10.2.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.7M Apr 23 10:58 mattermost-plugin-github-v0.11.0-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-github-v0.11.0-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 9.7M Jan 21 18:32 mattermost-plugin-github-v0.11.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.3M Sep 16  2019 mattermost-plugin-gitlab-v1.0.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.3M Apr 23 10:58 mattermost-plugin-gitlab-v1.0.1-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-gitlab-v1.0.1-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 9.4M Jan 21 18:32 mattermost-plugin-gitlab-v1.0.1.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.0M Apr 23 10:58 mattermost-plugin-jenkins-v1.0.0-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-jenkins-v1.0.0-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 9.0M Jan 21 18:32 mattermost-plugin-jenkins-v1.0.0.tar.gz
-rw-r--r-- 1 bediaa bediaa  11M Jul 16  2019 mattermost-plugin-jira-v2.0.6.tar.gz
-rw-r--r-- 1 bediaa bediaa  11M Jul 19  2019 mattermost-plugin-jira-v2.0.7.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.8M Aug 30  2019 mattermost-plugin-jira-v2.1.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.8M Sep 16  2019 mattermost-plugin-jira-v2.1.3.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.7M Jan 21 18:32 mattermost-plugin-jira-v2.2.2.tar.gz
-rw-r--r-- 1 bediaa bediaa  10M Apr 23 10:58 mattermost-plugin-jira-v2.3.2-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-jira-v2.3.2-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 8.7M Jun 15  2019 mattermost-plugin-nps-v1.0.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.7M Jul 19  2019 mattermost-plugin-nps-v1.0.1.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.7M Apr 23 10:58 mattermost-plugin-nps-v1.0.3-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-nps-v1.0.3-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 8.7M Jan 21 18:32 mattermost-plugin-nps-v1.0.3.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.6M Jul 19  2019 mattermost-plugin-welcomebot-v1.0.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.6M Sep 16  2019 mattermost-plugin-welcomebot-v1.1.0.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.6M Apr 23 10:58 mattermost-plugin-welcomebot-v1.1.1-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-welcomebot-v1.1.1-linux-amd64.tar.gz.sig
-rw-r--r-- 1 bediaa bediaa 8.6M Jan 21 18:32 mattermost-plugin-welcomebot-v1.1.1.tar.gz
-rw-r--r-- 1 bediaa bediaa 9.0M Sep 16  2019 mattermost-plugin-zoom-v1.0.7.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.7M Nov  6 09:06 mattermost-plugin-zoom-v1.1.1.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.8M Jan 21 18:32 mattermost-plugin-zoom-v1.1.2.tar.gz
-rw-r--r-- 1 bediaa bediaa 8.9M Apr 23 10:58 mattermost-plugin-zoom-v1.3.0-linux-amd64.tar.gz
-rw-r--r-- 1 bediaa bediaa  566 Apr 23 10:58 mattermost-plugin-zoom-v1.3.0-linux-amd64.tar.gz.sig
-rw-rw-r-- 1 bediaa bediaa 5.3M Jul 30  2018 mattermost-zoom-plugin-linux-amd64.tar.gz
-rw-rw-r-- 1 bediaa bediaa  24M Dec 13  2018 zoom-1.0.3.tar.gz
-rw-r--r-- 1 bediaa bediaa  24M Jan 31  2019 zoom-1.0.4.tar.gz
5

Error I get when navigating to the plugin marketplace

907×449

6

Just wanted to bump to see if anyone had any ideas? The error output doesn’t really give any hints on what the problem may be. All I get is a 500 error on command line debug as mentioned in original post.

7

Hi, @bediaa

Based on the statement here:

Does that mean that you were running on previous version prior to 5.22.1? If yes, what version was that?

I am trying to understand why the plugins directory is not available when you ran ls -lAh. Did you run it on root?

Since you are running mattermost under user bediaa, can you please provide the output of ls -lAh /opt/mattermost itself?

I want to make sure that the user bediaa has permission across all directories. If any of them are not under the bediaa, please run the following command:

sudo chown -R bediaa:bediaa /opt/mattermost/

Once done, please check if the error still persists.

8

Does that mean that you were running on previous version prior to 5.22.1? If yes, what version was that?

I don’t remember, but probably 5.20. I’ve kept it patched consistently since starting it in 2016. Almost every patch has been applied as they came out (sometimes missed a point release here and there).

I am trying to understand why the plugins directory is not available when you ran ls -lAh. Did you run it on root ?

The only user that has ever touched files in /home/bediaa/mattermost has been bediaa. All files are owned by bediaa. I ran sudo chown -R bediaa:bediaa /home/bediaa/mattermost and no change.

9

Bumping for visibility. Any ideas?

10

Hi, @bediaa

I am just trying to narrow down the potential root cause further. Is it possible for you to connect directly to Mattermost without the NGINX proxy and verify if the same issue persist?

If it connects successfully, let’s take a look at the NGINX configuration. There might be something that is misconfigured there based on Configuring NGINX as a proxy for Mattermost Server documentation.

11

Hmm, that’s a good idea. I did a SSH tunnel for 8065 to localhost and was able to login to the server completely unencrypted (bypassing nginx) using my local web browser (latest Firefox for Windows). Unfortunately, no change in behavior when trying to load the marketplace. Same error message in the webgui.

12

I had the same issue, but was able to fix it by adding:

"AllowedUntrustedInternalConnections": "api.integrations.mattermost.com" to config.js

I was now able to see all plugins from the marketplace, but got an error message when installing.
(the logs showed "err_where":"installExtractedPlugin","http_code":500,"err_details":"unable to remove old webapp bundle directory: client/plugins/jitsi: unlinkat client/plugins/jitsi: read-only file system"

So I also had to change the following settings in config.json from:

  "PluginSettings": {
    "Enable": true,
    "EnableUploads": false,
    "AllowInsecureDownloadUrl": false,
    "EnableHealthCheck": true,
    "Directory": "/var/lib/mattermost/",
    "ClientDirectory": "./client/plugins/",
    "Plugins": {},
    "PluginStates": {
      "com.mattermost.nps": {
        "Enable": true
      }
    },
    "EnableMarketplace": true,
    "EnableRemoteMarketplace": true,
    "AutomaticPrepackagedPlugins": true,
    "RequirePluginSignature": false,
    "MarketplaceUrl": "https://api.integrations.mattermost.com",
    "SignaturePublicKeyFiles": []
  },

to:

  "PluginSettings": {
    "Enable": true,
    "EnableUploads": false,
    "AllowInsecureDownloadUrl": false,
    "EnableHealthCheck": true,
    "Directory": "/var/lib/mattermost/plugins/",
    "ClientDirectory": "/var/lib/mattermost/client/plugins/",
    "Plugins": {},
    "PluginStates": {
      "com.mattermost.nps": {
        "Enable": true
      }
    },
    "EnableMarketplace": true,
    "EnableRemoteMarketplace": true,
    "AutomaticPrepackagedPlugins": true,
    "RequirePluginSignature": false,
    "MarketplaceUrl": "https://api.integrations.mattermost.com",
    "SignaturePublicKeyFiles": []
  },

After those 2 changes, I am now able to install and use plugins

2 Likes
13

zauberfisch, Where did you add that line to config.json? When I added it, mattermost failed to start until I took it back out.

14

AllowedUntrustedInternalConnections << this property is located in “ServiceSettings”. maybe default value is “”.