Mattermost, Inc.

Mattermost default plugins not visible after upgrade

Hi,
I tried to update our Mattermost production istance on Centos 7.6 (in VMWare environment) passing from 4.5 version to 5.16.3 following ufficial upgrade guide:

https://docs.mattermost.com/administration/upgrade.html

The upgrade itself works, but prepackaged plugins are not visible from system console, neither enabling or disabling plugins selecting right flag and saving…
I tried to googling for a solution and do the upgrade progress twice on a test environment clone of the production, but the result is always the same.
Could you please help me find a solution?

Looking at “server logs” into system console I found the message “Plugins have been disabled”:

{“level”:“info”,“ts”:1574068513.593584,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.5997508,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.8368235,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.838884,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.8433416,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.8435428,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.844091,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“error”,“ts”:1574068514.0615156,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:”/api/v4/plugins/webapp”,“request_id”:“7jegaxs7r7n8fkwp81fmps695e”,“ip_addr”:“172.16.20.99”,“user_id”:””,“method”:“GET”,“err_where”:“GetActivePluginManifests”,“http_code”:501,“err_details”:””}
{“level”:“error”,“ts”:1574068528.7409122,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:”/api/v4/plugins”,“request_id”:“nr6fmg8e5fr79e7oiscnkyg35r”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“GET”,“err_where”:“GetPlugins”,“http_code”:501,“err_details”:””}
{“level”:“error”,“ts”:1574068538.5922909,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:”/api/v4/plugins/statuses”,“request_id”:“87cw6mu5cp8dtfiw79u86ahxea”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“GET”,“err_where”:“GetPluginStatuses”,“http_code”:501,“err_details”:””}
{“level”:“error”,“ts”:1574069253.54626,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:"/api/v4/plugins/statuses",“request_id”:“6a44f8zcgjfspdm45174knpe4y”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“GET”,“err_where”:“GetPluginStatuses”,“http_code”:501,“err_details”:""}
{“level”:“error”,“ts”:1574069256.8254158,“caller”:“mlog/log.go:174”,“msg”:“An error occurred saving the configuration”,“path”:"/api/v4/config",“request_id”:“ny14r75mhiydtdzmsjeui4iuco”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“PUT”,“err_where”:“saveConfig”,“http_code”:500,“err_details”:“failed to persist: failed to write file: open /data/mattermost/versions/mattermost-4.5.0/config/config.json: permission denied”}
{“level”:“error”,“ts”:1574069260.660455,“caller”:“mlog/log.go:174”,“msg”:“An error occurred saving the configuration”,“path”:"/api/v4/config",“request_id”:“jt3foiw9n7d1d8neixy5cwjpxe”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“PUT”,“err_where”:“saveConfig”,“http_code”:500,“err_details”:“failed to persist: failed to write file: open /data/mattermost/versions/mattermost-4.5.0/config/config.json: permission denied”}
{“level”:“error”,“ts”:1574069274.0038836,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:"/api/v4/plugins/statuses",“request_id”:“qt7iiqtuktgtjb4jeqgfs533sa”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“GET”,“err_where”:“GetPluginStatuses”,“http_code”:501,“err_details”:""}

Thanks in advance
Marco

@mbarbaro Can you help with your settings from here https://docs.mattermost.com/administration/config-settings.html#plugin-management?

Hi @amy.blais,
here is a screenshot from our upgraded Mattermost instance.
Is that what you asked for or do you need to see an extract from our config.json ?
Thanks
Marco

Where does the link to the marketplace take you in the help text under Enable Marketplace when you click it?

Hi,
this is the link:
https://mattermost.com/pl/default-mattermost-marketplace.html
Thanks again,
Marco Barbaro
Direzione Sistemi Informativi

PAM Panorama S.P.A.

tel. 041-5495104

  • marco_barbaro@gruppopam.it

Da:
“Amy Blais via
Mattermost Inc.” mattermost@discoursemail.com

Per:
marco_barbaro@gruppopam.it

Data:
19/11/2019 02:19

Oggetto:
[Mattermost
Forum] [Trouble Shooting] Mattermost default plugins not visible after
upgrade

Hi Amy, do you have any hints to suggest me to solve the issue?
Was Mattermost 4.5 version too old to upgrade directly to 5.16.3 version? Should I have to upgrade to an intermediate version first?

Thanks again
Best Regards
Marco

@mbarbaro Can you help share your config.json settings related to plugins?

Sure @amy.blais here it is:

"PluginSettings": {
    "Enable": true,
    "EnableUploads": true,
    "Directory": "./plugins",
    "ClientDirectory": "./client/plugins",
    "Plugins": {},
    "PluginStates": {
        "jira": {
            "Enable": true
        }
    }
 }

Thanks
Marco

@mbarbaro, in your logs there’s one item that stands out:

{“level”:“error”,“ts”:1574069260.660455,“caller”:“mlog/log.go:174”,“msg”:“An error occurred saving the configuration”,“path”:"/api/v4/config",“request_id”:“jt3foiw9n7d1d8neixy5cwjpxe”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“PUT”,“err_where”:“saveConfig”,“http_code”:500,“err_details”:“failed to persist: failed to write file: open /data/mattermost/versions/mattermost-4.5.0/config/config.json: permission denied”} 

This sounds like the root of the issue. Is the config pointing back at the old 4.5.0 spot, but doesn’t have permission to update it?

Hi @jesse , thanks for your hint.
I modified config.json as suggested
and then I notice a new log error as if “client/plugins” dir
was not present even if it was into mattermost dir.
So I read about some threads highlighting
that mattermost was searching for “client” dir into “mattermost/bin”
dir and decided to try to create another “client” dir inside
“bin” and tried to restart mattermost service but it stopped
to work.
Looking at logs it was trying to find
for “root.html” file into “mattermost/bin/client” dir
!
I copied “root.html” from
mattermost/client into mattermost/bin/client and suddenly I was able to
enable plugins into Mattermost!
What do you think about this? Why Mattermost
was looking for “client” dir into “bin” dir in your
opinion?
Thanks again.
Best regards.
Marco Barbaro
Direzione Sistemi Informativi

PAM Panorama S.P.A.

tel. 041-5495104

  • marco_barbaro@gruppopam.it

Da:
“Jesse Hallam
via Mattermost Inc.” mattermost@discoursemail.com

Per:
marco_barbaro@gruppopam.it

Data:
21/11/2019 17:36

Oggetto:
[Mattermost
Forum] [Trouble Shooting] Mattermost default plugins not visible after
upgrade

Hi @jesse and @amy.blais is there a way to set Mattermost “working.dir”?
As I supposed, looking deeper at the product logs, the istance working dir starts from “bin”:

{“level”:“info”,“ts”:1574413783.6823297,“caller”:“app/server.go:219”,“msg”:“Current working directory is /data/mattermost/versions/mattermost-4.5.0/bin”}

Perhaps this is the start of the issues I had with plugins.
Could you please help me modifying that setting?
Thanks
Marco

Hey @mbarbaro – the working directory should just be the directory from which the mattermost binary is invoke. For example, if you’re in $HOME and you invoke the binary manually, the working directory will match $HOME.

Depending on your operating system and launching mechanism, there are ways of defining that directory before starting mattermost. For example, systemd uses WorkingDirectory as a configuration. Perhaps you can tune that?