Mattermost Peer-to-Peer Forum

[SOLVED] Server started but app not connecting and no logs


#1

I have reinstalled my server, downloaded latest mattermost, restored old DB and Data folder. Mattermost started for some hours and stopped responding

I rebooted the server and it started again, but no log… next day it stopped working again. So now its a daily routine for me to come in office and check if it is running. Then i go to server and check mattermost port is busy.

I did curl localhost:8065 and got response We're having trouble connecting to Mattermost. If refreshing this page (Ctrl+R or Command+R) does not work, please verify that your computer is connected to the internet.

I do have to reboot my server inorder to make it work… any idea how can I debug this


#2

I am on version 5.2 (latest one) I checked my configuration and loging was disabled. I have enabled it and now I can see the following…

$ sudo -u mattermost ./bin/mattermost 
2018-12-31T05:44:40.828Z        info    utils/i18n.go:78        Loaded system translations for 'en' from '/home/admin/mattermost/i18n/en.json'
2018-12-31T05:44:40.828Z        info    app/app.go:143  Server is initializing...
2018-12-31T05:44:40.830Z        info    sqlstore/supplier.go:207        Pinging SQL master database
2018-12-31T05:44:41.287Z        info    commands/server.go:82   Current version is 5.6.0 (5.6.2/Sat Dec 22 14:58:39 UTC 2018/3aed6efc6bb5a67fdb8237aac38d46fab3266e8d/2c2d33b22b4c2938938eade4b9914748c1be9d3b)
2018-12-31T05:44:41.287Z        info    commands/server.go:83   Enterprise Enabled: true
2018-12-31T05:44:41.287Z        info    commands/server.go:84   Current working directory is /home/admin/mattermost
2018-12-31T05:44:41.287Z        info    commands/server.go:85   Loaded config file from /home/admin/mattermost/config/config.json
2018-12-31T05:44:41.287Z        info    filesstore/localstore.go:33     Able to write files to local storage.
2018-12-31T05:44:41.288Z        info    app/license.go:44       License key from https://mattermost.com required to unlock enterprise features.
2018-12-31T05:44:41.289Z        info    app/plugin.go:130       Starting up plugins
2018-12-31T05:44:42.116Z        info    app/server.go:168       Starting Server...
2018-12-31T05:44:42.116Z        info    app/server.go:228       Server is listening on [::]:8065
2018-12-31T05:44:42.124Z        info    app/web_hub.go:75       Starting 4 websocket hubs
2018-12-31T05:44:42.131Z        info    sqlstore/post_store.go:1286     Post.Message supports at most 16383 characters (65535 bytes)
2018-12-31T05:44:42.166Z        info    jobs/workers.go:68      Starting workers
2018-12-31T05:44:42.166Z        info    jobs/schedulers.go:72   Starting schedulers.

But it just stays there and nothing happens.

Edit

After enabling DB log as well, i can see lots of queries running after starting scheduler… and then it seems mattermost is working now. will check again later when it stops working


#3

Thanks @aalasolutions for the report.

After enabling DB log as well, i can see lots of queries running after starting scheduler… and then it seems mattermost is working now. will check again later when it stops working

That’s a good plan. Moreover, I’d recommend you set your file log level (in System Console > General > Logging) to DEBUG, which will print high detail logging to debug issues.

https://docs.mattermost.com/administration/config-settings.html#file-log-level


#4

My server is stopped again now :slight_smile: here is the log file http://aalasolutions.com/mattermost.log I restarted mattermost again, and it keeps printing following things in output

sql-trace:07:14:45.266386 SELECT
                            *
                    FROM
                            Jobs
                    WHERE
                            Status = ?
                    ORDER BY
                            CreateAt ASC [1:"pending"] (642.541µs)
2019-01-01T07:14:59.060Z        debug   app/email_batching.go:110       Email batching job ran. 0 user(s) still have notifications pending.
sql-trace:07:15:00.267361 SELECT
                            *
                    FROM
                            Jobs
                    WHERE
                            Status = ?
                    ORDER BY
                            CreateAt ASC [1:"pending"] (621.455µs)
sql-trace:07:15:15.268345 SELECT
                            *
                    FROM
                            Jobs
                    WHERE
                            Status = ?
                    ORDER BY
                            CreateAt ASC [1:"pending"] (660.828µs)
2019-01-01T07:15:29.060Z        debug   app/email_batching.go:110       Email batching job ran. 0 user(s) still have notifications pending.
sql-trace:07:15:30.269230 SELECT
                            *
                    FROM
                            Jobs
                    WHERE
                            Status = ?
                    ORDER BY
                            CreateAt ASC [1:"pending"] (517.68µs)
sql-trace:07:15:45.270106 SELECT
                            *
                    FROM
                            Jobs
                    WHERE
                            Status = ?
                    ORDER BY
                            CreateAt ASC [1:"pending"] (551.333µs)
2019-01-01T07:15:59.060Z        debug   app/email_batching.go:110       Email batching job ran. 0 user(s) still have notifications pending.
sql-trace:07:16:00.270984 SELECT
                            *
                    FROM
                            Jobs
                    WHERE
                            Status = ?
                    ORDER BY
                            CreateAt ASC [1:"pending"] (543.703µs)

If I do curl localhost:8065 I see We're having trouble connecting to Mattermost


#5

after couple of reboots, it started to work again. This is latest log http://aalasolutions.com/mattermost_2.log


#6

I have reached the bottom of the issue. Issue is not with the reboots, and mattermost it self is working fine. I figured out that as soon as I logged in to Hosting Control Panel, my mattermost started working. It looks like my hosting panel is blocking direct access from app to the port.

I checked the firewall settings and I saw that my IP is in temporary allowed for 24 hours section. So after 24 hours it will stop working again.

Going to post this on hosting panel issue now…


#7

Glad you were able to narrow down the cause. Thanks @aalasolutions for posting it back, much appreciated :+1: