MySQL & Invalid session error on Mattermost

Troubleshooting
1

I updated Mattermost (team) to 5.7.0. Then I got a MySQL error.

mattermost[3293]: [mysql] 2019/01/17 21:44:41 packets.go:36: unexpected EOF
mattermost[3293]: {"level":"error","ts":1547729081.169309,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}
mattermost[3293]: [mysql] 2019/01/17 21:44:56 packets.go:36: unexpected EOF
mattermost[3293]: {"level":"error","ts":1547729096.1711783,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}
mattermost[3293]: [mysql] 2019/01/18 09:25:49 packets.go:36: unexpected EOF
mattermost[3293]: {"level":"error","ts":1547771149.360974,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}
mattermost[3293]: [mysql] 2019/01/18 09:26:04 packets.go:36: unexpected EOF
mattermost[3293]: {"level":"error","ts":1547771164.3638587,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}
mattermost[3293]: [mysql] 2019/01/18 09:26:19 packets.go:36: unexpected EOF
mattermost[3293]: {"level":"error","ts":1547771179.3648214,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}

I seem to be able to use Mattermost even though the above error occurred. It seems that the return after user interruption is slow. A process “connected” is displayed each time.
If I execute “systemctl restart mattermost”, the error is resolved once. I do not know the exact time, but as time goes by, the above error will occur again.

What is the cause of this (Mattermost) MySQL error?

Mattermost 5.7.0, CentOS 7.6, Nginx 1.14.1, PHP 7.3.1, MariaDB 10.2.21

2

Just a suspicion here since I don’t know your mattermost setup, but I get the feeling that there may network issues between your mattermost setup and mariadb. Are your mattermost server and mariadb on separate systems ? If so, you may want to check that there are no packet errors, lost packets, etc, between the systems. sar -n EDEV (as root) may be able to show if there are receiving or transmitting errors.

3

What does this mean?

4

I’m deleting all MySQL tables and deleting the data directory. And installed Mattermost 5.6.3. Even with 5.6.3, the above MySQL error occurred.

Compatibility problem with MariaDB 10.2.21? Or PHP 7.3? Or Redis 5? I have specified Redis in php_session_path.

5

When I run it, everything is “0.00”. Of course it is root.

7

I want to resolve MySQL errors I first wrote and the errors below. What is common is “Invalid Error”.

{"level":"info","ts":1548062112.917261,"caller":"web/handlers.go:99","msg":"Invalid session","error":"GetSession: Invalid session token=qtzwo6srk7ftfdb4k4eoimnsxe, err=<no value>, "}

Where is this a problem?

8

@Babbles, what @jchungvz meant by “Are your mattermost server and mariadb on separate systems ?” is whether your Mattermost instance and database instance are installed in a separate servers. Can you please confirm on that?

The invalid session error you shared looks similar to User gets logged out on page refresh. So, can you share the session related settings here so we can perhaps shed more light here?

9

Thanks for the supplement. I’m Mattermost and MariaDB are the same server. I may be influenced by Plesk’s involvement.

I can not use ‘localhost’ with mySQL DataSource in config.json, I have specified the IP address of the server instead of localhost.

"DataSource": "DB-USERNAME:DBPASSWORD@tcp(XXX.XXX.XX.XXX:3306)/DBNAME?charset=utf8mb4,utf8\u0026readTimeout=30s\u0026writeTimeout=30s"

Invalid Session

Instead of logging out, when I display the login screen of Mattermost surely an INFO level Invalid Session error occurs.

Since yesterday somehow MySQL error I first wrote has become very few.

{"level":"error","ts":1548220002.1904187,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}

This error does not change.
I sent an invitation mail to a new user and had him access it. Then the screen was empty after mail authentication. There were two people like this.

There seems to be no problem in the post itself after logging in. It is done smoothly so far.

10

Is this about that?

 "SessionLengthWebInDays": 30,
        "SessionLengthMobileInDays": 30,
        "SessionLengthSSOInDays": 30,
        "SessionCacheInMinutes": 10,
        "SessionIdleTimeoutInMinutes": 0,

That error has also occurred while the user is posting continuously. Either of the any web browser, PC, Mac, iPhone dedicated applications access method gives an error.

11

Hi @Babbles,

Does your server fulfill the hardware requirements?

12

When I check it from the /proc/cpuinfo command, I have 4 CPUs, each with one core. And the memory is 4GB. I estimate the number of users is about 20 people.

Is this error ignorable even at the error level? If I do not see the log, I can not feel such a big impact.

13

I’ve ignored this error, but is it still difficult to solve?
Below is logs that appears almost every day.

error jobs/schedulers.go:160 Failed to set next job run time {"error": "SqlJobStore.GetNewestJobByStatusAndType: Failed to get the latest job from status and type, Status=success, invalid connection"}

error jobs/jobs_watcher.go:70 Error occurred getting all pending statuses. {"error": "SqlJobStore.GetAllByStatus: Failed to get job, Status=pending, invalid connection"}

info mlog/log.go:166 Invalid session {"error": "SqlSessionStore.Get: There was an error looking for the session, sessionIdOrToken=TOKEN, invalid connection"}

error mlog/log.go:174 We encountered an error finding the session {"path": "/api/v4/users/status/ids", "request_id": "yy8u3oqhjtrq8q1zn1yspxb94r", "ip_addr": "IP-ADDRESS", "user_id": "", "method": "POST", "err_where": "SqlSessionStore.Get", "http_code": 500, "err_details": "sessionIdOrToken=TOKEN, invalid connection"}

Is this the reason why my DB size of MariaDB seems to swell more than I thought?

Mattermost 5.17.1, CentOS 7.7, nginx 1.16.2, MariaDB 10.2.29

14

error jobs/jobs_watcher.go:70 Error occurred getting all pending statuses. {“error”: “SqlJobStore.GetAllByStatus: Failed to get job, Status=pending, invalid connection”}

What is the cause of the above?
I tried adjusting the “SqlSettings” part of config.json, but it hasn’t changed at all.
What else should I doubt?

15

Is this error ignorable even at the error level?

16

any update. . ?

18

Hi shakasur and welcome to the Mattermost forums!

Do you need help with something here or did you just want to share your experience? Sorry, but I’m not sure so I’m better asking.