Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection

Troubleshooting
1

Hi.
I’m moving a MM instance from a dedicated host to a cPanel/Plesk server. I managed to get everything in place, at first glance everything seems fine, but after some minutes of runtime I see the errors below in logs:

Jan 15 13:21:59 host.com mattermost[130576]: [mysql] 2021/01/15 13:21:59 packets.go:36: unexpected EOF
Jan 15 13:21:59 host.com mattermost[130576]: {"level":"error","ts":1610713364.2011328,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses.","error":"SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}

I tried disabling TLS as suggested elsewhere (cannot find the link anymore) but it’s still throwing that message.

In Apache I added this to proxy:

  ProxyPreserveHost On

  # Set web sockets
  RewriteEngine On
  RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
  RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
  RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

  <Location />
        Require all granted
        ProxyPass http://127.0.0.1:8065/
        ProxyPassReverse http://127.0.0.1:8065/
        ProxyPassReverseCookieDomain 127.0.0.1 my.domain.com
  </Location>

What can be the problem? Thanks

2

Hello @maxxer! I use Apache as my reverse proxy as well. I see some differences between your reverse proxy configuration compared to mine, can you confirm if you are using an SSL certificate on your domain name or not, so I can share the correct configuration file of mine, and also see if I can run your configuration file?

3

Hi, thanks for the response.

I’m using SSL on the cPanel virtualhost. From Apache to Mattermost the connection is not secure.

4

Alright, same as mine then!
Here’s what I use mine as, although I replaced the domain name and all that with <REDACTED>:

<IfModule mod_ssl.c>
<VirtualHost *:443>
  # If you're not using a subdomain you may need to set a ServerAlias to:
  # ServerAlias www.mydomain.com
  ServerName <REDACTED>
  ServerAdmin <REDACTED>
  ProxyPreserveHost On
  DocumentRoot /opt/mattermost
  # Set web sockets
  ProxyRequests Off
RemoteIPHeader CF-Connecting-IP
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/api/v3/users/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RequestHeader set X-Forwarded-Proto "https"

<Location /api/v4/users/websocket>
    Require all granted
    ProxyPassReverse ws://127.0.0.1:8065/api/v4/users/websocket
    ProxyPassReverseCookieDomain 127.0.0.1 <REDACTED>
  </Location>

  <Location />
    Require all granted
    ProxyPassReverse https://127.0.0.1:8065/
    ProxyPassReverseCookieDomain 127.0.0.1 <REDACTED>
  </Location>
#  <Location /opt/chat_logs/.git>
#   Require all denied
#
#  </Location>

        ErrorDocument 401 /srv/teamhydrahacking.com/error/401
        ErrorDocument 403 /srv/teamhydrahacking.com/error/403
        ErrorDocument 404 /srv/teamhydrahacking.com/error/404
        ErrorDocument 500 /srv/teamhydrahacking.com/error/500

#<Directorymatch "^/.*/\.git/">

#  Order 'deny,allow'
#  Deny from all
#</Directorymatch>
RedirectMatch 404 /\.git
Include /etc/letsencrypt/options-ssl-apache.conf
SSLEngine on
SSLVerifyClient require
SSLVerifyDepth 1
        SSLCACertificateFile /root/origin-pull-ca.pem
        SSLCertificateFile /etc/letsencrypt/live/<REDACTED>/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/<REDACTED>/privkey.pem
</VirtualHost>
</IfModule>

Note that I have extra things in there for HTTPS configuration with Cloudflare authenticated origin pulls, and custom error pages etc

5

Thanks for the response. Unfortunately I’m getting the same error right after opening the login page:

Jan 16 08:50:04 host.com systemd[1]: Started Mattermost.
Jan 16 08:52:13 host.com mattermost[79088]: [mysql] 2021/01/16 08:52:13 packets.go:36: unexpected EOF
Jan 16 08:52:13 host.com mattermost[79088]: {"level":"error","ts":1610783533.0025039,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses.","error":"SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}
Jan 16 08:52:15 host.com mattermost[79088]: [mysql] 2021/01/16 08:52:15 packets.go:36: unexpected EOF
Jan 16 08:52:15 host.com mattermost[79088]: {"level":"error","ts":1610783535.3681574,"caller":"app/user.go:193","msg":"There was a error fetching if first user account","error":"SqlUserStore.Count: We could not count the users, invalid connection"}

But why an SQL error should be related to proxy config?

I forgot to say I’m on 5.21.0. Could it matter in some way?

6

For some reason the error message happens only when using Firefox, logging in with Chrome doesn’t print that in the logs.

Other than this chat is working, I tried (with the same user) writing to a channel and I see both windows updated…

7

Hmm well that could be part of Firefox blocking a backend connection to the server, which would obviously result in some issues. I do know that we are up to a major build number of 3 now though, so maybe upgrading would be useful as well?