We're Hiring!

Mattermost, Inc.

Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection

Hi.
I’m moving a MM instance from a dedicated host to a cPanel/Plesk server. I managed to get everything in place, at first glance everything seems fine, but after some minutes of runtime I see the errors below in logs:

Jan 15 13:21:59 host.com mattermost[130576]: [mysql] 2021/01/15 13:21:59 packets.go:36: unexpected EOF
Jan 15 13:21:59 host.com mattermost[130576]: {"level":"error","ts":1610713364.2011328,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses.","error":"SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}

I tried disabling TLS as suggested elsewhere (cannot find the link anymore) but it’s still throwing that message.

In Apache I added this to proxy:

  ProxyPreserveHost On

  # Set web sockets
  RewriteEngine On
  RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
  RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
  RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

  <Location />
        Require all granted
        ProxyPass http://127.0.0.1:8065/
        ProxyPassReverse http://127.0.0.1:8065/
        ProxyPassReverseCookieDomain 127.0.0.1 my.domain.com
  </Location>

What can be the problem? Thanks

Hello @maxxer! I use Apache as my reverse proxy as well. I see some differences between your reverse proxy configuration compared to mine, can you confirm if you are using an SSL certificate on your domain name or not, so I can share the correct configuration file of mine, and also see if I can run your configuration file?

Hi, thanks for the response.

I’m using SSL on the cPanel virtualhost. From Apache to Mattermost the connection is not secure.

Alright, same as mine then!
Here’s what I use mine as, although I replaced the domain name and all that with <REDACTED>:

<IfModule mod_ssl.c>
<VirtualHost *:443>
  # If you're not using a subdomain you may need to set a ServerAlias to:
  # ServerAlias www.mydomain.com
  ServerName <REDACTED>
  ServerAdmin <REDACTED>
  ProxyPreserveHost On
  DocumentRoot /opt/mattermost
  # Set web sockets
  ProxyRequests Off
RemoteIPHeader CF-Connecting-IP
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/api/v3/users/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RequestHeader set X-Forwarded-Proto "https"

<Location /api/v4/users/websocket>
    Require all granted
    ProxyPassReverse ws://127.0.0.1:8065/api/v4/users/websocket
    ProxyPassReverseCookieDomain 127.0.0.1 <REDACTED>
  </Location>

  <Location />
    Require all granted
    ProxyPassReverse https://127.0.0.1:8065/
    ProxyPassReverseCookieDomain 127.0.0.1 <REDACTED>
  </Location>
#  <Location /opt/chat_logs/.git>
#   Require all denied
#
#  </Location>

        ErrorDocument 401 /srv/teamhydrahacking.com/error/401
        ErrorDocument 403 /srv/teamhydrahacking.com/error/403
        ErrorDocument 404 /srv/teamhydrahacking.com/error/404
        ErrorDocument 500 /srv/teamhydrahacking.com/error/500

#<Directorymatch "^/.*/\.git/">

#  Order 'deny,allow'
#  Deny from all
#</Directorymatch>
RedirectMatch 404 /\.git
Include /etc/letsencrypt/options-ssl-apache.conf
SSLEngine on
SSLVerifyClient require
SSLVerifyDepth 1
        SSLCACertificateFile /root/origin-pull-ca.pem
        SSLCertificateFile /etc/letsencrypt/live/<REDACTED>/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/<REDACTED>/privkey.pem
</VirtualHost>
</IfModule>

Note that I have extra things in there for HTTPS configuration with Cloudflare authenticated origin pulls, and custom error pages etc

Thanks for the response. Unfortunately I’m getting the same error right after opening the login page:

Jan 16 08:50:04 host.com systemd[1]: Started Mattermost.
Jan 16 08:52:13 host.com mattermost[79088]: [mysql] 2021/01/16 08:52:13 packets.go:36: unexpected EOF
Jan 16 08:52:13 host.com mattermost[79088]: {"level":"error","ts":1610783533.0025039,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses.","error":"SqlJobStore.GetAllByStatus: Unable to get the jobs, Status=pending, invalid connection"}
Jan 16 08:52:15 host.com mattermost[79088]: [mysql] 2021/01/16 08:52:15 packets.go:36: unexpected EOF
Jan 16 08:52:15 host.com mattermost[79088]: {"level":"error","ts":1610783535.3681574,"caller":"app/user.go:193","msg":"There was a error fetching if first user account","error":"SqlUserStore.Count: We could not count the users, invalid connection"}

But why an SQL error should be related to proxy config?

I forgot to say I’m on 5.21.0. Could it matter in some way?

For some reason the error message happens only when using Firefox, logging in with Chrome doesn’t print that in the logs.

Other than this chat is working, I tried (with the same user) writing to a channel and I see both windows updated…

Hmm well that could be part of Firefox blocking a backend connection to the server, which would obviously result in some issues. I do know that we are up to a major build number of 3 now though, so maybe upgrading would be useful as well?