Mattermost, Inc.

Unable to save token - MM + Gitlab SSO

Summary

when using SSO authentication via Gitlab, error returns unable to save token
Affecting:
Web
Desktop App
Mobile

Steps to reproduce

Mattermost (Team - Selfhosted) 5.18.1 (Upgraded to 5.23) - now having css mime types issues
Gitlab 12.7.5
Aurora Mysql DB

Expected behavior

Login works

Observed behavior

Cannot access Mattermost
Error on mobile client: invalid state token …SqlTokenStore.GetByToken:Unable to get a token with this code

Been working happily for sometime, today, tokens not working.

image

Hi @moridinmhael.

I have a few questions:

  1. Are you using the bundled version of Mattermost that’s included as part of GitLab Omnibus?
  2. Did this error just start happening once you upgraded to 5.23?
  3. What are the exact steps you’re going through to get this error? I’m trying to figure out where in the code that this might be coming from.
  1. No, Standalone Mattermost
  2. This is occurring on 5.18.1 - before and after upgrade
  3. No changes from my side, to replicate, I just try and login.

System Design:
Gitlab in Datacentre, Mattermost hosting in AWS, SSO call is a remote call to Gitlab. As noted this has been working fine for months. However, I restored Mattermost database to an instance inside Datacentre, no there changes made, except for database connection string, and working as expected.
It appears to me, that the token call is time-sensitive/latency, albeit around 1 sec at worst.

Same Gitlab used in DC and from AWS, same account, same version, same database. Only difference being location and distance.

Yeah, it sounds like it might be something timeout-related. We had a similar issue in the past, but I don’t know if it ever got resolved (link). They were also seeing strange behaviour where the first login request would fail, but other requests shortly after would succeed. Is that happening for you too?

yes, the errors messages and observation are almost the same. Apart from being able to login on subsequent attempts. All attempts fail.
But behavior, setup and messages observed in the logs, appear the same.

Looking at your original post, I noticed that you mentioned two errors messages: “Unable to save the token” and “Unable to get a token with this code”. When are you seeing both of those?

Also, do you have any read replicas configured for your database? I’m wondering if we might be writing the token to one database and then trying to immediately read it back from the other before its had a chance to replicate the data between the two.

1 Like

yes, there is a read replica, however using the writer end-point for the db connection

The error messages appear on the webpage, after input of the authentication details on the GitLab SSO page, Mattermost will return one of those error messages, but cant determine which when.
I checked the session table in the DB, cleared it out, tried some connect. Sometimes, I see entries being written there, but still one of those errors appear, and does not proceed.

Interesting, to note, is that could still post messages to the API, and get email notification from Mattermost, so seems everything else is working, apart from the SSO authentication token portion