[Solved] Infinite refresh loop in browser clients, with "Invalid or expired session" log error spam

@teatinker @vayenampudi @ionbasa

Are any of you seeing the issue on our latest release, v4.1?

@gubbins thanks for your note.

  1. Do you use a reverse proxy? If so, is it NGINX or Apache?
  2. For GitLab Auth, do you use GitLab omnibus?
  3. For GitLab Auth, did you follow our GitLab setup guides? If not, were there any steps you took differently (if you still remember) https://docs.mattermost.com/deployment/sso-gitlab.html
  4. Finally, does it reproduce after session expiry, or does it appear to be unrelated to it?

@jasonblais:

  1. Yes, nginx on the same server
  2. gitlab enterprise 9.1.3
  3. Yes, except for selecting “api” in the scope (not sure if that was in the instructions I originally followed)
  4. Unrelated

I added the “api” scope. I’ll see if I can reproduce the issue again…

I have yet to roll out 4.1 to my team. That may happen this coming weekend or during labor day, since no one will be working. I’ll reply to this thread once that roll out happens and I can report my findings.

As to your other questions:

  • Apache reverse proxy for the current setup
  • No Gitlab at the moment

@jasonblais I am currently unable to reproduce the looping issue. It’s possibly fixed. Unfortunately two things changed at once so I don’t know which it was, if either:

  • some additional certificates were installed on the gitlab server to help with trust issues
  • I added the “api” scope to the application

I’ll keep a lookout on the logs, but, things do seem to have improved!

I’m still seeing this issue with with one user with Mattermost 4.1.0.

I can reproduce it after clearing cache and in Chrome incognito mode. When I try to login with the affected user id, I see no log messages in the server log. I cannot say if this error started after session expiry. Since the user cannot log in, I cannot access the sessions view in the account settings.

In the Chrome developer console are several “Invalid language tag” errors. The user probably set the language to german. I thought this error was fixed in 4.1.0?

Some details on our setup:

  • We use nginx as a reverse-proxy.
  • We don’t use gitlab.
  • We are running mattermost in a custom docker container with the binary from https://releases.mattermost.com.
  • We use http instead of https.

I spoke too soon - looking in the logs this morning I can see more examples of the log spamming from the refresh loop, so some users are still affected it seems.

Yeah, it sounds like similar to Gitlab SSO localization issue before which was fixed by setting currentLocale to en. Could you post the logs your seeing at Chrome console?

Could you post the logs and have it compared if still the same as to the original issue?

When trying to log in, I see this error five times in the chrome developer console:

channel_utils.js:314 Uncaught (in promise) RangeError: Invalid language tag: 
    at new Collator (native)
    at String.localeCompare (native)
    at o (http://chat.subshell.com/static/main.2b75939e681d6c1cdd03.js:1:582666)
    at Array.sort (native)
    at Array.sort (http://chat.subshell.com/static/main.2b75939e681d6c1cdd03.js:1:1168689)
    at x (http://chat.subshell.com/static/main.2b75939e681d6c1cdd03.js:1:582785)
    at k (http://chat.subshell.com/static/main.2b75939e681d6c1cdd03.js:1:583564)
    at s (http://chat.subshell.com/static/main.2b75939e681d6c1cdd03.js:1:579371)
    at http://chat.subshell.com/static/main.2b75939e681d6c1cdd03.js:1:530357
    at Set.forEach (native)

Thanks @teatinker,

  • could you post your Localization settings in config.json? Maybe something like this:
   "LocalizationSettings": {
       "DefaultServerLocale": "en",
       "DefaultClientLocale": "en",
       "AvailableLocales": ""
   },
  • for the user having issue, if you have access to database, could you check the locale value at Users table Locale column
  • lastly check if ./i18n and ./webapp/dist/i18n folders are present and have all *.json localization files
"LocalizationSettings": {
    "DefaultServerLocale": "en",
    "DefaultClientLocale": "de",
    "AvailableLocales": ""
},

sophora@chat:~/mattermost$ docker-compose exec mattermost bash
root@d585f39a4bd6:/# l mattermost/i18n/
de.json  en.json  es.json  fr.json  it.json  ja.json  ko.json  nl.json  pl.json  pt-BR.json  ru.json  tr.json  zh-CN.json  zh-TW.json
root@d585f39a4bd6:/# l mattermost/webapp/dist/i18n/
de.5faa82a9ccc75472d05c175397ffd87c.json  it.8b31f79c484d4933776b73ce89c303de.json  nl.84594be54be4fb3f7b5fad0479fb86ae.json     ru.c1d92c2191cb2df0b040bcfdeebf9880.json     zh-TW.a549614ffbfc6768903a5fa57b54ca89.json
es.09d2c8faca0ac660221e026e78eb85ba.json  ja.b116188e56540bff3d64638946c71429.json  pl.a450fad2143346314fdc32a837379ede.json     tr.3e4726d68cdbc3caaa1cf9d92f4b17f8.json
fr.92d4844afc25c26e25f9465ce6bca0ca.json  ko.424502101ff98085015ab7c6fa6dfae8.json  pt-BR.04ed576f25f0fb55c2e830d37e6056e2.json  zh-CN.82d1a57c484dcff4521c8c5d946ef9fe.json

mysql> select Locale from Users where Username = 'buege';
+--------+
| Locale |
+--------+
|        |
+--------+
1 row in set (0.00 sec)

While looking at the Users table, I noticed that the affected user is the only one where locale is not set. So this seems to be the culprit in my case.

Yeah, it should not be empty. Better to have it set to de same as your DefaultClientLocale. Let us know if that solves the problem.

I wonder how this user ended up with empty Locale. Was it created from usual user registration, bulk upload/import, switch from other auth service, or was changed only during server upgrade?

@jasonblais

Just wanted to chime in here on an update. I upgraded version 4.0.2 to 4.1.0 and then today to version 4.2.0

I haven’t seen this behavior happen anymore, and I omitted the following from my Apache2 config:

RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match

Everything seems fine for now. The issue though is still that on my end before the upgrade, this issue was very sporadic on when it would be encountered. So it’s hard to know certainly that it has been fixed. Only time will tell I guess.

I’ve tried forcing this issue to happen by manually restarting the MM Server (in the docker container) and manually restarting the reverse proxy server (Apache2), and I haven’t been able to reproduce it anymore.

Thank you for posting your feedback @ionbasa and pleased the issue seems solved now!

I’m upgraded to 4.2.0 a few days ago but I’m afraid I am still seeing this happen in the logs.

Hi @gubbins,

Has anything in your configuration changed that this issue would reappear after it seemed solved previously?

No - when I thought it was solved it turns out I just hadn’t waited long enough, as I commented here

Thanks @gubbins,

Could you perhaps post your logs as requested by @saturnino here so we can help troubleshoot further?

@lindy65 I don’t have a way to reproduce it right now, so until a user tells me about it and lets me capture some Chrome logs before they clear their caches, I have no way to capture anything unfortunately. I did report some here at the start.

@saturnino you said a potential fix is setting currentLocale to en - can you expand on what you mean? What action would I take?

@gubbins If you have access to database, you may check Users table and see to it that all users’ Locale column are not empty.

SELECT * FROM Users WHERE Locale="";

If there’s empty Locale, then I suggests to set it to en and observe if you still encounter the problem.