What do 502/503 mean in the context of Gitlab SSO?

Troubleshooting
1

I am trying to implement a Mattermost instance that connects exclusively to a Gitlab instance for authentication. Both systems are running directly on the same host (not in Docker or via Omnibus). Gitlab’s Rails based server is on port 8080 and Mattermost’s Go based server is available at port 8065 per defaults. Neither port is accessible outside of localhost. Both services are publicly reached via an Apache proxy on domains that use SSL.

I have used the http://127.0.0.1:/ format for both the Mattermost endpoints in the json config file and the Gitlab application callback URL’s.

Accessing Mattermost from the public URL works and displays the site just fine, but the only option is to create a team. Entering a team name produces:

We received an unexpected status code from the server (503)

Strangely submitting the form a second (and any subsequent) time produces a different code:

We received an unexpected status code from the server (502)

These messages are not very useful and it is unclear to me what part of the link is breaking down. Is Mattermost having a hard time reaching the Gitlab API (I am able to do so via curl from that box)? Or is the Gitlab callback not working (again running curl on the URL gives a result from the right service about invalid state as I would expect)?

Where do I troubleshoot from here?

2

Hi @alerque,

That’s really weird that the request is failing from Mattermost with 503/502s but works when you curl it manually.

Would you be able to post the section of your config.json file that has the GitLab SSO URLs?

I haven’t seen this issue before but if I can’t see anything obvious from your config.json further troubleshooting might require some extra code which either you or I can add around the requests to GitLab in the server code. It might just need some better error reporting which might make the issue obvious.

3

Thanks for the reply @jwilander.

I guess I should clarify what I mean by it “working” from curl. Since it seems these APIs communicate by passing POST vars back and forth, I wasn’t actually trying to replicate the traffic. I only made very basic GET requests with no parameters each direction and verified that the reply I got (error notwithstanding) was generated by the correct service. In other words the endpoints are reachable.

Here is the relevant part of my config.json file (obviously with the private data replaced with asterisks):

"SSOSettings": {
    "gitlab": {
        "Allow": true,
        "Secret" : "a**************************************************************2",
        "Id": "8**************************************************************9",
        "Scope": "",
        "AuthEndpoint": "http://127.0.0.1:8080/oauth/authorize",
        "TokenEndpoint": "http://127.0.0.1:8080/oauth/token",
        "UserApiEndpoint": "http://127.0.0.1:8080/api/v3/user"
    }
},

And here is the corresponding application setup in Gitlab:

My Apache config for the proxy also happens to be posted in this other topic.

Also for good measure here is what the Firefox Network console turns up if I load the page and submit two consecutive requests.

Note that is in fact the public URL of the Mattermost instance (proxied through Apache from its localhost port) and the matching Gitlab instance is where you might expect on the same host and proxied the same way. As this error occurs before a user is ever authenticated in Gitlab you should be able to replicate the error yourself.

5

Thanks @alerque,

That’s a fantastic response. I can’t see anything in your config settings that is causing the problem. That said, your response has given me enough information to try and reproduce your issue, so I’ll try to do that and add some debugging code to see if I can figure it out.

In the meantime, if you’d like to try enabling email sign-up (you don’t even have to configure a SMTP server), that might eliminate some possible causes of the issue. You can do this by setting DisableEmailSignUp to false and if you want to skip the SMTP server setup, set ByPassEmail to true in your config file.

The fact that the API call to /teams/create_with_sso/gitlab is returning the 502/503 errors is weird and has me thinking the problem might just be reaching any of the Mattermost APIs at all, and possibly nothing to do with the link between GitLab and Mattermost. So if email sign-up doesn’t work either then that all but confirms that is the problem.

6

You may very well be right about this being a problem with serving the Mattermost API at all. On poking this a little more with curl from the server the results don’t seem to be right to me. Of course I’m stabbing it in the dark and don’t know what the expected results are, but it feels like a problem internal to my Mattermost installation.

In particular, I’m sending requests to the 127.0.0.1:8065 address to sidestep any issues introduced by my front end proxy.

  • Requesting / gives the signup page as expected and a status header of 200. So far so good. The mattermost.log file also gives a normal looking entry (I have it turned up to DEBUG level).

  • Requesting /login/gitlab/complete as entered in the Gitlab application gives back an error page generated by Mattermost and status code of 500. This seems reasonable to me as it probably expects some callback vars and a previous state that doesn’t exist. So far so good.

  • Requesting /teams/create_with_sso/gitlab as mentioned in your post also returns an error page generated by Mattermost, but the status code is 404. Likewise the mattermost.log file has a 404 line:

    [2015/09/19 06:51:14 UTC] [EROR] /teams/create_with_sso/gitlab: code=404 ip=127.0.0.1

    Edit: It looks like the URL called by the page using at XHR request is actually /api/v1/teams/create_with_sso/gitlab. I tried that as well at it likewise returned 404.

Should that have been a valid URL? What URL’s can I test from CURL to make sure the API is actually firing on all cylinders (as opposed to just being available on the port)?

7

Ok, glad to know we’ve narrowed it down a bit.

Right now the code should be using a relative URL to hit the API (might good to make this a configurable URL). It should append /api/v1/your/call/here to the end of the domain and port of the page you are currently on. So if you’re at http://localhost:8065/someteam/channels/somechannel in your browser, then the request will be made to http://localhost:8065/api/v1/your/call/here.

The server would then show in the logs (with DEBUG level logging on):

[09/21/15 08:08:40] [DEBG] /api/v1/your/call/here

That would mark a successful start to the API call. It’s weird that your error log is missing the /api/v1/ part of the URL, but it explains why it’s a 404 (wrong URL).

You can try to curl the following:

curl -i http://127.0.0.1:8065/api/v1/users/me

That should return a 200 OK with a blank body (since no authentication is provided) and should give you a good idea as to whether you can correctly hit the API or not.

If that does work, then the APIs are working with your local address. That likely leaves the problem of the public URL not working with API calls. To test that you would curl using the public URL seen in your above screenshot:

curl -i https://mattermost.alerque.com/api/v1/users/me

If that produces the same result as the previous curl, then ok good, the proxy will have worked with the API request correctly. That would leave the problem that the Mattermost client is not constructing the URLs correctly.

To test that you can go into the file located here ‘platform/web/react/utils/client.jsx’ and search for /api/v1/teams/create_with_sso/ and replace it with https://mattermost.alerque.com/api/v1/teams/create_with_sso/ and save the file. You won’t need to restart the server, just wait ~10 seconds and reload the page in your browser. Try creating a team and see what happens.

If that works as well, then for some reason the URL construction is not working properly and I can help look into that. If any of the troubleshooting above fails before we get to this step, it’ll still help narrow down the issue and I can help further from there.

Sorry for the late response, I had a busy weekend but hopefully we can figure this out for you soon :slight_smile:

8

So here’s what I have:

  • curl -i http://127.0.0.1:8065/api/v1/users/me from the same box give 200 as expected.
  • url -i https://mattermost.alerque.com/api/v1/users/me from another box also gives 200 as expected (in fact it looks from the log like you hit this up a a few minutes before composing this post and got the same 200 result I did).
  • I don’t have a ‘platform/web/react/utils/client.jsx’ file in my installation. In fact the only thing in with ‘react’ in the name in the whole file site is ‘/usr/share/webapps/mattermost/web/static/js/react-with-addons-0.13.1.js’ (and a minified version). This is looking more like something is wrong with the package build process, but I’m not clear on what that might be.
9

I just edited the package recipe I’m using (which has been verified by others to work as a normal installation, but I haven’t found anybody using it with Gitlab SSO yet) to package web/react which was previously not packaged at all alongside the existing web/static and web/templates folder. The files now exist in my installation and I’ve edited client.jsx as you describe. There is no change in the behavior of the app, but I’m not convinced its even being used. Mutilating the syntax doesn’t throw any errors. It looks to me like everything from that source is wrapped up per this line in the output HTML:

 <script src="/static/js/bundle-0.7.1.min.js"></script>

Obviously that file is a little harder to work on, but it’s only minimized not obfuscated so I pulled it apart and found where that URL is and edited it. First of all, breaking the URL breaks the page so that’s promising (putting http instead of https threw an AJAX error about blocking mixed content types). At least here I’m editing code that’s actually being run.

That being said, putting the correct fully qualified https URL in there as you suggested yields exactly the same behavior as before, first request is 503, second and thereafter is 502.

10

Ok, that’s really weird.

I tried to investigate the issue by hitting your box with some curls and AJAX requests from the javascript console but they worked without issue just as you said. It’s only requests coming from the Mattermost app itself that are getting the 502/503 errors.

The only two things I can think to try at this point would be to:

  1. Temporarily turn off SSL on your site, and see if that makes a difference. If it doesn’t it will at least allow us to inspect the HTTP headers and also remove a possible problem from the equation.
  2. Take a look at your proxy settings to see if there could be anything related to proxy errors that you could change.

I did a quick look around to see if there are any possibly related issues out there, not sure if any of these are helpful:


http://blog.somepixels.net/en/502-proxy-error-uploading-from-apache-mod_proxy-to-tomcat-7/
http://stackoverflow.com/questions/11541802/browser-jquery-proxy-error

This guy seems to have had a very similar problem with GitLab although there is no response:
https://gitlab.com/gitlab-org/gitlab-ce/issues/841

11

I think we can rule out SSL or the Apache Proxy being an issue. I appear to have forgotten to mention this above but I created an SSH tunnel into this machine and connected by local browser to it so I was able to hit the localhost:8065 address directly. The app behaved exactly as it does over my public proxy including appearing to be fully functional right up to those specific requests which returned the same errors for me.

I will look through the posts you linked and see if I can find any clues, but I think — based on the information I see so far — that the proxy is the wrong end of the problem.

12

I’ve played with about every combination of settings and issues in the proxy related posts you linked (and several others) to no avail. Again I don’t think it’s proxy related because I can tunnel into the host and hit the 8065 port directly from my browser and I get the same error.

Additionally I’ve tried it without SSL enabled on a plain HTTP proxy configuration. In fact there are temporarily vhosts setup with or without it so you can try it too. Note that if your browser cached the permanent redirect header it got from the HTTP variant before it will be tricky to hit it until you convince it to forget that and actually try again. Using incognito/private mode or a different browser if you get redirected to HTTPS. I’ve had to disable this but can temporarily enable it again if there is something specific to be tested.

I wasn’t clear on what you thought could be done as far as inspecting headers that couldn’t be done with SSL enabled.

13

So I have an idea the Mattermost server process is completely crashing and getting restarted. I’ve been playing with it over a tunnel so that I can connect to localhost:8065 directly and I actually get slightly different results. It still errors at the same place, but if I watch the network manager the XMLHttpRequest that gets sent never actually returns anything. The request header goes out and the line is dropped. No response headers are returned at all.

In the log side of things, most of the URLs I hit just generate a [DEBG] /the/url line and that’s it. When that api call comes along I get the request line and then all the debug messages that show up when I restart the service from systemd: Server is initializing, Pinging sql master, Initializing routes, etc.

14

Also this isn’t Gitlab specific. I configured it for email signups and team creation and it dies any time the find or create team requests are made whether connected directly or through a proxy.

15

Mattermost 1.0.0-RC1 finally works for me. Whatever was wrong with my 0.7.1 installation that was causing the service to die has been fixed.