I am wondering how to configure mattermost behind an AWS load balancer
Mattermost mostly works using a ‘Web application load balancer’ but will get errors when clients try to upgrade the https:// connection to ws://
It doesn’t appear to load the page at all if I switch to using a Network Load Balancer
Has anyone successful got this to work? I’m trying to avoid having to run NGINX or similar as most solutions seem to point to.
@disnosec Do any of these troubleshooting tips help? https://docs.mattermost.com/install/troubleshooting.html#mattermost-error-messages
PS: Since you are using an Amazon ELB check that Idle Timeout is set to 120s. If it’s significantly lower it will cause an undesirable websocket disconnections.
Sorry, specifically it’s actually an Application Load Balancer. I was mistakenly referring to it as an ELB which AWS now refer to as a ‘classic’ load balancer. As far as I can tell they don’t enable websockets by default, I’ll need an .ebextensions config to force the nginx instance to Upgrade client connections.
It seems I was wrong and that they do indeed support websockets however it seems to be a little unstable, They work for a while on Android and Mattermost Classic IOS apps, but the newer IOS app never manages to connect via WS and constantly displays the ‘connecting…’ bar as stated here https://github.com/mattermost/mattermost-mobile/issues/742
We haven’t previously been able to get Mattermost working reliably with an “Application Load Balancer”. It seems to support WebSockets now, so it might work, but we recommend using an ELB instead.
@disnosec Have you tried with an ELB yet? If you have but would prefer to use the “Application Load Balancer”, we’d be curious to hear why?
The reason being, I was advised by AWS that the ALB should be used, over the ELB. Given that I’m only having issues with the IOS react native client (all other clients work fine, Android, Android Classic, IOS Classic, desktop/browser) I don’t think using an ALB should be an issue. Currently I’m just trying to MitM the IOS traffic but it’s proving difficult as I suspect it’s not proxy-aware. Enabling CORS from * is a ‘workaround’ but clearly introduces security risks. I’m just trying to determine what the IOS app is setting the Origin to that’s causing the upgrade rejection and then I’m happy to add a CORS entry for whatever it is, provided it’s static and won’t interfere with using a read-only mattermost config.
Try as I might I’m not able to intercept the TLS traffic from the IOS app in order to observe how the Origin for websocket upgrade is being misrepresented. While I can log onto the server via safari without and implicit proxy, using a Burp invisible listener and see the websocket origin etc. the traffic from the app(s) seems to be resisting the MitM, possibly it’s ignoring the burp suite CA? Either way I just keep getting the response. 961 493.001734922 172.17.0.1 10.42.0.57 TLSv1.2 73 Alert (Level: Fatal, Description: Handshake Failure)
Is there any other way to verify what the Origin the IOS app is setting for the websocket is? I know it’s supposed to be the ‘Server URL’ but that’s definitely set correctly.
Should also mention, I tried adding the * back into the CORS field with logging set to DEBUG - instantly connects the websocket on the IOS app but the logs simply read ‘[2017/12/01 05:34:15 UTC] [DEBG] GET - /api/v4/websocket’ so I still can’t check the origin statement
Do you have more than one domain that your instance can be accessed from? If not, you could try leaving the CORS setting blank (unless that’s causing issues).
I’m wondering if AWS would have any thoughts on the errors, since they recommended ALB over ELB?
No just the one domain. I’ve managed to successfully MitM the IOS mobile app using Burp but I can see no requests to /v4/websocket nor any websocket connection (with CORS set to *) but the app works fine. If I’m to open up the classic app I can see the websocket upgrade requests. Is there something unique about how the newer IOS app connects? Is it even using a websocket?
Traffic from the ‘classic’ IOS app shows this -
GET /api/v4/websocket HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: mattermost.site.com
Origin: https://mattermost.site.com
Cookie: ajs_anonymous_id=%2200000000000000000000000000%22; ajs_group_id=null; ajs_user_id=null; MMAUTHTOKEN=xo39bmwsgbrdsadasy7uxftbh; MMUSERID=7x81bkllkwtfi5ejkh65qc8shty
Pragma: no-cache
Cache-Control: no-cache
Sec-WebSocket-Key: uEWWtu2ZBZPg96ghffhNO2Q==
Sec-WebSocket-Version: 13
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B93
however no such request is made by the newer IOS app?
I think I may be onto something here, I setup an invisible listener and managed to capture the upgrade request from the modern IOS client-
GET /api/v4/websocket HTTP/1.1
Host: mattermost.site.com:443
Cookie: MMAUTHTOKEN=xo39bmwsgbrdsadasy7uxftbh; MMUSERID=7x81bkllkwtfi5ejkh65qc8shty
Sec-WebSocket-Protocol:
Sec-WebSocket-Key: uEWWtu2ZBZPg96ghffhNO2Q==
Sec-WebSocket-Version: 13
Upgrade: websocket
Origin: wss://mattermost.site.com:443
Connection: Upgrade
SOLUTION -
We can see that the modern app posts a get request with Origin including wss:// and a trailing :443
if my site is “https://foo.blah.com” then I must set a CORS entry for “wss://foo.blah.com:443” and the issue is resolved.
It may be worth changing the subject of the post to something including IOS given that it turned out that was the issue and not the AWS ELB/ALB