Bad token type [details: token_type=, response_body=] with LetsEncrypt https

Summary

Using apache2 vhosts for gitlab + mattermost on the same machine, letsencrypt for SSL, gitlab SSO fails with “Bad token type”

Steps to reproduce

On a clean Ubuntu 16.04, install gitlab-ce [ GitLab Community Edition 8.14.1 e45bf3d ] and letsencrypt. Have DNS for example.com, git.example.com, and chat.example.com all pointing to that same machine, and get certs for all three. Configure gitlab’s nginx to run on port 8001, and mattermost by default runs the service on localhost:8065. This is the /etc/gitlab/gitlab.rb:

external_url 'https://git.example.com'
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
nginx['enable'] = true
nginx['ssl_certificate'] = "/etc/letsencrypt/live/example.com/fullchain.pem"
nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/example.com/privkey.pem"
nginx['listen_port'] = 8001 

mattermost_external_url 'https://chat.example.com'
mattermost['enable'] = true
mattermost['service_use_ssl'] = true
mattermost_nginx['enable'] = false

Apache2 vhosts are used to redirect https://git.example.com and https://chat.example.com to the right ports:

<VirtualHost *:443>                                                                                 
  ServerName git.example.com                                                 

  <Proxy *>                                                                  
    Require all granted                                                      
  </Proxy>                                                                   

  SSLProxyEngine on                                                          
  SSLProxyVerify none                                                        
  SSLProxyCheckPeerCN off                                                    
  SSLProxyCheckPeerName off                                                  
  SSLProxyCheckPeerExpire off                                                
                                                                             
  ProxyRequests Off                                                          
  ProxyPass / https://127.0.0.1:8001/                                        
  ProxyPassReverse / https://git.example.com/                                
                                                                             
  RewriteEngine on                                                           
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f                       
  RewriteRule .* https://127.0.0.1:8001%{REQUEST_URI} [P,QSA]                
                                                                             
  # needed for downloading attachments                                       
  DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public              
                                                                             
  SSLEngine on                                                               
  SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem         
  SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem        
  Include /etc/letsencrypt/options-ssl-apache.conf                           
</VirtualHost>                                                               

and

<VirtualHost *:443>                                                                                 
  ServerName chat.example.com                                                
                                                                             
  SSLEngine on                                                               
  SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem         
  SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem        
  Include /etc/letsencrypt/options-ssl-apache.conf                           
                                                                             
  ServerSignature Off                                                        
  ProxyPreserveHost On                                                       
  ProxyRequests Off                                                          
                                                                             
  # Ensure that encoded slashes are not decoded but left in their encoded state.
  # http://doc.gitlab.com/ce/api/projects.html#get-single-project            
  AllowEncodedSlashes NoDecode                                               
                                                                             
  DocumentRoot /var/opt/gitlab/mattermost                                    
                                                                             
  RewriteEngine on                                                           
                                                                             
  RewriteCond %{REQUEST_URI}  ^/api/v1/websocket    [NC,OR]                  
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$           [NC,OR]                  
  RewriteCond %{HTTP:CONNECTION} ^Upgrade$          [NC]                     
  RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI}  [P,QSA,L]                
                                                                             
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f                       
  RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]               
  RequestHeader set X-Forwarded-Proto "https"                                
                                                                             
  <Location /api/v1/websocket>                                               
    Require all granted                                                      
    ProxyPassReverse http://127.0.0.1:8065                                   
    ProxyPassReverseCookieDomain 127.0.0.1 chat.example.com                  
  </Location>                                                                
                                                                             
  <Location />                                                               
    Require all granted                                                      
    ProxyPassReverse http://127.0.0.1:8065                                   
    ProxyPassReverseCookieDomain 127.0.0.1 chat.example.com                  
  </Location>                                                                
                                                                             
</VirtualHost>                                                               

Expected behavior

You can log in to Mattermost using Gitlab SSO

Observed behavior

Clicking on Gitlab Single-Sign-On sends me to Gitlab [ git.example.com ] to log in, which succeeds.
I get a prompt to “Authorize GitLab Mattermost to use your account?”
I click “authorize”, which sends me back to chat.example.com with the error:

GitLab Mattermost needs your help:
Bad token type
Go back to Mattermost

In the logfile /var/log/gitlab/mattermost/mattermost.log, I have the error

[EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=<###> uid= ip=<###> Bad token type [details: token_type=, response_body=]