Gitlab behind a reverse proxy

I am using GitLab for private projects. GitLab is run using the Docker image provided by GitLab. I can access my instance from outside via a reverse proxy (Apache).

The setup is simple:

  • GitLab Docker container is running on NUC and listens on port 7080 for HTTP connections
  • NUC is connected via OpenVPN to the server on AWS
  • Apache as a reverse proxy listening on port 443 for HTTPS
  • Apache terminates SSL: incoming requests are HTTPS, but forwarded as HTTP to GitLab
  • Apache forwards incoming requests to GitLab on Docker

Standard setup of GitLab in Docker with Apache as reverse proxy will give access to GitLab without problems. Start GitLab container, configure Apache, done. You can access GitLab from the internet, create repositories, clone, push, etc.

While the setup will work out of the box, you need to carry out additional configuration for GitLab to really make it work with SSL terminating. What is not working correctly is:

  • The external URL is not configured, so the URL in the repository clone dialog is not using HTTPS.
  • You cannot upload attachments in the Wiki
  • You cannot add pictures in the Wiki via copy & paste from the clipboard
  • Uploading files / images may work in the issues dialog, but not in the wiki, as the wiki is using different upload service.

Attaching an image from clipboard fails.

Problem

My external URL is https://gitlab.itsfullofstars.de. Setting this value as external URL in gitlab.rb. You configure GitLab by setting the parameters in the file gitlab.rb and then reconfigure GitLab.

## GitLab URL
##! URL on which GitLab will be reachable.
external_url 'https://gitlab.itsfullofstars.de'

Run reconfigure to enable the configuration.

gitlab-ctl reconfigure

Accessing gitlab.itsfullofstars.de:

This will set all parameters in all involved components of GitLab based on the values set in gitlab.rb. You can see the new value by looking at the automatically generated configuration file for the internal web server.

## GitLab settings
gitlab:
## Web server settings (note: host is the FQDN, do not include http://)
  host: gitlab.itsfullofstars.de
  port: 443
  https: true

The problem is: GitLab thinks it is running standalone, with direct access to the internet. There is not a specific parameter to inform that the requests are coming from a reverse proxy with SSL termination. Setting the values in gitlab.rb will result in an erroneous configuration:

  • SSL for internal GitLab web server (nginx) is enabled
  • Nginx is not listening on port 80, only on 443
  • My Apache reverse proxy is configured to connect to nginx port 80. Hence the Service Unavailable error.

Port 80 is not working any longer. Accessing GitLab directly via 192.168.x.x:7443 on HTTPS port (Docker mapping 7443 to 443).

Access will work. GitLab tries to get a new TLS certificate during the reconfiguration process, but fails, therefore the self signed certificate.

Attaching an image won’t work

Because of the external_url value, GitLab will redirect to gitlab.itsfullofstars.de. As the reverse proxy is not able to connect, it’s a 503 error.

Configuring the external GitLab URLs results in:

  • An incorrect HTTPS configuration due to wrong certificate
  • Adjustment of Apache reverse proxy: no longer SSL termination

I do not want to take of managing GitLabs internal TLS certificate. I want to access it via HTTP only and use Apache for SSL termination.

Solution

The solution is to configure the external URL and to let the internal nginx run on port 80 and no HTTPS.

Gitlab.rb

Configure a value for external_url.

vim config/gitlab.rb
external_url 'https://gitlab.itsfullofstars.de'
nginx['listen_port'] = 80
nginx['listen_https'] = false
gitlab-ctl reconfigure

GitLab HTTP server

Check the configuration for the internal GitLab web server. The server should be gitlab.itsfullofstars, the port 80 and protocol HTTP.

more data/gitlab-rails/etc/gitlab.yml
## GitLab settings

gitlab:
## Web server settings (note: host is the FQDN, do not include http://)
  host: gitlab.itsfullofstars.de
  port: 80
  https: false

Optional: Restart

Running reconfigure restarts the services, but if you want to be sure, restart GitLab.

gitlab-ctl restart

Apache configuration

My Apache configuration. Maybe not all parameters are needed, but it works.

<VirtualHost *:443>
  ServerName gitlab.itsfullofstars.de
  ProxyPreserveHost On
  ProxyRequests Off
  SSLProxyEngine on
  SSLEngine on
  SSLHonorCipherOrder on
  <Location />
    RequestHeader unset Accept-Encoding
    RequestHeader set Host "gitlab.itsfullofstars.de"
    RequestHeader add X-Forwarded-Ssl on
    RequestHeader set X-Forwarded-Proto "https"
    ProxyPass http://nuc:7080/
    ProxyPassReverse http://nuc:7080/
    Order allow,deny
    Allow from all
  </Location>
</VirtualHost>

Result

After executing the above steps, your configuration should be:

An external request is now for server gitlab.itsfullofstars.de. Apache does SSL termination, and nginx is accepting the request without either blocking it or trying to redirect to HTTPS.

Attaching an image to GitLab Wiki by pasting it from the clipboard


Links

Some resources I found while solving the issue for myself.

https://gitlab.com/gitlab-org/gitlab-ce/issues/27583

https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl

https://gitlab.com/gitlab-org/gitlab-ce/issues/52243

Let the world know

Solving reverse proxy error ERR_CONTENT_DECODING_FAILED

Configuring a reverse proxy is not an easy task. It involves some trial and error and dealing with unexpected errors. One of those errors is ERR_CONTENT_DECODING_FAILED. The web site won’t load in your browser and Chrome will show this error message:

Error ERR_CONTENT_DECODING_FAILED may show up in your browser when a resource is configured on your reverse proxy, and the backend communication is working. That is: the backend is returning data, but not in a form the browser expects. Like: browser expects a GZIP response, but receives plain text. Therefore the hint content decoding failed. Content received, but the browser is not able to decode / understand the data.

To solve this error, reset the Accept-Encoding request header in your Reverse Proxy configuration.

Apache

RequestHeader unset Accept-Encoding

http://httpd.apache.org/docs/current/mod/mod_headers.html 

Example Apache configuration section for a location named test.

<Location /test>
  RequestHeader unset Accept-Encoding
  ProxyPass https://0.0.0.0:443
  ProxyPassReverse https://0.0.0.0:443/
  Order allow,deny
  Allow from all
</Location>

NGINX

proxy_set_header Accept-Encoding "";

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header

Let the world know

SAPControl error. ASCS won’t start

It happened. SAP NetWeaver ABAP instance won’t start anymore. The DB gets up, but the CS simply won’t start. The log file startsap_ASCS01.log show a 401 error.

What happened? SAPstartsrv will start the instance and to do so, it needs to communicate with sapcontrol. To make this communication work a stream is going to be used. For make this work, sapcontrol uses a file located at /tmp. If now this file was deleted while sapcontrol was running, or the owner changed, sapcontrol will have problems in using these file to communicate. A solution to the problem can be found at SAP Note 1565645 – SAP composite note: sapcontrol and and SCN Wiki.

The basic solution is simple: Stop the service, delete any stream file in /tmp and the start the service again. Issue the following commands:

  • sapcontrol -nr 01 -function StopService -user gwdadm <password>
  • rm /tmp/.sapstream*
  • sapcontrol -nr 01 -function StartService <SID>

This creates the sapstream files again:

Important: These files need to be owned by <sid>adm and have the super user bit set.

To see if these three simple commands solved the startup problem of ASCS, run startsap and check the output. In case everything is back to normal, the output will show that the instance is started without error.

Let the world know

NWBC logon error with IE 10

Today I came across a strange error when logging in to NWBC, or to be accurate, trying to log on to NWBC. The IE developer toolbar showed me that the JS failed, caused by an error message: “The system cannot find the path specified”. This error only occurred with Internet Explorer. Clicking the error message revealed that the error occurred in the lightspeed.js file. The highlighted JS command was window.localStorage. Entering window.localStorage at the command line gave me the same error message: “The system cannot find the path specified”.

  • Root cause found!

It’s an enterprise controlled environment, so updating IE is not possible and even with Microsoft slowly killing IE, it will be around longer than you might think. Therefore, this error needs to be solved immediately. First action I did was to clear the cache, restarting IE and still: same result.

Next was to ensure that the security permissions allow window.localStorage. Internet Explorer Options -> Advanced -> Security section -> Enable DOM Storage

The option is activated, so window.localStorage should work. After googling a little bit around, I found a blog describing the same error. And with a solution! When deleting IE data, it is important to delesect the option “Preserve Favorites website data“.

After restarting IE, calling window.localStorage returned:

Now NWBC logon works again!

Let the world know