Gitlab behind a reverse proxy

Posted on June 13, 2019 by Tobias Hofmann

I am using GitLab for private projects. GitLab is run using the Docker image provided by GitLab. I can access my instance from outside via a reverse proxy (Apache).

The setup is simple:

GitLab Docker container is running on NUC and listens on port 7080 for HTTP connections
NUC is connected via OpenVPN to the server on AWS
Apache as a reverse proxy listening on port 443 for HTTPS
Apache terminates SSL: incoming requests are HTTPS, but forwarded as HTTP to GitLab
Apache forwards incoming requests to GitLab on Docker

Standard setup of GitLab in Docker with Apache as reverse proxy will give access to GitLab without problems. Start GitLab container, configure Apache, done. You can access GitLab from the internet, create repositories, clone, push, etc.

While the setup will work out of the box, you need to carry out additional configuration for GitLab to really make it work with SSL terminating. What is not working correctly is:

The external URL is not configured, so the URL in the repository clone dialog is not using HTTPS.
You cannot upload attachments in the Wiki
You cannot add pictures in the Wiki via copy & paste from the clipboard
Uploading files / images may work in the issues dialog, but not in the wiki, as the wiki is using different upload service.

Attaching an image from clipboard fails.

Problem

My external URL is https://gitlab.itsfullofstars.de. Setting this value as external URL in gitlab.rb. You configure GitLab by setting the parameters in the file gitlab.rb and then reconfigure GitLab.

## GitLab URL
##! URL on which GitLab will be reachable.
external_url 'https://gitlab.itsfullofstars.de'

Run reconfigure to enable the configuration.

gitlab-ctl reconfigure

Accessing gitlab.itsfullofstars.de:

This will set all parameters in all involved components of GitLab based on the values set in gitlab.rb. You can see the new value by looking at the automatically generated configuration file for the internal web server.

## GitLab settings
gitlab:
## Web server settings (note: host is the FQDN, do not include http://)
  host: gitlab.itsfullofstars.de
  port: 443
  https: true

The problem is: GitLab thinks it is running standalone, with direct access to the internet. There is not a specific parameter to inform that the requests are coming from a reverse proxy with SSL termination. Setting the values in gitlab.rb will result in an erroneous configuration:

SSL for internal GitLab web server (nginx) is enabled
Nginx is not listening on port 80, only on 443
My Apache reverse proxy is configured to connect to nginx port 80. Hence the Service Unavailable error.

Port 80 is not working any longer. Accessing GitLab directly via 192.168.x.x:7443 on HTTPS port (Docker mapping 7443 to 443).

Access will work. GitLab tries to get a new TLS certificate during the reconfiguration process, but fails, therefore the self signed certificate.

Attaching an image won’t work

Because of the external_url value, GitLab will redirect to gitlab.itsfullofstars.de. As the reverse proxy is not able to connect, it’s a 503 error.

Configuring the external GitLab URLs results in:

An incorrect HTTPS configuration due to wrong certificate
Adjustment of Apache reverse proxy: no longer SSL termination

I do not want to take of managing GitLabs internal TLS certificate. I want to access it via HTTP only and use Apache for SSL termination.

Solution

The solution is to configure the external URL and to let the internal nginx run on port 80 and no HTTPS.

Gitlab.rb

Configure a value for external_url.

vim config/gitlab.rb

external_url 'https://gitlab.itsfullofstars.de'
nginx['listen_port'] = 80
nginx['listen_https'] = false
gitlab-ctl reconfigure

GitLab HTTP server

Check the configuration for the internal GitLab web server. The server should be gitlab.itsfullofstars, the port 80 and protocol HTTP.

more data/gitlab-rails/etc/gitlab.yml

## GitLab settings

gitlab:
## Web server settings (note: host is the FQDN, do not include http://)
  host: gitlab.itsfullofstars.de
  port: 80
  https: false

Optional: Restart

Running reconfigure restarts the services, but if you want to be sure, restart GitLab.

gitlab-ctl restart

Apache configuration

My Apache configuration. Maybe not all parameters are needed, but it works.

<VirtualHost *:443>
  ServerName gitlab.itsfullofstars.de
  ProxyPreserveHost On
  ProxyRequests Off
  SSLProxyEngine on
  SSLEngine on
  SSLHonorCipherOrder on
  <Location />
    RequestHeader unset Accept-Encoding
    RequestHeader set Host "gitlab.itsfullofstars.de"
    RequestHeader add X-Forwarded-Ssl on
    RequestHeader set X-Forwarded-Proto "https"
    ProxyPass http://nuc:7080/
    ProxyPassReverse http://nuc:7080/
    Order allow,deny
    Allow from all
  </Location>
</VirtualHost>

Result

After executing the above steps, your configuration should be:

External URL is https://gitlab.itsfullofstars.de
The internal GitLab nginx is listening
- Port: 80
- Protocol: HTTP
- Server: gitlab.itsfullofstars.de

An external request is now for server gitlab.itsfullofstars.de. Apache does SSL termination, and nginx is accepting the request without either blocking it or trying to redirect to HTTPS.

Attaching an image to GitLab Wiki by pasting it from the clipboard

Links

Some resources I found while solving the issue for myself.

https://gitlab.com/gitlab-org/gitlab-ce/issues/27583

https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl

https://gitlab.com/gitlab-org/gitlab-ce/issues/52243

Response for preflight does not have HTTP ok status

Posted on November 26, 2018 by Tobias Hofmann

Issuing an AJAX request is more complex than you might think.

Issue

When you issue an AJAX request to a server in another domain (CORS), you may get the following error message:

Response for preflight does not have HTTP ok status.

Problem

The server is configured to allow CORS. The Apache configuration includes

Header set Access-Control-Allow-Origin "*"

The response header of the service contains the correct header value. With this set you can access the service via CORS.

Solution

Now, why does it not work?

You have to be aware that this works for simple CORS requests. For more complex requests that set custom headers, etc, the service may not work. This is due to the preflight mechanism of the browser that checks if the service accepts the request. Before issuing an AJAX request (e.g. GET or POST), an OPTIONS is triggered to check what the service is accepting. You can see this in the network tab of Chrome.

The request includes two headers:

Access-Control-Request-Headers
Access-Control-Request-Method

Before issuing the actual GET request, the browser is checking if the service is correctly configured for CORS. This is done by checking if the service accepts the methods and headers going to be used by the actual request. Therefore, it is not enough to allow the service to be accessed from a different origin, but also the additional requisites must be fulfilled.

Solution

To configure Apache to set the headers, add the missing headers. Include in your HTTP service configuration the header set directive.

Header always set Access-Control-Allow-Origin "*"
Header always set Access-Control-Allow-Methods "POST, GET, OPTIONS, DELETE, PUT"
Header always set Access-Control-Allow-Headers "append,delete,entries,foreach,get,has,keys,set,values,Authorization"

Still not working!

After setting the values, you may still not able to call the service, as the browser still reports an error. The response code is not 2xx. Returning a 200 HTTP code can be enforced in Apache config using a rewrite rule.

RewriteEngine On
RewriteCond %{REQUEST_METHOD} OPTIONS
RewriteRule ^(.*)$ $1 [R=200,L]

With this configuration, the service will now work with CORS. The first OPTIONS request will pass:

The following GET request will also pass:

ERR_CONTENT_DECODING_FAILED

Posted on September 18, 2018 by Tobias Hofmann

Configuring a reverse proxy is not an easy task. It involves some trial and error and dealing with unexpected errors. One of those errors is ERR_CONTENT_DECODING_FAILED. The web site won’t load in your browser will show this error message:

Example: browser expects a GZIP response, but receives plain text. Therefore the hint from your browser about content decoding failed. The content is received, but the browser is not able to decode / understand the data. If a plain text response is expected, but the received response from the backend is zipped, the browser cannot read the content.

To solve this error, reset the Accept-Encoding request header in your Reverse Proxy configuration.

Apache

Documentation

RequestHeader unset Accept-Encoding

Example

<Location /test>
    RequestHeader unset Accept-Encoding
    ProxyPass https://0.0.0.0:443
    ProxyPassReverse https://0.0.0.0:443/
    Order allow,deny
    Allow from all
</Location>

NGINX

Documentation

proxy_set_header Accept-Encoding "";

Solving reverse proxy error ERR_CONTENT_DECODING_FAILED

Posted on April 23, 2018 by Tobias Hofmann

Error ERR_CONTENT_DECODING_FAILED may show up in your browser when a resource is configured on your reverse proxy, and the backend communication is working. That is: the backend is returning data, but not in a form the browser expects. Like: browser expects a GZIP response, but receives plain text. Therefore the hint content decoding failed. Content received, but the browser is not able to decode / understand the data.

To solve this error, reset the Accept-Encoding request header in your Reverse Proxy configuration.

Apache

RequestHeader unset Accept-Encoding

http://httpd.apache.org/docs/current/mod/mod_headers.html

Example Apache configuration section for a location named test.

<Location /test>
  RequestHeader unset Accept-Encoding
  ProxyPass https://0.0.0.0:443
  ProxyPassReverse https://0.0.0.0:443/
  Order allow,deny
  Allow from all
</Location>

NGINX

proxy_set_header Accept-Encoding "";

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header

Block access from country by IP in Apache

Posted on July 13, 2017 by Tobias Hofmann

In this blog I will show how you can block access to your Apache hosted internet services, forbidding access to a whole country. The access is blocked based on the IP address of a client. In case of a VPN where the user connects to a VPN server in another country, the user will still be able to access the site.

The internet is a great to ensure freedom of speech. Anyone can raise his/her voice; use the information to be informed on what is happening in the world, let others know about something, share knowledge. You can do so by using a social site or by hosting your own site. The ease of access to information; be able to search it instantly; have huge amount of information able to be discovered by a large number of the world population. This is one of the true great contributions to really make the world a better place. Some countries don’t like this, applying censorship, access restriction, or worse. And basically, if you decide to block a country to access your site, it’s one step to the wrong direction.

Why would you block a whole country? Isn’t a great thing about the internet that it’s accessible from anywhere in the world, just using a browser? It’s not as simple. A few reasons to block a country can be:

Legal requirements. Your site is not in compliance with the countries law. For instance, maybe you are logging too much personal information?
The functionality is not meant for that country. You have a commercial service, and are not offering a payment option or a localized version.
You are popular in a country and flooded with a lot of requests, but these are just operational overhead for you as your site is not targeted for these users.
If you think hard enough, you can come up with a good reason.

After finding yourself in the situation to block a specific country, the question is: HOW? You can use a blocker in your web platform (WordPress plugin), or use Apache to do so. Using a .htpasswd file for this is not optimal due to performance. Better is to use a module. A quick Google search reveals that a good option is to use the GeoLite DB from MaxMind. And they also offer an Apache 2.4 module. The module works with Apache 2 and the HTTPD server available on Amazon AMI images.

Some references to projects used to set the country blocking up.

Libmaxmind db https://github.com/maxmind/libmaxminddb
Apache module https://github.com/maxmind/mod_maxminddb
GeoLite 2 DB http://dev.maxmind.com/geoip/geoip2/geolite2/

Steps

Steps for using GeoLite2 DB for blocking countries in Apache

Download GeoLite 2 DB
Install dependencies
Install Apache module
Configuration
Activation

1. Download GeoLite2 database

The GeoLite2 DB is available as a free and commercial license. The free version should be good enough for a private blog. You can get the free version from MaxMind site.

Select GeoLite 2 Country and binary format. Download the file using wget.

wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz

Unzip the file.

tar zxvf GeoLite2-Country.tar.gz

The actual DB file is close to 3 MB in size.

Copy it to a directory were the apache users can find it. A good default location is /usr/local/share in a new directory named GeoIP.

sudo mkdir /usr/local/share/GeoIP
sudo cp /home/ec2-user/geolite2db/GeoLite2-Country_20170704/GeoLite2-Country.mmdb /usr/local/share/GeoIP/

2. Install dependencies

Install libmaxmind

For the Apache module to work, the C library libmaxmind must be installed. This can be done by using yum.

sudo yum install libmaxminddb.x86_64 libmaxminddb-devel.x86_64

HTTPD devel files

Another dependency is the HTTP development files. These can also easily installed using yum.

sudo yum install httpd24-devel.x86_64

3. Install Apache module

The Apache module is available as source code from GitHub. For installation, download the latest release from GitHub. In my case, the latest release was version 1.1.0. Download the tar file.

Download the release to Linux using wgetand unzip it.

wget https://github.com/maxmind/mod_maxminddb/releases/download/1.1.0/mod_maxminddb-1.1.0.tar.gz

tar zxvf mod_maxminddb-1.1.0.tar.gz

Now you can compile and install the module. To do so, run

./configure
sudo make install

This should compile and put the files correctly into the right directory of HTTPD. If an error occurs during configuration, compilation or installation, look at the error message and good luck.

The directive to load the new module was automatically added to the file /etc/httpd/conf/httpd.conf

To test that the module can be loaded, restart HTTPD.

sudo service httpd restart

The service needs to start without error. This indicates that the module was successfully loaded. To validate this, check if the new module is actually loaded by HTTPD. To do so, list all loaded modules.

sudo httpd –M

Search for the maxmind module:

maxminddb_module (shared)

The new module is correctly loaded by HTTPD. Now we can configure Apache to make use of the module.

4. Configuration

Edit the HTTP config file and add the directive to block a specific country. The GitHub site of MaxMind contains an example that serves as a very good starting point.

MaxMindDBEnable On
MaxMindDBFile DB /usr/local/share/GeoIP/GeoLite2-Country.mmdb
MaxMindDBEnv MM_COUNTRY_CODE DB/country/iso_code
SetEnvIf MM_COUNTRY_CODE ^(RU|DE|FR) BlockCountry
Deny from env=BlockCountry

Using the above example, let’s adjust it to block Brazil. No worry, I won’t block Brazil, this is just a test as my IP currently is from Brazil, making it easier for me to test the setup. To block Brazil, check if MM_COUNTRY_CODE starts with BR: SetEnvIf MM_COUNTRY_CODE ^(BR) BlockCountry

MaxMindDBEnable On
MaxMindDBFile DB /usr/local/share/GeoIP/GeoLite2-Country.mmdb
MaxMindDBEnv MM_COUNTRY_CODE DB/country/iso_code
SetEnvIf MM_COUNTRY_CODE ^(BR) BlockCountry
Deny from env=BlockCountry

Add the above configuration snippet into a Location or Directory directive. This is because of the Deny command. This cannot be added directly under a virtual host.

<VirtualHost _default_:443>
  <Location />
    MaxMindDBEnable On
    MaxMindDBFile DB /usr/local/share/GeoIP/GeoLite2-Country.mmdb
    MaxMindDBEnv MM_COUNTRY_CODE DB/country/iso_code
    SetEnvIf MM_COUNTRY_CODE ^(BR) BlockCountry
    Order deny,allow
    Allow from al1
    Deny from env=BlockCountry
  </Location>
</VirtualHost>

5. Activation

To activate the configuration and to block Brazil, a restart of HTTPD is needed.

sudo service httpd restart

After HTTPD is successfully restarted, the new configuration is activated. To see if it is working, a basic test is to just access the site from an IP address that is blocked.

Test

My IP is from Brazil, accessing my site now should give me an access denied message.

It works!

Custom 503 error page for Apache

Posted on March 9, 2017 by Tobias Hofmann

A 5xx error code is returned by a web server when something went wrong: The server was not able to process the request. For a reverse proxy, a common 5xx error message is 503, meaning that the backend server is not reachable.

In the technical architecture of my blog site, the WordPress site with my blogs is hosted on a Raspberry Pi in my living room, while external access is through a reverse proxy hosted on Amazon EC2. If now the reverse proxy on EC2 cannot reach my Raspberry Pi, a 503 error message is given.

The root cause can be that the Raspberry Pi is turned off, there is no Internet connection available for some reason (power outage, provider problem), or something else. In case this happens, EC2 reverse proxy will throw an error and try to show the Apache standard 503 error page. The web page used to display the 503 is the same for all Apache installations worldwide. Giving your users a more personalized message can be a nice touch. For instance, including a statement that you are aware of the issue, it won`t take long to get solved, or a better explanation of what happened.

For this to work, you need to have

A custom 503.html file and
Configure Apache to use this web page.

Create custom 503 file

This is up to you. Internet and Google are your friend.

Apache configuration

Apache has the ErrorDocument directive. For an HTTP error code you specify a HTML file to be shown. Make the 503 HTML file created by you in the above section available on the web server.

/var/www/html/error/503.html

Important: the document root of Apache is /var/www/html. For accessing the file, the browser will call the URL /error/503.html. Reference it in the Apache configuration.

sudo vim /etc/httpd/conf/httpd.conf

Insert

ErrorDocument 503 /error/503.html

You are done in the case of a normal web server setup. The configuration shown so fare won`t work for a reverse proxy. A reverse proxy will forward all requests to the backend server, including the request for the 503 document. To not forward /error/503.html to the backend, put /error/ in a exception list. With this, every request to /error/ won`t be forwarded by Apache, and instead be served from the local web server. To exclude /error/ from the ProxyPass rule, add:

ProxyPass /error/ !

This exclusion must be before the other ProxyPass directives. A somewhat more complete example of a Apache configuration:

<VirtualHost _default_:443>
  DocumentRoot "/var/www/html"
  ProxyPass /error/ ! 
  ErrorDocument 503 /error/503.html
  SSLProxyEngine On
  ProxyPass / https://backend/
  ProxyPassReverse / https://backend/
  SSLEngine on
</VirtualHost>

Restart Apache

sudo service httpd restart

The next time the backend server is not reachable, the reverse proxy will serve the custom 503 error page to the users.

Get an A rating from SSL labs

Posted on February 11, 2016 by Tobias Hofmann

You should secure your web site using TLS. No, that`s not a typo, it`s TLS and not SSL. SSL is dead and should not be used anymore. Praise TLS. This may sound complicated at first, but it`s not. First step is to deactivate HTTP and activate HTTPS. How to do this depends on your web server. Luckily, there are a lot of good documentations on this available. For free. Thanks Internet. To help you evaluate your setup, there is an online service available that tests your HTTPS setup: ssllabs.com. Just enter your server name and you get a result: A to F. You have a secure site when you get an A. Problem with this documentation is that is shows you how to activate TLS, but not how to get to a setup secure enough to earn you an A rating from SSL labs. Now, what is a secure setup for TLS? You can argue here for eternity. So (too?) many parameters available. Let me try to show you what I did to get an A rating.

First, let`s take a look at the SSL labs service and check some sites to get an understanding of how the service works. SAP`s site (sap.com) get`s an A rating:

If something is wrong, the rating is downgraded and a justification of the rating is given. For instance, if you run a check against service.sap,com (155.56.89.225), you get a C rating.

The problems that caused the C rating are all related to protocol support. Don`t worry, service.sap.com is the old support site, support.sap.com get an A rating. Only showing this here to demonstrate the impact the supported protocols have on the rating. The other 3 criteria were rated equally. Some sites can get an even worse result (that site is not related to SAP).

While it is nice to know that an A rating is possible, how to get one for your own server? Let me show this using my very own web server as an example: https://www.itsfullofstars.de:8081

Request certificate

First step is to get a valid certificate. This is done by creating a CSR and send it to a CA. To create the CSR, you can use openssl.

openssl req -new -newkey rsa:4096 -nodes -keyout itsfullofstars.de.key -out itsfullofstars.de.csr -sha256

The output is a key file and a CSR file. The certificate is of 4096 bit strength and uses SHA-256 as signature. Send the CSR to your CA (I use StartSSL) and you get back the certificate (CRT), and normally also the intermediate certificate.

The server certificate is the CRT file. It is already in PEM format. To make this clear, I renamed it to .PEM. Uploading the certificate to the web server and activating it in the Apache configuration for HTTPS.

Base Apache configuration

SSLEngine on
SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
SSLCompression Off
SSLCipherSuite HIGH:MEDIUM:-RC4:-EXP:!kEDH:!aNULL
SSLCertificateFile /etc/ssl/certs/www.itsfullofstars.de.2016.pem
SSLCertificateKeyFile /etc/ssl/private/itsfullofstars.de.2016.key

Running SSL Labs test gives now a B rating.

Certificate chain

The rating is capped to B because of an incomplete certificate chain. Remember the 1_root_bundle.crt file delivered by the CA? That`s the intermediate CA certificate. That`s the certificate the web server is not providing, but should. Add the parameter SSLCertificateChainFile to Apache`s conf file.

SSLCertificateChainFile /etc/ssl/1_root_bundle.crt

Running SSL Labs test gives now an A- rating.

Already A- for just fixing the certificate chain problem. The report shows that I do get an A- and not better because the web server is not supporting forward secrecy (PFS). It’s not like I`ll need to have forward secrecy. I do not run an e-commerce site or let people log on.

What is PFS? It protects your users as it makes it really hard to decrypt the traffic. To decrypt the session, the session key must be known. In case the session key was created using a weak algorithm (e.g. RC4, RSA), all it takes is the server’s private key. If someone gets access to my server private key, an attacker can decrypt all traffic (even recorded one). Changing the algorithm to ephemeral Diffie Hellman makes this more secure, as the attacker needs to crack the session key. In my case the session key is exchanged with a 4096 bit certificate, should take them some time. As the session key is unique per session, the attacker will have to decrypt the key for each session. Just having the server’s private key is not enough.

Forward secrecy

While forward secrecy is a little bit of an overkill for my site, it`s possible to do and “it`s too much” does not count in regards to security. Therefore, I will activate forward secrecy on my server. Basically, PFS is done by activating the correct cipher suites and instruct the web server to ignore what the browser wants to do. This enforces the browser to use the ephemeral DH ciphers send by the server and those allow PSF.

SSLEngine on
SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
SSLCompression Off
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+AESGCM EECDH EDH+AESGCM EDH+aRSA HIGH !MEDIUM !LOW !aNULL !eNULL !LOW !RC4 !MD5 !EXP !PSK !SRP !DSS"
SSLCertificateFile /etc/ssl/certs/www.itsfullofstars.de.2016.pem
SSLCertificateKeyFile /etc/ssl/private/itsfullofstars.de.2016.key
SSLCertificateChainFile /etc/ssl/1_root_bundle.crt

Running SSL Labs test gives now an A rating.

Mission accomplished, my web site is now rated A by SSL labs.

Some resources

https://support.microsoft.com/en-us/kb/257591

https://www.digicert.com/ssl-support/ssl-enabling-perfect-forward-secrecy.htm

https://blog.qualys.com/ssllabs/2013/06/25/ssl-labs-deploying-forward-secrecy

http://www.heise.de/security/artikel/Forward-Secrecy-testen-und-einrichten-1932806.html

https://blog.qualys.com/ssllabs/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy

https://scotthelme.co.uk/perfect-forward-secrecy/

Operate GateOne behind Apache reverse proxy

Posted on January 28, 2016 by Tobias Hofmann

After installing GateOne on my Raspberry Pi 2 Debian system, I can log on to SSH via HTTP and browser. But only from my internal network, as the external accessible port is blocked by Apache. To add GateOne from outside, I either can disable Apache (no, won`t do it) or make GateOne accessible through Apache. I`ll use Apache as a reverse proxy for this.

Note: you`ll need Apache 2.4 for this, as GateOne uses Websocket for communication, and this is included only ootb with Apache 2.4.

Configuration

GateOne

I won`t use a subdomain for this example, so no new Apache virtual host will be use. This means that I have to use a URL prefix to access GateOne. The prefix is: /ssh. This must be configured in the GateOne configuration file:

sudo vim /opt/gateone/server.conf

Change the parameter url_prefix and restart GateOne

url_prefix = "/ssh"

To be able to access GateOne from external, the URL of the external server must be added to origin. In my case, this means that www.itsfullofstars.de is added.

origins = "https://www.itsfullofstars.de:8081;http://127.0.0.1;http://localhost

Apache

To make use of Apache as a reverse proxy, first the modules must be enabled. You can do this with a2enmod. Add also the web socket module

sudo a2enmod proxy_wstunnel
sudo a2enmod proxy_http

Edit the apache configuration to add a reverse proxy for /ssh. Do this for HTTP and WS. In case GateOne listens on TLS, do this for HTTPS and WSS.

ProxyPass /ssh ws://127.0.0.1:9080/ssh
ProxyPassReverse /ssh ws://127.0.0.1:9080/ssh
ProxyPass /ssh http://127.0.0.1:9080/ssh
ProxyPassReverse /ssh http://127.0.0.1:9080/ssh

Restart Apache.

sudo service apache2 restart

Now its possible to access GateOne through /ssh from external.

Load test your web applications part 2 – with Apache jMeter

Posted on August 19, 2013 by Tobias Hofmann

Note: 1st published at SCN on 19.8.2013

This blog is laying around on my hard drive for too much time. The previous version of this blog series was published over a year ago!. Originally named “Load test Web Dynpro Java with jMeter” this was actually the 1^st blog of the 3 part series. During the last year, a few blogs about jMeter were published, but as I do not want to leave the series unfinished, I am publishing now (finally) the last part of my load test your web application blog.

To make it easier a jMeter test plan is attached to this blog. This plan can be used in conjunction with a
WDJ application. Both were used for the creation of this blog.

With jMeter

JMeter is a great tool for testing your web applications. And you can even use it to test Web Dynpro Java applications. To facilitate the work of creating the tests, jMeter comes with a proxy to intercept browser requests and add them to the test. This allows for easy and fast recording of tests.

jMeter test for sample WDJ application:

As by default a WDJ application is accessible as anonymous – or Guest – user, authentication is not needed and a test can be executed several times against the WDJ application. Pretty easy but in real life you’ll hit real soon a hard limit. By default ,the HTTP Cookie Manager controller does not delete cookies with each iteration, or: JSESSIONID is not deleted.

Simulating users

Running the test and checking the user sessions in NWA shows that the same Guest session is used.

With that you may be tempted to run the test and to see the load generated by just 1 user. But try to run the test 50 times. After some calls, jMeter shows an error when executing the open WDJ app step.

The error message returned from the server is:

HTTP 403? Yes, 403. Re-using the same session will ensure that you`ll hit a well known limitation. You can now opt to implement SAP note 1012065, but it is better to end the session after each iteration. While only one Guest session is open, the maximum number of application sessions is exceeded: 23

Now let jMeter create a new session with each iteration.

The new sessions are created because the HTTP Cookie Manager deletes the JSESSIONID cookie with each iteration. Run the test 5 times, the open sessions in NWA will show

To access the report, go to: Availability & Performance -> Resource Monitoring -> Session Monitoring

If now a test with 500 iterations is run, 500 Guest sessions will be created. This may not a problem, but won’t reflect your reality where sessions are ended after the user navigates away. Logging in and off creates additional load, something worth to test too. Besides this, each session occupies some memory. Depending on your test case, this amount of memory can consume quite some substantial part of the free memory, compromising your test results.

The correct way to test WDJ applications is to end the user session after the test is executed, independently if Guest or named users are used. To end the session, all it takes is to delete the cookies each iteration and trigger a logout action: /irj/portal?logout_submit=1 [you have the portal installed in the same instance, right? If not, you’ll have to add your own logoff functionality to the application]

Add this step at the end of the test.

With a HTTP sampler as the last step, each iteration can be started with a new session without overloading the NetWeaver Java server with Guest or user sessions.

Load test your web applications part 1.5 – Apache ab

Posted on June 11, 2012 by Tobias Hofmann

Note: 1^st published on SCN on 11.6.2012

With ab, for Web Dynpro applications

In my blog about performance load testing with ab SAP Mentor Anton Wenzelhuemer raised a question if and how you can use ab to test a specific action/event in WD applications. That’s a good question as these events besides input validation normally trigger a connection to the backend and change the context node and attributes. And that is the part where you can improve the performance of the application.

Now, is it possible to test a user action with ab? To remind you, ab is used to test a single resource. The intent is to see how fast your web server can serve a HTML page and test how different configuration parameters affect the performance. You cannot test the flow of an action (load page, press button, get result). To find out if ab can be used to test a WD application, you have to find out what happens when you trigger the event.

Sample WDJ application

I build an example WDJ application using the steps outlined here: it’s and Web Dynpro Java application containing a table that shows the data of the get flight BAPI. For testing forms that require an input you cannot use ab (meaning: setting the input and call the form action), but you can use ab to simulate actually what happens after the input is set and the send button is clicked: the POST action of the browser.

[For testing the form from command line: to insert the airline Id and press the button, use curl, as curl allows interacting with forms. But curl is not a load testing tool (you may look at curl-loader).]

To make things easy for me I implemented a button (named: Generic) that sets the input of the airline to LH, triggers an event that causes the controller to call the BAPI and return the table content. Clicking this button I can record the POST action with Firebug:

http://nwce1:50000/webdynpro/resources/tobias.com/test~wdj~arfc/FlightListApp?SAPEVENTQUEUE=Button_Press%EE%80%82Id%EE%80%84MHDJ.New1CompView.Button1%EE%80%83%EE%80%82ClientAction%EE%80%84submit%EE%80%83%EE%80%82urEventName%EE%80%84BUTTONCLICK%EE%80%83%EE%80%81Form_Request%EE%80%82Id%EE%80%84…form%EE%80%85Async%EE%80%84false%EE%80%85FocusInfo%EE%80%84%40%7B%22sFocussedId%22%3A%20%22MHDJ.New1CompView.Button1%22%7D%EE%80%85Hash%EE%80%84%EE%80%85DomChanged%EE%80%84false%EE%80%85IsDirty%EE%80%84false%EE%80%83%EE%80%82EnqueueCardinality%EE%80%84single%EE%80%83%EE%80%82%EE%80%83&sap-wd-appwndid=ab98f4f2ae8f11e18027000c292b255f&sap-wd-cltwndid=ab98f4f1ae8f11e194f4000c292b255f&sap-wd-norefresh=X&sap-wd-secure-id=ab98f4f3ae8f11e18e55000c292b255f7276476579

Replaying the POST in the browser gives me the content WDJ returns to the browser:

Response in the browser

Response in firebug

The returned content is an XML file containing the data for the table. As you can see, WDJ return the table content as HTML. [Note: Whoever at SAP was or is or will be responsible for this: why? And please stop doing this. This should be done via AJAX and JSON / XML, and definitely you should not transfer HTML. That makes me wonder if the HTML transferred back changes between browsers.]

Conclusion: That POST triggers the server side event and returns the right content. This POST is what ab should send to the server.

How to get ab to send the exact same POST? For this ab offers 2 parameters that have to be used together: -p and –T. The parameter –p defines a file that contains the data to post and –T the content type. Saving the POST data in a file named postwdj.txt and set –T to application/x-www-form-urlencoded; charset=UTF-8.

Using only these two parameters won’t work with WD applications as WD checks the header for the user agent. The browser has to send a user agent that is supported by WD. To set the user agent for ab to IE (I use IE to be on the safe side as support for other browsers heavily depends on your SPS):

-H “User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”.

The next step is to authenticate the session if the application needs an authenticated user. How exactly you do authentication depends on your server configuration, you can enable basic authentication with ab or send the JSESSION (or MYSAPSSO2) cookie to the WD app: -C JSESSION=<value>

Setting the parameters:

ab -v 4 -n 1 -p postwdj.txt -T “application/x-www-form-urlencoded; charset=UTF-8”

-H “User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”

-C JSESSIONID=abc123

http://nwce1:50000/webdynpro/resources/tobias.com/test~wdj~arfc/FlightListApp

[Reminder: all these parameters I retrieved from what firebug recorded.]

Returns:

Did it worked? No. Why? “Request referes to an unkown session”. Unkown session? Yes, even with JSESSIONID as the AS Java session identifier, WD is more complex. Take a look at the complete POST data:

sap-wd-appwndid=09c61082af0811e1a8d7000c292b255f&sap-wd-cltwndid=09c61081af0811e1adf6000c292b255f&sap-wd-norefresh=X&sap-wd-secure-id=09c61083af0811e19571000c292b255f6987261425&SAPEVENTQUEUE=Button_Press…

There are several WD specific sessions involved. And these expire too. Make sure that the POST data you get from firebug is valid when you execute ab. With updated POST data:

Now, is that the result expected? Sometimes ab outputs the first lines of the response, sometimes not. To really know what the server returned I use Wireshark. Wireshark allows you to trace the TCP packages and to view the transmitted HTML. The image below shows the result of a not working POST, as the web page of the WD app is returned:

HTTP 200, but the POST was not triggered correctly, so the WD application returns the default view:

A Wireshark trace of a successful ab POST:

And the result at the command line when ab prints the returned message:

Running the test with 1 thread 100 times:

Worked fine

10 threads:

Worked

Running the test with 30 thread 100 times:

Worked

The problem is that too many connections to the application were made, and the session was never terminated, thus causing more and more memory consumption until my server crashed. There is a SAP note explaining this behavior and that in the end of the test run you should terminate the session. With ab this won’t work.

While ab allows you to test web and portal applications quite easily, it is not flexible enough for testing WD applications. Instead of being able to just run the test, you have to get the POST data and run the test shortly after. What ab is missing is to submit forms dynamically (like curl) or to build the POST data based on what the server expects. But for that you need to consider the flow of the application, something ab is not aware of. For testing complex web applications there is a handy tool also available from Apache: jMeter.