I am going to do some work on my AWS EC2 instance that hosts my web site https://www.itsfullofstars.de. More precisely: I did the work already and it worked out well, that’s why you can read the blog Before starting the work, I wanted to have a backup of my data. The data is saved on a EBS volume and is also the root / boot volume / disk of my EC2 instance.
AWS has a nice documentation on how to create and manage snapshots. As always with this kind of generic documentation, it contains a lot of information, or too much, as all possible cases are covered. To have a simpler reference, I’ll show in this blog how I created a snapshot.
EC2: Instance with root volume on EBS. OS: Linux
Data: Size: 8 GB, type: gp2, SSD
Task: Create a snapshot of the root device
Note that it seems that you can create a snapshot of a root volume while the instance is running. AWS states that you should stop the instance first:
„To create a snapshot for an Amazon EBS volume that serves as a root device, you should stop the instance before taking the snapshot.“
Yes, 3 steps is all it takes to take a snapshot of a EBS volume used as root volume in a EC2 Linux instance.
Go to your EC2 instance and stop it. You can also log on to your instance and issue a stop command there. I am using the AWS console as here I can do everything without having to switch to another tool.
Select Stop, not Terminate, and confirm your action. Oh, yes, do not forget: afterwards your server is not online and its services not accessible. Plan for some downtime, communicate it, etc.
Instance state switches to stopping, meaning that the server is going to shut down. This can take a few seconds.
After the instance is stopped, the state is stopped. Now you can start creating a snapshot of your root volume, as it is not accessed anymore.
To create a snapshot, follow the stops outlined by AWS documentation. Go to create snapshot section in AWS console. In case you do not have any snapshots created yet, the list will be empty.
Let’s create a snapshot. To start, click on Create Snapshot. This will open a wizard. I wanted to create a snapshot of a volume, so I selected as type Volume and selected the volume from the dropdown list. It’s a good idea to provide a description.
To start the creation process, click on Create Snapshot.
The snapshot will be created immediately. Be aware: this means that the snapshot request was created, not the actual snapshot. Taking the snapshot / copy of the volume will take some time.
You can see the status of the snapshot creation in the column Status of the snapshot. It will be in state pending until all data was transferred from the root volume to the snapshot file.
Taking the snapshot can take a few minutes, depending on the size of your EBS volume. Mine was 8 GB and it took like 5-7 Minutes to create the snapshot. This was an initial snapshot, no delta. Only when the status changes to completed, the process ended successfully.
After the snapshot it taken, you can start the EC2 instance again.
During startup, the status of your EC2 instance will be pending. After completing, it is running and if everything worked without errors, your server and the services are back online.
I am using GitLab for private projects. GitLab is run using the Docker image provided by GitLab. I can access my instance from outside via a reverse proxy (Apache).
The setup is simple:
GitLab Docker container is running on NUC and listens on port 7080 for HTTP connections
NUC is connected via OpenVPN to the server on AWS
Apache as a reverse proxy listening on port 443 for HTTPS
Apache terminates SSL: incoming requests are HTTPS, but forwarded as HTTP to GitLab
Apache forwards incoming requests to GitLab on Docker
Standard setup of GitLab in Docker with Apache as reverse proxy will give access to GitLab without problems. Start GitLab container, configure Apache, done. You can access GitLab from the internet, create repositories, clone, push, etc.
While the setup will work out of the box, you need to carry out additional configuration for GitLab to really make it work with SSL terminating. What is not working correctly is:
The external URL is not configured, so the URL in the repository clone dialog is not using HTTPS.
You cannot upload attachments in the Wiki
You cannot add pictures in the Wiki via copy & paste from the clipboard
Uploading files / images may work in the issues dialog, but not in the wiki, as the wiki is using different upload service.
Attaching an image from clipboard fails.
My external URL is https://gitlab.itsfullofstars.de. Setting this value as external URL in gitlab.rb. You configure GitLab by setting the parameters in the file gitlab.rb and then reconfigure GitLab.
## GitLab URL
##! URL on which GitLab will be reachable.
Run reconfigure to enable the configuration.
This will set all parameters in all involved components of GitLab based on the values set in gitlab.rb. You can see the new value by looking at the automatically generated configuration file for the internal web server.
## GitLab settings
## Web server settings (note: host is the FQDN, do not include http://)
The problem is: GitLab thinks it is running standalone, with direct access to the internet. There is not a specific parameter to inform that the requests are coming from a reverse proxy with SSL termination. Setting the values in gitlab.rb will result in an erroneous configuration:
SSL for internal GitLab web server (nginx) is enabled
Nginx is not listening on port 80, only on 443
My Apache reverse proxy is configured to connect to nginx port 80. Hence the Service Unavailable error.
Port 80 is not working any longer. Accessing GitLab directly via 192.168.x.x:7443 on HTTPS port (Docker mapping 7443 to 443).
Access will work. GitLab tries to get a new TLS certificate during the reconfiguration process, but fails, therefore the self signed certificate.
Attaching an image won’t work
Because of the external_url value, GitLab will redirect to gitlab.itsfullofstars.de. As the reverse proxy is not able to connect, it’s a 503 error.
Configuring the external GitLab URLs results in:
An incorrect HTTPS configuration due to wrong certificate
Adjustment of Apache reverse proxy: no longer SSL termination
I do not want to take of managing GitLabs internal TLS certificate. I want to access it via HTTP only and use Apache for SSL termination.
The solution is to configure the external URL and to let the internal nginx run on port 80 and no HTTPS.
Check the configuration for the internal GitLab web server. The server should be gitlab.itsfullofstars, the port 80 and protocol HTTP.
## GitLab settings
## Web server settings (note: host is the FQDN, do not include http://)
Running reconfigure restarts the services, but if you want to be sure, restart GitLab.
My Apache configuration. Maybe not all parameters are needed, but it works.
RequestHeader unset Accept-Encoding
RequestHeader set Host "gitlab.itsfullofstars.de"
RequestHeader add X-Forwarded-Ssl on
RequestHeader set X-Forwarded-Proto "https"
Allow from all
After executing the above steps, your configuration should be:
Recently I got some new hardware that I will use to run some useful software. To use the software from anywhere, I’ll need to have remote access. As I cannot do DMZ or port forwarding with my new internet provider, I decided to connect my home server using VPN to a access machine running on AWS.
The AWS EC2 Linux computer will serve as my entry point. Services running on the RP at home connected via VPN can be accessed from EC2. Other computers at my home cannot be accessed, as the IP is different and no route is configured.
This setup comes with several architectural questions to solve:
How to ensure the communication is secure?
How to guarantee the tunnel is up?
How to enable access from EC2 to the services running on the client?
The client must be assigned the same IP for the services be accessible from EC2
How to give access to the services from the internet?
The three top question will be answered in my next blogs about how to set up OpenVPN server and client. The first question is the easiest to answer: by using a VPN solution. I am going to use OpenVPN and this blog is about how to setup OpenVPN. I’ll cover the installation on the EC2 instance and on the Raspberry Pi, as well as the initial setup with the certificates, server and client configuration and how to connect. Starting the client and server as service keeps them running and in case the connection fails, an automatic reconnect is attempted. The EC2 instance can access the services running on the client automatically. The last two questions will be answered sometimes later.
Install OpenVPN on EC2
The OpenVPN software is available in yum on EC2 Linux AMI. You may need to enable the REPL repository. I assume you did this already. The packages to install a openvpn and easy-rsa.
sudo yum update
sudo yum install openvpn easy-rsa
This will also install a public key to install a package and ask for your permission to do so.
The easy-rsa package is needed to set up a certificate authority. In case you do have a CA available, you can use your CA to generate the certificates used by OpenVPN. For those that do not have a CA available, take the easy-rsa functionality.
The command above installs easy-rsa 3.x. With 3.x, the way how to use easy-rsa and to set up a CA and issue the certificates changed. You can see in detail how to use easy-rsa 3.x at the documentation available at the GitHub project site.
OpenVPN uses certificates, and easy-rsa issues those certificates. Basically, you have two components of easy-rsa to deal with:
Configuration of OpenVPN is put and read from /etc/openvpn. Easy-rsa software should be in a separate folder, like /home/ec2-user/easy-rsa, but to keep all in one place I’ll put easy-rsa inside the /etc/openvpn directory.
Note: for real productive usage, don’t do this. Separate easy-rsa executables and config files.
Copy easy-rsa to your selection location. For this, first find out where easy-rsa is installed.
repoquery -l easy-rsa
Location is /usr/share/easy-rsa/3.0.3. I’ll copy these files to /etc/openvpn/easy-rsa.
Follow the steps outlined at the easy-rsa git site. For the following steps, go into the directory where easy-rsa is installed.
sudo ./easyrsa init-pki
This will create the CA certificate to sign certificate requests. In other words: whoever gets access to the private key of the CA created in this step, can create new valid OpenVPN clients for your setup. Take care of the CA certificate and key.
sudo ./easyrsa build-ca
You’ll need to enter:
PEM pass phrase
The passphrase is used to unlock the private key and is an additional level of security. Even when someone gets a copy of the private key of your CA, without the pass phrase the key is not usable. The common name is used to identify the CA. I used the FQDN of my web server. After execution these two commands, the CA is initialized and can be used to issue certificates.
Generate Diffie-Hellman parameters.
sudo ./easyrsa gen-dh
Generate OpenVPN server certificate
The OpenVPN server needs a certificate issued by the CA to identify itself against the clients. This is a nice “feature” when using PKI. Server and client can validate the other side. Both need just to trust the CA certificate for this. The difference between the two certificates (client and server) is the included type. This is done by including an additional value in the certificate specifying the type of certificate:
TLS Web Server Authentication for the server and
TLS Web Client Authentication for the client
Which kind of certificate is going to be issued is specified by the easy-rsa command when creating the certificate request.
Generate certificate request
Create a certificate request containing the identity information of the server and let this request be signed by the CA. By specifying the server parameter, the request is for a server and the CA will include the value TLS Web Server Authentication in the extension.
sudo ./easyrsa gen-req server
As with the CA certificate, inform a pass phrase that adds additional security to the private key and a common name to uniquely identify the server. I used server as CN. Of course, it could also have been openvpn.mydomain.com or something else.
Send the request to the CA and sign it to issue a valid certificate. With that, the CA information is added to the CA, making it official and clients that connect to OpenVPN server will know if they can trust the server. Only when trust is verified, a connection will be established between the server and client.
sudo ./easyrsa sign-req server server
You’ll need to confirm the request by typing yes and the pass phrase.
The following certificate is needed to harden the overall security of OpenVPN. As OpenVPN is using TLS, it makes sense to add HMAC to validate integrity of the packages received. For this to work, a shared secret key is needed. This key will be written to a file named ta.key.
Take a sample configuration file as a template. Can be found in the doc folder of openvpn. The sample configuration file for the server is server.conf, and for the client, client.conf.
ls -1 /usr/share/doc/openvpn-2.4.4/sample/sample-config-files/
Copy server.conf to /etc/openvpn and edit the file.
sudo cp /usr/share/doc/openvpn-2.4.4/sample/sample-config-files/server.conf /etc/openvpn/
sudo vim /etc/openvpn/server.conf
Adjust the path to the ca, cert, key and dh files
These parameters inform OpenVPN where the certificates and Keys are stored. The CA cert ca.crt is used to validate the client certificates. They must be issued by this CA. The server.crt and server.key are used by the OpenVPN server to encrypt traffic and authenticate itselfs against clients. Diffie hellman dh.pem is used to provide Perfect Forward Secrecy.
Start OpenVPN server
To start the OpenVPN server and to test the current setup, run the following command:
sudo openvpn /etc/openvpn/server.conf
During startup, you need to provide the passphrase of the server certificate.
If all works, OpenVPN starts without erros: Initialization Sequence Completed. After this, the server is waiting for clients to connect.
If someone is reading my blogs for the last years you may remember that I have once written about setting up OpenVPN for accessing SUP on AWS. That blog was all about Windows and is outdated. I wrote it in 2012. But, as I published it once at SAP Community Network, it is not available anymore. SAP lost it during their last migration.
Let the world know