How to add a new disk to RAID5

I have a RAID5 consisting of three 10TB HDDs. This RAID5 has a total capacity of 20 TB.

I bought a new 10 TB HDD that I want to use to extend the RAID5: 4 HDDs with a total capacity of 30 TB. The file system on md0 is ext4. Currently, the RAID5 disks are sdc1, sdf1 and sde1. The additional disk is sdd1.

cat /proc/mdstat

The RAID5 is formatted with ext4 and available as md0.

mount

Steps

  1. Prepare new disk
  2. Add disk to RAID
  3. Grow RAID
  4. Extend ext4 files system.

Prepare new disk

First start with the preparation of the new disk. The disk is /dev/sdd and needs to have a partition. I use parted for this. First, create a label of type gpt.

parted -s -a optimal /dev/sdd mklabel gpt

Next is to create the partition using parted. This time, I am using the interface.

parted /dev/sdd

Add disk to RAID

The RAID is a software RAID on Linux, therefore mdadm is used to control the raid. To add a new disk, option –add is used and the raid and new disk are passed as parameters.

mdadm --add /dev/md0 /dev/sdd1

The result of the operation can be seen in mdstat.

cat /poc/mdstat

The new disk is added as a spare device. The (S) behind sdd1 means spare device. In case a device would fail, the spare device will take over automatically and a RAID rebuild will be triggered. This gives me less trouble in case a device fails, as I won’t have to do anything, but it won’t give me more space. The RAID5 is still at 20 TB.

Grow RAID

To make the RAID5 aware of the new disk and that it should be used for data storage, the RAID must be informed to use the new HDD using the grow command.

mdadm --grow --raid-devices=4 /dev/md0

The command informs the RAID that there are now 4 HDDs to be used, instead of 3. This command will trigger a RAID rebuild, as the information must be distributed to the HDDs.

This process will take some time. To learn how to increase the speed the sync, see my other blog about this topic.

The RAID5 consists now of 4 HDD, all working [UUUU]. The size of the RAID is still 20 TB. This is because the md0 has capacity of 30 TB, but the ext4 filesystem is still configured to make use of 20 TB.

Resize ext4 filesystem

To be able to use the 30TB available on the RAID5, you need to resize the file system. First, run an integrity check.

e2fsck -f /dev/md0

After the e2fsck ended without errors, the file system can be extended. This is done by using the tool resize2fs.

resize2fs /dev/md0

After resize2fs completes (can take a while), the size available is now 30TB:

mount /dev/md0 /mnt/md0/

Links

Let the world know

Monitor disk speed in Linux

Running a server allows you to do a lot of stuff from remote. Copying files is one of those tasks you can do from anywhere on the world while being logged on via SSH. For this task it is good to know the speed of read/write to get an idea if it’s working s expected. When sitting in front of your computer, you can see if a HDD is working, in Windows you see a MB/s indication, and in Linux? Not all copy commands show you the transfer rate by standard. Some disk intensive tasks won’t at all (RAID sync).

To monitor disk activities in Linux, several tools are available. One is iostat.

Installation

To install iostat in Debian, you must install the package sysstat

apt-get install sysstat

Execute

To run iostat, just enter iostat in the shell.

iostat

The output will list the captured read / write speed of the available devices. To get a continuous output of the disk activites, run iostat -y 1. This will update the output every second until you end the program.

iostat -y 1

Several options are available to control the output. To get the disk read / write in Mb and not in kB, add the -m flag

iostat -y 1 -m

Using iostat you can see the throughput oft he disks, even when you are running “hidden” tasks like a RAID sync or copy process in another session (screen).

Let the world know

Increase RAID sync rate

Scenario

  • The HDDs are in an external USB case.
  • RAID5 with 3 HDD (10TB)
  • Software RAID5 with mdadm and Debian Linux

Adding a new disk

When you add a new HDD to an existing RAID, a sync is started. In my case I added a 10TB disk to a RAID5. The sync started and as estimated time I got something in the range of days. The estimated time is listed in finish=5384 min.

This number goes up and down a little bit, but overall result is that the sync will need days. After checking the status again after a while, it still showed days: finish=3437min.

The main problem here Is the rate at which mdadm can sync the data. The value is between 30000K and 43000K. That’s not much given the size of the RAID. There are several tips available on the internet. What help me was to set the stripe_cache_size.

STRIPE_CACHE_SIZE

You set the size of stripe_cache_size for each RAID device (mdX). In case your RAID is md0:

echo 32768 > /sys/block/md0/md/stripe_cache_size

Result

The speed increased to 100000K/sec. That’s close to 3x faster than before. Time went down drastically.

Let the world know

DHCP Server on Linux with Raspberry Pi

My internet provider is Unitymedia. Their default router comes with a DHCP server. Honestly, it’s one of the worst products I ever had to work with. My private network is 192.168.0.x. The DHCP server of the Unitymedia box is distributing from time to time leases for 192.168.192.x. Changing my private network to 192.168.192.x one is not working, as then the DHCP server picks another address range. Advise from Unitymedia help desk was to reboot the box, which, of course, won’t solve the problem. Because of this error, some of my devices are in a different network: Chromecast won’t work, broken internet connection on smartphones, etc.

I do have a Raspberry Pi (RP) in 24/7 use. My idea is to run my own DHCP server on the RP. This not only solves the DHCP problem, but also gives me more control over the DHCP configuration.

Preparation

sudo apt-get update
sudo apt-get install isc-dhcp-server

This installs ISC DHCP server. As you can see in the output, starting the DHCP server failed.

sudo systemctl status isc-dhcp-server.service

The error is simply caused because the DHCP server is not configured. Let’s change that.

Configuration

Several parameters must be activated and configured.

sudo vim /etc/dhcp/dhcpd.conf

Lease time

default-lease-time 600;
max-lease-time 7200;

Activate DHCP server

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

Subnet

This configures what IP address are going to be distributed. My private network is 192.168.0.x with the router on 192.168.0.1. As DNS you can use whatever you want, as an example I am using Google DNS servers.

subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.150 192.168.0.240;
  option routers 192.168.0.1;
  option domain-name "itsfullofstars.de";
  option domain-name-servers 8.8.8.8, 8.8.4.4;
}

This will give DHCP clients an IP address between .150 and .240, with router .1, Google DNS and sets the domain name to my own.

Deactivate old DHCP server

To not have the DHCP server provided by Unitymedia box still issuing wrong IP address, I am going to deactivate the service via the web interface.

Start DHCP server

After installing and configuring the new DHCP server on RP and deactivating the one from the router box, it’s time to start the new DHCP server.

Result

To see if a IP address is assigned, use this command:

sudo systemctl status isc-dhcp-server.service

Android

Putting my Android device into flight mode and back makes it connect to Wifi again and obtain a new IP address via DHCP. In the DHCP status log, I can see the DHCPDISCOVER from the Android device and that it got the IP address 192.168.0.150 assigned.

Mac

As my Mac always got the wrong IP assigned, I changed it to manual configuration. Change the mode to DHCP, apply and deactivate / activating Wifi.

Soundbar

And my soundbar that got a strange IP address assigned by the Unitymedia router box? Works too!

Chromcast streaming shows the SoundBar is now in the same network.

Let the world know

Apt-get unable to connect to IPv6 address

Recently I had the problem that running apt-get update stalled while trying to connect to an IPv6 address. For instance, on a Raspberry Pi, the update process stalls while trying to connect to archive.raspberrypi.org. All other connections worked fine. Looking at the console output, a difference was that apt was trying to connect to an IPv6 address.

The problem was caused by:

100% [Connecting to archive.raspberrypi.org (2a00:1098:0:80:1000:13:0:8)]

A quick internet search showed that you can force apt to not use IPv6 and only IPv4. As the download worked for IPv4, this seems like a reasonable workaround.

Solution

You can pass a parameter to disable IPv4 to apt-get, or write it to apt config file to make it persistent.

Configuration file

Create a new configuration file. This makes it easy for you to keep the change during updates and to know that you configured this.

sudo vim /etc/apt/apt.conf.d/99disable-ipv6
Insert Acquire::ForceIPv4 "true";
Save
apt-get update

Parameter

To disable IPv6 just once while calling apt, the parameter is Acquire::ForceIPv4=true.

sudo apt-get -o Acquire::ForceIPv4=true update

Result

Loading the package data from archive raspberrypi.org is now ignored and apt-get update works again.

 

Let the world know

Setup OpenVPN server on Amazon EC2

Recently I got some new hardware that I will use to run some useful software. To use the software from anywhere, I’ll need to have remote access. As I cannot do DMZ or port forwarding with my new internet provider, I decided to connect my home server using VPN to a access machine running on AWS.

The AWS EC2 Linux computer will serve as my entry point. Services running on the RP at home connected via VPN can be accessed from EC2. Other computers at my home cannot be accessed, as the IP is different and no route is configured.

This setup comes with several architectural questions to solve:

  • How to ensure the communication is secure?
  • How to guarantee the tunnel is up?
  • How to enable access from EC2 to the services running on the client?
  • The client must be assigned the same IP for the services be accessible from EC2
  • How to give access to the services from the internet?

The three top question will be answered in my next blogs about how to set up OpenVPN server and client. The first question is the easiest to answer: by using a VPN solution. I am going to use OpenVPN and this blog is about how to setup OpenVPN. I’ll cover the installation on the EC2 instance and on the Raspberry Pi, as well as the initial setup with the certificates, server and client configuration and how to connect. Starting the client and server as service keeps them running and in case the connection fails, an automatic reconnect is attempted. The EC2 instance can access the services running on the client automatically. The last two questions will be answered sometimes later.

OpenVPN Server

Install OpenVPN on EC2

The OpenVPN software is available in yum on EC2 Linux AMI. You may need to enable the REPL repository. I assume you did this already. The packages to install a openvpn and easy-rsa.

sudo yum update
sudo yum install openvpn easy-rsa

This will also install a public key to install a package and ask for your permission to do so.

The easy-rsa package is needed to set up a certificate authority. In case you do have a CA available, you can use your CA to generate the certificates used by OpenVPN. For those that do not have a CA available, take the easy-rsa functionality.

Generate CA

The command above installs easy-rsa 3.x. With 3.x, the way how to use easy-rsa and to set up a CA and issue the certificates changed. You can see in detail how to use easy-rsa 3.x at the documentation available at the GitHub project site.

OpenVPN uses certificates, and easy-rsa issues those certificates. Basically, you have two components of easy-rsa to deal with:

  • CA software
  • Certificates

Configuration of OpenVPN is put and read from /etc/openvpn. Easy-rsa software should be in a separate folder, like /home/ec2-user/easy-rsa, but to keep all in one place I’ll put easy-rsa inside the /etc/openvpn directory.

Note: for real productive usage, don’t do this. Separate easy-rsa executables and config files.

Copy easy-rsa

Copy easy-rsa to your selection location. For this, first find out where easy-rsa is installed.

repoquery -l easy-rsa

Location is /usr/share/easy-rsa/3.0.3. I’ll copy these files to /etc/openvpn/easy-rsa.

sudo mkdir /etc/openvpn/easy-rsa
sudo cp -Rv /usr/share/easy-rsa/3.0.3/* .

Start easy-rsa

Follow the steps outlined at the easy-rsa git site. For the following steps, go into the directory where easy-rsa is installed.

cd /etc/openvpn/easy-rsa

Init PKI

sudo ./easyrsa init-pki

Build CA

This will create the CA certificate to sign certificate requests. In other words: whoever gets access to the private key of the CA created in this step, can create new valid OpenVPN clients for your setup. Take care of the CA certificate and key.

sudo ./easyrsa build-ca

You’ll need to enter:

  • PEM pass phrase
  • Common Name

The passphrase is used to unlock the private key and is an additional level of security. Even when someone gets a copy of the private key of your CA, without the pass phrase the key is not usable. The common name is used to identify the CA. I used the FQDN of my web server. After execution these two commands, the CA is initialized and can be used to issue certificates.

Diffie-Hellman

Generate Diffie-Hellman parameters.

sudo ./easyrsa gen-dh

Generate OpenVPN server certificate

The OpenVPN server needs a certificate issued by the CA to identify itself against the clients. This is a nice “feature” when using PKI. Server and client can validate the other side. Both need just to trust the CA certificate for this. The difference between the two certificates (client and server) is the included type. This is done by including an additional value in the certificate specifying the type of certificate:

  • TLS Web Server Authentication for the server and
  • TLS Web Client Authentication for the client

Which kind of certificate is going to be issued is specified by the easy-rsa command when creating the certificate request.

Generate certificate request

Create a certificate request containing the identity information of the server and let this request be signed by the CA. By specifying the server parameter, the request is for a server and the CA will include the value TLS Web Server Authentication in the extension.

sudo ./easyrsa gen-req server

Inform:

  • Pass phrase
  • Common Name

As with the CA certificate, inform a pass phrase that adds additional security to the private key and a common name to uniquely identify the server. I used server as CN. Of course, it could also have been openvpn.mydomain.com or something else.

Sign request

Send the request to the CA and sign it to issue a valid certificate. With that, the CA information is added to the CA, making it official and clients that connect to OpenVPN server will know if they can trust the server. Only when trust is verified, a connection will be established between the server and client.

sudo ./easyrsa sign-req server server

You’ll need to confirm the request by typing yes and the pass phrase.

TLS-AUTH

The following certificate is needed to harden the overall security of OpenVPN. As OpenVPN is using TLS, it makes sense to add HMAC to validate integrity of the packages received. For this to work, a shared secret key is needed. This key will be written to a file named ta.key.

Generate ta.key

cd /etc/openvpn
sudo openvpn --genkey --secret ta.key
sudo mv /etc/openvpn/ta.key /etc/openvpn/easy-rsa/private/ta.key

OpenVPN server configuration

Take a sample configuration file as a template. Can be found in the doc folder of openvpn. The sample configuration file for the server is server.conf, and for the client, client.conf.

ls -1 /usr/share/doc/openvpn-2.4.4/sample/sample-config-files/

Copy server.conf to /etc/openvpn and edit the file.

sudo cp /usr/share/doc/openvpn-2.4.4/sample/sample-config-files/server.conf /etc/openvpn/
sudo vim /etc/openvpn/server.conf

Adjust the path to the ca, cert, key and dh files

These parameters inform OpenVPN where the certificates and Keys are stored. The CA cert ca.crt is used to validate the client certificates. They must be issued by this CA. The server.crt and server.key are used by the OpenVPN server to encrypt traffic and authenticate itselfs against clients. Diffie hellman dh.pem is used to provide Perfect Forward Secrecy.

Start OpenVPN server

To start the OpenVPN server and to test the current setup, run the following command:

sudo openvpn /etc/openvpn/server.conf

During startup, you need to provide the passphrase of the server certificate.

If all works, OpenVPN starts without erros: Initialization Sequence Completed. After this, the server is waiting for clients to connect.

Note:

If someone is reading my blogs for the last years you may remember that I have once written about setting up OpenVPN for accessing SUP on AWS. That blog was all about Windows and is outdated. I wrote it in 2012. But, as I published it once at SAP Community Network, it is not available anymore. SAP lost it during their last migration.

Let the world know

Uncompressing a multi-part 7zip file in Debian

7zip is a popular compression program for Windows. It allows to effectively compress files, split them into several archives and to add protection by using a password. This all works fine if you are a Windows user. In case you now want to extract such a multi part password protected file in Linux, you’ll find out that this isn’t a standard use case. Uncompressing these files involves some work. 7zip is not made available for Linux by the developer. Gzip or zip won’t work with 7zip compressed files. But: an unofficial version is available and it is possible to extract 7zip files in Debian/Linux.

You have some options available for installing 7zip for Debian, like apt or by compilation. The version you get with apt is quite old: 9.2. In case the version of 7zip used to compress the file on Windows is higher than the one available for Debian, uncompressing may not work. An algorithm may be used that is not available on the lower version. In that case, 7zr will exit with an error and showing Unsupported Method.

Compilation from source

This option will give you the latest available version of 7zip for Linux. Especially useful when you try to unzip a file and get the message: Unsupported Method. To solve this, try to install a higher version of p7zip by downloading the source and compile p7zip.

Get the latest version of p7zip from SourceForge. Unzip it and then run make. After the compilation is done, you’ll have the executable 7za in the bin folder. This version should be able to work with files compressed by 7zip for Windows. Make sure to read the README.

Copy the correct makefile. 7zip provides several makefiles, for each target platform / architecture. In case of Linux, the default one should work. To start compilation, a simple make is sufficient.

make

This gives you the binary ./bin/7za

Unzip a file multi-part password protected file.

7za x h1.7z

APT

Install the 7zip program for Debian. This installs version 9.2.

sudo apt-get install p7zip

Let’s say we have 1 file that was zipped to file h1.7z using 7zip and splitter into 650 MB. 7zip produces 2 archives:

  • h1.7z.001
  • h1.7z.002

To list the archive:

7zr l h1.7z.001 -tsplit

We can see that the split archives contain one file named h1.7z. That is the zip file created by 7zip under Windows.

To unzip the file, use

7zr x h1.7z.001 -tsplit

Let the world know

A start job is running for dev-disk-by

Recently I restarted one of my Linux servers and the computer did not start up as expected. No external accessible service was running, like apache or SSH. This made the computer inaccessible from remote and left me in the dark. After a while, the server responded to ping, but nothing more.

After I connected the server to a display and keyboard, I could see the error message: “a start job is running for dev-disk-by […]”. After that, Linux gave me only the option to log on in rescue mode or to restart the system. A restart didn`t help. I checked the internet and found out that the message can be caused by a fstab entry.

Looking at the content of my /etc/fstab file I could see an old entry I once created for a test and never maintained (aka deleted). The system is trying to mount a partition that was not available / broken and the system stopped.

UUID=a6674495 -4249-9696-0d9c83 /mnt/disk btrfs defaults,noatime,auto 0 0

I commented out this line in fstab and restarted the system. Now the system was restarted correctly and all the services came up again.

Let the world know

Install SMP3 with Oracle DB

The following procedure for installing SMP3 with an Oracle DB is for Linux. For tests, you can use Oracle Express. Check your environment/company if you can use that version.

Prerequisites

Ensure that Oracle XE is up and running. It is important that the tnslistener is working! Run the listener and check the status:

/u01/app/oracle/product/11.2.0/xe/bin/lsnrctl status

Configure installation parameters

The steps are documented at SAP Help. You’ll have to edit the SilentInstall_Linux.txt file and adjust the installation parameters.

vim SilentInstall_Linux.txt

For Oracle, you’ll need to change these parameters (at the end, you’ll find a complete example file):

Activate that SMP3 uses an external DB

-V developerInstall="false"
-V productionInstall="true"
-V sqlaEmbeddedDB="false"
-V existDB="true"

Inform the Oracle XE connection parameters

-V existDBType="oracle-sid"
-V dbHostName="localhost"
-V dbPortNumber="1521"
-V dbLogin="gomobile"
-V dbPassword="secret"
-V dbDBName="XE"

Inform the JDBC driver location

-V jdbcDriver="/u01/app/oracle/product/11.2.0/xe/jdbc/lib/ojdbc6.jar"

Prepare Oracle DB

Form the above connection parameters you can see that SMP3 is going to use the user gomobile with the password secret to connect itself to Oracle XE. This means that the user with the password and a schema must be created in the DB. SMP3 comes with a SQL script for Oracle that does exactly that. The script is located at /db_tools/db/oracle/smp3/sql. The file is 001_SMP3_drop_and_create_user.DDL The file contains the SQL statements to create the user with the right permissions:

CREATE ROLE SY365_OBJOWNER;
GRANT CREATE SEQUENCE TO SY365_OBJOWNER;
GRANT CREATE SESSION TO SY365_OBJOWNER;
GRANT CREATE SYNONYM to SY365_OBJOWNER;
GRANT CREATE TABLE TO SY365_OBJOWNER;
GRANT CREATE VIEW TO SY365_OBJOWNER;
GRANT CREATE PROCEDURE TO SY365_OBJOWNER;
GRANT CREATE SEQUENCE TO SY365_OBJOWNER;
GRANT CREATE TRIGGER TO SY365_OBJOWNER;
GRANT CREATE INDEXTYPE TO SY365_OBJOWNER;
DROP USER GOMOBILE CASCADE;
CREATE USER GOMOBILE
IDENTIFIED BY secret
DEFAULT TABLESPACE USERS
TEMPORARY TABLESPACE TEMP
PROFILE DEFAULT
ACCOUNT UNLOCK;
-- 2 Roles for GOMOBILE
GRANT SY365_OBJOWNER TO GOMOBILE;
GRANT CREATE SESSION TO GOMOBILE;
GRANT CONNECT TO GOMOBILE;
ALTER USER GOMOBILE DEFAULT ROLE ALL;
-- 1 Tablespace Quota for GOMOBILE
ALTER USER GOMOBILE QUOTA UNLIMITED ON USERS;

You’ll have to add the command EXIT; at the end of the file

To run the SQL script, run:

sqlplus system/Sap123 @001_SMP3_drop_and_create_user.DDL > smp3.log
  • Note: Sap123 is the password for the user system.

Output is written to smp3.log

SQL*Plus: Release 11.2.0.2.0 Production on Wed Aug 24 21:37:08 2016
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
Role created.
Grant succeeded.
[…]
DROP USER GOMOBILE CASCADE
ERROR at line 1:
ORA-01918: user 'GOMOBILE' does not exist
User created.
Grant succeeded.
[…]
User altered.
User altered.

The error regarding DROP user is normal, as the user gomobile hasn’t been created before, so there is no user to drop.

Run installer

With the above steps done, SMP3 installer is ready to be run.

./SilentInstall_Linux.sh

The output will contain information regarding the Oracle DB:

dbg, existDBType:oracle-sid
WARNING: Selecting this option confirms SMP database is already created
dbg, jdbcDriver: /u01/app/oracle/product/11.2.0/xe/jdbc/lib/ojdbc6.jar
dbg, jdbcDriver fullFileName: /u01/app/oracle/product/11.2.0/xe/jdbc/lib/ojdbc6.jar
dbg, jdbcDriverFile: /sap/SAP/MobilePlatform3/Util/ojdbc6.jar
dbg, ojdbc6.jar will be renamed to ojdbc.jar in the installation
dbg, queryExit:oracle-sid localhost gomobile [pwd entered] 1521 XE
dbg, Ping succcesful: 0
dbg, smpDataExists:false
dbg, New node install

If everything works fine, you’ll get a confirmation message at the end of the installation.

Installation Successful

Validation

SAP Help contains some information on how to validate the installation. You can search for error message in the installation log, but when an error occurs, normally the installer stops. My preferred way to check SMP3 is to start the server and see if I can log on, create apps, etc. Base test is therefore to start SMP3 and to log on.

Let the world know

Install Oracle Express 11G R2 on CentOS 7

Preparations

Install some additional packages via yum to ensure that the installation and execution of the database will work. The list may differ, depending on the actual version of CentOS you are using, but the internet gave me back the following packages and you should be on the safe side.

yum update
yum install unzip libaio bc flex

Download

Before using the express edition, make yourself familiar with the license and usage restriction this edition is shipped with. If it still fits your needs, be aware that CentOS is not on the list of officially supported Linux distributions. You are on your own. Download Oracle Express 11G R2 from Oracle. It`s a 308MB file.

The downloaded file is a zipped RPM package; first step is to unzip the file.

unzip oracle-xe-11.2.0-1.0.x86_64.rpm.zip

This gives you a new directory called Disk1. This folder contains the installation RPM.

Installation

Go to the folder and install the RPM via rpm tool.

cd Disk1
rpm -ivh oracle-xe-11.2.0-1.0.x86_64.rpm

After the installation, you`ll be prompted to configure the database.

Configuration

Run the tool /etc/init.d/oracle-xe to configure the database.

/etc/init.d/oracle-xe configure

Configure the port

Specify the port of the listener

Inform the system user password. Be sure to note this down somewhere or to really remember it!!!

Specify if you want the database to be started at boot time.

The configuration should now start automatically and only take a few minutes to complete.

The database is install at /u01/app/oracle/product/11.2.0/xe/

The oracle_env script is in the folder bin.

Oracle Express 11G R2 is also started and using ps –ef you can see the processes running.

That`s it, Oracle Express 11G R2 is now installed on CentOS.

Let the world know