Increase RAID sync rate
Scenario
- The HDDs are in an external USB case.
- RAID5 with 3 HDD (10TB)
- Software RAID5 with mdadm and Debian Linux
Adding a new disk
When you add a new HDD to an existing RAID, a sync is started. In my case I added a 10TB disk to a RAID5. The sync started and as estimated time I got something in the range of days. The estimated time is listed in finish=5384 min.
This number goes up and down a little bit, but overall result is that the sync will need days. After checking the status again after a while, it still showed days: finish=3437min.
The main problem here Is the rate at which mdadm can sync the data. The value is between 30000K and 43000K. That’s not much given the size of the RAID. There are several tips available on the internet. What help me was to set the stripe_cache_size.
STRIPE_CACHE_SIZE
You set the size of stripe_cache_size for each RAID device (mdX). In case your RAID is md0:
echo 32768 > /sys/block/md0/md/stripe_cache_size
Result
The speed increased to 100000K/sec. That’s close to 3x faster than before. Time went down drastically.
7 Comments
Mike · October 26, 2022 at 16:48
Do I set the size of stripe_cache_size for my RAID device before adding disks or I can change while reshaping?
Tobias Hofmann · October 27, 2022 at 16:02
I was able to change the value during sync and the speed was adjusted accordingly on the fly.
Mike · October 28, 2022 at 06:09
Thank you Tobias for your reply. I have a follow up question. Will this also affect the transfer rate (read/write) of the RAID 5 and is there a way to calculate optimal size that you can assign to stripe_cache_size? I assume the hardware specs will be a factor. Thank you again!
john · November 12, 2023 at 18:18
when i tried to change the contents of /sys/block/md0/md/stripe_cache_size (which was 256) i got
Error writing lock file /sys/block/md0/md/.stripe_cache_size.swp: Permission denied .
this was while the resync was running
cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sde1[4] sdc1[1] sdb1[0] sdd1[3]
11720654784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[===============>…..] resync = 75.6% (2957317376/3906884928) finish=79.1min speed=200001K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk
Rosenfeld · December 23, 2023 at 14:10
I have entered the exact command you shared (changed only the name of the md) but before the command it was 52000K and after the command it is still 52000K. What would I have done wrong ?
Monitor disk speed in Linux | It`s full of stars! · March 5, 2019 at 11:02
[…] Running a server allows you to do a lot of stuff from remote. Copying files is one of those tasks you can do from anywhere on the world while being logged on via SSH. For this task it is good to know the speed of read/write to get an idea if it’s working s expected. When sitting in front of your computer, you can see if a HDD is working, in Windows you see a MB/s indication, and in Linux? Not all copy commands show you the transfer rate by standard. Some disk intensive tasks won’t at all (RAID sync). […]
How to add a new disk to RAID5 | It`s full of stars! · March 12, 2019 at 10:30
[…] This process will take some time. To learn how to increase the speed the sync, see my other blog about this topic. […]