Increase RAID sync rate

Published by Tobias Hofmann on

1 min read

Scenario

  • The HDDs are in an external USB case.
  • RAID5 with 3 HDD (10TB)
  • Software RAID5 with mdadm and Debian Linux

Adding a new disk

When you add a new HDD to an existing RAID, a sync is started. In my case I added a 10TB disk to a RAID5. The sync started and as estimated time I got something in the range of days. The estimated time is listed in finish=5384 min.

This number goes up and down a little bit, but overall result is that the sync will need days. After checking the status again after a while, it still showed days: finish=3437min.

The main problem here Is the rate at which mdadm can sync the data. The value is between 30000K and 43000K. That’s not much given the size of the RAID. There are several tips available on the internet. What help me was to set the stripe_cache_size.

STRIPE_CACHE_SIZE

You set the size of stripe_cache_size for each RAID device (mdX). In case your RAID is md0:

echo 32768 > /sys/block/md0/md/stripe_cache_size

Result

The speed increased to 100000K/sec. That’s close to 3x faster than before. Time went down drastically.

Let the world know

Tobias Hofmann

Doing stuff with SAP since 1998. Open, web, UX, cloud. I am not a Basis guy, but very knowledgeable about Basis stuff, as it's the foundation of everything I do (DevOps). Performance is king, and unit tests is something I actually do. Developing HTML5 apps when HTML5 wasn't around. HCP/SCP user since 2012, NetWeaver since 2002, ABAP since 1998.

9 Comments

Mike · October 26, 2022 at 16:48

Do I set the size of stripe_cache_size for my RAID device before adding disks or I can change while reshaping?

    Tobias Hofmann · October 27, 2022 at 16:02

    I was able to change the value during sync and the speed was adjusted accordingly on the fly.

      Mike · October 28, 2022 at 06:09

      Thank you Tobias for your reply. I have a follow up question. Will this also affect the transfer rate (read/write) of the RAID 5 and is there a way to calculate optimal size that you can assign to stripe_cache_size? I assume the hardware specs will be a factor. Thank you again!

john · November 12, 2023 at 18:18

when i tried to change the contents of /sys/block/md0/md/stripe_cache_size (which was 256) i got
Error writing lock file /sys/block/md0/md/.stripe_cache_size.swp: Permission denied .
this was while the resync was running

cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sde1[4] sdc1[1] sdb1[0] sdd1[3]
11720654784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[===============>…..] resync = 75.6% (2957317376/3906884928) finish=79.1min speed=200001K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk

    Kevin · July 21, 2024 at 08:26

    Don’t use joe or vi for editing that special file. Just use echo and pipe into that, like stated in the text:
    echo 32768 > /sys/block/md0/md/stripe_cache_size

Rosenfeld · December 23, 2023 at 14:10

I have entered the exact command you shared (changed only the name of the md) but before the command it was 52000K and after the command it is still 52000K. What would I have done wrong ?

John Talbot · August 4, 2024 at 04:20

My system is: Linux omsvault 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux
I do not see /sys/block/md0/md/stripe_cache_size. Instead I see:
ls /sys/block/md0/md
array_size dev-sdb1/ max_read_errors rd4@ sync_action
array_state dev-sdc/ metadata_version rd5@ sync_completed
bitmap/ dev-sdd1/ mismatch_cnt reshape_direction sync_force_parallel
bitmap_set_bits dev-sde/ new_dev reshape_position sync_max
chunk_size dev-sdg1/ raid_disks resync_start sync_min
component_size fail_last_dev rd0@ safe_mode_delay sync_speed
consistency_policy last_sync_action rd1@ serialize_policy sync_speed_max
degraded layout rd2@ suspend_hi sync_speed_min
dev-sda1/ level rd3@ suspend_lo uuid
Does this trick still work?

Monitor disk speed in Linux | It`s full of stars! · March 5, 2019 at 11:02

[…] Running a server allows you to do a lot of stuff from remote. Copying files is one of those tasks you can do from anywhere on the world while being logged on via SSH. For this task it is good to know the speed of read/write to get an idea if it’s working s expected. When sitting in front of your computer, you can see if a HDD is working, in Windows you see a MB/s indication, and in Linux? Not all copy commands show you the transfer rate by standard. Some disk intensive tasks won’t at all (RAID sync). […]

How to add a new disk to RAID5 | It`s full of stars! · March 12, 2019 at 10:30

[…] This process will take some time. To learn how to increase the speed the sync, see my other blog about this topic. […]

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.