Data Storage Management

RAID arrays

The same configuration is used on RTC and ICC. (AOC has a smaller /data array due to reduced storage needs.)

• /boot, /, and swap — XFS formatted, replicated as RAID1 on two 500 GB SSDs.

• /data — XFS formatted, replicated as RAID5 on four 2 TB SSDs (effective capacity: 6 TB).

Restoring failed drives

If a drive has been removed and reinserted, you should check to make sure it’s reassembled after reboot.

1. Check /proc/mdstat for missing drives

   $ cat /proc/mdstat
   Personalities : [raid1] [raid6] [raid5] [raid4]
   md124 : active raid5 sde1[2] sdd1[1] sdc1[0]
       5860144128 blocks super 1.2 level 5, 512k chunk, algorithm  2 [4/3] [UUU_]
       bitmap: 2/15 pages [8KB], 65536KB chunk
   
   md125 : active raid1 sda1[0] sdb1[1]
       470946816 blocks super 1.2 [2/2] [UU]
       bitmap: 2/4 pages [8KB], 65536KB chunk
   
   md126 : active raid1 sdb3[1] sda3[0]
       512000 blocks super 1.2 [2/2] [UU]
       bitmap: 0/1 pages [0KB], 65536KB chunk
   
   md127 : active raid1 sdb2[1] sda2[0]
       16776192 blocks super 1.2 [2/2] [UU]
   

   The [UUU_] is your hint that one of the four drives in /dev/md124 is missing. Now, which one is it?

2. Check for /dev/sdXX devices in lsblk that aren’t parents of /dev/mdXXX devices. (Note: Don’t count USB drives that aren’t supposed to be part of a RAID!)

   For example:

   $ lsblk
   NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   sda         8:0    0 465.8G  0 disk
   ├─sda1      8:1    0 449.3G  0 part
   │ └─md125   9:125  0 449.1G  0 raid1 /
   ├─sda2      8:2    0    16G  0 part
   │ └─md127   9:127  0    16G  0 raid1 [SWAP]
   └─sda3      8:3    0   501M  0 part
     └─md126   9:126  0   500M  0 raid1 /boot
   sdb         8:16   0 465.8G  0 disk
   ├─sdb1      8:17   0 449.3G  0 part
   │ └─md125   9:125  0 449.1G  0 raid1 /
   ├─sdb2      8:18   0    16G  0 part
   │ └─md127   9:127  0    16G  0 raid1 [SWAP]
   └─sdb3      8:19   0   501M  0 part
     └─md126   9:126  0   500M  0 raid1 /boot
   sdc         8:32   0   1.8T  0 disk
   └─sdc1      8:33   0   1.8T  0 part
     └─md124   9:124  0   5.5T  0 raid5
   sdd         8:48   0   1.8T  0 disk
   └─sdd1      8:49   0   1.8T  0 part
     └─md124   9:124  0   5.5T  0 raid5
   sde         8:64   0   1.8T  0 disk
   └─sde1      8:65   0   1.8T  0 part
     └─md124   9:124  0   5.5T  0 raid5
   sdf         8:80   0   1.8T  0 disk
   └─sdf1      8:81   0   1.8T  0 part
   

   In this example, sdf1 did not get reassembled into md124 and isn’t part of the /data RAID5 array.

3. Hot-add the dropped device to the array with mdadm $ sudo mdadm /dev/md124 --add /dev/sdf1

4. Repeat steps 1 and 2 and verify all drives are present in the RAID