LVM and Raid Scott Gilliland
Why aren't flat partitions enough? Redundancy Speed Live changes Multipath I/O Shared partitions
Raid Levels Raid-0: Block-level Striping Raid-1: Mirroring Raid-2: Byte striping Raid-3: Byte striping w/ parity disk Raid-4: Block-level striping with parity disk Raid-5: Block-level striping with distributed parity to cover the loss of any disk Raid-6: Block-level striping with distributed parity to cover the loss of any two disks
Raid Levels Raid-0: Block-level Striping Raid-1: Mirroring Raid-2: Byte striping Raid-3: Byte striping w/ parity disk Raid-4: Block-level striping with parity disk Raid-5: Block-level striping with distributed parity to cover the loss of any disk Raid-6: Block-level striping with distributed parity to cover the loss of any two disks
Raid 0 Reads and writes are split between disks For n disks, rates go up by n Loss of any disk looses all data In theory, you still have n-1/n of your data, but really, you lose your data. Great if you have lots of disks, but don't care about your data
Raid 1 N copies of all data Each write has to update all disks Ow, that's slow! Reads can be spread across all disks Like Raid 0, but without the data loss! Great if you only have two disks, and want redundancy Like if you're building a machine with old drives For more n 3, use Raid 5
Raid 5 Distributes the parity information across all disks Can loose any one disk Reads use n-1/n disks Writes write to all n disks Or worse a read of the whole stripe may be needed Stripe sizes matter Especially for large numbers of small,disparate reads Usually a safe bet for 3 or more disks
Raid 6 Usually defined as being Raid 5, but with twice the parity Can loose up to two disks Preferred over 'hot-spares' If you have an extra disk, you may as well use it How else will you know if your hot spare is bad?
Linux Software Raid Combines multiple block devices into one Disks or partitions Often called the MD system Stands for Multi Disk Modern arrays use a 'Superblock' Contains disk UID and array config Allows for auto-detection Actually at end of disk Configured via mdadm
Linux Software Raid (cont) Status from /proc/mdstat And more modernly /sys/md You can even stack Raid sets Raid 0+1, Raid 0+5 Can be 'reshaped' live Can be set to come up on boot But your bootloader will need a flat disk Or it will need to 'understand' RAID
Typical Raid 1 Two disks, one partition each Partitions mirrored as /dev/md0 Each has a bootloader, set to boot with root=/dev/md0 If boot disk fails, other disk can boot to the degraded array scott@galileo:~$ cat /proc/mdstat Personalities : [raid1] md0 : active raid1 hda1[0] hdc1[1] 78123968 blocks [2/2] [UU] unused devices: <none>
Typical Raid 5 A single or mirrored set of disks to boot A Raid 5 set as root, or as storage The mirrored set often covers all disks, as samesized disks will all have the same extra space Here, the spare space is striped together as /tmp -bash-3.2$ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid0] md1 : active raid0 sdc1[0] sdd1[1] sde1[2] sdf1[3] 10248192 blocks 256k chunks md0 : active raid5 sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb3[1] sda3[0] 4870987520 blocks level 5, 256k chunk, algorithm 2 [6/6][UUUUUU]
mdadm Seems to be the best way to change an array Create: # mdadm --create /dev/md0 --level=5 --raid-devices=3 \ /dev/sdc1 /dev/sdd1 /dev/sde1 Set as faulty, remove, add: # mdadm --manage /dev/md0 --fail /dev/sdc1 # mdadm --manage /dev/md0 --remove /dev/sdc1 # mdadm --manage /dev/md0 --add /dev/sdc1
LVM (and LVM2) Can be considered an alternative to traditional partitioning Can use any block device: partitions and full disks Allows for rearranging of storage, mirroring, striping, and snapshots With a daemon, can handle shared storage and multipath Ex: two servers with a shared SAN disk between them Does not do Raid 5 or 6. Or Raid 2, Raid 3, or Raid 4.
LVM Basics 3 main types of entities: PV Physical Volume This is a disk you want to store data on. VG Volume Group A pool of PV's on a machine. You usually just want one. LV Logical Volume An exposed block device. May be spanned, striped, mirrored, or a 'snapshot'.
LVM Snapshots Allows you to take a full-disk image at a single point in time Uses Copy-on-write Only needs additional space when the device is written Needs physical space allocated for changes Useful for taking backups that are selfconsistent Make snapshot, backup contents off-disk, erase snapshot
LVM configuration Command-line tools are simplest All the details in LVM(8) System-config-lvm is the Redhat GUI tool Only handles LVM, making MD+LVM configurations difficult EVMS tool has an LVM2 module Not actively maintained? Also has an MD module for software RAID. KVPM is a kde4-based manager Seems kinda beta, but works Can parse MD devices Handles filesystems
Command-line LVM Most of: [pv vg lv][change convert create display extend reduce remove rename resize s scan] lvm> pvs PV VG Fmt Attr PSize PFree /dev/md0 test lvm2 a- 1.91G 1.62G /dev/sdc2 test lvm2 a- 1.44G 974.00M /dev/sde2 test lvm2 a- 1.39G 1.20G lvm> vgs VG #PV #LV #SN Attr VSize VFree test 3 2 0 wz--n- 4.75G 3.77G lvm> lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert mirrored test mwn-a- 300.00M mirrored_mlog 100.00 striped test -wn-a- 400.00M
Command-line LVM PVS, VGS, and LVS can take additional fields scott@bravil:/usr/src/kvpm-0.4.5b$ sudo pvs -o pv_fmt,pv_size,dev_size,pv_free,pv_used,pv_name,pv_attr,pv_pe_count,pv_pe_alloc_count,pv_tags,pvseg_start,pvseg _size,pe_start,lv_name Fmt PSize DevSize PFree Used PV Attr PE Alloc PV Tags Start SSize 1st PE LV lvm2 1.44G 1.44G 724.00M 750.00M /dev/sdc2 a- 737 375 0 100 192.00K striped lvm2 1.44G 1.44G 724.00M 750.00M /dev/sdc2 a- 737 375 100 150 192.00K mirrored lvm2 1.44G 1.44G 724.00M 750.00M /dev/sdc2 a- 737 375 250 125 192.00K striped lvm2 1.44G 1.44G 724.00M 750.00M /dev/sdc2 a- 737 375 375 362 192.00K lvm2 486.00M 486.31M 486.00M 0 /dev/sdd1 a- 243 0 0 243 192.00K lvm2 1.39G 1.40G 978.00M 450.00M /dev/sde2 a- 714 225 0 100 192.00K striped lvm2 1.39G 1.40G 978.00M 450.00M /dev/sde2 a- 714 225 100 1 192.00K lvm2 1.39G 1.40G 978.00M 450.00M /dev/sde2 a- 714 225 101 125 192.00K striped lvm2 1.39G 1.40G 978.00M 450.00M /dev/sde2 a- 714 225 226 488 192.00K sudo lvs -o +segtype,stripes,stripesize,devices,seg_count,copy_percent LV VG Attr LSize Origin Snap% Move Log Copy% Convert Type #Str Stripe Devices #Seg Copy% mirrored test mwn-a- 300.00M mirrored_mlog 31.33 mirror 2 0 mirrored_mimage_0(0),mirrored_mimage_1(0) 1 31.33 striped test -wn-a- 900.00M striped 2 4.00K /dev/sdc2(0),/dev/sde2(0) 2 striped test -wn-a- 900.00M striped 2 4.00K /dev/sdc2(250),/dev/sde2(101) 2
KVPM
System-config-lvm