Recovering from Hard Drive Disasters. Theodore Ts'o Senior Technical Staff Member IBM

Size: px
Start display at page:

Download "Recovering from Hard Drive Disasters. Theodore Ts'o Senior Technical Staff Member IBM tytso@alum.mit.edu"

Transcription

1 Recovering from Hard Drive Disasters Theodore Ts'o Senior Technical Staff Member IBM

2 Goals of this Class Learn what to do when your data disappears What not to do --- how to not make things worse Gain an understanding of how data is stored on modern systems Be able to recover as much data as possible after a data loss disaster Avoid losing risks of data loss in the first place

3 Logistics Tutorial runs from 10:00 13:00 Break at 11:30 Feel free to ask me questions But I reserve the right to defer the answer until later in the session or to the break/end of the class.

4 Agenda How data can get lost? What to do when data is lost? How is data stored on disk? How to recover from accidents? How to avoid losing data in the first place

5 How does data get lost? User goofs (PEBCAK) Software Bugs Hardware Failures

6 User goofs Accidental file deletion (rm *.o or rm * -rf) Accidental file overwrite Accidental filesystem corruption Accidental partition table corruption

7 Software failures Mistakes caused by software instead of humans. :-) Kernel bugs will tend to cause filesystem or partition corruptions Userspace bugs will tend to cause file-level corruptions or deletions Challenge: discover the data corruption as soon as possible after it happens To avoid incorrect calculations/results Better chance of recovering data

8 Hardware failures Data corruption caused by hardware problems Memory errors Bits flipped during transfer SCSI has a parity check IDE UDMA has a CRC check non-ide PIO/DMA has no data integrity checks at all (Neither does the PCI bus) Controller errors Hard disk crashes Caused by mechanical shocks Caused by dirt or small particles of iron oxide impacting the hard drive heads

9 What to do when data is lost? Don't Panic

10 What to do when data is lost? First, do no harm Determine what happened Create backups if necessary (very often via an image backup) dd if=/dev/hda1 of=/dev/hdb1 bs=1k conv=sync,noerror Try to recover the data

11 How is data stored on disk? Low level hard drive details Partitioning Logical Volume Manager Filesystem structure

12 How hard drives work --- a quick lesson The components of a hard drive What is a head crash? Performing touch and go's Digression: Increasing hard drive life Cascade failures Lies and Fantasies

13 The components of a hard drive One or more iron-oxide covered platters, located inside a sealed enclosure. Spindle motor, which spins the platters at 3600rpm-10,000rpm Read-write heads which fly on an air cushion only a few nanometers above the surface of the platter. A (stepper) motor which moves the heads to the correct track. A controller which accepts ATAPI/SCSI commands and tells the motors and read/write heads what to do.

14 What is a head crash? Springs push the heads towards the platter; the interaction between the spinning platters and the heads creates an air cushion which pushes the heads just above the platters When the head scrapes against the platter, this is called a head crash. Causes of head crashes Sudden power failure Physical shock Contaminants impacting the head

15 Performing Touch and Go's On a controlled shutdown, the heads are moved to a landing zone before the platters stop spinning for a controlled head crash. When the drive starts spinning, the head drags along the platter until the air cushion is established. The probability of failure increases geometrically, with P failure reaching 50% after 50,000 landing/takeoff cycles

16 Digression: Increasing Hard Drive Life For disks designed for use in servers, all that needs to be done is to reduce the number of head landing/takeoffs. Disks designed for desktop do not have spindle motors designed for continuous operation. Other disks (for example on the ipod) may have heat dissipation problems if run continuously

17 Cascade failures When the head smashes against the spinning platter this can scrape off iron oxide which will then fly with great speed around the sealed enclosure. There are devices which attempt to catch these particles, but if there are enough of them, some will impact some of the heads, causing further head crashes. This can cause an exponentially growing number of disk errors, leading to a complete failure of the hard drive.

18 Lies and Fantasies Disks used to have a very simple geometry. Very old disks had an even number of (data) heads Older disks used an odd number of heads --- one side of the platter used as a control surface instead of a stepper motor There were a fixed number of sectors per track Addressed via a (Cylinder, Head, Sector) scheme Modern disks use much more complicated layout But keep a CHS addressing scheme only for backwards compatibility Even the Linear addressing scheme is a fantasy due to bad block sparing schemes.

19 Partitioning Goal: to divide a single large disk into smaller units (i.e. for multiple filesystems) Many different schemes MSDOS (MBR) partitioning BSD disk label Solaris partition tables EFI GUID Macintosh partition tables Ultrix Amiga etc.

20 MBR Partition Table Format Flags (active/bootable) 1 byte Starting location (C,H,S) 3 bytes Ending location (C,H,S) 3 bytes Type 1 byte Starting (relative) LBA 4 byte Number of LBA 4 byte Magic number (0xAA55) Boot loader (446 bytes) Partition entries (16 bytes each)

21 Extended partitions Only room for 4 partitions in the MBR. One of these four partitions can be an extended partition An extended partition contains an Extended Boot Record (EBR) which looks just like an MBR, except Contains at most two entries The first entry must be a valid non-ebr partition The second entry (if present) must be another EBR partition Works like a linked list

22 Extended Partition Example hda1 hda2 hda3 hda5 hda6

23 Problems with MBR One-byte type field, which is often ambiguous/wrong Nested MBR structure very fragile Partition location stored twice Beginning and End (C,H,S) Linear sector number and size Fields are too small for modern disk sizes

24 GUID Partition Table Sometimes also called the EFI partition table, since it is specified by the Extended Firmware Interface specifications Used by the IA64 architecture, but is not IA-64 specific. Features Stored in two locations on disk for redundancy Allows for large partitions Partitions may be named with a human-readable label

25 GUID Partition Header Contains the following information Signature (8 bytes) EFI PART Revision (4 bytes) 0x Header Size (4 bytes) HeaderCRC32 (4 bytes) Reserved (4 bytes) Must be zero MyLBA (8 bytes) AlternateLBA (8 bytes) First Usable LBA (8 bytes) Last Usable LBA (8 bytes) Disk GUID (16 bytes) Paritition Entry array LBA (8 bytes) Number of Partition Entries (4 bytes) Size of Partition Entry (4 bytes) Partition Entry Array CRC32 (4 bytes)

26 GUID Partition Entry Contains the following information: Partition type GUID (16 bytes) Unique partition GUID (16 bytes) Starting LBA (8 bytes) Ending LBA (8 bytes) Attribute bits (8 bytes) Partition name (72 bytes)

27 GUID Partition Table Layout LMBR GPT GPT Header Entries 0.. n Partition GPT GPT Entries Header 0.. n LBA 0 LBA 1 LBA 2 LBA n First Usable Block Last Usable Block

28 Logical Volume Managers Logical extension of partition tables Disk > Volume Group Partition > Logical Volume Additional features Allows a partition to span multiple disks Partitions do not have to be contiguous; Logical Volumes may be resized without moving data Logical volumes may be moved from one disk to another while they are in use Snapshot volumes allow filesystems to be frozen for backup or consistency checking

29 Theory behind LVM's Volume Group contains Physical Volumes and Logical Volumes Physical Volume Typically a hard drive, but may be an individual partition, or a software raid device Physical Extent Each PV is divided into a large number of PE's. A typical extent size is 4MB Logical Volume Some number of PE's are combined to form logical volumes, which work like partitions

30 LVM Implementations LVM1 by Heinz Mauelshagen Used in Linux 2.4 ; removed in Linux 2.5 EVMS by IBM Patches available for Linux 2.4/2.5 Supports multiple LVM formats: LVM1, AIX, OS/2, etc. Could create snapshot volumes The Device Mapper Interface In Linux 2.6; patches exist for Linux 2.4 LVM2 and EVMS2 both use the Device Mapper Interface

31 Filesystems Just as the partition table divide the disk into multiple partitions, a filesystem divides a partition into multiple files Filesystem types: Disk-based filesystems Examples: FAT, ext2, etc. Network-based filesystems Examples: NFS, AFS, Coda, etc.

32 Disk-based Filesystems FAT-based filesystems Inode-based filesystems Log-structured filesystems

33 FAT-based filesystems A family of filesystems first found in MS-DOS, and extended in Windows 95, 98, NT, 2000, etc. Disk divided into clusters of sectors (blocks) Basic data structure: File Allocation Table (FAT) An array for each cluster in the disk Indicates whether the cluster is free, reserved, or the next location in the cluster chain. (Essentially, a linked list)

34 FAT cluster chains Files and directories are FAT cluster chains. In order to access a file/directory, you must know its first cluster Root directory stored in a fixed location Each directory entry contains the first cluster for each file or directory Implication: Random access is hard! One of the reasons why file fragmentation is so disastrous on FAT filesystems

35 (Major) FAT Variants FAT16: supports at most 64k clusters To support larger disks, the cluster size had to get bigger; this caused allocation inefficiencies. FAT32: supports 2 32 clusters VFAT (virtual fat) Originally referred to its implementation: implemented FAT using protected-mode (32-bit) code under Windows Usually refers to its long-file-name support (stores 13 characters per FAT directory for backwards compatibility)

36 Inode-based filesystems Primary example: BSD Fast Filesystem (FFS) and its derivitives Solaris ufs, ext2, etc. Many other inode-based filesystems SGI's xfs NTFSv5

37 Primary characteristics of FFS class filesystems The concept of an inode: separation of the filename (to be stored in the directory) from the rest of the file metadata allows hard links Fixed inode table Division of filesystem into groups Bitmap allocation tables for blocks and inodes Generally, use of simple linked list for directories

38 Ext2 Filesystem Layout Boot BG #0 BG #1... BG #N Super Block FS descriptors Block Bitmap Inode Bitmap Inode Table Data blocks

39 Ext2 Inode structure Mode Owners Size Timestamps... data data data direct blocks indirect d. indirect t. indirect data data data data data data data data

40 Ext2 Directory Layout Inode Table I1 i2 I3 I3 Directory name1 name2 name3 name4

41 Other inode-based filesystems Other more modern filesystems depart from some of the design choices of the traditional FFS Dynamically allocated inode table Use of B-trees For directories For logical->physical block mapping For allocation Focus on contiguous allocation of blocks for performance reasons Add journalling for recovery purposes

42 Reiserfs Developed by Hans Reiser Hans believes that filesystems and databases should be unified No file parsing; just use store each row as directory, and each field as an individual file! Therefore, reiserfs has very good small file support Entire filesystem is one gigantic B-tree, with tunable (pluggable) hash functions Filesystem consistency checker a little weak. Data corruption after unclean shutdowns?

43 JFS Port by IBM of OS/2's JFS filesystem (not AIX's JFS!) Features: Journaling B-tree directories OS/2 compatibility Does not seem to have grabbed much community enthusiasm.

44 XFS Port of SGI's XFS to Linux Uses B-tree's everywhere Directories Free-space (two b-trees) Extents Optimized for scalability to very large filesystems Delayed allocation Very, very, very big code-base

45 Ext3 Based on existing ext2 code Adds journaling support Can use a hashed b-tree in directories for fast lookups Other new features that are coming soon Preallocation reservation On-line resizing

46 Log-structured filesystems Examples: BSD's lfs Important distinction Update-in-place filesystems all of the fillesystems we've covered up until now When you modify a block in a file, you determine where on disk the data is stored, and you overwrite it. Log-structured filesystems The complete opposite is true: an allocated block is never overwritten; only replaced. Blocks are always written at the end of the "log"

47 Updating files with an LFS The basic rule is that blocks with an LFS are never overwritten, only replaced. So to modify a block in a data file: A new block must be allocated and data written to it. The block containing the inode (which contains a pointer to the data block) must be replaced with a new one. The directory entry pointing to the inode must be replaced with a new one. The parent directory must likewise be updated, etc. on up to the root directory.

48 The Log Cleaner LFS treats the entire disk as circular log. There must always be free space at the end of the log to write replacement blocks, inodes, directories, etc. The job of the log cleaner is to coalesce free space by copying used blocks to the head of the log in order to defragment the free space. This is critical to the LFS's performance; the cleaner can slow down the overall system performance if it has to run while the system is busy.

49 LFS Performance Issues LFS performance assumptions Disk seeks are very expensive compared to write operations A lot of caching can make up for dismal read performance Writing is very fast since no seeks are required, even though the root directory must be rewritten each time any data is written to disk, since it's done as a sequential block write.

50 LFS Read Performance Getting LFS Read performance to acceptable speeds is the hard problem with LFS. LFS tends to scatter directories all over the disk, and file fragmentation is extremely common. One solution: have a huge amount of memory dedicated for caching commonly used data structures, to avoid needing to hit the disk. Make the FS cleaner smart enough to be able to reorganize the FS so that reads can be fast. (This requires enough system inactivity so that the LFS has time to run.)

51 Other non-update-in-place Filesystems Other related filesystems that aren't log-structured, but still have the property that allocated blocks are never-overwritten. Shadow-Page Filesystems Uses a maping table which maps logical blocks to physical blocks. Examples: Network Appliances WAFL The Tux2 filesystem uses a "phase-tree" approach.

52 How to recover from accidents? Don't Panic

53 How to recover from accidents? Questions to ask yourself: How did this happen? What is the lowest level that was corrupted? How important is the data? When was the last time backups were performed? Create a plan of attack before you do anything else

54 What has been corrupted/lost? Low-level hard drive data (RAID configuration/data) Partition table LVM Filesystem corruption Application-level file corruption

55 Kernel log messages Hardware problems will often cause error messages sent to the console or the system logs. Examples hda: dma_intr: status=0x51 { DriveReady SeekCompleteError } hda: dma_intr: error=0x40 { UncorrectableError } LBAsect = , sector 1204 end_request: I/O error, dev 03:03 (hda), sector 1204

56 More examples Kernel error logs Bad CRC error messages can mean a problem with the IDE controller cable hdd: dma_intr: status=3d0x51 { DriveReady SeekComplete Error } hdd: dma_intr: error=3d0x84 { DriveStatusError BadCRC } SCSI errors SCSI disk error : host 0 channel 0 id 1 lun 0 return code = I/O error: dev 08:13, sector 49272

57 In case of hardware errors Examine the logs to see how many errors are reporting Is this an isolated case or a prelude to massive hard drive failure? Check the S.M.A.R.T. reports If it appears the disk might be failing, do a complete backup doing anything else! An image backup can be the best way to grab as much data as possible before the disk fails entirely

58 Performing Image Backups Using DD dd if=/dev/hda1 of=/dev/hdb1 bs=1k conv=sync,noerror Using dd_rescue A more refined tool Faster Provides progress bar information Can be interrupted and restarted more easily Can be found at:

59 Hard Drive Data Recovery Services There are companies that specialize in recovering data from crashed drives. Some will do only logical data recovery (i.e., what we're talking about in this class) Others can recover data from physically damaged disk Questions to ask yourself How valuable is your data? Do you have alternatives? How confident are you? How much time do you have? Questions to ask Do you have a clean room? How much do you charge? Do you charge if no data can be recovered?

60 Partition table problems If the partition table is corrupt, then the filesystem cannot be found --- either by the kernel or by the userspace utilities If the filesystem cannot be found, this is the first time thing you check. fdisk -l /dev/hda Device Start End Blocks Id System /dev/hda1* Linux /dev/hda Linux swap /dev/hda Linux /dev/hda Extended /dev/hda Linux /dev/hda Linux /dev/hda Linux

61 Backing up the partition table Tape a printed copy of fdisk -l to the disk drive Backing up the MBR/partition table to a file dd if=/dev/hda of=mbr count=1 bs=512 To restore just the partition table (without disturbing the boot code) dd if=mbr of=/dev/hda bs=1 count=64 skip=446 seek=446 If lilo was installed, there may be a backup copy in /boot/boot.0300 (for IDE disks) or /boot/boot.0800 (for SCSI). Don't rely on this, though!

62 Using Gpart to find the partition table For when you don't have a backup of the partition table. To use gpart to scan a disk: gpart /dev/hda To write the new partition table to a file gpart -W /tmp/part.img /dev/hda If you're really brave: gpart -b /mbr.hda.bak -W /dev/hda /dev/hda Gpart doesn't always work; MBR backup's are a much safer bet!!

63 Common LVM Problems LVM Metadata corruption Extremely unlikely, since by default two copies of the metadata are stored on each Physical Volume There is also a copy stored in /etc/lvm. Good idea to back this up periodically May be written back to a physical disk using the vgcfgrestore(8) program One or more missing physical volumes (PV's) in a Volume Group e.g, caused by hardware failure LVM is not a substitute for RAID Data loss is very likely

64 Recovering from a missing PV Simplest solution: upgrade to LVM2 (if you are not using it already) Link /dev/ioerror to /dev/zero Execute the command: vgchange partial -ay This will mount the volume group in read only mode, and allow you to recover those LV's partially span the missing disks To permanently remove the missing PV's, use the command: vgreduce removemissing vg_name

65 Filesystem corruption problems Can be reported by e2fsck during boot time If serious enough, the boot sequence will abort, so a system administrator can look at the system Can be reported by the kernel (check system logs) EXT3-fs error (device md(9,2)): ext3_readdir: bad entry in directory # : rec_len %% 4!= 0 - offset=0, inode= , rec_len=20275, name_len=73 When the kernel discovers a corruption problem, it can either continue and ignore the error, remount the filesystem read-only, or panic. Controlled via: tune2fs -e <continue remount-ro panic> /dev/hda1

66 Running e2fsck Primary tool for fixing filesystem inconsistencies in the ext2/ext3 filesystems. Strong emphasis on not losing user data. In general the default answer of yes is safe, but sometimes human intervention can do better. After running e2fsck, sometimes files will be missing or corrupted. E2fsck will always ask before doing actions that remove directory entries or inodes. Usually this means there was nothing left to save, but it's important for the administrator to know about the damage/missing files.

67 Common filesystem corruptions Inaccurate filesystem accounting (pass 5) Inaccurate entries in the inode/block bitmaps Number of free inodes and directories per block group and in the entire filesystem Number of links in an inode Directory entries pointing at invalid inodes Allocated inodes not referenced by a directory Blocks claimed by multiple inodes Corrupted directory entries Invalid fields in inode structure

68 The lost+found directory Where e2fsck will place orphaned inodes (both files and directories) It is up to the system administrator to determine where the files originally came from, and to restore them to the correct location. This is easier for directories, since the filenames in the directory are often a strong hint. The locate database can extremely useful for this purpose For files, use the file command to determine the file type, and then inspect the contents of the file to determine

69 Directory entries pointing at invalid inodes These will be deleted by e2fsck Important that the system administrator determine how to recover these files from backups, or by reloading the appropriate.rpm/.dpkg

70 Blocks claimed by multiple inodes Usually caused by inaccurate block allocation bitmaps E2fsck detects these problems during pass 1, and will start additional passes (passes 1b, 1c, and 1d) These passes can take a large amount of memory and time if there are a large number of blocks and inodes that are corrupted in this fashion. If this the case, it may be caused by part of the inode table being written to the wrong place. E2fsck will offer to make copies of the affected blocks so that each inode has its own copy of the blocks. Or the affected inode can be removed.

71 Corrupted inode table Generally caused by hardware errors Data written to the wrong place Garbage written to the right place This is can lead to a large amount of damage, since there is no redundancy to the information in the inode table. Signs this may have happened Large numbers of inodes with invalid block numbers or garbaged fields Large numbers of multiply claimed blocks If you suspect this may have happened, perform an image level backup immediately.

72 Finding data without the inode Suppose the inode table has been trashed Hardware failure Administrator accidentally (or on purpose) runs mke2fs on the (wrong) disk Critical data is on the filesystem that wasn't backed up. What then? The data is still there, but we have no idea where it might be.

73 Undeleting files To protect against user errors, use user-level delete/undelete programs that don't really unlink() files. But if a file is really unlinked, it may be possible to save it via an undeletion process. If you are using ext2 (and not ext3) If the blocks or inode has not be reused since the file was deleted. unmount the filesystem or remount it read-only immediately Use the debugfs's lsdel to find the deleted file. Then use debugfs's dump command to write the recovered file.

74 Undeleting files using debugfs # debugfs /var/tmp/test.img debugfs 1.35 (28-Feb-2004) debugfs: lsdel Inode Owner Mode Size Blocks Time deleted / 624 Wed Jun 16 20:55: deleted inodes found. debugfs: dump <12> /tmp/recovery debugfs: quit

75 Desperation searches Looking for a string somewhere on disk. This will list block numbers that contain the requested regular expression grep -ab regexp /dev/hda1 \ awk -F: '{printf("%x\n", ($1+1023)/1024);}' Replace 1023,1024 above with 4095,4096 if the filesystem is using a 4k blocksize. Use the lde program to examine the block numbers, and save the requested blocks to a recovery file. Ext2 tries to store files contiguously; this is a help.

76 How to avoid losing data in the first place Backups, backups, backups! SMART Checksum tests RAID

77 Backups Do them. Enough said. Automatic backups means you won't forget. Disk space is cheap. Reason to do backups Recovery from hardware failure Recovery from user oops Recovery from buggy programs File corruption may not be discovered until several days or weeks later

78 Sample backup scripts These scripts are used to backup my church's fileserver Incremental scripts run automatically out of / etc/cron.daily So if the file has been existence for at least one day, we can always recover some older version of the file Backup files are periodically written to CD-R

79 backup-full-data #!/bin/sh export PATH="/sbin:/usr/sbin:/bin:/usr/bin" BACKUP_DIR="/u1/backup" SOURCE_DIR="/home" TAR_NAME="backup-full-data" DATE=`date +%Y-%m-%d-%H%M` FN="$TAR_NAME-$DATE.tar.bz" cd $SOURCE_DIR echo $FN >.$TAR_NAME.timestamp tar cf -. bzip2 > $BACKUP_DIR/$FN exit 0

80 backup-incr-data export PATH="/sbin:/usr/sbin:/bin:/usr/bin" BACKUP_DIR="/u1/backup" SOURCE_DIR="/home" TAR_NAME="backup-incr-data" DATE=`date +%Y-%m-%d-%H%M` FN="$TAR_NAME-$DATE.tar.bz" TMP="/tmp/backup-incr-$$" cd $SOURCE_DIR echo $FN > $SOURCE_DIR/.$TAR_NAME.timestamp find. -type f -newer.backup-full-data.timestamp > $TMP tar -cf - --files-from $TMP.*.timestamp bzip2 > \ $BACKUP_DIR/$FN rm $TMP exit 0

81 S.M.A.R.T. Acronym for Self-Monitoring, Analysis, and Reporting Technology S.M.A.R.T attributes Vendor-defined metrics of drive health Reallocated sector count read channel margin seek error rate seek time performance

82 S.M.A.R.T. attributes ATTRIBUTE_NAME WORST TYPE WHEN_FAILED VALUE THRESH UPDATED RAW_VALUE Raw_Read_Error_Rate Pre-fail Always Throughput_Performance Pre-fail Offline - 0 Spin_Up_Time Pre-fail Always Start_Stop_Count Old_age Always - 38 Reallocated_Sector_Ct Pre-fail Always FAILING_NOW 1887 Seek_Error_Rate Pre-fail Always - 0 Seek_Time_Performance Pre-fail Offline - 0 Power_On_Hours Old_age Always Spin_Retry_Count Pre-fail Always - 0 Power_Cycle_Count Old_age Always - 38 Power-Off_Retract_Count Old_age Always - 38 Load_Cycle_Count Old_age Always - 38 Temperature_Celsius Old_age Always - 34 Reallocated_Event_Count Old_age Always Current_Pending_Sector Old_age Always - 0 Offline_Uncorrectable Old_age Offline - 0 UDMA_CRC_Error_Count Old_age Always - 0

83 Automatic S.M.A.R.T. monitoring You can check S.M.A.R.T. health by using the smartctl command smartctl -H /dev/hda smartctl -A /dev/hda Better to have this be done automatically via the smartd daemon.

84 /etc/smartd.conf One line per device. For example, to monitor all attributes, enable automatic online data collection, automatic Attribute autosave, and do a short self-test every day at 2am, and a long self test Saturdays at 3am: /dev/hda -a -o on -S on \ -s (S/../.././02 L/../../6/03) \ -m

85 Checksum tests Goals of doing checksums Detect malicious data modification Requires crypto checksums Example tool: integrit Detect accidental data corruption Cyclic Redundancy Checks (CRC's) are sufficient Example tool: cfv (command-line file verify)

86 The CFV tool Available from Implemented in Python To create a checksum file: cfv -C -rr -f checksum.sfv.gz /u1 To verify a checksum file: cfv -f checksum.sfv.gz

87 Backing up ext2/3 metadata using e2image If the ext2 inode table is lost, there are very few recovery options. E2image will back up critical ext2/3 filesystem metadata to a file. Make sure you put the file on a different filesystem than the one you are backing up!! Usage is very simple: e2image /dev/hda3 /u1/hda3.e2i A typical e2image file for a 30 gigabyte filesystem is approximately will take ~30 seconds to create, and will consume megabytes

88 Using the e2image file Run dumpe2fs or debugfs with the -i flag For example: dumpe2fs -i /u1/hda3.e2i In e2fsprogs 1.36, the debugfs command will accept the -d flag to specify the original device: debugfs -d /dev/hda3 -i /u1/hda3.e2i This will cause debugfs to use /dev/hda3 to find data, directory, and indirect blocks (which are not stored in the e2i file). This is enough to use the debugfs ls, cat, and dump commands

89 Restoring the e2image metadata to the filesystem Very dangerous if you write the image file to the wrong filesystem! Command: e2image -I /dev/hda3 /u1/hda3.e2i This will allow you to (mostly) recover from a mke2fs If you remember to run e2image before running mke2fs The root directory still ends up getting trashed

90 RAID Acronym for Redundant Array of Inexpensive Disks Original idea by Patterson, Gibson, and Katz of UC Berkeley in 1987 Different levels of RAID RAID 0 data striping RAID 1 data mirroring RAID 5 data striping across multiple drives, with a rotating parity stripe

91 Software vs. Hardware RAID Advantages of Software RAID Cheaper Simpler (managing hardware RAID controllers can be tricky) You can use RAID across different host bus adaptors Advantages of Hardware RAID Performance Booting off a RAIDed volumes Cross-OS compatibility Battery backup means no need to rebuild after an unclean shutdown

92 Using Linux's software raid The /etc/raidtab file Creating a raid device

93 The /etc/raidtab file raiddev /dev/md0 raid-level 5 nr-raid-disks 3 nr-spare-disks 1 persistent-superblock 1 parity-algorithm left-symmetric chunk-size 32 device /dev/sda3 raid-disk 0 device /dev/sdb1 raid-disk 1 device /dev/sdc1 raid-disk 2 device /dev/sdd1 spare-disk 0

94 Creating a raid device Very simple, once the /etc/raidtab file is created: mkraid /dev/md0 Change the partition type of the raid components to be 0xFD (so that autodetection works correctly) Create your filesystem on /dev/md0 If you are using ext2/ext3, add use the RAID stride option, and set it to the chunk size divided by the blocksize. (i.e, if you are using a chunk size of 32k and 4k blocksize, set stride to 8): mke2fs -R stride=8 /dev/md0

95 Managing a RAID array To look at the status of an array mdadm --detail /dev/md0 To simulate a failure mdadm --set-faulty /dev/md0 /dev/sdb1 To remove a failed disk from the array mdadm remove /dev/md0 /dev/sdb1 To add a new disk to the array mdadm add /dev/md0 /dev/sdb1

96 Monitoring a RAID array In your init scripts, add the following (if your distributions hasn't done this already) mdadm /dev/md0 This will send mail automatically when a disk fails, or some other RAID event happens

97 Raid Caveats Remember RAID is not a substitute for backups! Raid does not protect against bus or media corruption!!

98 Conclusion The best defense against data loss 1)Backups 2)Vigilance against impending hardware failures 3)Recovery techniques DON'T PANIC If you know how the data corruption started, you'll know what level to start addressing the problem. With enough care, you can always usually get your data back.

99 Legal Statement This work represents the view of the authors and does not necessarily represent the view of IBM. IBM is a registered trademark of International Business Machines Corporation in the United States and/or other countries. Linux is a registered trademark of Linus Torvalds. Other company, product, and service names may be trademarks or service marks of others.

Typing some stupidities in text files, databases or whatever, where does it fit? why does it fit there, and how do you access there?

Typing some stupidities in text files, databases or whatever, where does it fit? why does it fit there, and how do you access there? Filesystems, LVM, MD Typing some stupidities in text files, databases or whatever, where does it fit? why does it fit there, and how do you access there? Filesystem - Introduction Short description of

More information

LVM2 data recovery. Milan Brož mbroz@redhat.com. LinuxAlt 2009, Brno

LVM2 data recovery. Milan Brož mbroz@redhat.com. LinuxAlt 2009, Brno LVM2 data recovery Milan Brož mbroz@redhat.com LinuxAlt 2009, Brno Linux IO storage stack [ VFS ] filesystem [ volumes ] MD / LVM / LUKS / MPATH... [ partitions ] legacy partition table recovery from the

More information

File Systems Management and Examples

File Systems Management and Examples File Systems Management and Examples Today! Efficiency, performance, recovery! Examples Next! Distributed systems Disk space management! Once decided to store a file as sequence of blocks What s the size

More information

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System CS341: Operating System Lect 36: 1 st Nov 2014 Dr. A. Sahu Dept of Comp. Sc. & Engg. Indian Institute of Technology Guwahati File System & Device Drive Mass Storage Disk Structure Disk Arm Scheduling RAID

More information

CS 153 Design of Operating Systems Spring 2015

CS 153 Design of Operating Systems Spring 2015 CS 153 Design of Operating Systems Spring 2015 Lecture 22: File system optimizations Physical Disk Structure Disk components Platters Surfaces Tracks Arm Track Sector Surface Sectors Cylinders Arm Heads

More information

Linux Software Raid. Aug 2010. Mark A. Davis

Linux Software Raid. Aug 2010. Mark A. Davis Linux Software Raid Aug 2010 Mark A. Davis a What is RAID? Redundant Array of Inexpensive/Independent Drives It is a method of combining more than one hard drive into a logic unit for the purpose of: Increasing

More information

Linux Filesystem Comparisons

Linux Filesystem Comparisons Linux Filesystem Comparisons Jerry Feldman Boston Linux and Unix Presentation prepared in LibreOffice Impress Boston Linux and Unix 12/17/2014 Background My Background. I've worked as a computer programmer/software

More information

Recovering Deleted Files in Linux

Recovering Deleted Files in Linux Recovering Deleted Files in Linux Brian Buckeye and Kevin Liston Most systems administrators have experienced a situation where a vital file has accidentally been deleted without a recent backup. In this

More information

Storage Administration Guide

Storage Administration Guide www.suse.com/documentation Storage Administration Guide SUSE Linux Enterprise Server 10 SP3/SP4 March 6, 2011 Legal Notices Novell, Inc. makes no representations or warranties with respect to the contents

More information

Filesystems Performance in GNU/Linux Multi-Disk Data Storage

Filesystems Performance in GNU/Linux Multi-Disk Data Storage JOURNAL OF APPLIED COMPUTER SCIENCE Vol. 22 No. 2 (2014), pp. 65-80 Filesystems Performance in GNU/Linux Multi-Disk Data Storage Mateusz Smoliński 1 1 Lodz University of Technology Faculty of Technical

More information

Lecture 18: Reliable Storage

Lecture 18: Reliable Storage CS 422/522 Design & Implementation of Operating Systems Lecture 18: Reliable Storage Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of

More information

LVM and Raid. Scott Gilliland

LVM and Raid. Scott Gilliland LVM and Raid Scott Gilliland Why aren't flat partitions enough? Redundancy Speed Live changes Multipath I/O Shared partitions Raid Levels Raid-0: Block-level Striping Raid-1: Mirroring Raid-2: Byte striping

More information

TELE 301 Lecture 7: Linux/Unix file

TELE 301 Lecture 7: Linux/Unix file Overview Last Lecture Scripting This Lecture Linux/Unix file system Next Lecture System installation Sources Installation and Getting Started Guide Linux System Administrators Guide Chapter 6 in Principles

More information

COS 318: Operating Systems

COS 318: Operating Systems COS 318: Operating Systems File Performance and Reliability Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall10/cos318/ Topics File buffer cache

More information

Redundant Array of Inexpensive/Independent Disks. RAID 0 Disk striping (best I/O performance, no redundancy!)

Redundant Array of Inexpensive/Independent Disks. RAID 0 Disk striping (best I/O performance, no redundancy!) 1 Data storage A Simple Overview 1. HW: SCSI/IDE/SATA/SAS (SAN!) 2. HW/SW: RAID 3. SW: Logical volumes 4. SW: Journalling filesystems 5. SW/NET: Networked filesystem (NAS!) DAS-NAS-SAN 2 RAID Redundant

More information

Creating a Cray System Management Workstation (SMW) Bootable Backup Drive

Creating a Cray System Management Workstation (SMW) Bootable Backup Drive Creating a Cray System Management Workstation (SMW) Bootable Backup Drive This technical note provides the procedures to create a System Management Workstation (SMW) bootable backup drive. The purpose

More information

Chapter 2 Array Configuration [SATA Setup Utility] This chapter explains array configurations using this array controller.

Chapter 2 Array Configuration [SATA Setup Utility] This chapter explains array configurations using this array controller. Embedded MegaRAID SATA User's Guide Areas Covered Before Reading This Manual This section explains the notes for your safety and conventions used in this manual. Chapter 1 Overview This chapter introduces

More information

Storage. Presenter:! Robert Wang

Storage. Presenter:! Robert Wang Storage Presenter:! Robert Wang Linux s Abstraction (vfs) (file systems) (physical devices) Storage Device Disk Drive Multiple Drives RAID! Redundant Array of Independent/Inexpensive Disks! Software or

More information

Data storage, backup and restore

Data storage, backup and restore , backup and restore IMT3292 - System Administration November 25, 2008 A Simple Overview 1 HW: SCSI/IDE/SATA/SAS (SAN!) 2 HW/SW: 3 SW: Logical volumes 4 SW: Journalling filesystems 5 SW/NET: Networked

More information

EMC DATA DOMAIN DATA INVULNERABILITY ARCHITECTURE: ENHANCING DATA INTEGRITY AND RECOVERABILITY

EMC DATA DOMAIN DATA INVULNERABILITY ARCHITECTURE: ENHANCING DATA INTEGRITY AND RECOVERABILITY White Paper EMC DATA DOMAIN DATA INVULNERABILITY ARCHITECTURE: ENHANCING DATA INTEGRITY AND RECOVERABILITY A Detailed Review Abstract No single mechanism is sufficient to ensure data integrity in a storage

More information

NSS Volume Data Recovery

NSS Volume Data Recovery NSS Volume Data Recovery Preliminary Document September 8, 2010 Version 1.0 Copyright 2000-2010 Portlock Corporation Copyright 2000-2010 Portlock Corporation Page 1 of 20 The Portlock storage management

More information

Software RAID on Red Hat Enterprise Linux v6

Software RAID on Red Hat Enterprise Linux v6 Software RAID on Red Hat Enterprise Linux v6 Installation, Migration and Recovery November 2010 Ashokan Vellimalai Raghavendra Biligiri Dell Enterprise Operating Systems THIS WHITE PAPER IS FOR INFORMATIONAL

More information

1 File Management. 1.1 Naming. COMP 242 Class Notes Section 6: File Management

1 File Management. 1.1 Naming. COMP 242 Class Notes Section 6: File Management COMP 242 Class Notes Section 6: File Management 1 File Management We shall now examine how an operating system provides file management. We shall define a file to be a collection of permanent data with

More information

COS 318: Operating Systems. File Layout and Directories. Topics. File System Components. Steps to Open A File

COS 318: Operating Systems. File Layout and Directories. Topics. File System Components. Steps to Open A File Topics COS 318: Operating Systems File Layout and Directories File system structure Disk allocation and i-nodes Directory and link implementations Physical layout for performance 2 File System Components

More information

RAID Technology Overview

RAID Technology Overview RAID Technology Overview HP Smart Array RAID Controllers HP Part Number: J6369-90050 Published: September 2007 Edition: 1 Copyright 2007 Hewlett-Packard Development Company L.P. Legal Notices Copyright

More information

Acronis Disk Director 11 Advanced Server. Quick Start Guide

Acronis Disk Director 11 Advanced Server. Quick Start Guide Acronis Disk Director 11 Advanced Server Quick Start Guide Copyright Acronis, Inc., 2000-2010. All rights reserved. Acronis and Acronis Secure Zone are registered trademarks of Acronis, Inc. "Acronis Compute

More information

Shared Storage Setup with System Automation

Shared Storage Setup with System Automation IBM Tivoli System Automation for Multiplatforms Authors: Markus Müller, Fabienne Schoeman, Andreas Schauberer, René Blath Date: 2013-07-26 Shared Storage Setup with System Automation Setup a shared disk

More information

TestDisk Step By Step CGSecurity

TestDisk Step By Step CGSecurity This Recovery example guides you through TestDisk step by step to recover a missing partition and repair a corrupted one. Translation of this TestDisk manual to other languages are welcome. Example Problem

More information

Rsync: The Best Backup System Ever

Rsync: The Best Backup System Ever LinuxFocus article number 326 http://linuxfocus.org Rsync: The Best Backup System Ever by Brian Hone About the author: Brian Hone is a system administrator and software developer at

More information

Optimizing Ext4 for Low Memory Environments

Optimizing Ext4 for Low Memory Environments Optimizing Ext4 for Low Memory Environments Theodore Ts'o November 7, 2012 Agenda Status of Ext4 Why do we care about Low Memory Environments: Cloud Computing Optimizing Ext4 for Low Memory Environments

More information

File System Management

File System Management Lecture 7: Storage Management File System Management Contents Non volatile memory Tape, HDD, SSD Files & File System Interface Directories & their Organization File System Implementation Disk Space Allocation

More information

Intel Entry Storage System SS4200-E Array Recovery

Intel Entry Storage System SS4200-E Array Recovery Intel Entry Storage System SS4200-E Array Recovery These procedures are provided as is with no warranty whatsoever, including any warranty of merchantability, fitness for any particular purpose, or any

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Introduction to I/O and Disk Management

Introduction to I/O and Disk Management Introduction to I/O and Disk Management 1 Secondary Storage Management Disks just like memory, only different Why have disks? Memory is small. Disks are large. Short term storage for memory contents (e.g.,

More information

Installing a Second Operating System

Installing a Second Operating System Installing a Second Operating System Click a link below to view one of the following sections: Overview Key Terms and Information Operating Systems and File Systems Managing Multiple Operating Systems

More information

Chapter 12: Mass-Storage Systems

Chapter 12: Mass-Storage Systems Chapter 12: Mass-Storage Systems Chapter 12: Mass-Storage Systems Overview of Mass Storage Structure Disk Structure Disk Attachment Disk Scheduling Disk Management Swap-Space Management RAID Structure

More information

Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition

Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition Chapter 11: File System Implementation 11.1 Silberschatz, Galvin and Gagne 2009 Chapter 11: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation

More information

CSE 120 Principles of Operating Systems

CSE 120 Principles of Operating Systems CSE 120 Principles of Operating Systems Fall 2004 Lecture 13: FFS, LFS, RAID Geoffrey M. Voelker Overview We ve looked at disks and file systems generically Now we re going to look at some example file

More information

How to Choose your Red Hat Enterprise Linux Filesystem

How to Choose your Red Hat Enterprise Linux Filesystem How to Choose your Red Hat Enterprise Linux Filesystem EXECUTIVE SUMMARY Choosing the Red Hat Enterprise Linux filesystem that is appropriate for your application is often a non-trivial decision due to

More information

Storage and File Systems. Chester Rebeiro IIT Madras

Storage and File Systems. Chester Rebeiro IIT Madras Storage and File Systems Chester Rebeiro IIT Madras 1 Two views of a file system system calls protection rwx attributes Application View Look & Feel File system Hardware view 2 Magnetic Disks Chester Rebeiro

More information

Areas Covered. Chapter 1 Features (Overview/Note) Chapter 2 How to Use WebBIOS. Chapter 3 Installing Global Array Manager (GAM)

Areas Covered. Chapter 1 Features (Overview/Note) Chapter 2 How to Use WebBIOS. Chapter 3 Installing Global Array Manager (GAM) PRIMERGY RX300 S2 Onboard SCSI RAID User s Guide Areas Covered Chapter 1 Features (Overview/Note) This chapter explains the overview of the disk array and features of the SCSI array controller. Chapter

More information

EaseUS Partition Master

EaseUS Partition Master Reviewer s Guide Contents Introduction... 2 Chapter 1... 3 What is EaseUS Partition Master?... 3 Versions Comparison... 4 Chapter 2... 5 Using EaseUS Partition Master... 5 Partition Manager... 5 Disk &

More information

Practical Online Filesystem Checking and Repair

Practical Online Filesystem Checking and Repair Practical Online Filesystem Checking and Repair Daniel Phillips Samsung Research America (Silicon Valley) d.phillips@partner.samsung.com 1 2013 SAMSUNG Electronics Co. Why we want online checking: Fsck

More information

Review. Lecture 21: Reliable, High Performance Storage. Overview. Basic Disk & File System properties CSC 468 / CSC 2204 11/23/2006

Review. Lecture 21: Reliable, High Performance Storage. Overview. Basic Disk & File System properties CSC 468 / CSC 2204 11/23/2006 S 468 / S 2204 Review Lecture 2: Reliable, High Performance Storage S 469HF Fall 2006 ngela emke rown We ve looked at fault tolerance via server replication ontinue operating with up to f failures Recovery

More information

The Logical Volume Manager (LVM)

The Logical Volume Manager (LVM) Page 1 WHITEPAPER The Logical Volume Manager (LVM) This document describes the LVM in SuSE Linux. It is freely distributable as long as it remains unchanged. SuSE has included a Logical Volume Manager

More information

Two Parts. Filesystem Interface. Filesystem design. Interface the user sees. Implementing the interface

Two Parts. Filesystem Interface. Filesystem design. Interface the user sees. Implementing the interface File Management Two Parts Filesystem Interface Interface the user sees Organization of the files as seen by the user Operations defined on files Properties that can be read/modified Filesystem design Implementing

More information

Support for Storage Volumes Greater than 2TB Using Standard Operating System Functionality

Support for Storage Volumes Greater than 2TB Using Standard Operating System Functionality Support for Storage Volumes Greater than 2TB Using Standard Operating System Functionality Introduction A History of Hard Drive Capacity Starting in 1984, when IBM first introduced a 5MB hard drive in

More information

technology brief RAID Levels March 1997 Introduction Characteristics of RAID Levels

technology brief RAID Levels March 1997 Introduction Characteristics of RAID Levels technology brief RAID Levels March 1997 Introduction RAID is an acronym for Redundant Array of Independent Disks (originally Redundant Array of Inexpensive Disks) coined in a 1987 University of California

More information

Chip Coldwell Senior Software Engineer, Red Hat

Chip Coldwell Senior Software Engineer, Red Hat Chip Coldwell Senior Software Engineer, Red Hat Classical Storage Stack: the top layer File system examples: ext3, NFS, GFS built on Linux VFS ( Virtual File System abstraction) defines on-disk structure:

More information

File System Forensics FAT and NTFS. Copyright Priscilla Oppenheimer 1

File System Forensics FAT and NTFS. Copyright Priscilla Oppenheimer 1 File System Forensics FAT and NTFS 1 FAT File Systems 2 File Allocation Table (FAT) File Systems Simple and common Primary file system for DOS and Windows 9x Can be used with Windows NT, 2000, and XP New

More information

Data Storage and Backup. Sanjay Goel School of Business University at Albany, SUNY

Data Storage and Backup. Sanjay Goel School of Business University at Albany, SUNY Data Storage and Backup Sanjay Goel School of Business University at Albany, SUNY Data Backup 2 Data Backup Why? Files can be accidentally deleted Mission-critical data can become corrupt. Natural disasters

More information

Table of Contents. Software-RAID-HOWTO

Table of Contents. Software-RAID-HOWTO Table of Contents The Software-RAID HOWTO...1 Jakob Østergaard jakob@unthought.net and Emilio Bueso bueso@vives.org...1 1. Introduction...1 2. Why RAID?...1 3. Devices...1 4. Hardware issues...1 5. RAID

More information

How you configure Iscsi target using starwind free Nas software & configure Iscsi initiator on Oracle Linux 6.4

How you configure Iscsi target using starwind free Nas software & configure Iscsi initiator on Oracle Linux 6.4 How you configure Iscsi target using starwind free Nas software & configure Iscsi initiator on Oracle Linux 6.4 Download the software from http://www.starwindsoftware.com/ Click on products then under

More information

File-System Implementation

File-System Implementation File-System Implementation 11 CHAPTER In this chapter we discuss various methods for storing information on secondary storage. The basic issues are device directory, free space management, and space allocation

More information

Recovering Data from Windows Systems by Using Linux

Recovering Data from Windows Systems by Using Linux Recovering Data from Windows Systems by Using Linux Published by the Open Source Software at Microsoft, May 27 Special thanks to Chris Travers, Contributing Author to the Open Source Software Lab Most

More information

Acronis Disk Director 11 Advanced Server. User's Guide

Acronis Disk Director 11 Advanced Server. User's Guide Acronis Disk Director 11 Advanced Server User's Guide Copyright Acronis, Inc., 2000-2010. All rights reserved. Acronis and Acronis Secure Zone are registered trademarks of Acronis, Inc. "Acronis Compute

More information

FAT32 vs. NTFS Jason Capriotti CS384, Section 1 Winter 1999-2000 Dr. Barnicki January 28, 2000

FAT32 vs. NTFS Jason Capriotti CS384, Section 1 Winter 1999-2000 Dr. Barnicki January 28, 2000 FAT32 vs. NTFS Jason Capriotti CS384, Section 1 Winter 1999-2000 Dr. Barnicki January 28, 2000 Table of Contents List of Figures... iv Introduction...1 The Physical Disk...1 File System Basics...3 File

More information

USB 2.0 Flash Drive User Manual

USB 2.0 Flash Drive User Manual USB 2.0 Flash Drive User Manual 1 INDEX Table of Contents Page 1. IMPORTANT NOTICES...3 2. PRODUCT INTRODUCTION...4 3. PRODUCT FEATURES...5 4. DRIVER INSTALLATION GUIDE...6 4.1 WINDOWS 98 / 98 SE... 6

More information

Using the Software RAID Functionality of Linux

Using the Software RAID Functionality of Linux Using the Software RAID Functionality of Linux Kevin Carson Solution Architect Hewlett-Packard (Canada) Co. 2004 Hewlett-Packard Development Company, L.P. The information contained herein is subject to

More information

CHAPTER 17: File Management

CHAPTER 17: File Management CHAPTER 17: File Management The Architecture of Computer Hardware, Systems Software & Networking: An Information Technology Approach 4th Edition, Irv Englander John Wiley and Sons 2010 PowerPoint slides

More information

TUXERA NTFS for Mac USER GUIDE 2/13. Index

TUXERA NTFS for Mac USER GUIDE 2/13. Index 2/13 Index 1. Introduction to Tuxera NTFS for Mac... 3 1.1 What is a file system driver?... 3 1.2 How does it work?... 3 2. Installation... 5 2.1 Installing Tuxera NTFS for Mac... 5 2.2 Uninstalling Tuxera

More information

PIONEER RESEARCH & DEVELOPMENT GROUP

PIONEER RESEARCH & DEVELOPMENT GROUP SURVEY ON RAID Aishwarya Airen 1, Aarsh Pandit 2, Anshul Sogani 3 1,2,3 A.I.T.R, Indore. Abstract RAID stands for Redundant Array of Independent Disk that is a concept which provides an efficient way for

More information

DELL RAID PRIMER DELL PERC RAID CONTROLLERS. Joe H. Trickey III. Dell Storage RAID Product Marketing. John Seward. Dell Storage RAID Engineering

DELL RAID PRIMER DELL PERC RAID CONTROLLERS. Joe H. Trickey III. Dell Storage RAID Product Marketing. John Seward. Dell Storage RAID Engineering DELL RAID PRIMER DELL PERC RAID CONTROLLERS Joe H. Trickey III Dell Storage RAID Product Marketing John Seward Dell Storage RAID Engineering http://www.dell.com/content/topics/topic.aspx/global/products/pvaul/top

More information

Unix System Administration

Unix System Administration Unix System Administration Caleb Phillips Data Redundancy CSCI 4113, Spring 2010 Where are We? Advanced Topics NTP and Cron Data Redundancy (RAID, rsync, backups...) Division of Labor (NFS, NIS, PAM, LDAP,...)...

More information

This chapter explains how to update device drivers and apply hotfix.

This chapter explains how to update device drivers and apply hotfix. MegaRAID SAS User's Guide Areas Covered Before Reading This Manual This section explains the notes for your safety and conventions used in this manual. Chapter 1 Overview This chapter explains an overview

More information

The BRU Advantage. Contact BRU Sales. Technical Support. Email: brusales@tolisgroup.com T: 480.505.0488 F: 480.505.0493 W: www.tolisgroup.

The BRU Advantage. Contact BRU Sales. Technical Support. Email: brusales@tolisgroup.com T: 480.505.0488 F: 480.505.0493 W: www.tolisgroup. The BRU Advantage The following information is offered to detail the key differences between the TOLIS Group, Inc. BRU data protection products and those products based on the traditional tar and cpio

More information

BackTrack Hard Drive Installation

BackTrack Hard Drive Installation BackTrack Hard Drive Installation BackTrack Development Team jabra [at] remote-exploit [dot] org Installing Backtrack to a USB Stick or Hard Drive 1 Table of Contents BackTrack Hard Drive Installation...3

More information

Adaptable System Recovery (ASR) for Linux. How to Restore Backups onto Hardware that May not be Identical to the Original System

Adaptable System Recovery (ASR) for Linux. How to Restore Backups onto Hardware that May not be Identical to the Original System Adaptable System Recovery (ASR) for Linux How to Restore Backups onto Hardware that May not be Identical to the Original System Storix, Inc. 7801 Mission Center Ct. Ste. 250 San Diego, CA 92108 1-619-543-0200

More information

This is when a server versus a workstation is desirable because it has the capability to have:

This is when a server versus a workstation is desirable because it has the capability to have: Protecting your Data Protecting your data is a critical necessity of having your DemandBridge Software and data programs loaded on a computer that has the ability to integrate redundant devices such as

More information

These application notes are intended to be a guide to implement features or extend the features of the Elastix IP PBX system.

These application notes are intended to be a guide to implement features or extend the features of the Elastix IP PBX system. Elastix Application Note #201201091: Elastix RAID Setup Step By Step Including Recovery Title Elastix Raid Setup Step By Step Including Recovery Author Bob Fryer Date Document Written 9 th January 2012

More information

Introduction. What is RAID? The Array and RAID Controller Concept. Click here to print this article. Re-Printed From SLCentral

Introduction. What is RAID? The Array and RAID Controller Concept. Click here to print this article. Re-Printed From SLCentral Click here to print this article. Re-Printed From SLCentral RAID: An In-Depth Guide To RAID Technology Author: Tom Solinap Date Posted: January 24th, 2001 URL: http://www.slcentral.com/articles/01/1/raid

More information

MANAGING DISK STORAGE

MANAGING DISK STORAGE 1 Chapter 12 MANAGING DISK STORAGE Chapter 12: MANAGING DISK STORAGE 2 CHAPTER OVERVIEW Understand disk-storage concepts and terminology Distinguish between basic and dynamic storage Identify the types

More information

PhotoRescue Manual. DataRescue 2001-2005

PhotoRescue Manual. DataRescue 2001-2005 PhotoRescue Manual DataRescue 2001-2005 Since you are looking at this guide, you probably have lost pictures you care about. Our aim, with this guide, is to help you recover them. If you aren't an experienced

More information

On Benchmarking Popular File Systems

On Benchmarking Popular File Systems On Benchmarking Popular File Systems Matti Vanninen James Z. Wang Department of Computer Science Clemson University, Clemson, SC 2963 Emails: {mvannin, jzwang}@cs.clemson.edu Abstract In recent years,

More information

Ryusuke KONISHI NTT Cyberspace Laboratories NTT Corporation

Ryusuke KONISHI NTT Cyberspace Laboratories NTT Corporation Ryusuke KONISHI NTT Cyberspace Laboratories NTT Corporation NILFS Introduction FileSystem Design Development Status Wished features & Challenges Copyright (C) 2009 NTT Corporation 2 NILFS is the Linux

More information

4 Backing Up and Restoring System Software

4 Backing Up and Restoring System Software 4 Backing Up and Restoring System Software In this Chapter... Planning a Backup Strategy, 4-3 Preparing for Disaster Recovery, 4-4 Creating Boot Recovery Diskettes, 4-5 Making a Full Backup Tape, 4-8 Restoring

More information

SAN Conceptual and Design Basics

SAN Conceptual and Design Basics TECHNICAL NOTE VMware Infrastructure 3 SAN Conceptual and Design Basics VMware ESX Server can be used in conjunction with a SAN (storage area network), a specialized high speed network that connects computer

More information

Which filesystem should I use? LinuxTag 2013. Heinz Mauelshagen Consulting Development Engineer

Which filesystem should I use? LinuxTag 2013. Heinz Mauelshagen Consulting Development Engineer Which filesystem should I use? LinuxTag 2013 Heinz Mauelshagen Consulting Development Engineer TOP Major on-disk local Linux filesystems Features, pros & cons of each Filesystem tools Performance/scalability

More information

RAID HARDWARE. On board SATA RAID controller. RAID drive caddy (hot swappable) SATA RAID controller card. Anne Watson 1

RAID HARDWARE. On board SATA RAID controller. RAID drive caddy (hot swappable) SATA RAID controller card. Anne Watson 1 RAID HARDWARE On board SATA RAID controller SATA RAID controller card RAID drive caddy (hot swappable) Anne Watson 1 RAID The word redundant means an unnecessary repetition. The word array means a lineup.

More information

Update: About Apple RAID Version 1.5 About this update

Update: About Apple RAID Version 1.5 About this update apple Update: About Apple RAID Version 1.5 About this update This update describes new features and capabilities of Apple RAID Software version 1.5, which includes improvements that increase the performance

More information

Getting Started With RAID

Getting Started With RAID Dell Systems Getting Started With RAID www.dell.com support.dell.com Notes, Notices, and Cautions NOTE: A NOTE indicates important information that helps you make better use of your computer. NOTICE: A

More information

GPT hard Disk Drives. For HP Desktops. Abstract. Why GPT? April 2011. Table of Contents:

GPT hard Disk Drives. For HP Desktops. Abstract. Why GPT? April 2011. Table of Contents: GPT hard Disk Drives For HP Desktops April 2011 Table of Contents: Abstract... 1 Why GPT?... 1 GPT vs MBR... 2 Bootable vs Data Drives and UEFI BIOS... 4 OS Support... 6 Storage Driver Support... 6 Imaging

More information

Chapter 10: Mass-Storage Systems

Chapter 10: Mass-Storage Systems Chapter 10: Mass-Storage Systems Physical structure of secondary storage devices and its effects on the uses of the devices Performance characteristics of mass-storage devices Disk scheduling algorithms

More information

HP-UX System and Network Administration for Experienced UNIX System Administrators Course Summary

HP-UX System and Network Administration for Experienced UNIX System Administrators Course Summary Contact Us: (616) 875-4060 HP-UX System and Network Administration for Experienced UNIX System Administrators Course Summary Length: Classroom: 5 days Virtual: 6 hrs/day - 5 days Prerequisite: System Administration

More information

Considerations when Choosing a Backup System for AFS

Considerations when Choosing a Backup System for AFS Considerations when Choosing a Backup System for AFS By Kristen J. Webb President and CTO Teradactyl LLC. October 21, 2005 The Andrew File System has a proven track record as a scalable and secure network

More information

Data Integrity: Backups and RAID

Data Integrity: Backups and RAID Data Integrity: Backups and RAID Introduction Keeping your data safe and reliable TM Backups Types of backups Strategy Tools RAID (Redundant Array of Independent Disks) Types of RAID What type to Use Disk

More information

File Systems for Flash Memories. Marcela Zuluaga Sebastian Isaza Dante Rodriguez

File Systems for Flash Memories. Marcela Zuluaga Sebastian Isaza Dante Rodriguez File Systems for Flash Memories Marcela Zuluaga Sebastian Isaza Dante Rodriguez Outline Introduction to Flash Memories Introduction to File Systems File Systems for Flash Memories YAFFS (Yet Another Flash

More information

CS3210: Crash consistency. Taesoo Kim

CS3210: Crash consistency. Taesoo Kim 1 CS3210: Crash consistency Taesoo Kim 2 Administrivia Quiz #2. Lab4-5, Ch 3-6 (read "xv6 book") Open laptop/book, no Internet 3:05pm ~ 4:25-30pm (sharp) NOTE Lab6: 10% bonus, a single lab (bump up your

More information

Journaling the Linux ext2fs Filesystem

Journaling the Linux ext2fs Filesystem Journaling the Linux ext2fs Filesystem Stephen C. Tweedie sct@dcs.ed.ac.uk Abstract This paper describes a work-in-progress to design and implement a transactional metadata journal for the Linux ext2fs

More information

Defining Digital Forensic Examination and Analysis Tools Using Abstraction Layers

Defining Digital Forensic Examination and Analysis Tools Using Abstraction Layers Defining Digital Forensic Examination and Analysis Tools Using Abstraction Layers Brian Carrier Research Scientist @stake Abstract This paper uses the theory of abstraction layers to describe the purpose

More information

1. Introduction to the UNIX File System: logical vision

1. Introduction to the UNIX File System: logical vision Unix File System 1. Introduction to the UNIX File System: logical vision Silberschatz, Galvin and Gagne 2005 Operating System Concepts 7 th Edition, Feb 6, 2005 Logical structure in each FS (System V):

More information

SCSI device drivers are provided for the following operating systems: Microsoft Windows NT Server 4.0 Novell NetWare 5.1

SCSI device drivers are provided for the following operating systems: Microsoft Windows NT Server 4.0 Novell NetWare 5.1 This section describes how to install and configure the Dell small computer system interface (SCSI) device drivers included with your Dell PowerEdge 1400 computer system. These device drivers are designed

More information

Flash vs. Hard disks FFS problems New problems. SSDs und LogFS. Jörn Engel. Lazybastard.org. November 12, 2009

Flash vs. Hard disks FFS problems New problems. SSDs und LogFS. Jörn Engel. Lazybastard.org. November 12, 2009 Lazybastard.org November 12, 2009 Market Research SmartMedia Approach Modern SSDs The good Lower random access times More IO/s More robust against mechanical failures Less noise Lower power consumption

More information

Recovering Data from Windows Systems by Using Linux

Recovering Data from Windows Systems by Using Linux Recovering Data from Windows Systems by Using Linux Published by the Open Source Software Lab at Microsoft. November 2007. Special thanks to Chris Travers, Contributing Author to the Open Source Software

More information

Data recovery Data management Electronic Evidence

Data recovery Data management Electronic Evidence Data recovery Data management Electronic Evidence 2 RAID, SAN and Virtual systems RAID, SAN and Virtual systems Data Loss Scenarios: RAID Recovery drive failure Deleted VM recovery Reinstall of ESX on

More information

Oracle Cluster File System on Linux Version 2. Kurt Hackel Señor Software Developer Oracle Corporation

Oracle Cluster File System on Linux Version 2. Kurt Hackel Señor Software Developer Oracle Corporation Oracle Cluster File System on Linux Version 2 Kurt Hackel Señor Software Developer Oracle Corporation What is OCFS? GPL'd Extent Based Cluster File System Is a shared disk clustered file system Allows

More information

Chapter 11: File System Implementation. Operating System Concepts 8 th Edition

Chapter 11: File System Implementation. Operating System Concepts 8 th Edition Chapter 11: File System Implementation Operating System Concepts 8 th Edition Silberschatz, Galvin and Gagne 2009 Chapter 11: File System Implementation File-System Structure File-System Implementation

More information

Lab 2 : Basic File Server. Introduction

Lab 2 : Basic File Server. Introduction Lab 2 : Basic File Server Introduction In this lab, you will start your file system implementation by getting the following FUSE operations to work: CREATE/MKNOD, LOOKUP, and READDIR SETATTR, WRITE and

More information