Linux System Administration and Shell Scripting

Transcription

1 Linux System Administration and Shell Scripting MENDOZA, John Robert T. Science Research Specialist II Advanced Science and Technology Institute October 1-5, 2012 ASTI Training Room

2 House Rules Always ask questions. :)

3 MENDOZA, John Robert T. 3 years of experience in Government service 3 years of experience in open-source networking technologies Cloud and HPC operator Open-source software advocate Likes BSD, Linux System administrator Licensed ECE professional

4 Virtualization allows multiple operating systems to run simultaneously on a computer system.

5 Virtualization Terminologies Virtual Machine (VM) Environment created by the virtualization software which contains emulated hardware devices. Host Physical machine / Operating System where the virtualization software is installed. Has all the real physical hardware devices installed.

6 Virtualization Terminologies Guest An operating system running on a virtualized platform. Has all the necessary hardware components to run (processor, memory, storage, etc.) Hypervisor (Virtual Machine Manager) software that manages hardware resources used by virtual machines. Allocates computing resources to avoid contention between VMs.

7 Virtualization There are different kinds of virtualization technology: Container-based (Kernel-level) Paravirtualization Full (a. k. a. Native, Hardware-assisted)

8 Container-based (System-level) Multiple instances of the same operating system. Provides the required isolation and security to run multiple applications or copies of the same OS (but different distributions of the OS) on the same server. Does not use VMMs. Examples: OpenVZ, Linux V-Server, FreeBSD Jails, Solaris Zones

9 Paravirtualization Requires modified version of the operating system to run as guest. Examples: Xen and User-mode Linux (UML)

10 Full Virtualization (HVM, Native) Allows unmodified versions of operating system to run on the host. Can run almost all operating systems available. Uses a full set of emulated hardware devices (storage, networking, etc.) Examples: VMWare, VirtualBox, Xen, Microsoft Hyper-V, KVM

11 Importance of Virtualization Virtualization use-cases: Running multiple operating systems simulaneously. Consolidating hardware infrastructure. Software testbed. Easier software installations.

12 VirtualBox is a cross-platform virtualization application.

13 VirtualBox Supported host operating systems: (Minimum) Windows XP Mac OS X 10.6 Solaris 10 Linux Hosts Ubuntu 8.04 Debian 5 Oracle Linux 4 Red Hat Linux 4 Fedora Core 5

14 Get VirtualBox

15 VirtualBox Documentation

16 VirtualBox: Creating a VM

17 VirtualBox: Creating a VM SupportedGuest OperatingSystems: WindowsNT4.0 Windows2000/XP/Server2003/ Vista/Server2008/Windows7/ Windows8/Server2012 DOS/Windows3.x/95/98/ME Linux2.4/2.6 Solaris10(u6andhigher),Solaris11 (includingsolaris11express) FreeBSD OpenBSD3.7orlater OS/2Warp4.5 MacOSX

18 VirtualBox: Creating a VM

19 VirtualBox: Creating a VM

20 VirtualBox: Creating a VM

21 VirtualBox: Creating a VM

22 VirtualBox: Creating a VM

23 Exercises

24 Introduction to Linux

25 Richard Stallman (March 16, Present) Founded the Free Software Foundation (FSF) in 1985 Developed the GNU/Linux Operating System Popularized the concept of copyleft Created Emacs! Authored GNU Public License (GPL)

26 Linus Benedict Torvalds (December 28, Present) Developed the Linux kernel in 1991 Owns the Linux trademark Working in Open Source Development Lab (OSDL) in Oregan

27

28 Anatomy of an OS Operating systems consist of the following: Kernel System programs Application programs

29 Parts of the Kernel The kernel has several components: Process Management Memory Management Hardware device drivers Filesystem drivers

30 Parts of the Kernel User Programs / System Programs User-mode System Calls Process Management Memory Management Kernel Network Services Virtual FS Management Network TCP/IP Drivers Servides File System Drivers Ethernet Card Drivers SATA Disk Driver Ethernet NIC SATA Hard Drives Kernel Hardware

31 What is Linux? Operating system developed in 1991 by Linus Torvalds Licensed under the GNU Public License (GPL) 2-nd Most popular Linux distribution (Desktop) Mostly used operating system for publicly accessed servers. (>60% usage share)

32 Features Free (as in free speech, and not as in 'free beer') Portability (to any hardware platform) Robust (high uptimes expected) Secure (no viruses!) Scalable (high-perfomance! Baby!) Community-supported (open-source) Cons: Not user-friendly (only for pros) Not unified (too many distributions!) Open-source(?)

33 Who uses Linux?

34 Linux Distributions Red Hat Enterprise Linux (RHEL) Debian Community Enterprise Operating System (CentOS) Ubuntu Fedora SuSe

35 Linux Distributions Red Hat Enterprise Linux (RHEL) Debian Community Enterprise Operating System (CentOS) Ubuntu Fedora SuSe

36 Ubuntu ubuntu oǒ'boǒntoō Founded by Mark Shuttleworth in 2004 Has a release cycle of every 6 months Ubuntu is an ancient African word meaning 'humanity to others' First release version was 4.10 (Warty Warthog); Latest version is (Precise Pangolin)

37 Ubuntu Funded by Canonical Group Ltd. Has two editions: Desktop and Server Long-Term Support (LTS) version every 4th release, i.e. every two (2) years Most popular operating system on the cloud: Amazon and Rackspace Licensed under the Ubuntu License Policy Derived from Debian Linux

38 Hardware Requirements Recommended Minimum Requirements: 300 Mhz x386 processor 128 MB of RAM 500 MB Storage (Base System) Monitor, Graphics card, Optical Drive/USB and Keyboard :P

39 Where to Get

40 Ubuntu Ubuntu is distributed in different types of media images: Desktop Server Alternate

41 Installation

42 Procedure Download an ISO image of Ubuntu Server bit. Mount the ISO image on the CD/DVD Drive of the virtual machine. Boot.

43 Installation Screens Boot Menu Select a Language English Select Location Install Ubuntu Server Philippines Configure the Keyboard Detect Keyboard Layout: NO Country of Origin of Keyboard: English (US) Keyboard Layout: English (US)

44 Installation Screens Configure the network Do not configure the network at this time Specify PREFERRED hostname Setup Users and Passwords Full name of the new user (optional) Username for the account Password of the new user (at least 8 characters with at least 3 character classes: upper case, lower case, and symbols, enter twice) Encrypt Home Directory: NO

45 Hard Disk Structure Hard disks consists of platters or discs. These platters have concentric bands which are called tracks. (Seagate boasts of 340,000 tracks per linear inch width in their 3TB disks) Sections within each track is called sectors. Sectors are the smallest physical storage unit in a disk and is always 512 bytes in size. Clusters (allocation units) are group of sectors.

46 Disk Partitions Partitions are contiguous set of blocks on a drive treated in an independent disks. Partitioning is the method of dividing a single hard drive into multiple logical disks. Partition table is an index that relates the sections of the hard drive to partitions. Contains the number of partitions and identifier of what operating systems are installed in the partitions. Contains information on the beginning and end of the partitions

47 Disk Partitions There are different types of partition: Primary Extended A subpartition of the primary which is a workaround for the limitation There can only be one extended partition in a disk Has its own file system identifier Logical Partitions created within the extended partition There can be 15 logical partitions in an extended partition Swap Original design of the partition table permits only 4 primary partitions per disk Extended memory for OS

48 Disk Partitions Points to ponder: Partitions can't be moved but can be resized Partitions can't overlap Partitions should not have gaps between adjacent partitions Theoretically, a disk can have a maximum of 18 partitions At least one primary partition is require per disk There can only be one type of filesystem per partition Always place the swap partition at the end of the disk

49 Disk Partitions The first sector in every partition is reserved as the boot sector The first sector of the disk is called the Master Boot Record (MBR) and contains the partition table.

50 Disk Nomenclature There is a special nomenclature used when referring to disk drives hd refers to IDE drives sd refers to SATA, SCSI and USB drives cd refers to optical drives

51 Disk Nomenclature Letters are suffixed to the device names to indicate their order of detection by the operating system (for IDE devices, this depends on the controller where the disk is connected) hda hdd IDE hard disk drives sda sdz SATA hard disk drives NOTE: Name assignments do not have any relationship with the SATA/SCSI port where the disk is connected.

52 Disk Nomenclature Numbers are then suffixed to indicate the partitions and its type hda{1-4} first four primary/extended partition hda{5-20} logical partitions

53 Filesystems data structures that an operating system uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk Relationship of Disks, Partitions and Filesystems Filesystems need partitions Partitions need Disks

54 Types of Filesystems Most common filesystem types: File System Original OS Max File Size Max Vol Size UFS2 FreeBSD 512 GB to 32 PB 1YB NTFS Microsoft Windows NT 16 EB 16 EB EXT4 Linux 16 TiB 1 EiB

55 Filesytem Mounting Before a filesystem is used, it should be mounted onto a directory called mount points. Typical mount points: /root /home /var /tmp /usr

56 Quiz What filesystem allows sharing of filesystem between computers for file and data access? What filesystem did Windows 95 use? What is the default filesystem for Mac OSes?

57 Exercises

58 Installation Screens Partition Disks Partitioning Method: Manual Select disk to Partition: sda Create a new partition table on the device: YES Select FREE SPACE Select Create a new partition Follow these file system layout: / /var /home swap 8 GB 6 GB 4 GB 1 GB primary logical logical logical ext4 ext4 ext4 bootable=yes Finish partitioning and write changes to disk

59 Installation Screens Installing the Base System (may take some time) Configuring TaskSel No automatic updates Software selection Software to install: OpenSSH server Select and Install Software (again, this takes time)

60 Installation Screens Install the GRUB Boot loader on a hard disk Install the GRUB Boot loader to the master boot record: YES Finish the installation Continue and reboot. :)

61 Logical Volume Manager (LVM) LVM provides a higher-level view of the disk storage on a computer system than the traditional view of disks and partitions Allows management of storage volumes with user-defined volume groups A mean of storage virtualization Can concatenate, stripe together or combine partitions which can be resized or moved

62 Benefits of LVM Storage consolidation Modes of operation: Linear and Striping Dynamic storage allocation Snapshots Online servicing

63 Elements of LVM Volume Group (VG) Physical Volume (PV) Logical Volume (LV) Physical Extent (PE) Logical Extent (LE)

64 Elements of LVM

65 Redundant Array of Inexpensive Disks (RAID) combine multiple small, independent disk drives into an array of disk drives which yields performance exceeding that of a Single Large Expensive Drive (SLED) MTBF of the array is equal to the MTBF of the individual disk. Mean Time Between Failure (MTBF) is the predicted elapsed time between inherent failures of a system during operation Implementing RAID benefits performance and redundancy

66 Redundant Array of Inexpensive Disks (RAID) There are three ways to create RAID volumes: Hardware Software FakeRaid

67 Redundant Array of Inexpensive Disks (RAID) There are different types of RAID architecture: Striping (RAID 0) Mirroring (RAID 1) Parity (RAID 5) Striping and Mirroring (RAID 1+0, RAID 0+1)

68 Redundant Array of Inexpensive Disks (RAID) RAID 0 (Striping) Data are striped to member disk devices Provides the best performance in read and write operations Needs minimum of 2 disks No redundancy (can't tolerate disk failure) Capacity can be computed as S * N Where: S = single disk capacity (smallest) N = number of disk

69 Redundant Array of Inexpensive Disks (RAID) RAID 1 (Mirroring) Data is mirrored or replicated on member disk Read and write is comparable to a single disk performance Requires a minimum of 2 disks Provides 1+1 redundancy (able to survive a disk failure) Capacity equals S/N

70 Redundant Array of Inexpensive Disks (RAID) RAID 5 (Parity) and RAID 6 (Double Parity) Parity is calculated and stored across the disk members Read performance is excellent; write performance is somewhat slower (parity calculation) Requires a minimum of 3 disks (for RAID5); 4 disks (for RAID6) Can tolerate 1 disk failure (RAID5); 2 disk failure (RAID6) Capacity equals: RAID 5: (N-1) * S RAID 6: (N-2) * S

71 Redundant Array of Inexpensive Disks (RAID) There is no substitute for backups!

72 Configure LVM on Installation Partition Disk Create a partition table on the disks Create a 100MB partition on the disk Create a Physical Volume (PV) Create a logical partition on the disk Use it as physical volume for LVM Create Volume Group (VG) Name it as UbuntuTraining Select the created logical partition to include in the volume group Create Logical Volumes (LV) Mount as /boot, ext4 formatted, bootable Create this logical volumes UbuntuRoot 8 GB UbuntuVar 4 GB UbuntuHome UbuntuSwap 3 GB 2 GB

73 Configure LVM on Installation Format and mount the LV to filesystems UbuntuRoot / ext4 UbuntuVar /var ext4 UbuntuHome /home ext4 UbuntuSwap swap

74 Configure LVM on Installation Extending the volume group Create a partition table on the second disk Delete the UbuntuRoot LV Extend the volume group UbuntuTraining: Add the partition of the second disk Create the new UbuntuRoot logical volume with all the capacity available

75 Configure RAID on Installation Partition Disks Create the prescribed partitions on the first disk instead of formatting them with ext4, select use physical volume for RAID Create the same layout on the second disk Create RAID volumes by matching the partitions from the first and second disk to create a new device, md. Format and mount the RAID devices as described

76 Configure RAID on Installation Detach the disk drives from the virtual machine one at a time and test if it boots!

77 Booting Up For The First Time (Post-Installation)

78 Boot Sequence Basic Input/Output System (BIOS) Boot Loader (Grub) Kernel and Init Runlevels Getty and Login Console

79 Basic Input/Output System (BIOS) Tests and checks the available devices in the system Looks for the devices to boot the system When booting from a disk drive, reads the content of the first sector (MBR) The MBR contains instructions on how to load the boot-loader (which is by default installed in the MBR)

80 GRUB Boot Loader First software program that runs when a computer starts and is responsible for loading and transferring control to the operating system kernel software GRUB stands for GRand Unified Bootloader Supports two boot methods: Direct loading and Chain Loading Provides a CLI, pre-os environment which displays the available operating systems installed in the disk partitions

81 Kernel and Init First application process executed by the kernel after the boot-loader passes on the boot sequence Located in /sbin Initializes its configuration file located in /etc/inittab Spawns all other processes in the operating system (mother of all processes)

82 Init Runlevel runlevel is a configuration of what applications/processes will be running on the operating system at boot time The init process looks in the inittab for the configured runlevel. It then looks into the script directory configured in the inittab, /etc/rcx.d Provides the concept of operation modes

83 Init Runlevel The run levels are: 0 - halt (Do NOT set initdefault to this) 1 - Single user mode (runlevel S) 2 - Multiuser, without NFS (The same as 3, no networking) 3 - Full multiuser mode 4 - unused 5 - X reboot (Do NOT set initdefault to this)

84 Getty and Login Console Getty is a process which prints tty lines and executes the login process from the virtual consoles or virtual terminals (VT). There is only one getty process per virtual terminal.

85 Virtual Consoles interface for the user to interact with the operating system provided by two main peripheral devices, keyboard and the monitor display. By default, there are 7 virtual terminals available for the user. Switching virtual terminals are done by these keystroke combinations, <CTRL> + <ALT> + <F1-F7>

86 Ubuntu System Administration End of Day 1