LBNC and IBM Corporation Document: LBNC-Install.doc Date: Path: D:\Doc\EPFL\LNBC\LBNC-Install.doc Version: V1.0

Transcription

1 LBNC Compute Cluster Installation and Configuration Author: Markus Baertschi Owner: Markus Baertschi Customer: LBNC Subject: LBNC Compute Cluster Installation and Configuration Page 1 of 14

2 Contents 1. Introduction Configuration Overview Physical Placement in Rack Power Cabling Network Cabling Other Cabling Network Configuration Hosts and Addresses Generic Network Information Disk and Storage Setup Blades DS3200 Disk Disk System Management Disk sharing via NFS OS Installation Disk Partitioning Installed Package Groups User Management Backup and TSM configuration General Considerations Storage Configuration Backup Policies Backup Schedules Users and Passwords System Users and Passwords How to Guides Add a User Change / Reset a Users Password Increase the size of a Directory Restore a file from the backup Open Topics Hardware Configuration Notes SAS Connectivity Modules Nortel Ethernet Switch Module...14 Subject: LBNC Compute Cluster Installation and Configuration Page 2 of 14

3 1. Introduction This document resumes the initial configuration of the LBNC compute cluster. Subject: LBNC Compute Cluster Installation and Configuration Page 3 of 14

4 2. Configuration Overview 2.1 Physical Placement in Rack Placement Item Description Empty (23U) (future expansion) 20 Console Console Screen and Keyboard 19 TS3100 Tape Library Master Node Node 01 Node 02 Node 03 Node 04 DS3200 EXP3000 Empty (2U) Empty (2U) UPS Spare (9x) BladeCenter Master Node 4 Compute Nodes SATA Disk Controller 12 x 1TB SATA SATA Expansion Unit 12 x 1TB SATA (future disk expansion) (future disk expansion) Uninterruptible power supply (UPS) Subject: LBNC Compute Cluster Installation and Configuration Page 4 of 14

5 2.2 Power Cabling Schematic drawing of the power connections. Single power devices (console, tape library) are not shown. Critical ones (console) connect to the UPS, uncritical (tape) connect to mains. UPS mains Power Outlet Bladecenter DS3200 EXP3000 UPS Device Plugs Plug Remarks into Front-End Wall IEC- 16/3 Single Phase 16 AMP PDU 309 Universal PDU FE-PDU C20 6 x C13 BladeCenter FE-PDU C20 (4x) DS3200 Univ-PDU C13 (2x) EXP3000 Univ-PDU C13 (2x) TS3100 Univ-PDU C13 Console Univ-PDU C Subject: LBNC Compute Cluster Installation and Configuration Page 5 of 14

6 2.4 Network Cabling Device Plugs into Port Remarks BC-Switch Wall 1 Redundant connection via 2 cables? UPS-Switch BC-Switch 2 1 x RJ45 DS3200 BC-Switch 4/5 2 x RJ45 TS3200 BC-Switch 6 1 x RJ45 BC-MM UPS-Switch Bladecenter Management Module The Bladecenter Management Module is connected to the Hub built in the UPS because communication does not work if it is connected to the switch module in the Bladecenter. 2.5 Other Cabling Device Plugs into Remarks Console BC MM Video, Keyboard, Mouse BC SAS DS x SAS EXP3000 DS x SAS Subject: LBNC Compute Cluster Installation and Configuration Page 6 of 14

7 3. Network Configuration 3.1 Hosts and Addresses IP Address Hostname Description lbncmm1 Bladecenter Management Module lbncnet1 Nortel GB Ethernet Switch lbncsas1 SAS Connectivity Module lbncsas2 SAS Connectivity Module lbncds1a DS3200 Controller A lbncds1b DS3200 Controller B lbncts1 TS3100 Tape Library lbncups UPS Controller lbncm1 Master Node lbncn01 Computer Node lbncn02 Computer Node lbncn03 Computer Node lbncn04 Computer Node Generic Network Information Item Value Description Netmask Netmask of the EPFL backbone Default Gateway Default Gateway to the EPFL Campus Network DNS Server Name Servers DNS Server Subject: LBNC Compute Cluster Installation and Configuration Page 7 of 14

8 4. Disk and Storage Setup 4.1 Blades The master node boots of the RAID array for reliability. Its internal disk is used as TSM disk storage pool (working storage for backups). Compute nodes boot off the local disk. 4.2 DS3200 Disk The DS3200 disk subsystem is configured for good reliability (RAID5), good capacity (8+1 arrays) and decent performance (3 arrays). Array - LUN Capacity Description array1 - master-boot 200G Boot disk for the master node (boot on SAN to get RAID5 protection) array1 - data1 3.5T Data array2 - data2 8T Data array3 - data3 8T Data hot spare 1T Spare disk for to reduce exposure in case of disk failure Disk System Management The disk system is managed using a Java application installed on the master node. The graphical application needs the X windowing system to work and must be run as user root. The management application can be installed on other computers and control the disk subsystem remotely, if desired. The application can be downloaded from the IBM DS3200 support page on the web. lbncm1:~ # echo $DISPLAY lbncpc16.epfl.ch:0 lbncm1:~ # /opt/ibm_ds4000/client/smclient 4.3 Disk sharing via NFS NFS shares to configure Subject: LBNC Compute Cluster Installation and Configuration Page 8 of 14

9 5. OS Installation 5.1 Disk Partitioning We partition the system disk on the compute nodes as follows: Partition Size Mount point Description /dev/sda1 50M /boot Boot partition /dev/sda2 8G swap Paging space /dev/sda3 remaining / Root (with OS, etc) On the master node the partitioning scheme is different, as we boot from the SAS RAID array: Partition Size Mount point Description /dev/sdb1 50M /boot Boot partition /dev/sdb2 8G swap Paging space /dev/sdb3 100G / Root (with OS, etc) /dev/sdb4 remaining LVM User Storage (LVM) /dev/sda1 147G /tsmstgp1 TSM storage pool (on the single internal disk) /dev/sdxx 20T LVM User Storage (LVM) 5.2 Installed Package Groups 5.3 User Management How do we manage users? maerkl denervaud Subject: LBNC Compute Cluster Installation and Configuration Page 9 of 14

10 6. Backup and TSM configuration 6.1 General Considerations Due to the small size of this installation we will implement TSM in a simple and effective way, but without compromising data security. One single tape storage pool Daily TSM database backup to local disk Daily backup of the Linux OS, the TSM binaries and the TSM database backup to tape. This backup can be used to restore everything. 6.2 Storage Configuration Storage Pool Size Description backup_disk 100G Disk storage pool backup_lto4 10T Tape storage pool Database 10G TSM database 6.3 Backup Policies We use a single backup policy for all data. Policy Domain: LBNC Policy Set: LBNC Management Class: LBNC Parameters Value Description VERExists 10 How many versions of an active file we keep in storage. VERDeleted 3 How many versions of a deleted file we keep in storage. RETExtra 100 How many days we keep old versions of files in storage. RETOnly 400 How many days we keep a deleted file in storage. RETVer (archive) 800 How long we keep archived data (default, can be overridden when archiving). 6.4 Backup Schedules Item Hour Description system backup 07:00 System backup of the OS and the TSM database purge_disk 18:00 Copy backup data from disk to tape backup 01:00 Backup user data Subject: LBNC Compute Cluster Installation and Configuration Page 10 of 14

11 7. Users and Passwords 7.1 System Users and Passwords Please see the separate confidential document. 7.2 How to Guides Add a User All users should be in the group 'lbnc' to have access to the /lbnc directory. The password must be set twice, once for the login and once for the Windows access. The 'yast2' tool is graphical if there is an X server and $DISPLAY is set. The 'rdist' command updates the nodes with changed files (configured in 'distfile') lbncm1:~ # root lbncm1:~ # yast2 users lbncm1:~ # smbpasswd -a maerkl New SMB password: <password> Retype new SMB password: <password> Added user maerkl. lbncm1:~ # rdist lbncn04: updating host lbncn04 lbncn03: updating host lbncn03 lbncn03: /etc/passwd: updating lbncn03: /etc/sudoers: updating lbncn03: updating of lbncn03 finished lbncm1:~ # exit Change / Reset a Users Password Use yast2 as in add a user Increase the size of a Directory The script 'chfs' can increase the size of filesystems. lbncm1:~ # df /home Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/datavg-home % /home lbncm1:~ # chfs -a size=+1m /home lvextend -L+1M /dev/datavg/home Rounding up size to full physical extent 4.00 MB Extending logical volume home to GB Logical volume home successfully resized mount -o remount,resize /home Subject: LBNC Compute Cluster Installation and Configuration Page 11 of 14

12 lbncm1:~ # df /home Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/datavg-home % /home Show the remaining disk space lbncm1:/srv/www/htdocs # vgdisplay --- Volume group --- VG Name datavg System ID Format lvm2... Total PE Alloc PE / Size / TB Free PE / Size / 6.25 TB VG UUID OVldIe-MGWK-wAE1-4TAx-tTW0-fGls-zypKE Restore a file from the backup The command 'dsmc restore' restores data form tape. It will take minutes to load the tape and restore the file. Using the commandline has the big advantage that you don't need the TSM administrative password as for the Web-GUI. markus@lbncm1:~> rm dsmerror.log markus@lbncm1:~> dsmc restore dsmerror.log IBM Tivoli Storage Manager Command Line Backup/Archive Client Interface Client Version 5, Release 5, Level 0.0 Client date/time: 03/09/ :19:57 (c) Copyright by IBM Corporation and other(s) 1990, All Rights Reserved. Node Name: LBNCM1 Session established with server SERVER1: Linux/x86_64 Server Version 5, Release 5, Level 0.0 Server date/time: 03/09/ :19:57 Last access: 03/09/ :19:32 Restore function invoked. ** Interrupted ** ANS1114I Waiting for mount of offline media. Restoring 183 /home/markus/dsmerror.log [Done] Restore processing finished. Total number of objects restored: 1 Total number of objects failed: 0 Total number of bytes transferred: 207 B Data transfer time: 0.00 sec Network data transfer rate: 18, KB/sec Aggregate data transfer rate: 0.00 KB/sec Elapsed processing time: 00:01:03 markus@lbncm1:~> ls -l dsmerror.log -rw-r--r-- 1 markus users :11 dsmerror.log Subject: LBNC Compute Cluster Installation and Configuration Page 12 of 14

13 8. Open Topics Topics to be discussed Topic Description Solution Hostnames How should we name the hosts? ok Network Cabling Single or double connection to EPFL single ok backbone? Diode Master node accessible from outside yes ok EPFL? Disk management One big directory (LVM) lvm ok Three medium directories (8/8/3T) Many small directories (LVM) NFS NFS Shares to use? one? User Local local, ca 20 management NIS LDAP Backup - What to backup and what not to backup - Backup Policies & retention Compute job management How are compute jobs managed (batch system)? Monitoring? Nagios? yes ftp server couple GB no backup /lbnc-old need simple batch system -> openpbs lbnc home lbnc-old notifications website 7 TB 1 TB 4 TB nagios tsm restore Subject: LBNC Compute Cluster Installation and Configuration Page 13 of 14

14 9. Hardware Configuration Notes 9.1 SAS Connectivity Modules The SAS configuration module Web-interface seems not to work for our needs. I had to modify the Zonig on the command line to add the blade 1 port to the zone 10 with the external ports. 9.2 Nortel Ethernet Switch Module The spanning tree protocol (stp) has to be switched off globally. If this is not done the EPFL backbone switches the Ethernet port off. Subject: LBNC Compute Cluster Installation and Configuration Page 14 of 14