How to build a Beowulf Cluster Applied to Computational Electromagnetic Carlos Henrique da Silva Santos Leonardo André Ambrosio Hugo Enrique Hernández Figueroa Department of Microwaves and Optics (DMO) School of Electrical and Computer Engineering (FEEC) State University of Campinas (UNICAMP)
Summary Motivation; Objectives; 1. Preview: Operation Systems and Network concepts; 2. Installing Linux Fedora distribution; 3. Configuring a computer cluster; 4. HPC cluster applications in electromagnetism; Conclusions; Main references; How to Build a Cluster Carlos Henrique 2
Motivations Different concepts over High Performance Computing (HPC) applied to Computational Electromagnetism; Low computational costs solutions for HPC; New computational technologies and resources to build a HPC cluster; Some large computational electromagnetic application. How to Build a Cluster Carlos Henrique 3
Objectives Presenting some computational concepts: Operating system layers Network structures; Linux distributions and services (NFS and SSH); Sophisticated and easy ways to build a computer cluster, focusing on data security; How to configure services and connections. How to Build a Cluster Carlos Henrique 4
1 st Part - Concepts Operating System Linux Network How to Build a Cluster Carlos Henrique 5
Operating Systems Kernel Hardware Kernel System Calls Program How to Build a Cluster Carlos Henrique 6
Operating System: Structure Memory management; Disk and file systems; Networking; Security: Internal: Programs running locally; External: Network connections; Graphical user interface; Device Driver; Process management. Firewall How to Build a Cluster Carlos Henrique 7
Operating System: Process Process: A program in execution; An instance of a program running in a computer; The entity that can be assigned and executed on a processor (program code and a dataset); Identifier State Priority Program counter Memory pointers Context data I/O Status information Accounting information... How to Build a Cluster Carlos Henrique 8
Modern Unix System a.out coff elf File mappings Exec switch NFS FFS Device mappings Disk driver Anonymous mappings virtual memory framework block device switch Common facilities vnode/vfs interface Scheduler framework s5fs RFS Time sharing process Tape driver Streams System processes Network driver tty driver How to Build a Cluster Carlos Henrique 9
Why Linux is necessary? Linux is a free operating system; It was concepted by Linus Torvalds to attempt his necessities, without intentions to get money; The development style adopted is the collective help, where someone manages the colective efforts of a group to improve the system; Millions of people are gratuitously contributing to develop Linux, for pleasure or to have a better operating system. How to Build a Cluster Carlos Henrique 10
Operating System: Linux It is a multiprocessor and multiuser system, which means that many users have the possibility to execute many processes into the same computer using Linux in the same time; Free and open source software, basing on the GPL. Furthermore, most of the computational tools for this operational system are free too; It has some different Kernel to provide better solutions depending on the application; It is one of the best operational system for network and HPC applications; Free tutorial, manual and help on the Internet. How to Build a Cluster Carlos Henrique 11
Networking Application: Computer Cluster 0 1 How to Build a Cluster Carlos Henrique 12
Network: Services, Protocols and Layers Protocols: are related to the packets sent between peer entities on different machines; Services: is a set of primitives (operations) that a layer provides to the layer above it. In other words, is related to the interface between layers. Machine 1 Layer k+1 Service provided by layer k Protocol Layer k Machine 2 Layer k+1 Layer k Layer k-1 Layer k-1 How to Build a Cluster Carlos Henrique 13
Network: OSI Reference Model layers 7 Application Application protocol Application APDU Interface 6 Presentation Presentation protocol Presentation PPDU 5 Session Session protocol Session SPDU 4 Transport Transport protocol Communication subnet boundary Transport TPDU Internet subnet protocol 3 Network Network Network Network Packet 2 Data link Data link Data link Data link Data link 1 Physical Physical Physical Physical Bit How to Build a Cluster Carlos Henrique 14
Network: Open Systems Interconnection (ISO-OSI) Reference Model Deals with connecting open system, which are open for communication with other systems; OSI model has seven layers, based on: 1. A layer should be created where a different abstraction is needed; 2. Each layer should perform a well-defined function; 3. Each layer function should be defined in order to provide an international standardization protocol; 4. The layer boundaries should be chosen to minimize the information flow across the interfaces; 5. The number of layer should be large enough that distinct functions and small enough that the architecture does not become widely. How to Build a Cluster Carlos Henrique 15
2 nd Part Installing Fedora RedHat How to Build a Cluster Carlos Henrique 16
Choosing installation version i386 ppc Intel x86-compatible processors, including Intel Pentium and Pentium- MMX, Pentium Pro, Pentium-II, Pentium-III, Celeron, Pentium 4, Xeon, Core Duo, and Centrino/Centrino Duo; VIA C3/C3-m and Eden/Eden-N; and AMD Athlon, AthlonXP, Duron, AthlonMP, Sempron, and Turion PowerPC processors, such as those found in Apple Power Macintosh, G3, G4, and G5, and IBM pseries systems x86_64 64-bit AMD processors such as Athlon64/FX/X2, Turion64, Opteron; and Intel 64-bit processors such as EM64T How to Build a Cluster Carlos Henrique 17
Pre-installation How to Build a Cluster Carlos Henrique 18
Installation: System properties How to Build a Cluster Carlos Henrique 19
Installation: Operating System Resource How to Build a Cluster Carlos Henrique 20
3 rd Part Cluster Configuring Linux Network SSH NFS Installing MPI Libraries How to Build a Cluster Carlos Henrique 21
TCP: Transport Control Protocol Transport Layer at OSI Model It is connection oriented, which means that all problems no solved in IP level will be solved here; Some problems that must be solved by TCP: Lost of packages or destructions occurs during the transmission; Package expedition out of order or duplicated. How to Build a Cluster Carlos Henrique 22
TCP: Transport Control Protocol The TCP specify the data package format and patterns recognized in reliable changes between two computers, like the procedures that were applied in computers to ensure the data communication; Procedures to be considered: Distinguishing between multiples destinies in a determined machine; Error recovering, like some lost packages or duplicated. How to Build a Cluster Carlos Henrique 23
TCP: Transport Control Protocol Available multiple programs runs in one machines communicate; Multiplex the TCP input traffic between the applications programs; Uses numerical port to identify the last destination in a machine. Each port is identified by a low integer number; Example: 128.10.2.3.25 specifies the TCP port number 25, in the machine with IP address 128.10.2.3. How to Build a Cluster Carlos Henrique 24
IP: Internet Protocol Network Configuration - Structure http://www.informabr.com.br/ipsubnet.htm How to Build a Cluster Carlos Henrique 25
Linux Network Configuration: shell Mode Identify network device in use; Open the file: /etc/sysconfig/network-scripts/ifcfg-ethx DEVICE=ethx BOOTPROTO=none ONBOOT=yes TYPE=Ethernet IPADDR=10.0.0.100 NETWORK=10.0.0.0 BROADCAST=10.0.0.255 NETMASK=255.255.255.0 GATEWAY=10.0.0.1 DNS1=10.0.0.1 # Identify the device # Type of boot (none, static ou dhcp) # Starting on machine boot # Type of network # IP address of the machine at intranet # Network Masquerade # Range of IP address in the network # Defines the subnet # Package destination to network exit # Import the DNS specified on /etc/resolv.conf How to Build a Cluster Carlos Henrique 26
Shell Script Network Address Translation modprobe ip_conntrack modprobe ip_conntrack_ftp modprobe ip_tables modprobe ipt_state modprobe iptable_nat modprobe ip_nat_ftp modprobe ipt_limit echo 1 > /proc/sys/net/ipv4/ip_forward echo 1 > /proc/sys/net/ipv4/tcp_syncookies echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts echo 1 > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses for f in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 >$f done for f in /proc/sys/net/ipv4/conf/*/accept_redirects; do echo 0 >$f done for f in /proc/sys/net/ipv4/conf/*/send_redirects; do echo 0 >$f done for f in /proc/sys/net/ipv4/conf/*/accept_source_route; do echo 0 >$f done iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE How to Build a Cluster Carlos Henrique 27
Main Network Services Requested FTP/SFTP (File Transfer Protocol/SSH File Transfer Protocol): to share and access files in different machines or networks; SSH: Secure Shell: remote access control; NFS (Network File System): sharing files in a network How to Build a Cluster Carlos Henrique 28
SSH: Secure Shell O SSH has been defined on IANA beyond the RFC 4250; A free and open-source available is OpenSSH, which supports the versions 1.3, 1.5 e 2.0 of the SSH protocol; Shell command: ssh cepofslave The first time that a connection for a different is required, the following message is exhibit: The authenticity of host cepofslave can t be established. DSA key fingerprint is 95:90:3a:3a:bc:f3:9a:9b:01:5d:b3:07:38:e2:11:0c. Are you sure you want to continue connecting (yes/no)? How to Build a Cluster Carlos Henrique 29
NFS: Network File System It is a model to structure data file distribution in some servers, mounting a virtual directory.; Making possible to users to centralize their folders in a server; These tools is important in High Performance Computing structures that access disc. Computer cluster is an evident example. How to Build a Cluster Carlos Henrique 30
NFS: Network File System To allow client access in NFS servers the following Daemons are necessary: nfsd: to receive NFS clients requests; mountd: daemon to mount NFS partitions and to execute nfsd requests; portmap: allows to NFS clients dicover which NFS server port is using. How to Build a Cluster Carlos Henrique 31
NFS: Network File System A client can connect a shared data file, he needs of permission to do it. For this reason, is important to certify if he is allowed in the file /etc/exports; In the /etc/exports each line represents one file system to be exported. A remote system can be specify just only one time by the file system, and only one standard input can be defined. IP range identification /home/cluster 10.0.0.0/255.255.255.0(rw,no_root_squash) Identifying IPs /home/cluster 10.0.0.100(rw,no_root_squash) /home/cluster 10.0.0.101(rw,no_root_squash) How to Build a Cluster Carlos Henrique 32
Configure Cluster: Server NFS Step 0 Step 1 Step 3 Step 4 Creating one user called cepof adduser cepof passwd cepof /etc/exports /home/cepof 10.0.0.101 (rw, no_root_squash) /home/cepof 10.0.0.102 (rw, no_root_squash) /home/cepof 10.0.0.103 (rw, no_root_squash) /etc/hosts 10.0.0.101 cepofslave1 10.0.0.102 cepofslave2 10.0.0.103 cepofslave3 /etc/hosts.equiv cepofslave1 cepofslave2 cepofslave3 How to Build a Cluster Carlos Henrique 33
Configure Cluster: Server Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Reboot NFS service service nfs restart Change to cepof user su cepof Creating the cryptographic key to access ssh-keygen -t dsa Creating authorization file cat.ssh/id_dsa.pub > authorized_keys Permission of the directory.ssh chmod 700.ssh File access permission chmod 644.ssh/authorized_keys How to Build a Cluster Carlos Henrique 34
Configure Cluster: Slave nodes Step 0 Creating cepof user adduser cepof Passwd cepof Step 1 /etc/hosts 10.0.0.101 cepofslave1 10.0.0.102 cepofslave2 10.0.0.103 cepofslave3 Step 2 /etc/fstab 10.0.0.1:/home/cluster /home/cluster nfs exec,dev,suid,rw 1 1 How to Build a Cluster Carlos Henrique 35
Configure Cluster: Slave nodes Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 FTP connection on the Server sftp cepof@10.0.0.1 Getting the public key get.ssh/id_dsa.pub Ending connection exit Creating the authorization key file cat.ssh/id_dsa.pub > authorized_keys Permissions of the.ssh directory chmod 700.ssh Permissions of the authorization file chmod 644.ssh/authorized_keys How to Build a Cluster Carlos Henrique 36
Example: Hello World #include <unistd.h> #include <stdio.h> #include "mpi.h" int main(int argc, char ** argv) { int my_rank; int p; int source; int dest; int silen = 128; int gherr; char hname[128]; char message[800]; MPI_Status status; int tag = 50; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &p); gherr = gethostname( hname, silen); if (my_rank!= 0) { sprintf(message, "Greetings from process %d on %s!", my_rank, hname); dest = 0; MPI_Send(message, strlen (message)+1, MPI_CHAR, dest, tag, MPI_COMM_WORLD); } else { printf ("Messages received by process %d on %s.\n\n", my_rank, hname); for (source = 1; source < p; source++) { MPI_Recv(message, 800, MPI_CHAR, source, tag, MPI_COMM_WORLD, &status); printf("%s\n", message); } } MPI_Finalize(); return 0; } How to Build a Cluster Carlos Henrique 37
Parallel Process Schematic Process Process 0 0 (Mater (Mater Node) Node) Process Process 1 1 Process Process 2 2 Process Process 3 3 How to Build a Cluster Carlos Henrique 38
Creating boot configuration file vi lamhosts Add every IP or hosts of the cluster machines 10.0.0.1 10.0.0.101 10.0.0.102 lamboot v lamhosts mpicc hello.c o hello mpirun C hello LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University n-1<13229> ssi:boot:base:linear: booting n0 (10.0.0.1) n-1<13229> ssi:boot:base:linear: booting n1 (10.0.0.102) n-1<13229> ssi:boot:base:linear: booting n2 (10.0.0.103) n-1<13229> ssi:boot:base:linear: booting n3 (10.0.0.104) n-1<13229> ssi:boot:base:linear: booting n4 (10.0.0.105) n-1<13229> ssi:boot:base:linear: booting n5 (10.0.0.106) n-1<13229> ssi:boot:base:linear: booting n6 (10.0.0.107) n-1<13229> ssi:boot:base:linear: booting n7 (10.0.0.108) n-1<13229> ssi:boot:base:linear: booting n8 (10.0.0.110) n-1<13229> ssi:boot:base:linear: booting n9 (10.0.0.111) n-1<13229> ssi:boot:base:linear: booting n10 (10.0.0.112) n-1<13229> ssi:boot:base:linear: booting n11 (10.0.0.113) n-1<13229> ssi:boot:base:linear: booting n12 (10.0.0.114) n-1<13229> ssi:boot:base:linear: finished How to Build a Cluster Carlos Henrique 39
4 th Part Applications of HPC Electromagnetic Applications How to Build a Cluster Carlos Henrique 40
Cluster Application: Human Head SAR Simulation Slice 1 Slice 2 Slice 3 Air Skin Bone Blood Brain Muscle Temporal Muscle y x Slice 4 Slice 5 Ω Ω1 Ω2 Ω 3 How to Build a Cluster Carlos Henrique 41
Cluster Application: Human Head SAR Simulation SAR calculus considering energy and mass; SAR = σ E 2ρ i 2 = 2 J i [ mw 2ρσ / g ] Applying the Poynting vector SAR = t W m = t W V [ mw / g] Local average of the SAR in terms of the simulation time (N max ) SAR = σ 1 2ρ N max [ ] 2 2 E x + E y [ mw / g] How to Build a Cluster Carlos Henrique 42
Cluster Application: Human Head SAR Simulation Performance Time (seconds) Tempo Despendido (segundos) 300 250 200 150 100 50 0 Tempo Despendido pela Simulação SAR na Cabeça Humana Time of the Human Head SAR Simulation (dimensions variations) 1 2 3 4 5 Quantidade de Processos number of process 200x200 400x400 600x600 800x800 1000x100 Efficiency of the Human Head SAR Parallel Simulation Eficiência da Simulação SAR na Cabeça Humana Percentual de Eficiência Efficiency 1,2 1 0,8 0,6 0,4 0,2 0 200x200 400x400 600x600 800x800 1000x1000 Dimensions Dimensões dos of the Domínios computational Computacionais domain 1 Processo 2 Processos 3 Processos 4 Processos 5 Processos How to Build a Cluster Carlos Henrique 43
Cluster Application: SAR Simulation varying the time steps SAR Simulation Eficiência Efficiency da in Simulação the Human Head SAR na Cabeça (400x400) Humana (400x400) Eficiência efficiency 1,2 1 0,8 0,6 0,4 0,2 0 1000 2000 3000 4000 5000 1 Processo 1 process 2 Processos 2 processes 3 Processos 3 processes 4 Processos 4 processes 5 Processos 5 processes Quantidade Number de Passos of time de steps Tempo How to Build a Cluster Carlos Henrique 44
Cluster Application: SAR Simulation Slice 1 Slice 2 Slice 3 Slice 4 Slice 5 How to Build a Cluster Carlos Henrique 45
Cluster Application: Metamaterial simulations Metamaterial Tempo das Simulações Simulation Metamaterial Time 2D Time (seconds) Time(seconds) Tempo Despendido (segundos) 2000 1500 1000 500 0 1 2 3 4 5 number of process Quantidade de Processos 3000x323 AEL 1 AEL2 AEL3 AEL10 1552 814 596 468 Eficiência do Processamento Paralelo Metamaterial Simulation Time AEL8 457 1,2 1 efficiency Eficiência 0,8 0,6 0,4 0,2 0 1 2 3 4 5 3000x323 Efficiency = T T sequential parallel number of process Quantidade de Processos How to Build a Cluster Carlos Henrique 46
CePOF Cluster 0 1 Public Key - DSA Internet Router Gateway Firewall Intranet How to Build a Cluster Carlos Henrique 47
CePOF Cluster: New Cluster Paradigm 15 nodes: 2 AMD Opteron 246 processors 4 GB of memory RAM 73 GB of HD SCSI Switch 3COM 2824 (Gigabit Network) Patch Cord Kat. 6 No break APC SURT 10000, GREE Air-Conditioner with 42.000 Btus S.O. Virtual S.O. Virtual Hw. Virtual Hw. Virtual Virtual Machine Windows 2003 Server A2210 (AMD Opteron) How to Build a Cluster Carlos Henrique 48
Conclusions Computer Cluster allows high performance computing (HPC) with low costs; Free software and microcomputer reused (Beowulf cluster); Complex structure to configure, to use and to keep; Choose better Operational System distribution in order to improve the computational performance; Making possible to simulate and analyze new electromagnetic structures and many other applications. How to Build a Cluster Carlos Henrique 49
References A. S. Tanenbaum, Redes de Computadores 3ª. Edição, Editora Campus, 1996. C. A. F. Oliveira, Cluster Beowulf- uma solução de baixo custo, Revista CienteFico, Ano IV, V. I, Salvador, 2004. C. H. S. Santos, H. E. Hernández-Figueroa, Método FDTD 2D Paralelo para Simulações de Estruturas Metamateriais, 12º Simpósio Brasileiro de Microondas e Optoeletrônica (SBMO) e 7º Congresso Brasileiro de Eletromagnetismo (CBMAG), Belo Horizonte, 2006. C. H. S. Santos, L. A. Ambrosio, H. E. Hernández Figueroa, The Benefits of Heterogeneous Beowulf Cluster on the Human Head SAR Simulation, International Microwave & Optoelectronics Conference (IMOC), IEEE-SBMO, Brasília (2005) C. H. S. Santos, Computação Paralela Aplicada a Problemas Eletromagnéticos Utilizando o Método FDTD, dissertação de mestrado, Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas (Unicamp), Campinas, 2005. J. J. Dongarra, et al, Numerical Linear Algebra for High-Performance Computers, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, 1998. M. A. F. Batista, L. N. F. Guimarães, Algoritmos Genéticos em Ambientes Paralelos, V Worcap INPE, São José dos Campos, 2005. M. Pitanga. Construindo Supercomputadores em Linux Segunda Edição, Editora Brasport, São Paulo, 2004. N. S. Simões, S. F. Souza, L. Muniz, F. Fardim Junior, A. F. Souza, Reis N. C. Junior, A. M. P. Valli, L. Catabriga, Instalação e Configuração de Clusters de Estações de Trabalho: Experiência do Laboratório de Computação de Alto Desempenho do Departamento de Informática da UFES, IV Workshop em Sistemas Computacionais de Alto Desempenho - WSCAD'2003, São Paulo, 2003. P. S. Pacheco, Parallel Programming with MPI, Morgan Kaufmann Publishers, San Francisco, 1997. P. Veríssimo, L. Rodrigues, Distributed Systems for System Architects, Kluwer Academic Publishers, Massachusetts, 2001. Introdução ao MPI, Centro Nacional de Processamento de Alto Desempenho-SP, http://www.cenapad.unicamp.br/servicos/treinamentos/mpi.shtml (acessado em: 18/04/2007). S. Humphries, Field Solutions on Computer, CRC Press LLC, Florida, 2000. Curso de MPI, Centro Nacional de Processamento de Alto Desempenho-NE. http://www.beowulf.org/overview/history.html (acessado em: 18/04/2007). http://www.beowulf.org/overview/faq.html#17 (acessado em: 18/04/2007). H. Zima, B. Chapman, Supercompilers for Parallel and Vector Computers, ACM Press, New York, 1991. How to Build a Cluster Carlos Henrique 50
Thank you! Carlos Henrique da Silva Santos henrique@dmo.fee.unicamp.br Leonardo André Ambrosio leo@dmo.fee.unicamp.br Hugo Enrique Hernández Figueroa hugo@dmo.fee.unicamp.br How to Build a Cluster Carlos Henrique 51
Network: Data Communication Client process Client machine (1) Connect request (2) ACK Server machine (3) Request data System calls (4) Reply Server process Operating system Kernel Protocol stack Drivers (5) Disconnect (6) Disconnect Kernel Protocol stack Drivers How to Build a Cluster Carlos Henrique 52