Linux for Scientific Computing

Transcription

1 Linux for Scientific Computing Bill Saphir Berkeley Lab Things you should know if you re thinking about using Linux for Scientific Computing Bill Saphir Berkeley Lab

2 Random thoughts on things you should know if you re thinking about using Linux for Scientific Computing Bill Saphir Berkeley Lab Why? Scientific research is one of the first areas where Linux has had a major impact on production, mission-critical computing.

3 Features of scientific computing Floating point performance is everything (well, almost everything) Users write their own codes Legacy Fortran is common Full-featured user-friendly GUI interface not required Goal is science, not computer science. Who are the foolish zealots? Data analysis in experimental physics Cern, Fermilab, SLAC, Brookhaven, DESY; Astrophysics Parallel computing on clusters Sometimes called Beowulf clusters Mini-supercomputers Thousands of random apps powered by graduate students

4 Outline Why Linux? Hardware Computer architecture Processors Benchmarks Serial computation Compilers Libraries Parallel computation SMP Clusters Why Linux? Access to cheap hardware Control Availability of software Convergence Access to cheap graduate students Alternative to NT

5 Computer architecture for HPSC P P P pppppp pppppp pppppp M M M M Von Neumann Faster Processors Memory bandwidth important. Cache not so important Vector Processors no longer with us (in US) P P P P P P P P M M M M M SMP DMP Processor support in Linux These supported processors are useful for scientific computing: x86 Alpha Sparc/Sparc64 PowerPC MIPS Coming up: Power 3 Merced

6 Which processor? Three important criteria Cost Performance Availability of software Measuring Performance Peak Linpack STREAM (memory bandwidth) SPEC NPB and NSB

7 Current Peak Rates Name MHz Flop/cycle Peak Mflop/s Alpha Alpha Power Sparc PIII R10K Linpack The Linpack benchmark solves a dense linear algebra problem -- BLAS 3 Can be run in serial or parallel Because BLAS 3 can be blocked, Linpack effectively runs in cache and gets a very high percentage of peak. Linpack is important for two reasons: Good basic test of whether a machine (parallel) can run or not Basis of Top 500 list (

8 STREAM Benchmark Measures memory bandwidth Developed by John McCalpin 4 Tests Copy (A = B) Scale (A = s*b) Add (A = B + C) Triad (A = B + s*c) Stream Results Processor MHz Peak (Mflop/s) Triad (2*MW/s) Alpha Alpha Pentium II Ultrasparc (UE10K) MIPS (O2K) Power ~250 Cray C

9 SPEC95 SPEC = Standard Performance Evaluation Corporation SPECint95 8 integer-intensive codes written in C SPECfp95 10 floating point-intensive codes written in Fortran All are scientific computations. SpecFP 95 Processor MHz SPECfp95 SPECint95 Alpha Power Ultrasparc MIPS Athlon PIII/ Alpha

10 Multiprocessor machines x86/alpha/sparc/mips all available in SMPs Cache coherent shared memory Single copy of operating system Well-supported by Linux up to about 4 processors OS support is not the limiting factor. Memory bandwidth is. Low-end SMPs share memory through a bus Nearly saturated by one processor. Two or more processes compete for memory bandwidth. Expect 1.5x speedup max on Intel or current Alpha. Software Compilers Libraries 3rd party software

11 Compilers Old standbys, available on all platforms C: gcc C++: g++ Fortran 77: g77 Open source but: g++ doesn t handle complex C++ (e.g. heavy use of expression templates) g77 is Fortran 77 only no parallelization for SMPs generated code is not very fast x86 Compilers Portland Group ( Fortran 90/95/OpenMP parallelization/better performance (~10%)/HPF Kuck and Associates ( Better C++/OpenMP parallelization NAG ( Fortran 90/95/Tends to be picky Absoft ( F90/95/Includes IMSL (RH 5.2?) Fujitsu ( C/C++/F90/F95

12 Alpha compilers Compaq/DEC compilers are available Better performance (optimized for Alpha) Full Fortran 90 (available now) C/C++ later NAG Fortran 95 Other compilers Ultrasparc Nothing more available MIPS Nothing more available Power-3 IBM is looking into putting AIX compilers under Linux

13 NAS Parallel Benchmarks Developed at NASA Ames Numerical Aerodynamic Simulation facility. Designed to measure performance of parallel computers 8 codes: 5 kernels and 3 pseudo-applications represent a CFD workload. 5 sizes: S, W, A, B, C. Two versions NPB 1: pencil and paper (algorithm specified) NPB 2: specified by source code NAS Serial Benchmarks (NPB 2-serial) are single processor versions of NPB 2. A Few NSB results Proc MHz Cmplr OS FP Avg ds DEC Tru ds DEC Tru xp 500 DEC Tru xp 500 DEC Linux xp 500 gcc Linux DEC Tru PII 400 PGI Linux 53.4 Celeron 400 PGI Linux 45.1 see

14 Basic Free Numerical Libraries There are many free libraries. Some of the more important (and industrial strength) ones are: Optimized BLAS for x86 FFTW: Fastest Fourier Transform in the West Don t use numerical recipes! Basic libraries - Commercial X86 NAG ( IMSL ( Alpha Compaq Portable Math Library (CPML) -- libm replacement Compaq Extended Math Library (CXML)

15 More software Two excellent sources of information. Scientific Applications on Linux at Kachinatech: Steven Baum s Linux List Parallelism 2 types of concurrency in parallel applications Embarassing parallelism Little/no coupling between tasks Independent processes can be executed in parallel seti@home; analysis of event data from colliders; monte carlo simulations. Everything else parallelism is fine-grained data distribution is fine-grained frequent communication main application focus of the rest of this talk

16 Parallelism Three viable programming models Compiler-generated parallel code SMP only Not (yet?) widely used with Linux Threads SMP only Not widely used for scientific computing Message passing Distrbuted memory or SMP Widely used on clusters Non-viable alternatives: HPF, distributed shared memory Compiler parallelization Compiler detects concurrency in loops and distributes work in a loop to different threads. for (i = 0; i < ; i++) a[i] = c[i] * (b[i+1] - 2b[i] + b[i-1]); Requires cache-coherent shared memory in general Compiler is usually assisted by compiler directives. OpenMP is the standard for Fortran and C KAI Portland group

17 Message Passing Programming model: Separate processes with separate address spaces Communication by cooperative send/receive Mixed MPI/threads possible in theory, but not supported in Linux implementations. MPI (Message Passing Interface) is the industry standard. PVM should be used only when MPI can t do the job. Hardware Distributed memory (cluster) Shared memory Mix of shared/distributed Clusters A cluster is a collection of interconnected computers used as a unified computing resource. (Pfister) Clusters can offer High performance Large capacity High availability Incremental growth Clusters used for Scientific computing Making movies Commercial servers (web/database/etc)

18 Beowulf clustering Clustering of x86-based Linux machines for scientific computing was popularized by the Beowulf project at Caltech/JPL. Beowulf-class is a slippery term, but usually implies: Off-the-shelf parts Low cost LAN Open source OS National labs are looking at highly-scalable non-beowulf clusters for next generation of supercomputing. How to build a cluster Building/maintaining a cluster is a lot of work Type of cluster depends on the type of job. Tightly coupled applications have more stringent requirements. Expect a flood of software and documentation to appear over the next year that makes it much easier to put together clusters.

19 Architecture Switch Network setup Private network Cluster security/setup/administration much easier Application cannot interact with outside world Public network Security/setup/administration difficult. IP addresses needed. Interaction possible Firewall Most flexible Experts only

20 Local install or diskless? Local install Most natural if you re used to installing desktops N separate copies of Linux to maintain Works best in completely homogeneous system Diskless install 1 copy of Linux to maintain Requires special tools to manage For many applications, scales up to 32 or 64 nodes Node classification Interactive nodes Attached to external network Compile/edit/debug Fileserver nodes Global file systems (e.g. home directories) Remote filesystems for diskless clients Other service nodes Batch server/yp server/security server Compute nodes Space-shared by parallel applications

21 Other cluster infrastructure YP (NIS) for user management BOOTP for IP address management Global filesystem. Necessary and expected Most important unsolved problem of clusters. No viable solution except NFS See Other Hardware Network Fast ethernet. By far the most common. Gigabit ethernet. Expensive, not much faster Myrinet. Network of choice for high-end clusters. $1500- $2000/node. Scalable. New networks on horizon: Giganet, Servernet II Virtual Interface Architecture may make high performance networks more accessible and available. Serial console management Cyclades, Rocketport (comtrol.com) multiport serial cards

22 Other software MPI Get MPICH from LAM is another free implementation, but no compelling reason to use it. PBS Batch management system developed at NASA Ames Space shares the cluster; manages multi-user system Easily integrated with MPICH Task Farms How would you do things differently for a task farm (embarassingly parallel application)? Consider MOSIX to transparently load balance processes Switched network not necessary

23 Good news More good news

24 Bad News Downright Ugly

25 The Network Matters More info on clusters How to Build a Beowulf. Sterling Becker, et. al. MIT Press, 1999 In Search of Clusters. Gregory Pfister. Prentice Hall, 1998 (2nd edition) The Beowulf mailing list: subscribe to beowulf-request@beowulf.gsfc.nasa.gov HOWTO:

26 Open source presentation