Message-passing Multiprocessors. Server-based systems

Transcription

1 Message-passing Multiprocessors. In recent years, there has been interest (and even commercially available machines) which do not use common memory at all, but instead, communicate by passing messages. the two techniques are equivalent: the question for designers of MIMD machines comes down to performance and convenience as well as the issue of how the parallel machine is constructed, there is the additional issue of how one sets about programming it it may be easier to program a shared memory machine as a message-passing system Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 137 Server-based systems One does not normally think of server-based systems as MIMD machines because they are not collaborating on one task (unless one considers providing a computing service as a task). These server-based systems distribute the processing involved at a functional level: one machine will provide X-service (i.e. the hardware and software resources for interacting with the X-protocol), another will provide CPU service, another will provide file service, and another provide printing service (etc.). Communications is via an Ethernet (which provides a shared 10 or 100 Mbit/s link). Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 138

2 Loosely coupled vs. Tightly coupled distributed memory multiprocessors Functionally distributed systems (like the Department s server system) are loosely coupled These machines have many processing elements which collaborate on the solution of some problem The task is executed in parallel, rather than being primarily executed serially, with only occasional interchanges of data between processes. (This is really the definition of a tightly-coupled multiprocessor) Another phrase used is the grain of the processing how much time is spent in each process between each communication with another process? Small grain multiprocessing: short time between communications Large grain multiprocessing: long time between communications Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 139 NUMA systems Some systems use non-uniform access: shared memory with an interconnecting system These generally use current high performance processors, each with local cache and memory, as well as fast access on to a high speed network Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 140

3 Transputers and distributed memory multiprocessors The transputer range of processors was manufactured by INMOS, (became part of Thomson CSF) they consisted of a complete microprocessor system (CPU, optional FPU, memory) and four high-speed serial links, all on a single chip. The idea was that one could connect these processors together with a minimum of additional hardware. and so build one s own low-cost distributed memory multiprocessor The programming language, Occam, was designed especially to support this type of architecture and it had good theoretical foundations as well. This could be a functionally distributed system or a general purpose system such systems were manufactures by Meiko and Parsytec, amongst others. Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 141 What went wrong? Many things Faster general purpose processors came along more quickly than expected allowing better performance on a single processor and Inmos were not able to upgrade their processor range fast enough functionally distributed systems were straightforward but general purpose ones required general-purpose routing software in addition this slowed down the passing of messages Link-switching chips were introduced but it still didn t catch on! Meiko produced a similar machine based on high-speed SUN microprocessors. Company now called Quadrics, still producing high performance massage passing supercomputers Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 142

4 Parallel Software There remains the difficult problem of producing parallel software it s hard to produce sequential software and much hardware to produce efficient parallel software parallelization of existing software could be the answer but this is also very difficult using high-level descriptive (rather than imperative) languages could be a solution functional languages are descriptive (rather than imperative) and could be implemented in parallel directly the reduction machine logical languages e.g prolog again a possibility: the logical inference machine Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 143 Programming MIMD machines Special medium- and high- level languages have been designed Medium level: Occam designed for the transputer uses message-passing as a primary construct in the language makes parallelism clearly evident inside the program but hard to use thinking in parallel is difficult Occam syntax was unusual as well Parallel Java or Parallel C(++) or Parallel Pascal has familiar syntax plus additional keywords familiar but still difficult No clear solution is in sight but there is plenty of research Java Grande, Parallel Java for example Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 144

5 Real machines: The Cray T3 series Seymour Cray was the pioneer of supercomputers - but even he ran into financial difficulties. His company is now owned by Silicon Graphics (Sgi) EPCC have a Cray T3E 368 Digital EV5.6 processors (like Alpha processor) Each has between 128 and 256Mb Cray s T3E Mhz processors local memory 256M to 2Gb per processor 3D interconnect allows global memory with low latency Software: supports both global memory and distributed memory models F90, C++, plus a great deal of OS software. EPCC are running a 1 year M.Sc. In High Performance Computing Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 145 Real Machines: There are quite a lot of manufacturers of parallel computers: see Beowulf cluster A cheap alternative for parallel processing Multi-PC parallel (MIMD) machine Relatively new idea, mid s Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 146

6 Each node is a commodity PC, running Linux, with CPU, memory, hard disc and network interface, but only one display. Nodes are connected through a high-speed Ethernet switch. Large clusters may use hundreds of PCs, and more and/or faster networking systems.see "The do-it-yourself supercomputer", W.W. Hargrove, F.M. Hoffman and T.Sterling, Scientific American, August 2001 Node 1 Node 2 Node 3 To external server Ethernet Network Node 4 Node 7 Node 5 Node 6 Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 147 Why Beowulf? Simplicity easy to build: Linux software for supporting multiple processor tasks is available (and free) cheap to build: all the components are commercially available Problem: CPU/memory link is much faster than network link 3.2GB/s vs 100Mb/s large grain parallelism : need to choose applications carefully: Examples: Optimisation: searching parameter spaces running a task with a different set of parameters on a number of different processors network usage is only required in distributing parameters, and collecting results genetic algorithm searches macroeconomics neural compartmental modelling and other tasks where the spatial characteristics of the task can be distributed to different processors, and the network communication kept low. Copyright 1999, 2002 Leslie Smith 31R6 - Computer Design Slide 148