Systems and Networks for Astronomy. Marco de Vos ASTRON Director of R&D

Size: px

Start display at page:

Download "Systems and Networks for Astronomy. Marco de Vos ASTRON Director of R&D (devos@astron.nl)"

Drusilla Clarke
8 years ago
Views:

1 Systems and Networks for Astronomy Marco de Vos ASTRON Director of R&D

2 ASTRON NWO Knowledge Institute for Astronomical Instrumentation Our mission: Making astronomical discoveries happen, through innovative instruments and software

3 Making discoveries happen Strong Astronomical Science program Frontline observing facilities WSRT and LOFAR Hosting JIVE Working towards SKA Advanced Technology Lab Development of new instruments Innovation & Technology program Technology Transfer (through holding)

towards SKA Advanced Technology Lab Development of new instruments

4 Drente-light

5 Voorbeeld: de Crabnevel

9 Time-travellers

10 Constructive criticism Waarneming Experiment Model Hypothese

11 Cosmological Redshift - Hydrogen line is seen at: 1.4 MHz 14 MHz 140 MHz 1.4 GHz Big Bang protons Hydrogen First Stars Sun & Earth Now Opaque Invisible Formation of Earth

12 The many faces of the supernova remnant Casseopeia A X-ray infrared optical radio

13 De melkweg op meerdere golflengtes

15 From steel to silicon

17 LOFAR top-level architecture MHz RCU Board A/D Filter Σ RSP Board Beamformer Filter Σ GbE WAN Central Processing Facilities MHz Backplane & RF Shield Filter Filter Σ Σ Output control Distributed Beamforming RSP board 24 Station GbE switch (24 ports) GbE switch fabric (231 outputs) Sync. Buffering Delay Correlator / Beamformers (Blue Gene /L) Ionosphere Calibration RFI Mitigation Storage calibration Image creation User applications WAN fibre connections Station 77 Archive Export and GRID

switch (24 ports) GbE switch fabric (231 outputs) Sync.

18 Streaming towards science 0.5 Tb/s 1.5 Top/s 40 Gb/s 16 Tb/s 0.8 Tb/s 43 Tflop/s

19 RTS 1 First light from CS1a

20 Embedded networks

21 Station based processing Input data rate: ~ 460 Gbps Output data rate: ~ 2 Gbps Processing capacity: ~ 1.5 Tmul/s Storage capacity: 96 Gbyte Used for 1 beam & station cross correlation 1GbE 10 GbE Buffer Beamformer Filter Buffer Beamformer Filter Buffer Beamformer Filter Buffer Beamformer Filter Receiver Receiver Receiver Receiver

22 Thunderstorm Events Does the Electric field of the atmosphere influence CR radio signal? For E>100 V/cm E-field force dominates B-field: Fair weather: E=1 V/cm Thunderstorms: E=1 kv/cm Select thunderstorm periods from meteorological data: Clear radio excess during thunder storms B-field effect dominates under normal conditions >90% duty cycle possible Thunderstorm events control sample Buitink et al. (LOPES coll.) 2005 & 2006 in prep.

23 ITS Spatial Filter Experiment ITS sky map at MHz, no RFI. Two strong sources (Cas A, Cyg A) visible ITS sky map at MHz, strong RFI at (Az,el) = (-1.3,0) rad ITS sky map at MHz, fixed null at (Az,el) = (-1.3,0) rad ITS observation, 26 Feb. 2004, 60 antennas, df=10 khz, 6.75 s integration A.J. Boonstra, March 15, 2004

24 37 Tbps raw data (0.5 Tbps per station) 10 Gbps/station

25 Station FTS-2 SFP; hot pluggable 1GB/s optical module for usage in switches XFP; hot pluggable 10GB/s optical module for usage in switches 1000Base-SX; fibre connection up to 550m over multi mode fibre 1000Base-LX; fibre connection up to 10 km over single mode fibre Hub 1GbE switch (44x 1000BASE-T, 4x 1000BASE-LX) 1GbE switch (44x 1000BASE-T, 4x 1000BASE-LX) 1/10GbE switch (22x 1000BASE-T, 2x 10GBASE-LX) 10GBASE-LX, LAN-PHY Interface to Surfnet Base-ZX; fibre connection up to 100km over single mode fibre CWDM; Coarse Wavelength Division Multiplexing. Up to 20 channels over one fibre DWDM; Dense Wavelength Multiplexing. Up to 160 channels over one fibre RUG/RC CEP 1Gb/s electric 100Mb/s electric SURFNET-6 10GBASE-LX, LAN-PHY Interface to Surfnet-6 1/10GbE switch (22x 1000BASE-T, 2x 10GBASE-LX) 10Gb/s optic 1Gb/s optic

27 Streaming supercomputing Tbyte/day 10 Tbyte/day BG/L Rack BG/L Rack GbE switch GbE switch BG/L Rack BG/L Rack 10 GbE switch 10 GbE switch GbE switch 10 GbE switch GbE switch 10 GbE switch GbE switch 10 GbE switch 250 Tbyte/day Cluster of servers general purpose nodes Infiniband interconnect Cluster of servers 4BG RAM/node Infiniband interconnect Cluster of servers 10 TB RAID per node Infiniband interconnect Cluster of servers general purpose nodes Infiniband interconnect

28 CEP Performance BG/L Rack BG/L Rack GbE switch GbE switch x 8Gbps 8Gbps Store: Store: Gbps Gbps BG/L Rack 10 GbE switch 10 GbE switch GbE switch 10 GbE switch GbE switch 10 GbE switch GbE switch 10 GbE switch 2 T-ops Transpose 15 T-ops ~300 Gbps BG/L Storage: Rack >500 TB Cluster of servers 4BG RAM/node Infiniband interconnect Cluster of servers 10 TB RAID per node Infiniband interconnect x 10Gbps 3 T-ops Cluster of servers general purpose nodes Infiniband interconnect Within correlator: 20 Tbps 5 T-flops Cluster of servers general purpose nodes Infiniband interconnect Products <1 <1 Gbps Gbps

31 Storage system Infiniband cluster with DAS 10 GbE switch 10 GbE switch GbE switch 10 GbE switch GbE switch 10 GbE switch GbE switch 10 GbE switch GbE switch GbE switch GbE switch On-Line Input Section Infiniband Cluster Auxiliary processing section Infiniband cluster with co-processor boards Storage Off-line Off-line processing section General purpose Infiniband cluster Long term Archiving & Data Export GbE switch On-line pipeline processing Detailed Design On-Line pipeline processing section 4 BlueGene racks From Station Processing

32 Compute Card

33 Node Board

34 Bluegen (half) Rack

35 Blue Gene/L chip and network

36 The Double Hummer FPU Two pipelines 32 registers; 128 bits load cross pipeline register access Dedicated instructions for: complex arithmetic float/double conversions

37 Programming the correlator Tiled execution for maximum register usage 98% of peak performance achieved Correlator is main consumer good scaling till full LOFAR needs

38 Subsystems: Blue Gene/L 6 racks 128 IO nodes and 1024 compute nodes Internal Tree and Torus (175 MB/s * 6 directions) Compute node Dual core 512 MB RAM IO node 1 GbE connection Connected to 8 compute nodes Diskless 1 Service Node (p650 + DB2) TCP/IP UDP/IP 8 Front-End nodes for development MPI Sockets on GbE IO Node Sockets on Tree MPI on Torus Compute Node

39 IO with the BG/L system IO-rich configuration 8 compute nodes behind one GbE connection Compute nodes are part of global network

40 IO tree and Torus network

41 Subsystems: Blue Gene/L (2) How partitions work IO Node CN 32 nodes partition IO Node CN IO Node CN IO Node CN 128 nodes partition IO Node CN Partitions are isolated from each other Each partition executes 1 programme

42 Blue Gene/L installation April 2005

43 What does a cluster look like 40 racks with computers 5 with network components Kilometers of fibre and copper cables! All under floor

44 TFlopCorrelator demo Real-time correlation of predefined analog signals Performance: 97.5% of theoretical maximum

45 Figure 5 : Initial results of streaming data processing throughput measurements. The throughput bandwidth through a cluster node is measured as function of package size and amount of processing that is performed to the data stream. The cluster node is a Dual Xeon running Linux and is connected to the input and output nodes using Infiniband 4X. A throughput of 200 Mbyte/s is observed when 2 floating point operations are applied to each input byte (equivalent to slightly more than 2 complex multiplications per complex<float> input sample).

46 CS1 Imaging Pipeline cd Imaging observation mode Specification GUI specification Deterministic RFI fringe control Input Section Filter Correlator data storage Storage flagger MS Selfcal (BBS) MS Imager (Aips++) Image (intermediate) (intermediate) Storage LSM Image Viewer (AIPS++) Selfcal Strategy and configuration Graphical Interface user

47 Monitoring and Control LCU 1 LCU 2 LCU 100 LCU LCU CCU 1..6 CTU 1 CTU 2 CTU 16 MAC utilises a distributed database system (SCADA) Local Control Units (LCU) continue operation on their own Requires only TCP/IP and low bandwidth connection (100 Mbps)

48 The European VLBI Network

49 Very Long Baseline Interferometry

50 Post-2005: JIVE data processing centre 30 Gbps 2 Tbps Russia China USA 1-10 Gbps South Africa

51 First e-vlbi results

55 The path to the SKA ~ 0.3 GHz ~ 1 GHz EoR Detector Radio Fish-eye (dense) aperture arrays Radio Camera small (~ 8m) dishes

56 Reference Design Inner core Station Wide-angle radio camera + radio fish-eye lens

57 Dynamic Modelling Global Sky Model Instrument Model Environment Model Predict (simulated data) Sensor Data Solve

58 New operational models 22:00 18:00 12:00 03:47 06:00 input section Trigger Input data Handling Transient detection Transient follow-up Empty (Application cancelled) Transient detection on-line and aux EOR 32 MHz Empty (Application cancelled) Survey 8 MHz Tied array pulsar observation Survey 8 MHz Student beam Maintenance / testing off-line Storage Write EOR data Read UV Selfcal (observation form yesterday) Write transient Storage services Write survey Pulsar Processing (near on-line) Write survey Read Transient Read UV Automated Selfcal Survey observation Transient Analysis

59 RUG/EoR Science Center UvA/Transients Science Center KUN/UHECR Science Center RUL/Survey Science Center NITG/Geo Science Center KNMI/Geo Science Center Joint LOFAR Operations Center SurfNet6 Geant Bonn Science Center

60 e-vlbi Telescopes E-LOFAR Stations SKA Observatory SKA Pathfinders Science Centers

61 Astronomy makes things happen

Ionospheric Research with the LOFAR Telescope

Ionospheric Research with the LOFAR Telescope Leszek P. Błaszkiewicz Faculty of Mathematics and Computer Science, UWM Olsztyn LOFAR - The LOw Frequency ARray The LOFAR interferometer consist of a large