The Murchison Widefield Array Data Archive System. Chen Wu Int l Centre for Radio Astronomy Research The University of Western Australia

Transcription

1 The Murchison Widefield Array Data Archive System Chen Wu Int l Centre for Radio Astronomy Research The University of Western Australia

2 Agenda Dataflow Requirements Solutions & Lessons learnt Open solution Data ingestion Data distribution Data storage & access Data processing 1

3 Quick Overview On the ground Main Goals: Detection of hydrogen signature from EOR, transient detection, all sky survey, Solar MWA Overview Low frequency interferometer MHz (30 deg FOV) Science goals Detection of hydrogen, Signature from EOR, Transient detection, All sky survey, Solar observations 2048 dipoles antennas Low frequency interferometer MHz 2048 dipoles 16 elements per tile to make 128 tiles (8128 baselines) Virtual beam steering 16 elements per tile to make 128 tiles ( baselines) Longest baseline 3km Dense core short baselines Virtual beam steering Operation started in July 2013 Longest baseline 3km Metal spiders 1 2

4 Tier 0 (MRO) Online processing Online archive 400 MB/s NGAS Client Receiver Correlator PFB Tile Beam former Data Capture Tier 2 Tier 1 Science Archive 10Gbps 1Gbps 10Gbps Mirrored Archive AARNet MIT, USA VUW, New Zealand RRI, India MS Backup 1.44 TB/night Images 18 GB/night QA Raw Visibilities Catalog Catalog NGAS Client Long-term archive ( 3 PB / year) RV MS Measurement Set Raw RV Pawsey, Perth (Disk + Tape) VO local storage Visibilities 7 TB / node RTS images Web CASA pipeline Sky model RTS pipeline Fornax Processing (96 nodes) Scripting UI Image (cutout) & Image cubes 3

5 High-level Requirements of MWA Archive High throughput data ingestion ~400MB/s visibility, MB/s vis + image cube Efficient data distribution to multiple locations Australia / New Zealand / USA / India Secure and cost-effective storage of 8 10 TB of data collected daily Fast access to science archive by astronomers from 3 continents Intensive re-processing of archived data on GPU clusters calibration, imaging, etc. Continuous growth data volume (3PB/year), data variety (visibility, image cube, catalogue FITS, CASA, HDF5, JPEG2000) environment (Cortex à Pawsey) 4

6 Open Solution NGAS Openness Open source (L-GPL) originated from ESO for data archiving by Knudstrup and Wicenec (2000) Used by the astronomy community VLT, & La Silla, ALMA, evla, MWA Plugin-based - almost every feature is implemented as a plugin plugged into a light kernel Loss-free operation consistency checking, replication, fault-tolerance (e.g. power outage, disk failure, etc.) Object-based data storage Span multiple file systems distributed across multiple sites (e.g. VLT > 100 million objects) native HTTP interface (POST, GET, PUT, etc.) and location-transparent (Web admin UI) Scalable architecture Horizontal scalability add another NGAS instance Low cost, hardware-neutral deployment, Linux machine + Disk arrays Support mobile storage media with energy efficiency Ingestion, Staging, Long-term Archive, Temporary storage, Proxy, Processing, etc. Versioning 5

7 Open Solution NGAS The code base of the NGAS core software contains about 40,000 lines of Python code For the MWA Archive, we have made some optimisations (e.g.) Multiple stream concurrent data transfer across the Pacific ocean (> 60MB/s) New plugins (Proxy Archive) flexible data path routing New plugins for interacting with the Hierarchical Storage Management systems A new running mode NGAS Data mover A new in-archive processing framework (similar to MapReduce) 4,000 lines of C++ code to integrate the NGAS Client into the DataCapture system Openness allows us to Fully understand what is precisely happening on each section of the data path Make necessary changes / optimizations when necessary Make changes quickly in an agile fashion for verification and test 6

8 Data Ingestion Involves: Client push (synchronous) or Server pull (asynchronous) Find data type-specific storage medium with available capacity Receive data stream, and compute checksum (CRC) on the fly Data is stored temporarily at the Staging Area Data Archive Plug-In is invoked Quality check / compression Move file to the targeted volume / directory (e.g. MWA-DAPI) Register file in NGAS DB If replication defined, trigger file delivery threads file removal scheduling 7

9 Data Ingestion simulation simulators, each produces data at a rate of 16 MB/s to 2 NGAS servers running on 1 Supermicro node, and each NGAS server ingests data from 6 simulators Archiving simulation throughput 1200 Total archive throughput - MB/s A - 1Gbps bandwidth for commissioning B - 12 clients / 4 servers on 6 / 2 Fornax nodes C - aggregated data producing rate for 12 clients D - 12 clients / 4 servers on 6 idataplex / 1 Supermicro E - 12 clients / 2 servers on 6 idataplex / 1 Supermicro F - 24 clients / 4 servers on 6 / 1 Fornax nodes G - 24 clients / 4 servers on 24 / 2 Fornax nodes H - 24 clients / 4 servers on 12 / 2 Fornax nodes I - 24 clients / 4 servers on 28 Fornax nodes J - aggregated data producing rate for 24 clients Simulated data rate per client - MB/s 8

10 Data Ingestion field test field test Data ingestion at the MRO Online Archive. Each one of the five parallel streams records an ingestion rate over 75 MB/s. The aggregated ingestion rate amounts to 382 MB/s 9

11 Efficient Data Distribution Subscription-based data distribution Which subscriber has what data and since when? We are using HTTP to transfer 5TB per day from Western Australia to Boston, USA. (> 50 MB /s) HTTP is only an application protocol but not a transport / network protocol 10

12 Multi-tiered data storage More than 500TB since 23 Aug 2013 M&C DB Observation Portal MWA Pawsey Hierarchical Storage Management (DMF) Tape Library x32 x32 Tape Library x2 (CSIRO - library + cables + op9cs) DM-PUT DM-GET Stage Fast Disk Storage Bulk Disk Storage CXFS Release Stage Archive API POSIX FE Nodes Science DB 10 Gbps VO (TAP, SIAP) 11

13 Project A - # of accesses Project A - # of accesses Project B - # of accesses Project B - # of accesses Overall Interaccess Overall Inter-access Project C Interaccess Project C Inter-access 12

14 Data processing on Fornax +,-.(/#012,-0,3#-(45-#52-6 Fornax Cluster Long-term archive (045 7"82$(9":-+ ()*+,$ EHF G,1-#$-, G&,-#$%> /),+;)6 ;"<*(9":-+ ()*+,# +)$ +)# JobMAN ;".<3,-(9":-+!""$!""#!"%& Process!%"& RAM #$$# &' &( )% )! )" )* )& )) )+ ), )' )( +% +! +" +*! "!$""! " Distribute #$$#!"%% Local storage 7 TB / node CACHE &' &( )% )! )" )* )& )) )+ ), )' )( +% +! +" +*!"%- Data movement dominates the processing cycle!"%' =-5->"<.-$,(9":-+!"%. Disks + Tape!%"& 73+,#-(9":-+ NGAS Short Term Storage Data Processing Pipeline!)11#!)112!)113!/01#!)11&!)11'!)11.!)11- Disk Cache Stage! "!)11$ &' &( )% )! )" )* )& )) )+ ), )' )( +% +! +" +*!/01$ #$$#!%"& &' &( )% )! )" )* )& )) )+ ), )' )( +% +! +" +*! " #$$#!%"& &' &( )% )! )" )* )& )) )+ ), )' )( +% +! +" +*! " #$$#!%"& F(G,1-#$-, (+:9,19 Tape Libraries! 1 13

15 Local file system Global file systems 14

16 Data re-processing Web UI 15

17 12 April 2013: First image from the full MWA (105 tiles) Credit: MWA Science Commissioning Team 150 MHz (8 minutes of data) over some 35 degrees of the sky 16

18 Thank you for your attention 17