German Copernicus Data Access and Exploitation Platform BiDS 16, Teneriffa, Spain,

Transcription

1 DLR.de Chart 1 German Copernicus Data Access and Exploitation Platform BiDS 16, Teneriffa, Spain, Christoph Reck Gina Campuzano Klaus Dengler Torsten Heinen Mario Winkler DLR Oberpfaffenhofen German Aerospace Center (DLR) Earth Observation Center (EOC),

2 COPERNICUS: Initial constellation complete Sentinel-1A, Sentinel-2A, Sentinel-3A Sentinel-3 first image Released 02/03/2016 2:21 pm Copyright Copernicus data (2016)

3 DLR.de Chart 3 and more data to come ESA Data Hub all Sentinels user products Yearly volume estimate [TB] Average Data Rate [Mbit/s] February 2016: 14 monthssentinel-1 PAC operations: 1 Petabyte; > 750,000 data sets, (data sets to Level-0 products to valueadded Level-2 products) in the EOC long-term data archive. This amount stored since December 2014 for only S1A is equivalent to all the radar data from the Envisat satellite that were generated during its mission lifetime of over ten years.

4 DLR.de Chart 4 Deutsches Satelliten Daten Archiv Sentinel PAC Archive 50 (+33) PetaByte storage capacity ~ 1,5 PetaByte of product data per year per Sentinel

5 DLR.de Chart 5 OGC-Compliant Spatial Data Services Interoperable data discovery, viewing, and download Data discovery, viewing, and download Google Earth Apps on mobile devices EOWEB GeoPortal Community portals Geoportal.DE, WDC-RSAT, GEOSS, Desktop GIS Data access services Discovery CSW-ISO Viewing WMS Download WFS/WCS Processing WPS Data preparation and storage EOC Geoservice Long-term archiving Data Catalog EOC Earth observation data library

6 DLR.de Chart 6 Big Data Computing at EOC: GeoFarm - in-house private cloud for internal large scale EO computing open for projects with partners (no public cloud ) platform for demonstration of cloud technologies for EO DLR precursor for larger installations at the envisaged Copernicus Center GeoFarm Extension 2016 > 4300 Cores > 33 TB RAM > 1,9 PB Storage (HDD & SSD) Copernicus DLR > 2,7 PB Sentinel product data/ Yr * > 2 * 10 Gbit network connection > 50 PB long term archive capacity **

7 DLR.de Chart 7 German Copernicus Data and Exploitation Platform Objectives CODE-DE (Copernicus Data and Exploitation Platform Deutschland) is to establish, configure, and operate the software systems and infrastructure to exploit the possibilities of the continuous data stream of free, full and open Copernicus Sentinel data and service information covered by the following elements: Ingestion and Archive Search and Access Processing Value Added Products Portal and User Management Monitoring and Reporting

8 DLR.de Chart 8 CODE-DE: Data and Exploitation Infrastructure Copernicus- Products National Missions Long-Term Archive Copernicus & ` value-added Products Financing: TBytes / Jahr Global EU DE S S S Summe Online-Archive Processing (orchestration, processors, toolboxes) Access (search, visualization, download) Portal CSW Open Search W*S http(s)

9 DLR.de Chart 10 Key Functional Aspects Fast (> 100 Mbyte/sec) and parallel access (> 20 applications) Based on best-practices and OGC Standards Avoid data duplication Provide processing capacity collocated to the data Portal functions (FAQ, Service Marketplace, Catalog Client, etc.) Follow usability best-practices User management (with quotas, priorities and external federations) Security (avoid compromising data and systems) Reporting usage

10 DLR.de Folie 11 Architecture Users Internet (via DFN) Public Processing Cloud(s) data access UFTP clientd Internet Catalogue Client HTTP HTTPS Service Marketplace EIP DMZ EGP Client Drupal HMA CSW Discovery Services FEDEO ECSW EOWEB OpenSearch register metadata Ingestion & Eviction Service pull + push WMS Visualisation Services Metadata Extraction trigger write pull WFS Download Service Storage 150 TB WCS GPFS access control HTTPS NGINX trigger access control access IP-Multicast Distribution Service UFTP Processing Cloud WPS SSH HTTP PaaS vsphere (Hosted Proc.) Hadoop VM servers... Hadoop Cluster... Copernicus data Access and Exploitation Platform EOC-Production Interfaces Long-Term Archive psm delivery Data Hub Relay DHuS HTTP(S) Apache Storage 150 TB Governance User Management Monitoring Reporting Functional element SW Component Toolbox or Plug-in (to be developed)

11 DLR.de Folie 12 Hardware Infrastructure 2 x 12 Cores E5-2680v3 128 GB Ram GPFS Server VM Client(s) VM 2 x 12 Cores E5-2680v3 128 GB Ram GPFS Server VM Client(s) VM... 2 x 12 Cores E5-2680v3 128 GB Ram GPFS Server VM Client(s) VM Clustered NFS 2 x 10 Gbit/s x2 x2 x2 x2 x2 x2 HPC Cluster 2 x Xeon 6 Core 2.6 GHz 48 GB Ram 2 x Xeon 6 Core 2.6 GHz 48 GB Ram... 2 x Xeon 6 Core 2.6 GHz 48 GB Ram Storage Head Dual Controller Storage Extension 1 Storage Extension 2 Storage Extension 3 up to 16 x 8 Gbit/sec FC connections up to 2 x 16 x 6 Gbit/sec SAS connections up to in Disks > 1 PByte

12 DLR.de Chart 13 Prototype Performance Measurements KBytes/s 2,300 2,100 1,900 1,700 1,500 1,300 Reading files in parallel dd if=$filename of=/dev/null bs=1g The upward error bars depict the maximum value of 40 runs, whereas the lower error bar shows the standard deviation from the average. 1, parallel transfers KBytes/sec 1,200 1, DHuS HTTPS remote access over 10GBit/sec network wget -q " > /dev/null parallel transfers

13 DLR.de Folie 14 Catalog Client

14 DLR.de Folie 15 > DFD-INF Abteilungsbesprechung > C.Reck CODE-DE > Processing Platform Calvalus (Brockman Consult) access by users, data consumers Hadoop cluster virtual machines for internal services, hosted services, and projects VM servers HDFS distributed storage I/O host computing and EO data cluster for systematic and hosted processing storage for VMs, EO data, data access SAN access to data hub

15 Temporal Feature Extraction Germany Example: Landsat 8, RGB NDVI max-mean-min RGB NDBI-max, NDVI-max and NDWI-mean

16 Temporal Feature Extraction Germany Base Products : Thematic Masks

17 Temporal Feature Extraction Germany Base Products : Thematic Masks

18 Temporal Feature Extraction Germany Base Products : Thematic Masks

19 Temporal Feature Extraction Germany Base Products : Thematic Masks

20 DLR.de Chart 21 Conclusion Features Fast catalogue with HMA CSW and OGC OpenSearch interfaces Flexible dataset browsing with OGC Web Map Service (WMS) High performance data access using HTTP protocol Advanced data access using OGC Web Coverage Service (WCS) Parallel file system on an online storage attached network (SAN) Hadoop and Docker processing infrastructure Option for historical data access from the long-term-archive cross-cutting services like Governance, User Management, Monitoring and Reporting and Network infrastructure complement the architecture. These form a streamlined simple, scalable and performant architecture covering standardized interfaces from Discovery over Visualization to Download for web users and machine-to-machine applications.

21 DLR.de Chart 22 Thank you for your attention!