Data and Information Management for EO Data Centers Eberhard Mikusch German Aerospace Center - Mexico, 23. 04 2008
Earth Observation System Environment at DLR/DFD 10010 00101 Radar 10010 00101 Atmospheric 10010 00101 Optical Multi-Mission Mission Production Chains Distributed Sites Archive C-AF Oberpfaffenhofen, Data Volume [TByte] 130 120 110 100 90 80 70 60 50 40 30 20 10 AIR_RS TerraSAR-X GEMOS SRTM XSAR_2 XSAR_1 ERS_SAR SEASAT ENV_SCIA ENV_MIPAS ENV-AATSR HyMAP ROSIS DAIS ZKI MODIS NOAA GOME METOP-GOME2 MSG METEOSAT (SOLEMI) METEOSAT ERS_GOME ERS_ORB Neustrelitz 0 1996.06 1996.12 1997.06 1997.12 1998.06 1998.12 1999.06 1999.12 2000.06 2000.12 2001.06 2001.12 2002.06 2002.12 2003.06 2003.12 2004.06 2004.12 2005.6 2005.12 2006.6 2006.12 2007.6 Robot Archives Oberpfaffenhofen Folie 2
DFD Archive Volume Statistics - Oberpfaffenhofen Archive C-AF Oberpfaffenhofen, Data Volume [TByte] 130 120 110 100 90 80 70 60 50 40 30 20 10 AIR_RS TerraSAR-X GEMOS SRTM XSAR_2 XSAR_1 ERS_SAR SEASAT ENV_SCIA ENV_MIPAS ENV-AATSR HyMAP ROSIS DAIS ZKI MODIS NOAA GOME METOP-GOME2 MSG METEOSAT (SOLEMI) METEOSAT ERS_GOME ERS_ORB 0 1996.06 1996.12 1997.06 1997.12 1998.06 1998.12 1999.06 1999.12 2000.06 2000.12 2001.06 2001.12 2002.06 2002.12 2003.06 2003.12 2004.06 2004.12 2005.6 2005.12 2006.6 2006.12 2007.6 Folie 3
Support for Numerous Missions and Services EnMAP TanDEM-X TerraSAR-X National Missions European Missions International Missions ENVISAT METOP Mission examples Airborne Imaging TERRA CHAMP SRTM Archive C-AF Oberpfaffenhofen, Data Volume [TByte] 130 120 110 100 90 80 70 60 50 40 30 20 10 0 1996.06 1996.12 1997.06 1997.12 1998.06 1998.12 1999.06 1999.12 2000.06 2000.12 2001.06 2001.12 1996 2008 2002.06 2002.12 2003.06 2003.12 2004.06 2004.12 2005.6 2005.12 2006.6 2006.12 2007.6 AIR_RS TerraSAR-X GEMOS SRTM XSAR_2 XSAR_1 ERS_SAR SEASAT ENV_SCIA ENV_MIPAS ENV-AATSR HyMAP ROSIS DAIS ZKI MODIS NOAA GOME METOP-GOME2 MSG METEOSAT (SOLEMI) METEOSAT ERS_GOME ERS_ORB Folie 4
Variety of Products and Processing Atmospheric Data 10110010 00010101 10111001 01000100 700 600 500 O 3 O 3 300 240 Optical Sensor Data 10110010 00010101 10111001 01000100 Radar Data 10110010 00010101 10111001 01000100 Folie 5
Data and Information Management System - Functionality Online Access Delivery Cataloguing & Archiving Order Management 10110010 00010101 10111001 01000100 700 600 500 O 3 O 3 240 300 Production Folie 6
Data and Information Management System - Components Scientific & Commercial Users Related Systems DIMS User Information Services EOWEB MMFI TSXX Product & Order Interface Order Management Product Library Inventory Archive Archive Online/ Offline Delivery Operator Interface Operators Production Control MMFI TSXX Multi-Mission Facility Infrastructure TerraSAR-X Comm. Service Segment Unified Meteorological Archive and Retrieval Facility Atmospheric Optical Radar Processing Systems Folie 7
Data and Information Management System - Scenarios Order-Driven Scientific & Commercial Users Related Systems DIMS Catalogue Order User Information Services EOWEB MMFI TSXX Product & Order Interface Order Management Processing on Request Subscription Product Library Inventory Archive Archive Online/ Offline Delivery Operator Interface Operators Ingestion Re-Processing Production Control Data-Driven MMFI TSXX Multi-Mission Facility Infrastructure TerraSAR-X Comm. Service Segment Unified Meteorological Archive and Retrieval Facility Atmospheric Optical NRT Service Radar Processing Systems Folie 8
DIMS Multi-Mission Support Related Systems MMFI TSXX DIMS DIMS Order Management Product & Order Interface User Information Services EOWEB Product Library Inventory Archive Archive Ozone Production Control SAF Online/ Offline Delivery WDC - RSAT WDC - RSAT Ozone SAF SRTM PAF SRTM PAF MODIS PAF Operator Interface ENVISAT VA MMFI TSXX Multi-Mission Facility Infrastructure TerraSAR-X Comm. Service Segment Unified Meteorological Archive and Retrieval Facility TerraSAR-X PGS TerraSAR-X Atmospheric PGS Optical Future...... TanDEM-X, EnMAP, TanDEM-X, EnMAP, Radar PAF PDS SAF VA WDC RSAT Processing and Archiving Facility Payload Data Handling Segment Satellite Application Facility Unified Meteorological Archive and Retrieval Facility Value Added World Data Center Remote Sensing of the Atmosphere Folie 9
Essential Tasks of a Data Center Product Management Processing Management sensor = X-SAR sensorparameters headingangle = 39.5 deg radarfrequency = 9.601*10 9 Hz code = Geocoded Dataset temporalcoverage start = 2000-02-14T11:14:37 stop = 2000-02-14T11:15:01 lat1 = 41.35 lon1 = 11.30 mission = SRTM SAR Processor Geocoding Processor Product Processing System sensor mission = X-SAR SRTM sensorparameters = X-SAR sensorparameters code headingangle = Geocoded= Dataset 39.5 deg temporalcoverage radarfrequency headingangle = 39.5 = 9.601*10 deg 9 Hz code radarfrequency lat1 start = Geocoded = 41.35 2000-02-14T11:14:37 Dataset = 9.601*10 9 Hz temporalcoverage lon1 stop = 2000-02-14T11:15:01 11.30 temporalcoverage lat1 start = 41.35 2000-02-14T11:14:37 lat1 lon1 start stop = 41.35 2000-02-14T11:15:01 11.30 2000-02-14T11:14:37 code lon1 stop = Geocoded = 2000-02-14T11:15:01 11.30 Dataset temporalcoverage lat1 = 41.35 sensorparameters lon1 start = 11.30 2000-02-14T11:14:37 mission headingangle stop = 2000-02-14T11:15:01 SRTM = 39.5 deg radarfrequency = 9.601*10 9 Hz SAR Processor PSM Geocoding Processor Metadata Browse Data Primary Data Folie 10
Essential Tasks of a Data Center Product Management Processing Management sensor = X-SAR sensorparameters headingangle = 39.5 deg radarfrequency = 9.601*10 9 Hz code = Geocoded Dataset temporalcoverage start = 2000-02-14T11:14:37 stop = 2000-02-14T11:15:01 lat1 = 41.35 lon1 = 11.30 mission = SRTM SAR Processor Geocoding Processor Product Processing System sensor mission = X-SAR SRTM sensorparameters = X-SAR sensorparameters code headingangle = Geocoded= Dataset 39.5 deg temporalcoverage radarfrequency headingangle = 39.5 = 9.601*10 deg 9 Hz code radarfrequency lat1 start = Geocoded = 41.35 2000-02-14T11:14:37 Dataset = 9.601*10 9 Hz temporalcoverage lon1 stop = 2000-02-14T11:15:01 11.30 temporalcoverage lat1 start = 41.35 2000-02-14T11:14:37 lat1 lon1 stop start = 41.35 2000-02-14T11:15:01 11.30 2000-02-14T11:14:37 code lon1 stop = Geocoded = 2000-02-14T11:15:01 11.30 Dataset temporalcoverage lat1 = 41.35 sensorparameters lon1 start = 11.30 2000-02-14T11:14:37 mission headingangle stop = 2000-02-14T11:15:01 SRTM = 39.5 deg radarfrequency = 9.601*10 9 Hz SAR Processor PSM Geocoding Processor Metadata Browse Data Primary Data Folie 11
DLR/DFD Distributed Robot Archives Neustrelitz 4 Fully Fully automated, automated, extendable extendable robot robot libraries libraries 4 Petabyte Petabyte capacity (currently 2.2PB) capacity (currently 2.2PB) 4 High High network network bandwidth bandwidth (600MBit/s) (600MBit/s) 4 Long-term Long-term data data preservation preservation 4 Long-time Long-time operational operational experience experience 70TByte of data Quantum AML/J Oberpfaffenhofen 130TByte of data Quantum AML/2 Folie 12
Hierarchical Storage Management 1st Copy migrate Quantum AML-2 write read stage 2nd Copy (remote) Disk Cache Online Nearline 2200 TByte archiving capacity, 10 TByte disk cache Quantum i2000 Folie 13
EOWEB User Information Service - Interactive User Interface User registration Product catalogue and browse Product ordering for archived data and satellite acquisitions Example: Catalogue Order, SRTM Folie 14
Geo Data Access Improving EOWEB Data Access Geo Data Browser GIS Web Sites with dynamic geo info Visualisation Evaluation Standardized access via OGC web mapping interfaces Bridge between Product Library and Geo Information Systems Post-Processing Functions: e.g. extraction, projections, format conversions DIMS OGC W*S Interface EOWEB User Inform. Services Product Library Inventory Archive Online Access Post- Processing Long-Term Archiving OGC: Open GIS Consortium Folie 15
Data and Information Management System - Operations Summary Scientific & Commercial Users Related Systems MMFI TSXX Product & Order Interface DIMS User Information Services EOWEB DLR-DFD DIMS operation characteristics Order Online/ Management 50 processing systems Delivery 50 Product Library Offline 60.000 product Inventory transfers Archive Archive per month 60.000 product transfers per month 100 100 product types 1,300,000 products 8 Mio. Mio. files files 220 220 TByte of of data data Operator Interface Operators Production Control MMFI TSXX Multi-Mission Facility Infrastructure TerraSAR-X Comm. Service Segment Unified Meteorological Archive and Retrieval Facility Atmospheric Optical Radar Processing Systems Folie 16
Conclusion Uploading Operating Tool Processing System Management Processing Algorithm User Information Services Product Library Online/Offline Product Generation Delivery Order Management Production Control Processing System Management Processing Algorithm DLR-DFD has developed a Data and Information Management System - DIMS to provide building blocks for data centers / ground segments sustainable access to archived data and satellite acquisitions a scalable multi-mission facility configurable for new missions/sensors comprehensive functionality for digital product management: archiving, cataloguing, processing, ordering and delivery DIMS components are in operational use at DLR-DFD distributed ground segment sites (since 2000) at ESA distributed multi-mission facility infrastructure (7 sites) at Infoterra TerraSAR-X commercial user service segment 10010 00101 10010 00101 Rada r Optical 10010 00101 Atmospheric Folie 17