NOAA National Data Center. + CLASS Overview

Transcription

1 An Overview of the NOAA National Data Center and CLASS Landscape + CLASS Overview Kenneth ths. Casey, NODC On Behalf of the CLASS Operations Working Group (COWG) + Kern Witcher, CLASS Program Manager Presentation to the DAARWG, 27 June 2012

2 First, recall

3 The NOAA National Data Centers National Oceanographic Data Center Understanding our Oceans and Coasts National Geophysical Data Center Understanding our World National Climatic Data Center Understanding our Climate

4 The NOAA National Data Centers Based on NODC s Levels of Stewardship Across the three Data Centers the words vary a little, but all focus on stewarding environmental data now and for the future.

5 Comprehensive Large Array data Stewardship dh System (CLASS) Designed originally for large volume satellite data sets IT infrastructure supporting the lowest level of stewardship NESDIS has mandated its use by all three Data Centers Even the lowest levels of stewardship require Even the lowest levels of stewardship require non IT domain knowledge and expertise!

6 and the landscape in which we are operating

7 Consolipalooza l Consolimaggedon Consolitopia Consolidation has even hit the popular p culture!

8 Many Forces At Play

9 Consolidation Free Body Diagram Federal Data Center Consolidation Initiative FDCCI Initiated by the Administration/OMB in 2010 Reduce hardware, software, real estate, cooling costs CLASS Integration into Data Center Operations Initiated by NESDIS in 2011 Support common archival storage needs of the Data Centers NESDIS Data Centerconsolidation Initiated by NESDIS in 2012 Explore options to reduce admin and IT costs Other Initiatives Other Initiated by individual Data Centers Respond to various pressures $

10 Consolidation Landscape NESDIS Data Center Consolidation FDCCI FDCCI FDCCI NODC NCDC NGDC Other Other Other CLASS Integra tion CLASS Integra tion CLASS Integra tion CLASS

11 So, what are we doing about it?

12 Activities Underway FDCCI Server virtualization Reducing ITfootprints Data calls/surveys NESDIS Data Center Consolidation Deputies meeting to form options Will report to NESDIS Management CLASS Integration

13 Aerospace Corp. Review Notional look at the way forward: Containing Costs, Enhancing Mission Data Providers and Users Science Mission Data Providers and Users Data Stewardship/Management Data Stewardship/Management Software Development and IT Data Services Software Development and IT Data Services IT Infrastructure Information Technology IT Infrastructure, O&M 13

14 Evolution of the NOAA Archive Architecture ls unde er discu ussion DRAFT Detai FY2012 FY2013 FY2014 FY2015 FY2016 Phase I Phase II Phase III NODC NCDC NGDC MOB Cloud NCDC NGDC NODC NPP GCOM W Jason Pilot IRODS Data Center Migration Concurrent CLASS Initiatives JPSS GOES R Metadata Access Path Dt Data Net Nt HPSS Archive Path On Hold Programs S t a g i n g Access Dissemination Stewardship M2M Archive Storage Service

15 Data Centers Data Migration Plan Approaches and milestones for integrating CLASS into NOAA Data Center operations by FY15 Three Phases: Archival Storage: use CLASS for safe, long term storage Access Services: expands CLASS to include access capabilities expected by Consumers and functions needed for Data Center stewardship Operations: compare levels of service and decommission local Data Center services when appropriate p

16 A metaphor, if you will Working to integrate CLASS into our archive operations is a bit like our day job. You get up, pull on your boots, and make the best of it you can. But you are not really very happy in your day job, so you start exploring alternatives. Maybe you tk take some online classes at night, learn some new skills, invest in a startup you do something to change the game, live the dream, or expand your horizons the CLASS Cloud Access Pilot is just that sort of thing. freestock photo muddy boots image free stockphoto muddy boots image

17 CLASS Cloud Access Pilot Why: To test the cost effectiveness effectiveness, scalability, performance, and agility of a Cloud solution for access to archived data Who: Three NOAA Data Centers and CLASS, reported on by CLASS Operations Working Group What: At least one data collection from each Data Center, including most/all /llof NODC s data

18 CLASS Cloud Access Pilot When: This FY, some overlap into next Where: A commercial provider of Cloud IaaS: Amazon Web Services (S3/EC2). Government Cloud also being examined. How: Three parallel activities Populate Cloud storage with DC held data Load Data Center access services to the Cloud Populate Cloud storage with CLASS held data Test the Cloud based data access services with a select group of real world users

19 CLASS Cloud Access Pilot Common Storage Services using Cloud Common Storage Services using Cloud Technologies

20 CLASS Cloud Access Pilot Traditional Data Center NOAA Google UMS Managed by User Managed by Vendor iaas paas saas/

21 CLASS Cloud Access Pilot Happy Users Find Access FTP TDS LAS DAP Data Virtually Organized in logical directories (e.g., symbolic links) Cloud IaaS Data Physically Organized in Accessions 1. The three arrows pointing into the cloud from CLASS and the DCs can develop at different rates. 2. Eventually, the DC to Cloud arrow could go away, and CLASS could manage the synchronization of data to the cloud access layer. 3. For now, discovery services like Geoportal could run locally at DCs, but could also eventually move into the cloud. Discovery services could point to the cloud access layer, the existing DChosted access mechanisms, or both. Discovery NODC, NGDC, NCDC DC local holdings sent to CLASS CLASS Data Centers Data Migration Plan

22 Status in Brief Complicated landscape with lots of forces at play Progress on Archival Storage Phase Next few months of CLASS Cloud Access Pilot critical ii to refine Services Phase

23 Brief Highlights from each NOAA National Data Center

24 NODC Facing 26% proposed FY13 budget cut; examining centralizing IT in MS, admin in MD Good progress on Archival Storage phase... on track to have our data in CLASS by end of calendar year Excited about CLASS Cloud Access Pilot and running a cloud computing pilot as well

25 NGDC Working with CLASS on generic Ingest strategies for efficient data migration Working with CLASS on Machine to Machine Search & Access Implementations: Data Center interfaces need to talk to CLASS for querying and placing orders Working with CLASS on defining data stewardship needs related to the archive holdings

26 NCDC Internal ac archival astorage migration/consolidation gato dato plan under way Taking the opportunity to clean up the 700+ individual datasets in NCDC s legacy archive NCDC to take lead for GOES R Access function, to lead Access PDR in Spring 2013 New NCDC website/portals due Summer 2012 Implementing configuration management of CDRs and other NCDC data products

27 Now CLASS Overview

28 Comprehensive Large Array-data Stewardship System Oerie Overview Presented to DAARWG Kern Witcher Kern Witcher CLASS Program Manager June 27, 2012

29 2 9 Background Refresher

30 CLASS Level I Requirements (preliminary) As an enterprise solution, CLASS will reduce anticipated cost growth associated with storing environmental datasets by: Providing common services for acquisition, security, and project management for the IT system supporting NOAA Archives Consolidating stove-pipe, legacy archival storage (See Note 3 below) systems thereby reducing the number of archival storage related IT projects for NOAA to manage and data systems for customers to access. Relieving data owners of archival storage-related system development and operations issues Note 3: Archival storage provides the services and functions for the storage, maintenance and retrieval of archival information packets. Archival storage functions include receiving archival information packets from ingest and adding them to permanent storage, managing the storage hierarchy, refreshing the media in which archive holdings are stored, performing routine and special error checking, providing disaster recovery capabilities, and providing archival information packets to access to fulfill orders

31 Current CLASS Architecture Direct Connectivity to: ESPC- NOAA Environmental Satellite Processing Center National Ice Center NOAA Coastal Watch JPSS Interface Data Processing Segment (IDPS) Functions Test and Integration Environment Development Team Fairmont, WV CLASS NCDC, Asheville NC CLASS NSOF, Suitland MD CLASS NGDC, Boulder CO Operations Ingest Storage (Disk & Tape) Public Access Satellite Landing Zone Users Replication via NOAA Science Network(N-WAVE) Users Development Environment 3 1

32 Recent Accomplishments and Current Efforts 3 2

33 Accomplishments Since Last Briefing 2.0 Core Release 6.0 Linux Migration in final Test and Integration. Release is scheduled for July 8,2012 Completed Server Virtualization ti Study 3.0 Data Center Migration Aerospace completed Phase I ( as is analysis) of Data Center Con Ops Completed Data Center Requirements Documents Completed Data Center Interface Control Documents 4.0 JPSS Successfully supported the Launch of NPP Have ingested 803Tb of NPP data (1.606Pb across both nodes) 5.0 Goes-R Completed Archive and Access PDR Completed Interface CDR Completed Receipt Node Design Machine to Machine (M2M) API in final design In Final Stages of HPSS design 3 3

34 CLASS Suomi-NPP Data Flow Svalbard, Norway C3S IDPS to SDS RDRs (~1.3TB/day) IDPS SDS IDPS to GRAVITE RDRs, VIIRS M Band EDR, OMPS Auxiliary Files (~1.4TB/day) IDPS to CLASS RDRs, SDRs, TDRs EDRs, ARP, IP, Ancillary Data, Signature Files, Release Packages (~3.2TB/day) CLASS to SDS SDRs, TDRs EDRs, ARP, IP, Ancillary Data, Signature Files, Release Packages (~3.3 3 TB/day) CLASS NSOF STAR GRAVITE CLASS to GRAVITE SDRs, EDRs, Ancillary Data, Auxiliary Files, Mission Notices (~2.5TB/day) GRAVITE to CLASS RIPs (~1.5TB/day) ~4.7TB/day NWAVE 10Gb/s replication ~4.7TB/day CLASS to STAR SDRs, TDRs EDRs, ARP, IP, Ancillary Data, Signature File (~3.3 TB/day) CLASS NGDC CLASS NCDC Public Subscriptions Public Ad hoc orders NPP Sensors NPP Segments ATMS Advance Technology Microwave Sounder C3S Command, Control & Communications CrIS Cross-track Infrared Sounder Segment OMPS Ozone Mapping and Profiler Suite GRAVITE Government Resource for Algorithm VIIRS Visible and Infrared Imaging Radiometer SuiteVerification, Independent Testing & Evaluation IDPS - Interface Data Processing Segment SDS Science Data Segment STAR NOAA Center for Satellite Applications & Research 3 4 ARP Application Related Product EDR Environmental Data Record IP Intermediate Product RDR Raw Data Record RIP Retained Intermediate Product SDR Sensor Data Record TDR Temperature Data Record

35

36 3 6 Current Program Projections

37 CLASS Program Milestones Fiscal Year FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 CLASS GOES-R Campaign SRR Archive & Interface PDRAccess PDR Interface CDR Archive & Access CDR ORR Launch 10/15 NPP Campaign JPSS-1 Launch FOR NCT3 J-1 SRR NCT4 ORR J-1 PDR IDPS IDPS Blk IDPS IDPS Blk 1.5 J CDR Blk 1.5 Blk 2.0 CDR CDR Ops Ops ORR 6/15 Launch 10/16 Climate Model Data SDS v1 CDR v1 CDR v2 Nexrad Campaign Charter ICD ORR NDE Campaign ICD PDR CDR NPP Launch Jason-3 Campaign SRR 7/11 SDR 10/11 PDR CDR ORR 2/12 6/12 9/12 Launch 4/13 Data Center Migration R3 R4 PDR CDR CLASS Releases Completed Milestone R5.2 Current Milestone R5.3 R5.4NEAAT R5.5 2 NEAAT NEAAT NEAAT R6.0 4/14 10/14 4/15 10/15 4/16 Development Construction Integration/ Test Operations 10/ Notes: CDR: Critical Design Review FOR: Flight Operations Review PDR: Preliminary Design Review SDR: Systems Definition Review SRR: System Requirements Review ORR: Operational Readiness Review NCT: NPP Connectivity Test ICD: Interface Control Document

38

39 CLASS Projected Ingest and Delivery 39

40 NOAA Enterprise Archive System (CLASS) Cumulative Total Volume by Data Type Actual Volumes Projected Volumes Petabytes Migration Completed FY NEXRAD Model In-Situ Satellite Actual Volumes Projected Volumes 44.7% Pet tabytes Est. FY 15 Legacy System Migration Completed 27.9% 19.8% NEXRAD Model InSitu/Misc % Satellite FY 40

41 41 Near and Long Term Plans

42 Near-Term Plans 2.0 Core Initiate Server Virtualization Expect at least a 30% reduction of servers. 3.0 Data Center Migration Initiate Aerospace Phase II ( to be state) of Data Center Con Ops Complete Cloud Pilot Project Complete NexRad Pilot Project 4.0 JPSS Update the CLASS IRD and ICD to support the Systems Requirements Review for the IDPS Block 1.5 and 2.0 releases 5.0 Goes-R Complete Archive and Access CDR Procure and deploy the GOES-R receipt node into the test and integration environment Update HPSS documentation and begin hardware procurement based on IBM recommendations. Finalize the M2M design and begin implementation 42

43 CY12 Software Releases 4/1 5/1 6/1 7/1 8/1 9/1 10/1 11/1 12/1 D Build 6.0 Linux Build NexRad E V Build Build 6.1 DB Performance Enhancement Build 6.2 Data Center Migration I & T Test Begins 2/3/13 P A L Linux Migration PaL unavailable for external testing O P 4/30 6/25 7/24 11/5 S NDE, GCOM-W and various NPP fixes NexRad Pilot 6.2 Data Center Migration Linux Transition Note: Please refer to REMEDY for complete release details 6.1 DB Performance enhancements: Changes to Schema, Updates to M2M

44 CLASS- Looking Forward Present CLASS Future CLASS Access Services Climate. Gov Ordering Discovery P R O D U C E R Ingest Preservation Planning Data Management Archival Storage Administration Access requests results C O N S U M E R Satellite Mod del Insitu Pre-ingest Services Data Gateway Data Gateway Data Gateway gest erface In Inte Access Interface NOAA Enterprise Archival and Storage System Admin. Interface Diss s. Interfa ace Dissemination Services Commercial Cloud Services Private Cloud Services Direct Delivery Web Accessible Folders Administration and Preservation 44

45 Final Thoughts CLASS does an outstanding job of meeting it s original intended purpose (Large Satellite Data) The program recognizes that for CLASS to become a true NOAA enterprise solution, it must evolve. The challenge is how to evolve in a manner that will not disrupt our current operational missions and accomplish this under a constrained budget environment DAARWG could be a valuable resource for providing an IV&V function for data and algorithm integrity 45