NIST Big Data PWG & RDA Big Data Infrastructure WG: Implementation Strategy: Best Practice Guideline for Big Data Application Development

Transcription

1 NIST Big Data PWG & RDA Big Data Infrastructure WG: Implementation Strategy: Best Practice Guideline for Big Data Application Development Wo Chang National Institute of Standards and Technology (NIST) March 18, 2014

2 Agenda NIST Big Data PWG & RDA s BDI-WG Collaboration Overview NBD-PWG and BDI-WG Tasks and Deliverables Given: NBD Use Cases and Reference Architecture (RA) BDI-WG will Utilize Available Provisioning Tools to Build BD Apps. (number of well-known resources provisioning/configuring tools): Ubuntu (OS)/OpenStack (IaaS) using Juju and charms Dell (HW)/OpenStack (Iaas) using Crowbar and barclamps Microsoft System Center using set of integrated products Puppet Labs (Server/Client automation tool) using Puppet and puppies Apply Provisioning Tool to rest of RA Components Seek Volunteers to Implement Real-World Big Data applications Q/A 2

3 RDA Big Data Infrastructure WG (RDA + NIST Big Data PWG) The focus of the Research Data Alliance (RDA) s BDI-WG is to establish best practices implementation guidelines for how to deploy and manage big data applications using NIST Big Data Public Working Group s Reference Architecture and other big data architectures along with best technologies available today to meet the ever challenging big data application demands. The rate at which data volumes, speeds, and complexity are growing requires new forms of computing infrastructure to enable breakthrough discoveries and innovation in ways that enhance economic security and improve quality of life. 3

4 NBD-PWG and BDI-WG Tasks and Deliverables NBD-PWG Tasks Select/identify unique Big Data scenarios from the submitted use cases Work with use case submitters to identify unique Big Data scenarios and requirements for actual implementations. This includes security and privacy concerns and requirements Work with the BDI-WG to develop high-level interface calls between RA key components by learning how the low-level implementation work Establish preliminary high-level interface abstraction between RA key components BDI-WG Tasks Work with NBD-PWG to understand the RA structure and identified use cases Identify best Big Data technologies and solutions to implement the identified use cases Work with the use case submitters to implement their use cases Document the implementation process 4

5 51 Use Cases ( 1. Government Operations (4): National Archives & Records Administration, Census Bureau 2. Commercial (8): Finance in Cloud, Cloud Backup, Mendeley (Citations), Netflix, Web Search, Digital Materials, Cargo shipping (e.g. UPS) 3. Defense (3): Sensors, Image Surveillance, Situation Assessment 4. Healthcare & Life Sciences (10): Medical Records, Graph & Probabilistic Analysis, Pathology, Bioimaging, Genomics, Epidemiology, People Activity Models, Biodiversity 5. Deep Learning & Social Media (6): Driving Car, Geolocate Images, Twitter, Crowd Sourcing, Network Science, NIST Benchmark Datasets 6. The Ecosystem for Research (4): Metadata, Collaboration, Language Translation, Light Source Experiments 7. Astronomy & Physics (5): Sky Surveys, Large Hadron Collider at CERN, Belle Accelerator II (Japan) 8. Earth, Environmental & Polar Science (10): Ice Sheet Scattering, Earthquake, Ocean, Earth Radar Mapping, Climate Simulation, Atmospheric Turbulence, Subsurface Biogeochemistry, AmeriFlux & FLUXNET gas sensors 9. Energy (10): Smart Grid 5

6 51 Use Cases ( 6

7 Use Case: Event Discovery, Extending Published Studies Event discovery query based on heuristics Study performed using 1 year of satellite data Can we extend such studies to 40 years of satellite data and allow the studies to be done in minutes/hours? NBD PWG + RDA BDI WG Implementation Strategy, Wo Chang, NIST, March 18,

8 Use Case: Scope the Problem Five common research approaches in Atmospheric Science Event Analysis: Documenting in detail the processes for an event using multiple data and models Event Climatologies: Finding representativeness of an event by capturing its characteristics such as spatio/temporal distribution, intensity, cycles, durations, correlations with other events, etc. Synoptic Climatologies: Creating a composite evolution of an event addressing what happened before, at the time of, and after the event Forecast Methods: Creating new methods to predict an event Modeling Simulation Studies: Suite of simulations where the model s physics or initial conditions are altered to evaluate their relative importance Tools supporting Event Analytics on Big Data would support four of the five common research approaches 8

9 Use Case: Event Analytics for Big Data Research Approach Value Proposition Event Analysis Event Climatologies Synoptic Climatologies Forecast Methods Find interesting events hidden in the data Easily allow users to find representativeness of a type of an event within the data Support causality analysis with other events Allow researchers to develop new heuristic rules to define an event 9

10 INFORMATION VALUE CHAIN System Orchestrator Data Provider K E Y : DATA SW Service Use Data Flow Analytic Tools Transfer DATA SW Big Data Application Provider Collection Curation Analytics Visualization Big Data Framework Provider Processing Frameworks (analytic tools, etc.) Horizontally Automate Scalable Vertically Scalable Platforms (databases, etc.) Horizontally Scalable Provisioning Vertically Scalable Infrastructures Horizontally Scalable (VM clusters) Vertically Scalable Physical and Virtual Resources (networking, computing, etc.) DATA SW Access DATA SW Data Consumer Security & Privacy Management IT VALUE CHAIN 10

11 Provisioning Tool: Ubuntu JuJu with Charms Charms ( Scalable application services defined Charms give Juju its power. They encapsulate application configurations, define how services are deployed, how they connect to other services and are scaled. Charms are easily shared and there are 100s of Charms already rated and reviewed in our Charm store. 11

12 Provisioning Tool: Ubuntu JuJu with Charms using GUI NBD PWG + RDA BDI WG Implementation Strategy, Wo Chang, NIST, March 18,

13 Provisioning Tool: Dell Crowbar with Barclamps Modular Design: Barclamps APIs, User Access, & Ecosystem Partners Nagios Ganglia Dashboard Dell Crowbar Ops Management Cloud Infrastructure & Dell IP Extensions Core Components & Operating Systems Physical Resources Nova Swift Glance Compellent EqualLogic Crowbar DNS Logging Deployer Provisioner NTP BIOS IPMI Network RAID Dell Cloud Solutions * Crowbar has potential to service other programs beyond OpenStack 13

14 Provisioning Tool: Dell Crowbar with Barclamps using Manager 14

15 Provisioning Tool: Puppet with the Master/Agent Dataflow Puppet Dataflow 15

16 Provisioning Tool: Puppet with the Master/Agent Dataflow 16

17 Provisioning Tool: Puppet with the Master/Agent Dataflow 17

18 Provisioning Tool: Puppet with the Master/Agent Dataflow 18

19 Seek Volunteers to Implement Real-World Big Data Applications Real-World Applications Cargo Shipping Radar Data Analysis Particle Physics Pathology Imaging/digital pathology Atmospheric Turbulence 19

20 Seek Volunteers to Implement Real-World Big Data Applications Seek Volunteers Help to implement select real-world Big Data use cases 2-3 students per team per project with professor/faculty approval Two months of fast-paste volunteering commitment (no compensation) Project description, computing resources, datasets, domain experts will be provided Send to with the following information: Recommendation letter of approval by a professor/faculty member List of student names with respective computer and analytic skills Agreement to commit two months volunteering Returns Hands-on implementation of Big Data application experience Recognition on NIST and RDA websites Student names will be included on BDI-WG Best Practice Implementation Guide 20

21 Seek Volunteers to Implement Real-World Big Data Applications Q/A 21