Overview NIST Big Data Working Group Activities

Transcription

1 Overview NIST Big Working Group Activities and Big Architecture Framework (BDAF) by UvA Yuri Demchenko SNE Group, University of Amsterdam Big Analytics Interest Group 17 September 2013, 2nd RDA Plenary

2 Outline Overview NIST Big Working Group (NBD-WG) activities and deliverables Proposed Big Architecture Framework (BDAF) Models and Big Lifecycle Big Infrastructure (BDI) Discussion: Liaison and information exchange with NIST BD-WG Disclaimer: Presented here information about NIST Big Working Group (NBD-WG) and images from the NBD-WG working documents are not official position of the NBD-WG and are solely the authors opinion. CWG- BDA NIST BD-WG and UvA BDAF Slide_2

3 NIST Big Working Group (NBD-WG) Deliverables target September September initial draft documents 30 September Workshop and F2F meeting Activities: Conference calls every day 17-19:00 (CET) by subgroup - Big Definition and Taxonomies Requirements (chair: Geoffrey Fox, Indiana Univ) Big Security Reference Architecture Technology Roadmap BigdataWG mailing list and useful documents Input documents Big Reference Architecture Requirements for 21 usecases NIST BD-WG and UvA BDAF 3

4 NIST Proposed Reference Architecture (before July 2013) Obviously not data centric Doesn t make data (lifecycle) management clear [ref] NIST Big WG mailing list discussion NIST BD-WG and UvA BDAF 4

5 Big Ecosystem Reference Architecture (By Microsoft) [ref] Initial contribution July 2013 [ref] Big Ecosystem Reference Architecture (Microsoft) NIST BD-WG and UvA BDAF 5

6 Capability Service Abstraction Cloud C. Framework Transitional Framework SaaS PaaS IaaS Software Platform Cluster Hardware (Storage, Networking, etc.) Management Security & Privacy Management System Management Lifecycle Management NIST Reference Architecture version 0.0 (August 2013) Sources V A R I E T Y V E L O C I T Y V O L U M E Capability Management Service Abstraction Transformation Collection Curation Analytic Visualization Access Usage Service Abstraction Usage R E P O R T RENDER IN G R E T R I E V E NIST BD-WG and UvA BDAF 6

7 NIST Reference Architecture version 0.1 (September 2013) I N F O R M AT I O N F L O W / V A L U E C H A I N System Manager or Vertical Orchestrator System Service Abstraction DATA DATA SW SW Capabilities Service Abstraction Provider Capabilities Provider Service Abstraction Big Framework Transformation Provider Collection Curation Scalable Applications (analytic tools, etc.) Legacy Applications Scalable Platforms (databases, etc.) Analytics Visualization Access Usage Service Abstraction DATA SW Legacy Platforms Scalable Infrastructures (VM cluster, etc.) Legacy Infrastructures Consumer Hardware (Storage, Networking, etc.) K E Y : DATA SW Service Use Big Information Flow SW Tools and Algorithms Transfer I T S TA C K / V A L U E C H A I N NIST BD-WG and UvA BDAF 7

8 Big Architecture Framework (BDAF) by the University of Amsterdam Big definition: from 5+1Vs to 5 parts Big Architecture Framework (BDAF) components NIST BD-WG and UvA BDAF 8

9 Improved: 5+1 V s of Big Variety Structured Unstructured Multi-factor Probabilistic Linked Dynamic Volume Terabytes Records/Arch Tables, Files Distributed 6 Vs of Big Velocity Batch Real/near-time Processes Streams Value Generic Big Properties Volume Variety Velocity Acquired Properties (after entering system) Value Changing data Changing model Linkage Variability Trustworthiness Authenticity Origin, Reputation Availability Accountability Correlations Statistical Events Hypothetical Veracity Variability Veracity NIST BD-WG and UvA BDAF 9

10 Big Definition: From 5+1V to 5 Parts (1) (1) Big Properties: 5V Volume, Variety, Velocity, Value, Veracity Additionally: Dynamicity (Variability) (2) New Models Lifecycle and Variability linking, provenance and referral integrity (3) New Analytics Real-time/streaming analytics, interactive and machine learning analytics (4) New Infrastructure and Tools High performance Computing, Storage, Network Heterogeneous multi-provider services integration New Centric (multi-stakeholder) service models New Centric security models for trusted infrastructure and data processing and storage (5) Source and Target High velocity/speed data capture from variety of sensors and data sources delivery to different visualisation and actionable systems and consumers Full digitised input and output, (ubiquitous) sensor networks, full digital control NIST BD-WG and UvA BDAF 10

11 Big Definition: From 5V to 5 Parts (2) Refining Gartner definition Big ( Intensive) Technologies are targeting to process (1) highvolume, high-velocity, high-variety data (sets/assets) to extract intended data value and ensure high-veracity of original data and obtained information that demand cost-effective, innovative forms of data and information processing (analytics) for enhanced insight, decision making, and processes control; all of those demand (should be supported by) new data models (supporting all data states and stages during the whole data lifecycle) and new infrastructure services and tools that allows also obtaining (and processing data) from a variety of sources (including sensor networks) and delivering data in a variety of forms to different data and information consumers and devices. (1) Big Properties: 5V (2) New Models (3) New Analytics (4) New Infrastructure and Tools (5) Source and Target NIST BD-WG and UvA BDAF 11

12 Big Nature: Origin and consumers (target) Big Origin Science Telecom Industry Business Living Environment, Cities Social media and networks Healthcare Big Target Use Scientific discovery New technologies Manufacturing, processes, transport Personal services, campaigns Living environment support Healthcare support NIST BD-WG and UvA BDAF 12

13 Big Nature: Origin and consumers (targets) Scietific Discovery New Technology Manufactur Transport Personal services, campaigns Living Environmnt, Infrastruct, Utility Science Telecom Industry Business Living environment, Cities Social media, networks Healthcare support Healthcare Rich information on usecases is available from the NIST document store NIST BD-WG and UvA BDAF 13

14 Moving to -Centric Models and Technologies Current IT and communication technologies are host based or host centric Any communication or processing are bound to host/computer that runs software Especially in security: all security models are host/client based Big requires new data-centric models location, search, access variability and lifecycle integrity and identification centric security and access control NIST BD-WG and UvA BDAF 14

15 Defining Big Architecture Framework Existing attempts don t converge to consistent view: ODCA, TMF, NIST See Big Architecture Framework (BDAF) by UvA Architecture Framework and Components for the Big Ecosystem. Draft Version Architecture vs Ecosystem Big undergo a number of transformations during their lifecycle Big fuel the whole transformation chain sources and data consumers, target data usage Multi-dimensional relations between models and data driven processes Infrastructure components and data centric services Architecture vs Architecture Framework (Stack) Separates concerns and factors Control and Management functions, orthogonal factors Architecture Framework components are inter-related NIST BD-WG and UvA BDAF 15

16 Big Architecture Framework (BDAF) for Big Ecosystem (BDE) (1) Models, Structures, Types formats, non/relational, file systems, etc. (2) Big Management Big Lifecycle (Management) Model Big transformation/staging Provenance, Curation, Archiving (3) Big Analytics and Tools Big Applications Target use, presentation, visualisation (4) Big Infrastructure (BDI) Storage, Compute, (High Performance Computing,) Network Sensor network, target/actionable devices Big Operational support (5) Big Security security in-rest, in-move, trusted processing environments NIST BD-WG and UvA BDAF 16

17 Big Architecture Framework (BDAF) Aggregated Relations between components (2) Col: Used By Row: Requires This Models & Structures Managmnt & Lifecycle Big Infrastruct & Operations Big Analytics & Applications Big Security Models Structrs Managmnt & Lifecycle Big Infrastr & Operations Big Analytics & Applicatn Big Security NIST BD-WG and UvA BDAF 17

18 Big Ecosystem:, Lifecycle, Infrastructure Consumer Source Collection& Registration Filter/Enrich, Classification Analytics, Modeling, Prediction Delivery, Visualisation Big Target/Customer: Actionable/Usable Target users, processes, objects, behavior, etc. Big Source/Origin (sensor, experiment, logdata, behavioral data) Big Analytic/Tools Federated Access and Delivery Infrastructure (FADI) Storage General Purpose Compute General Purpose High Performance Computer Clusters Storage Specialised bases Archives (analytics DB, In memory, operstional) Management categories: categories: metadata, categories: metadata, (un)structured, metadata, (un)structured, (non)identifiable (un)structured, (non)identifiable (non)identifiable Intercloud multi-provider heterogeneous Infrastructure Security Infrastructure Network Infrastructure Internal Infrastructure Management/Monitoring NIST BD-WG and UvA BDAF 18

19 Big Infrastructure and Analytic Tools Big Target/Customer: Actionable/Usable Target users, processes, objects, behavior, etc. Big Source/Origin (sensor, experiment, logdata, behavioral data) Big Analytic/Tools Analytics: Refinery, Linking, Fusion Analytics Applications : Link Analysis Cluster Analysis Entity Resolution Complex Analysis Federated Access and Delivery Infrastructure (FADI) Analytics : Realtime, Interactive, Batch, Streaming Storage General Purpose Compute General Purpose High Performance Computer Clusters Storage Specialised bases Archives Management categories: categories: metadata, categories: metadata, (un)structured, metadata, (un)structured, (non)identifiable (un)structured, (non)identifiable (non)identifiable Intercloud multi-provider heterogeneous Infrastructure Security Infrastructure Network Infrastructure Internal Infrastructure Management/Monitoring NIST BD-WG and UvA BDAF 19

20 Consumer Analitics Application Transformation/Lifecycle Model Model (1) Model (1) (inter)linking? Persistent ID Identification Privacy, Opacity Storage Model (3) Common Model? Variety and Variability Semantic Interoperability Model (4) Source Collection& Registration Filter/Enrich, Classification Analytics, Modeling, Prediction Delivery, Visualisation repurposing, Analitics re-factoring, Secondary processing Does Model changes along lifecycle or data evolution? Identifying and linking data Persistent identifier Traceability vs Opacity Referral integrity NIST BD-WG and UvA BDAF 20

21 Scientific Lifecycle Management (SDLM) Model DB Lifecycle Model in e-science User Researcher discovery Curation (including retirement and clean up) recycling Re-purpose archiving Raw Experimental Structured Scientific linkage to papers archiving Project/ Experiment Planning collection and filtering analysis sharing/ publishing End of project Linkage Issues Persistent Identifiers (PID) ORCID (Open Researcher and Contributor ID) Lined Re-purpose Clean up and Retirement Ownership and authority Detainment Links Open Public Use Metadata & Mngnt NIST BD-WG and UvA BDAF 21

22 Evolutional/Hierarchical Model Actionable Papers/Reports Archival Usable Processed (for target use) Processed (for target use) Processed (for target use) Classified/Structured Classified/Structured Classified/Structured Raw Common Model? interlinking? Fits to Graph data type? Metadata Referrals Control information Policy patterns NIST BD-WG and UvA BDAF 22

23 Additional Information Existing proposed Big architectures NIST BD-WG and UvA BDAF 23

24 Industry Initiatives to define Big (Architecture) Open Center Alliance (ODCA) Information as a Service (INFOaaS) TMF Big Analytics Reference Architecture Research Alliance (RDA) All data related aspects, but not Infrastructure and tools LexisNexis HPCC Systems NIST BD-WG and UvA BDAF 24

25 ODCA INFOaaS Information as a Service Using integrated/unified storage New DB/storage technologies allow storing data during all lifecycle [ref] Open Center Alliance Master Usage model: Information as a Service, Rev org/docs/information_as_a_servic e_master_usage_model_rev1.0.p df NIST BD-WG and UvA BDAF 25

26 ODCA Example INFOaaS Architecture Core and Information Components Integration and Distribution Components Presentation and Information Delivery Components Control and Support Components NIST BD-WG and UvA BDAF 26

27 TMF Big Analytics Architecture [ref] TR202 Big Analytics Reference Model. Version 1.9, April NIST BD-WG and UvA BDAF 27

28 LexisNexis Vision for Analytics Supercomputer (DAS) [ref] [ref] HPCC Systems: Introduction to HPCC (High Performance Computer Cluster), Author: A.M. Middleton, LexisNexis Risk Solutions, Date: May 24, 2011 NIST BD-WG and UvA BDAF 28

29 LexisNexis HPCC System Architecture ECL Enterprise Control Language THOR Processing Cluster ( Refinery) Roxie Rapid Delivery Engine [ref] HPCC Systems: Introduction to HPCC (High Performance Computer Cluster), Author: A.M. Middleton, LexisNexis Risk Solutions, Date: May 24, 2011 NIST BD-WG and UvA BDAF 29

30 IBM GBS Business Analytics and Optimisation (2011). b6acfbd608c0/document/aa78f77c-0d57-4f41-a923-50e5c6374b6d/media&ei=yrknubjmnm_liwkqhocqbq&usg=afqjcnf_xu6aifcahlf4266xxnhkfkatlw&sig2=j8jifv_md5dnzfql0spvrg&bvm=bv ,d.cGE September 2013, RDA- NIST BD-WG and UvA BDAF 30