@TECHTrain. Linkedin/Groups: Corporation

Transcription

1 @TECHTrain Big Data for Government Symposium Linkedin/Groups: Technology Training i Corporation

2 NIST Big Data g Public Working g Group Wo Chang, NIST, wchang@nist.gov Ro b e r t Marcus, E T S t r a t e g i e s Chaitanya B a r u, UC San u UC San Diego ttp://bigdatawg.nist.gov November, 20, 2013

3 AGENDA h h Why Big Data? Why NIST? NBD PWD Charter and Deliverables O Overall Workplan ll W k l Subgroup Charter and Deliverab bles Definitions & Taxonomies Su ubgroup Requirements & Use Case Su ubgroup Security & Privacy Subgroup Reference Architecture Subg group Technology Roadmap Subgroup Next Steps/Future Activities g y roup p ISO/IEC JTC 1 Big Data Study Gr

4 hy Big Data? Why NIST? hy Big Data? There is a broad agreement am mong commercial, academic, and government ders about the remarkable potential of Big Data to spark innovation, fuel commerce, and drive gress. hy NIST? (a) Recommendation from January , 2013 Cloud/Big Data Forum and (b) A lack consensus on some important, fundamental quesstions is confusing potential users and holding k progress. Questions such as: What are the attributes that define Big Data soluutions? How is Big Data different from the traditional datta environments and related applications that we h have encountered thus far? d h f What are the essential characteristics of Big Data a environments? How do these environments integrate with curren ntly deployed architectures? What are the central scientific, technological, and d standardization challenges that need to be addressed to accelerate the deployment of robustt Big Data solutions? NBD PWG is being launched to address these queestions and is charged to develop consensus efinitions, taxonomies, secure reference architeccture, and technology roadmap for Big Data that an be embraced by all sectors.

5 C HARTER ( M 0001 ) P W G ) is t o form a communit y he focus o f the (NBD P f i n t e r e s t f r o m industry, academia, and go g v e r n m e n t, with the goal o f developing a c o n s e n s u s definitions d, a xo no mi es, secure reference a r c h i t e c t u r es e, and ec h n o l o g y roadmap. The aim is t o creat e e vendor eutral, technology and i n f r a s t r u c t u r e a gnos g t ic eliverables t o enable big d a t a stake hold e r s t o pick nd cc h o o s e best analytics t o o l s f o r their p r o c e s s i n g nd nd visualization r e q u i r e m e n t s o n the mo s t suitable ompu ti n g platforms and clusters while a llowing e s and f l o w alue added f r o m big d a t a service provid er f da d t a be b t w e e n t he h s t a ke k h o lde ld r s in i a co hes h iiv e a nd d ecure manner. NBD PWG NBD PWG CHARTER AND DELIVERABLES DELIVERABLES: W o r k i n g D r 1 Big D a t a Definitions Big D a t a Ta xo no mi es 3. Big D a t a Requirement s 4. Big D a t a S e c u r i t y and Requirements 5. Big D a t a A r c h i t e c t u r e s 6. Big D a t a Reference A rc hi t e c t u re s 7. Big D a t a S e c u r i t y and Reference A r c h i t e c t u r e 8. Big D a t a Techn o lo g y R ad Ro dmap LAUNCHED DATE: June 26,, TARGET DATE: September 27, 2013

6

7 Requirements and Use Cases Techn nology Road dmap SUBGROUPS NBD PWG Reference Architecture Definitions & Taxonomies Security and Privacy AN D THEIR SCOPES A ND DELIVERABL E S

8 Scope (M M0020) Requirements and Use Cases Geoffrey Fox, U. Indiana Joe Paiva, VA g y, Tsegereda Beyene, Cisco The focus is to form a community of interest from industry, academia, and government, with the goal of developing a consensus list of Big Data requirements across all t k h lders. This includes gathering and understanding hi i l d th i d d t di stakeholde various usee cases from diversified application domains. Tasks Gather i nput from all stakeholders regarding Big Data ments. requirem Analyze/prioritize a list of challenging general ments that may delay or prevent adoption of Big requirem Data deployment Develop p a comprehensive list of Big Data requirements nitions and nomies Requirements BD BD WG Security and Privacy

9 equirements and Use Case S Subgroup y documents: M0105 use cases, M01125, requirements, M0152 working d Use Case Template Goals, Description Goals Description Data Characteristics, Data Types Data Analytics C Current Solutions S l i Security & Privacy Lifecycle Management and Data Quali f y g ity y System Management and Other issuess

10 equirements and Use Case S Subgroup 51 Use Cases Received ( g.nist.gov/usecases.php) 1. Government Operation (4): National Archivves and Records Administration, Census Bureau 2. Commercial (8): Finance in Cloud, Cloud Backup, Mendeley (Citations), Netflix, Web Searc Digital Materials, Cargo shipping (as in UPS) 3. Defense (3): Sensors, Image surveillance, Situation Assessment 4. Healthcare and Life Sciences (10): Medical records, Graph and Probabilistic analysis, Pathology, Bioimaging, Genomics, Epidemiiology, People Activity models, Biodiversity 5. Deep Learning and Social Media (6): Driving Car, Geolocate images, Twitter, Crowd Sourc Network Science, NIST benchmark datasets 6. The Ecosystem for Research (4): Metadata,, Collaboration, Language Translation, Light so experiments 7. Astronomy and Physics (5): Sky Surveys compared to simulation, Large Hadron Collider a CERN, Belle Accelerator II in Japan 8. Earth, Environmental and Polar Science (10): Radar Scattering in Atmosphere, Earthquak Ocean, Earth Observation, Ice sheet Radar s scattering, Earth radar mapping, Climate simulation datasets, Atmospheric turbulenc l d h b l ce identification, Subsurface Biogeochemistry d f b f h (microbes to watersheds), AmeriFlux and FL LUXNET gas sensors 9. Energy (10): Smart grid

11 equirements and Use Case S Subgroup Two step process is used for requirement exttraction: 1. Extract specific requirements and map to reference architecture based on each application s characteristics such as: a. data sources (data size, file formats, rate of grow, at rest or in motion, etc.) b. data lifecycle management (curaation, conversion, quality check, pre analytic p processing, etc.) g, ) c. data transformation (data fusion n/mashup, analytics), d. capability infrastructure (softwaare tools, platform tools, hardware resources such as storage and networking), g g),, and, e. data usage (processed results in text, table, visual, and other formats). f. all architecture components informed by Goals and use case description g. Security & Privacy has direct map 2. Aggregate all specific requirements into h high level generalized requirements which ar vendor neutral and technology agnostic. 437 requirements were extracted from 51 U Use Cases 35 aggregated general requirements into 7 7 categories

12 equirements and Use Case S Subgroup

13 Scope (M M0018) Definitions and Taxonomies Nancy Grady, SAIC Natasha Balac, SDSC g, Eugene Luster, R2AD The focus iss to gain a better understanding of the principles of Big Data. I It is important to develop a consensus based common language a and vocabulary terms used in Big Data across stakeholdeers from industry, academia, and government. In ers from industry academia and government In addition, it is also critical to identify essential actors with roles and respon nsibility, and subdivide them into components and sub componentts on how they interact/ relate with each other according t di to their similarities and differences. h i i il i i d diff Tasks F For Defin D fi itions: Compile terms used from all stakeholders ii C il d f ll k h ld g the meaning of Big Data from various standard bodies, regarding domain applications, and diversified operational environments. nitions and nomies Requirements BD BD WG For Taxon nomies Identify key actors with their roles and nomies: Identify key actors with their roles and responsib bilities from all stakeholders, categorize them into componeents and subcomponents based on their similarities and differencees Develop B Big Data Definitions and taxonomies documents Security and Privacy

14 efinitions and Taxonomies S Subgroup (M0024, M0142) y documents: M0024 ongoing discusssion, M0142 working draft Big Data Definitions, v1 (Developed from JJan , 2013 NIST Cloud/BigData Workshop) Big Data refers to digital data volum Bi D f di i l d l me, velocity l i and/or variety d/ i that: h enable novel approaches to f frontier questions previously inaccessible or impractical using current or conventional methods; and/or exceed the storage capacity y or analysis capability of current or conventional methods an l h d nd systems. d differentiates by storing and d analyzing population data and not sample sizes not sample sizes.

15 efinitions and Taxonomies S Subgroup (M0024, M0142) ta Science is the extraction of actionablle owledge directly from data through a prrocess discovery hypothesis and analytical discovery, hypothesis, and analytical pothesis analysis. ta Scientists is a practitioner who has ficient knowledge of the overlapping reegimes expertise in business needs domain expertise in business needs, domain owledge, analytical skills and programm ming p pertise to manage the end to end scient g tific f thod process through each stage in thee big ta lifecycle (through action) to deliver va alue.

16 efinitions and Taxonomies S Subgroup (M0024, M0142) Big Data Taxonomies (M0202) Actors S t R l System Roles Sensors Data Provider makes available data internal and/or external to the system Applications Data Consumer uses the output of th system Software agents Individuals System Orchestrator governance, governance requirements, monitoring Organizations Big Data Application Provider instantiates application Hardware resources Big Data Framework Provider provid resources Service abstractions

17 efinitions and Taxonomies S Subgroup (M0024, M0142) Big Data Taxonomies (M0202)

18 efinitions and Taxonomies S Subgroup (M0024, M0142) Big Data Taxonomies (M0202)

19 Scope (M M0021) Reference Architecture Orit Levin, Microsoft James Ketner, AT&T p, g Don Krapohl, Augmented Intelligence The focus iss to form a community of interest from industry, academia, and government, with the goal of developing a consensus based approach to orchestrate vendor neutral, technologyy and infrastructure agnostic for analytics tools and computing ti environments. i t Th he goal is to enable Big Data l i t bl Bi D t stakeholdeers to pick and choose technology agnostic analytics tools for proocessing and visualization in any computing platform and clusterr while allowing value added from Big Data service providers and the flow of the data between the stakeholders in cohesive an nd secure manner. Tasks Gather an nd study available Big Data architectures representing various sttakeholders, different data types, use cases, and document the architectures using the Big Data taxonomies model based upo on the identified actors with their roles and responsib bilities. bilities nitions and nomies Requirements BD BD WG Security and Privacy Ensure th hat the developed Big Data reference architecture and the Security a and Privacy Reference Architecture correspond and complem p ent each other.

20 eference Architecture Subgrroup y documents: M0100, M0151 white p paper, M0123 working draft Data Processing Flow D t P i Fl M0039 Data Transfor D t T f rmation Flow ti Fl M IT Stackk IT St M0047

21 eference Architecture Subgrroup d Bi D A hi ndors Big Data Architectures

22 eference Architecture Subgrroup hat the Baseline Big Data RA IS superset of a traditional data system representation of a vendor neutral and echnology agnostic system h l ti t functional architecture comprised of ogical roles pplicable to a variety of business models l bl fb d l o Tightly integrated enterprise systems o Loosely coupled vertical industries IS NOT A business architecture representing internal vs. external functional boundaries A deployment architecture A detailed IT RA of a specific p system implementation All of the above will be developed in the next stage in the context of g f specific use cases.

23 eference Architecture Subgrroup Diagram

24 Scope (M M0019) Security and Privacy Arnab Roy, CSA/Fujitsu Nancy Landreville, U. MD, Akhil Manchanda, GE The focus iis to form a community of interest from industry, academia, and government, with the goal of developing a consensus secure reference architecture to handle security and privacy p cy issues across all stakeholders. y This includes gaining an n understanding of what standards are available or under deveelopment, as well as identifies which key organizatioons are working on these standards. Tasks Gather i nput from all stakeholders regarding security and p privacy c y concerns in Big Data processing, storage, and g p g g services. Analyze/prioritize a list of challenging security and requirements that may delay or prevent adoption privacy r of Big D f ata deployment d l nitions and nomies Requirements BD BD WG Security and Privacy p a Security and Privacy Reference Architecture Develop that sup pplements the general Big Data Reference A hit cture Architec t

25 ecurity and Privacy Subgrou up y documents: Google Doc ongoing d discussion, M0110 requirements rking draft, M0xxx architecture & taxonomies Requirements Scope Infrastructure Security Data Privacy Data Management Integrity and Reactive Security Requirements Use Cases Studied Retail (consumer) Healthcare Media (social media and communications) Government (military, justice systems, etc.) Marketing Architecture & Taxonomies. 2.. Privacy Provenance System Health

26 Scope (M M0022) Th f is to form a community of interest from industry, t f it f i t t f i d t The focus is academia, and government, with the goal of developing a consensus vision with recommendations on how Big Data shoul move forwa ard by performing a good gap analysis through the materials gggathered from all other NBD subgroups. f g p This includes setting stan ndardization and adoption priorities through an Carl Buffington, USDA/Vistronix understandding of what standards are available or under development as part of the recommendations. Dan McClary, Oracle Technology Roadmap y, David Boyd, Data Tactic Tasks Gather in nput from NBD subgroups and study the taxonom mies for the actors roles and responsibility, use case and requ uirements, and secure reference architecture. Gain und derstanding of what standards are available or unde developm ment for Big Data Perform a thorough gap analysis and document the findings g g p y g Identify w what possible barriers may delay or prevent adoption n of Big Data nitions and nomies Requirements BD BD WG Security and Privacy Documeent vision and recommendations

27 chnology Roadmap Subgro oup y documents: M0087 working draft nputs from other subgroups Definitions and Taxonomies Requirements and Use Cases Security and Privacy Reference Architecture Potential Standards Group with Big Daata related activities (M0035) Capabilities and Technology Readinesss Big Data Decision Framework Big Data Mapping and Gap Analysis Big Data Strategies Adoption Implementation Reso rcing Resourcing

28 ubgroups Working Draft Outtline ntact: ebsite: n NBD PWG: NBD PWG htt //bi d t i t ov/newuser.php / h ocuments: Working Drafts g Big Data Definitions and Taxonomiess (M0142) Big Data Requirements (M0245) Bi D S Big Data Security and Privacy Requir i d P i R irements (M0110) (M ) Big Data Architectures White Paper S Survey M0151) Big Data Reference Architectures (M0226) Big Data Security and Privacy Reference Architectures (M0110) Big Data Technology Roadmap (M00 087) ST Big Data Workshop Slides: k h ld h bigdatawg.nist.gov/workshop.php d k h h

29

30 t Steps/ ure vities

31 O/IEC JTC 1 Big Data Study Group T d R f Terms and References 1. Survey the existing ICT landscape for key technologgies and relevant standards/models/studies /use cases and scenarios for Big Data from JTC 1, ISO, IEC and d other standards setting organizations, 2. Identify key terms and definitions commonly used in fy y f y n the area of Big Data, f g, 3. Assess the current status of Big Data standardizatioon market requirements, identify standards gaps, and propose standardization priorities to serve as a basiis for future JTC 1 work, and 4. Provide a report with recommendations and other p potential deliverables to the 2014 JTC 1 Plenary. Focus Areas: o Big Data Architecture/Infrastructure o Big Data Security & Privacy o Big Data Analytics o Big Data Applications & Tools o Big Data Management Meetings: three face to face meetings + telecon o January January US o April Europe o July Asia Format: 4 days (2 days workshop short papers, and 2 days meeting) Workshop papers will go into NIST Special Publication k h ll l bl High quality papers may go into ACM Conference Proceeeding (in process) Report due in September, 2014