BigData Management and Analytics in the Cloud** In the Contenxt of the Materials Genome Initiative

Transcription

1 BigData Management and Analytics in the Cloud** In the Contenxt of the Materials Genome Initiative Mladen A. Vouk* North Carolina State University, Raleigh, NC (*) Professor and Head of Computer Science and Associate Vice-Provost for Information Technology (**) This work was supported in part by the DOE SciDAC grant DE-FC02-07-ER25484, by the U.S. Army Research Office (ARO) under grant W911NF managed by NCSU Secure Open Systems Initiative, NSF grant # , and by the IBM Shared University Research Program MGI_Cloud/15May2012/v6c 1

2 BigData Size -From gigabytes to exaabytes depends on the domain sometimes equates to important or key data. Content - Tends to include raw data vast amounts of it, added to that is meta-data: processed, derived data (usually reduced in size compared to raw data), software, lab books, data about data loss of information? Structured but mostly unstructured. Data Representation in MGI appears to be an open issue as is Interoperability. Similarly open is metadata is/should be collected and maintained? What standards are needed? Classification, ontology, taxonomy, etc. Data Management. Security, domain specificity, export/import controls, data location, tools, automation of workflows, contribution modes (e.g., GenBank), sustainability. Computations to data or data to Computations? Data Quality and Data Usability. MGI specific Analytics General and MGI specific 2

3 HPD/HPC/GP Cloud Subsystems Import, Export Exchange Inter-op, Portability System Analytics (Data, Process, W/F, System) Bare-Metal & Virtualized Resources & Services Provenance Meta-Data (all levels) HPC/HPD Workflow Services Domain Services & Other Attributes Reliability & Fault Tolerance, QA, Trust, Client (End-User) Portal Network API Authentication Authorization Accounting Privacy, Security, Licenses, Intlect. Prop. Clouds of Clouds Cloud Broker Help Desk, Training, Education Data,, Data, Data Where is my Data Who is using it? What about IP? MGI_Cloud/15May2012/v6c 3

4 Provenance Provenance is about meta-data (data about data). For example, it is the history (lineage) of data, resource, or object ownership and access to it, and any transformations applied to the data/object/resource. t covers such information as code execution history and conditions under which it was executed, and so on. Provenance information may be found in a variety of place (including logs, audit-trails, etc.). There are at least the following (non-exclusive) categories Data provenance focuses on data flows, data history, inputs, outputs, and data transformations that occur in a cloud. Process provenance focuses on control and event flows (sequence diagrams) occurring during cloud operation and/or workflow execution Cloud and Workflow Evolution is about different versions and implementations of the cloud, and workflows running in the cloud, i.e. about evolution of such structures and forms. System Information is data about specific run-time system environments in which a computational workflow executes MGI_Cloud/15May2012/v6c 4

5 System Services (Data and Computational) Cloud enables end-users Hardware as a Service (HaaS) On demand access to a explicit (specific) computational, storage and networking product and/or equipment configuration possibly at a particular site (Location as a Service - LaaS) Infrastructure as a service (IaaS) On demand access to user specified hardware, interconnects, and storage capabilities, performance and services which may run on a variety of hardware products - NIST Platform as a Service (PaaS) - On-demand access to user specified combination hypervisors, operating system, and middleware that enables user required applications and services that are running on either Haas and/or IaaS - NIST Application as a Service (AaaS) - On-demand access to user specified application(s) Software as a Service (SaaS) - may encompass anything from PaaS through AaaS - NIST Facilities as a service (FaaS), Cloud as a Service, Security as a Service, Portability-as-a -Service, Storage & Location as a Service, e.g., where are my data stored? I wish my data to be stored within 100 miles of X needs HaaS control MGI_Cloud/15May2012/v6c 5

6 Example of an Academic and Research Cloud VCL Home Page Home Page Open Source (Apache) In production since 2004 Offers HPC, HPD, and GP services IBM Blade Servers 40,000+ users 250,000+ service reservations/year plus over 14 million HPC CPU hours, over 500,000 GB of storage (~500TB) Windows & Linux platforms About 1,800 service images (except for about 20 or so, these are user generated and maintained services). CCT/L1/V1/Jan10 MGI_Cloud/15May2012/v6c 6

7 VCL User Services Non-Persistent and Persistent Actual sole-use bare-metal, or virtual images HaaS, IaaS, PaaS, AaaS, [SaaS, CaaS] Undifferentiated Resources Single Seat (VCL-Desktop) Multiple Synced Seats (VCL-Class) Servers (VCL-Server) Aggregates (VCL-Cloudlets) HPC Clusters (VCL-HPC) Differentiated Resources VCL Agent H EC2 Labs, Other clouds, IBM e.g., System Z (mainframes) Storage (e.g., corporate, ) Supercomputers MGI_Cloud/15May2012/v6c 7

8 VCL Top Level Architecture VCL Manager & Scheduler Internet Authentication Service Provenance Node Manager #1 Provenance Storage Image Repository Node Manager #2 Provenance VCL Database Node Manager #n Provenance Image Repository Image Repository University Labs Supercomp. Mainframe Provenance Storage Virtual or Real Differentiated Resources Appliances MGI_Cloud/15May2012/v6c Virtual or Real Undifferentiated Resources, and Virtual System Pools 8

9 Databases Streams Sensors Big Data Analytics sub-cloud In Beta-Testing With pharma VCL Analytics Images Hadoop-based processing College of Management, CIMS P. Mugge, M. Kowalenko igold Dashboard Web Clouds VCL Crawler Images VCL Data Banks IBM Content Analytics front end IBM BigInsights back end Natural Language processing MGI_Cloud/15May2012/v6c 9

10 Analytics Cloud under the hood Conceptdriven Analytics W/F Generation Wizard Run-time Manager and Scheduler W/F Engine Workflow control plane Synchronous & Asynchronous Services Knowledge creation & Integration, Social Networking, Provenance, Tracking & Meta-Data (DBs and Portals) Execution: Plane - Heavy duty in-cloud computations, flows, services Supercomputers Analytics Enabled Resources and Images Clusters Active Storage v-facilities (testing, labs, etc.) Other cloud devices and resources MGI_Cloud/15May2012/v6c 10

11 A Possible Cloud of Clouds MGI-Cloud Architecture MGI- Classified MGI- Industry MGI-Gov Labs MGI- Public MGI- Academic -Private MGI- Academic - Private MGI v-facilities MGI_Cloud/15May2012/v6c 11

12 v-facilities, e.g. NCSU s v-btec Reactor Lab v-reactor Lab Remote 3D with haptic feedback Local Next Generation Virtual Computing Lab (VCL) Services MGI_Cloud/15May2012/v6c Remote students Ms. BTEC-Watson Advanced computing, networking, storage, & visualization support 12

13 Smart v-facilties (e.g., v-btec) ngapplications ngclouds (e.g., VCL O/S) ngnetworks NCSU Path Physical Partners and v-partners Services and Quality (Analytics driven) Ad-hoc/Initial Reactive Repeatable Defined Managed Pro-active Optimized MGI_Cloud/15May2012/v6c 13

14 More information Henry E. Schaffer, Samuel F. Averitt, Marc I. Hoit, Aaron Peeler, Eric D. Sills, and Mladen A. Vouk, NCSUs Virtual Computing Lab: A Cloud Computing Solution," IEEE Computer, pp , July Mladen Vouk, Andy Rindos, Sam Averitt, John Bass, Michael Bugaev, Aaron Peeler, Henry Schaffer, Eric Sills, Sarah Stein, Josh Thompson, Matthew P. Valenzisi, Using VCL Technology to Implement Distributed Reconfigurable Data Centers and Computational Services for Educational Institutions, IBM Journal of Research and Development, Vol. 53, No. 4, pp. 2:1-18, 2009 ( Mladen Vouk, Eric Sills and Patrick Dreher, Integration of High-Performance Computing into Cloud Computing Services, in Handbook of Cloud Computing, Editors Borko Furht, Armando Escalante, Springer New York, Dordrecht, Heidelberg, London, pp , 2010 MGI_Cloud/15May2012/v6c 14