An Overview of Big Data Technology and Security Implications

Size: px
Start display at page:

Download "An Overview of Big Data Technology and Security Implications"

Transcription

1 An Overview of Big Data Technology and Security Implications Peter Mell Senior Computer Scientist NIST Information Technology Laboratory

2 Disclaimer The ideas herein represent the author s notional views on big data technology and do not necessarily represent the official opinion of NIST. Any mention of commercial and not- for- profit entities, products, and technology is for informational purposes only; it does not imply recommendation or endorsement by NIST or usability for any specific purpose. NIST Information Technology Laboratory

3 Presentation Outline Section 1: Introduction and Definitions Section 2: Big Data Taxonomies Section 3: Security Implications and Areas of Research Section 4: MapReduce and Hadoop Section 5: Notable Implementations Appendix A: Seminal Research Results Appendix B: Overview of Big Data Framework Types NIST Information Technology Laboratory

4 Section 1: Introduction and Definitions NIST Information Technology Laboratory

5 Big Data the Data Deluge The world is creating ever more data (and it s a mainstream problem) Mankind created data 150 exabytes in 2005 (exabyte is a billion gigabytes) 1200 exabytes in exabytes in 2020 (expected by IBM) Examples: U.S. drone aircraft sent back 24 years worth of video footage in 2009 Large Hadron Collider generates 40 terabytes/second Bin Laden s death: 5106 tweets/second Around 30 billion RFID tags produced/year Oil drilling platforms have 20k to 40k sensors Our world has 1 billion transistors/human Credit: The data deluge, Economist; Understanding Big Data, Eaton et al. 5

6 A Quick Primer on Data Sizes 6

7 Predictions of the Industrial Revolution of Data Tim O Reilly Data is the new raw material of business Economist Challenges to achieving the revolution It is not possible to store all the data we produce 95% of created information was unstructured in 2010 Key observation Relational database management systems (RDBMS) will be challenged to scale up or out to meet the demand Credit: Data data everywhere, Economist; Extracting Value from Chaos, Gantz et al. 7

8 Industry Views on Big Data O Reilly Radar definition: Big data is when the size of the data itself becomes part of the problem EMC/IDC definition of big data: Big data technologies describe a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high- velocity capture, discovery, and/or analysis. IBM says that three characteristics define big data: Volume (Terabytes - > Zettabytes) Variety (Structured - > Semi- structured - > Unstructured) Velocity (Batch - > Streaming Data) Microsoft researchers use the same tuple Credit: Big Data Now, Current Perspectives from O Reilly Radar (O Reilly definition); Extracting Value from Chaos, Gantz et al. (IDC definition); Understanding Big Data, Eaton et al. (IBM definition) ; The World According to LINQ, Meijer (Microsoft research) 8

9 Notional Definition for Big Data Big Data Big data is where the data volume, acquisition velocity, or data representation limits the ability to perform effective analysis using traditional relational approaches or requires the use of significant horizontal scaling for efficient processing. Big Data Big Data Science Big Data Framework Big Data Infrastructure 9

10 More Notional Definitions Big Data Science Big data science is the study of techniques covering the acquisition, conditioning, and evaluation of big data. These techniques are a synthesis of both information technology and mathematical approaches. Big Data Frameworks Big data frameworks are software libraries along with their associated algorithms that enable distributed processing and analysis of big data problems across clusters of compute units (e.g., servers, CPUs, or GPUs). Big Data Infrastructure Big data infrastructure is an instantiation of one or more big data frameworks that includes management interfaces, actual servers (physical or virtual), storage facilities, networking, and possibly back- up systems. Big data infrastructure can be instantiated to solve specific big data problems or to serve as a general purpose analysis and processing engine. 10

11 Big Data Frameworks are often associated with the term NoSQL Structured Storage NoSQL Origins First used in 1998 to mean No to SQL Reused in 2009 when it came to mean Not Only SQL Groups non- relational approaches under a single term The power of SQL is not needed in all problems Specialized solutions may be faster or more scalable NoSQL generally has less querying power than SQL Common reasons to use NoSQL Ability to handle semi- structured and unstructured data Horizontal scalability NoSQL may complement RDBMS (but sometimes replaces) RDBMS may hold smaller amounts of high- value structured data NoSQL may hold vast amounts of less valued and less structured data Credit: NoSQL Databases, Strauch; Understanding Big Data, Eaton et al. RDBMS NoSQL 11

12 Common Tradeoffs Between Relational and NoSQL Approaches Relational implementations provide ACID guarantees Atomicity: transaction treated an all or nothing operation Consistency: database values correct before and after Isolation: events within transaction hidden from others Durability: results will survive subsequent malfunction NoSQL often provides BASE Basically available: Allowance for parts of a system to fail Soft state: An object may have multiple simultaneous values Eventually consistent: Consistency achieved over time CAP Theorem It is impossible to have consistency, availability, and partition tolerance in a distributed system the actual theorem is more complicated (see CAP slide in appendix A) Credit: Principles of Transaction- Oriented Database Recovery, Haerder and Reuter, 1983; Base: An ACID Alternative, Pritchett, 2008; Brewer s Conjecture and the Feasibility of Consistent, Available, Partition- Tolerant Web Services, Gilbert and Lynch 12

13 CAP Theorem with ACID and BASE Visualized ACID with eventual availability Partition Tolerance BASE with eventual consistency Consistency Availability Small data sets can be both consistent and available 13

14 Section 2: Big Data Taxonomies NIST Information Technology Laboratory

15 Big Data Characteristics and Derivation of a Notional Taxonomy Volume Velocity Variety (semi- structured or unstructured) Requires Horizontal Scalability Relational Limitation Big Data No No No No No No No No Yes No Yes Yes, Type 1 No Yes No Yes Maybe Yes, Type 2 No Yes Yes Yes Yes Yes, Type 3 Yes No No Yes Maybe Yes, Type 2 Yes No Yes Yes Yes Yes, Type3 Yes Yes No Yes Maybe Yes, Type 2 Yes Yes Yes Yes Yes Yes, Type 3 Types of Big Data: Type 1: This is where a non- relational data representation required for effective analysis. Type 2: This is where horizontal scalability is required for efficient processing. Type 3: This is where a non- relational data representation processed with a horizontally scalable solution is required for both effective analysis and efficient processing. In other words, the data representation is not conducive to a relational algebraic analysis. 15

16 NoSQL Taxonomies Remember that big data frameworks and NoSQL are related but not necessarily the same some big data problems may be solved relationally Scofield: Key/value, column, document, graph Cattel: Key/value, extensible record (e.g., column), document Strauch: Key/value, column, document (mentions graph separately) Others exist with very different categories Consensus taxonomy for NoSQL: Key/value, column, document, graph Notional big data framework taxonomy: Key/value, column, document, graph, sharded RDBMSs Credit: NoSQL Databases, Strauch; NoSQL Death to Relational Databases(?), Scofield; Scalable SQL and NoSQL Data Stores, Cattel 16

17 Notional Big Data Framework Taxonomy Conceptual Structures: Key Value Stores Schema- less system Key Value Column- oriented databases Storage by column, not row Name Height Eye Color Bob 6 2 Brown Nancy 5 3 Hazel Graph Databases Uses nodes and edges to represent data Often used for Semantic Web Data Relationships Data Document Oriented Database Stores documents that are semi- structured Includes XML databases Key Key Structured Document Structured Document Sharded RDBMS RDBMS RDBMS RDBMS 17

18 Comparison of NoSQL and Relational Approaches Performance Horizontal Scalability Flexibility in Data Variety Complexity of Operation Functionality Key-Value stores high high high none variable (none) Column stores Document stores Graph databases high high moderate low minimal high variable (high) high low variable (low) variable variable high high graph theory Relational databases variable variable low moderate relational algebra Matches columns on the big data taxonomy Credit: NoSQL Death to Relational Databases(?), Scofield (column headings modified from original data for clarity) 18

19 Notional Suitability of Big Data Frameworks for types of Big Data Problems Horizontal Scalability Flexibility in Data Variety Appropriate Big Data Types Key-Value stores high high 1, 2, 3 Column stores high moderate 1 (partially), 2, 3 (partially) Document stores variable (high) high 1, 2 (likely), 3 (likely) Graph databases variable high 1, 2 (maybe), 3 (maybe) Sharded Database variable (high) low 2 (likely) 19

20 Section 3: Security Implications and Areas of Research NIST Information Technology Laboratory

21 Hypothesis: Big Data approaches will open up new avenues of IT security metrology Revolutions in science have often been preceded by revolutions in measurement, - Sinan Aral, New York University Arthur Coviello, Chairman RSA Security must adopt a big data view The age of big data has arrived in security management. We must collect data throughout the enterprise, not just logs We must provide context and perform real time analysis There is precious little information on how to do this Credit: Data, data everywhere, Economist; talks- about- building- a- trusted- cloud- resilient- security 21

22 Big Data is Moving into IT Security Products Several years ago, some security companies had an epiphany: Traditional relational implementations were not always keeping up with data demands A changed industry: Some were able to stick with traditional relational approaches Some partitioned their data and used multiple relational silos Some quietly switched over to NoSQL approaches Some adopted a hybrid approach, putting high value data in a relational store and lower value data in NoSQL stores Credit: This is based on my discussions with IT security companies in 12/2011 at the Government Technology Research Alliance Security Council 2011

23 Security Features are Slowly Moving into Big Data Implementations Many big data systems were not designed with security in mind Tim Mather, KPMG There are far more security controls for relational systems than for NoSQL systems SQL security: secure configuration management, multifactor authentication, data classification, data encryption, consolidated auditing/reporting, database firewalls, vulnerability assessment scanners NoSQL security: cell- level access labels, kerberos- based authentication, access control lists for tables/column families Credit: Securing Big Data, Cloud Security Alliance Congress 2011, Tim Mather KPMG 23

24 Public Government Big Data Security Research Exists Accumulo Accumulo is a distributed key/value store that provides expressive, cell- level access labels. Allows fine grained access control in a NoSQL implementation Based on Google BigTable 200,000 lines of Java code Submitted by the NSA to the Apache Foundation Credit: apps/

25 What further research needs to be conducted on big data security and privacy? Enhancing IT security metrology Enabling secure implementations Privacy concerns on use of big data technology 25

26 Research Area 1: The Computer Science of Big Data 1. What is a definition of big data? What computer science properties are we trying to instantiate? 2. What types of big data frameworks exist? Can we identify a taxonomy that relates them hierarchically? 3. What are the strengths, weaknesses, and appropriateness of big data frameworks for specific classes of problems? What are the mathematical foundations for big data frameworks? 4. How can we measure the consistency provided by a big data solution? 5. Can we define standard for querying big data solutions? With an understanding of the capabilities available and their suitability for types of problems, we can then apply this knowledge to computer security. 26

27 Research Area 2: Furthering IT Security Metrology through Big Data Technology 1. Determine how IT security metrology is limited by traditional data representations (i.e., highly structured relational storage) 2. Investigate how big data frameworks can benefit IT security measurement What new metrics could be available? 3. Identify specific security problems that can benefit from big data approaches Conduct experiments to test solving identified problems 4. Explore the use of big data frameworks within existing security products What new capabilities are available? How has this changed processing capacity? 27

28 Research Area 3: The Security of Big Data Infrastructure 1. Evaluate the security capabilities of big data infrastructure Do the available tools provide needed security features? What security models can be used when implementing big data infrastructure? 2. Identify techniques to enhance security in big data frameworks (e.g., data tagging approaches, shadoop) Conduct experiments on enhanced security framework implementations 28

29 Research Area 4: The Privacy of Big Data Implementations Big data technology enables massive data aggregation beyond what has been previously possible Inferencing concerns with non- sensitive data Legal foundations for privacy in data aggregation Application of NIST Special Publication privacy controls 29

30 Potential Future Milestones Area 1 (not security specific) Publication on harnessing big data technology Definitions, taxonomies, and appropriateness for classes of problems Area 2 (security specific) Publication on furthering IT security metrology through big data technology Research papers on solving specific security problems using big data approaches Area 3 (security specific) Publication on approaches for the secure use of big data platforms Area 4 (privacy specific) Not yet identified 30

31 Section 4: MapReduce and Hadoop NIST Information Technology Laboratory

32 MapReduce Dean, et al. Seminal paper published by Google in 2004 Simple concurrent programming model and associated implementation Model handles the parallel processing and message passing details Simplified coding model compared to general purpose parallel languages (e.g., MPI) Three functions: Map - > Parallel sort - > Reduce Map: Processes a set of key/value pairs to produce an intermediate set of key/value pairs Parallel sort: a distributed sort on intermediate results feeds the reduce nodes Reduce: for each resultant key, it processes each key/value pair and produces the result set of values for each key Approachable programming model Handles concurrency complexities for the user Limited functionality Appears to provide a sweet spot for solving a vast number of important problems with an easy to use programming model Credit: MapReduce: Simplified Data Processing on Large Clusters, Dean et al. 32

33 MapReduce Diagram from Google s 2004 Seminal Paper e 33

34 Storage, MapReduce, and Query (SMAQ) Stacks Query Efficient way of defining computation Platform for user friendly analytical systems Map Reduce Distributes computation over many servers Batch processing model Storage Distributed and non- relational Credit: 2011 O Reilly Radar, Edd Dumbill 34

35 Hadoop Widely used MapReduce framework The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model hadoop.apache.org Open source project with an ecosystem of products Core Hadoop: Hadoop MapReduce implementation Hadoop Distributed File System (HDFS) Non- core: Many related projects Credit: 35

36 Hadoop SMAQ Stack (select components) Query Pig (simply query language) Hive (SQL like queries) Cascading (workflows) Mahout (machine learning) Zookeeper (coordination service) Hama (scientific computation) Map Reduce Hadoop Map Reduce implementation Storage HBase (column oriented database) Hadoop Distributed File System (HDFS, core Hadoop file system) 36

37 Alternate MapReduce Frameworks BashReduce Disco Project Spark GraphLab Carnegie- Mellon Storm HPCC (LexisNexis) 37

38 Section 5: Notable Implementations (both frameworks and infrastructure) NIST Information Technology Laboratory

39 Google File System Ghemawat, et al. Design requirements: performance, scalability, reliability, and availability Design assumptions: Huge files Expected component failures File mutation is primarily by appending Relaxed consistency <- think of the CAP theorem here Master has all its data in memory (consistent and available!!) All reads and writes occur directly between client and chunkservers For writes, control flow is decoupled from pipelined data flow Credit: The Google File System, Ghemawat et al. 39

40 Google File System Architecture 40

41 Google File System Write Control and Data Flow Master assigned a primary replica Client pipelines data through replicas Client contacts primary to instantiate the write 41

42 Big Table Chang, Dean, Ghemawat, et al. Big table is a sparse, distributed, persistent multi- dimensional sorted map It is a non- relational key- value store / column- oriented database Row keys- table data is stored in row order Model supports atomic row manipulation Tablets- subsets of tables (a range of the rows) Unit of distribution/load balancing Column families group column keys Access control done on column families Each cell can contain multiple time stamped versions Uses Google File System (GFS) for file storage Distributed lock service- Chubby Implementation- Lightly loaded master plus tablet servers Internal Google tool Credit: Bigtable: A Distributed Storage System for Structured Data, Chang et. al. 42

43 Big Table Storage Paradigm Column family Column family Column key Row key Timestamp 43

44 Hbase Open source Apache project Modeled after the Big Table research paper Implemented on top of the Hadoop Distributed File System Use HBase when you need random, realtime read/ write access to your Big Data hosting of very large tables - - billions of rows X millions of columns - - atop clusters of commodity hardware. HBase is an open- source, distributed, versioned, column- oriented store modeled after Google's Bigtable Credit: 44

45 Dynamo DeCandia, Hastorun,, Vogels Werner Vogels, Amazon CTO 2007 paper describing Amazon s implementation that had been in production use for one year Dynamo is a key- value eventually consistent database designed for scalability and high availability Key lookup only (no relational schema or hierarchical namespace) No central point of failure (i.e., nodes are symmetric) Handles nodes of differing capability (i.e., heterogeneity) Offers incremental scalability (i.e., you can add one node at a time) Writes rarely rejected, reads may return multiple versions Strong focus on SLA performance guarantees (for 99.9% of operations) Credit: Dynamo: Amazon s Highly Available Key- value Store, DeCandia et al. 45

46 Dynamo Techniques and Advantages 46

47 Dynamo Configurability and Production Implementation the primary advantage of Dynamo is that it provides the necessary knobs to tune their instances Dynamo paper Users controls three variables: N, number of hosts on which to replicate data R, minimum number of hosts that must participate in a read low value could increase inconsistency W, minimum number of hosts that must participate in a write low value could decrease durability Amazon Dynamo infrastructure Dial- in the desired requests per second (elasticity) Data stored in solid state drives for low latency access Automatic replication across availability zones Zero administration burden or even features Developers choose between eventually consistent and strongly consistent reads 47

48 Apache Cassandra Described as a big table model running on an Amazon dynamo- like infrastructure Distributed database management system No single point of failure Designed for use with commodity servers Key- value with column indexing system Rows are keys that are randomly partitioned among servers Keys maps to multiple values Values from multiple keys may be grouped into column families Each key s values are stored together (like a row oriented RDBMS and yet the data has a column orientation) Cassandra Query Language (SQL like querying) Initially developed by Facebook and then open sourced Credit: 48

49 Questions and Comments Peter Mell NIST Senior Computer Scientist NIST Information Technology Laboratory

50 Appendix A: Seminal Research Results NIST Information Technology Laboratory

51 CAP Theorem Brewer, 2000 CAP: Consistency, Availability, Partition- tolerance It is impossible to have all three CAP properties in an asynchronous distributed read/write system Asynchronous nature is key (i.e., no clocks) Any two properties can be achieved Delayed- t consistency possible for partially synchronous systems (i.e., independent timers) All three properties can be guaranteed if the requests are separated by a defined period of lossless messaging. most real world systems are forced to settle with returning most of the data most of the time. Take away: Distributed read/write systems are limited by intersystem messaging and may have to relax their CAP requirements Credit: Brewer s Conjecture and the Feasibility of Consistent, Available, Partition- Tolerant Web Services, Gilbert and Lynch 51

52 BASE vs. ACID Pritchett 2008 CAP: Consistency, Availability, Partition Tolerance For distributed systems, we must have partitioning ACID (atomicity, consistency, isolation, durability) Focuses on consistency, not availability Properties are transparently provided by the database BASE (basically available, soft state, eventually consistent) Focuses on availability, not consistency Requires independent analysis of each application to achieve these properties (i.e., this is harder than ACID) Promotes availability by avoiding two phase commits over the network Application code may provide idempotence to avoid 2 phase commits Credit: BASE: An Acid Alternative, Pritchett 52

53 Origin of SQL Codd 1970 Invented by Edgar F. Codd, IBM Fellow A Relational Model of Data for Large Shared Data Banks published in 1970 Relational algebra provides a mathematical basis for database query languages (i.e. SQL) Based on first order logic [His] basic idea was that relationships between data items should be based on the item's values, and not on separately specified linking or nesting. This notion greatly simplified the specification of queries and allowed unprecedented flexibility to exploit existing data sets in new ways - Don Chamberlin, SQL co- inventor Credit: A Relational Model of Data for Large Shared Data Banks, Codd 53

54 The Notion of CoSQL Meijer, et al SQL and key- value NoSQL (called cosql) are mathematical duals based on category theory SQL and CoSQL can transmute into each other Duality between synchronous ACID and asynchronous BASE Transmute data to take advantage of either ACID or BASE monads and monad comprehensions provide a common query mechanism for both SQL and cosql Microsoft s Languge- Integrated Query (LINQ) implements this query abstraction There could be a single language, analogous to SQL, used for all cosql databases!! Credit: A co- Relational Model of Data for Large Shared Data Banks, Meijer 54

55 Notional Big Data Taxonomy w/ Mappings to Mathematical Systems Big Data Predicate Logic Big Data Science Big Data Platforms Codd Relational Algebra Meijer Category Theory NoSQL SQL ACID Key Value BASE Document Stores Graph Databases Column Oriented CAP Theorem Brewer Pritchett 55

56 Appendix B: Overview of Big Data Framework Types NIST Information Technology Laboratory

57 Overview Key Value Stores Key value stores are distributed associative arrays (collections of key/value pairs) Implementing maps or dictionaries Enables storage of schema- less data Simple operations: add(key,value), set(key,value), get(key), delete(key) Values may be complex and unstructured objects Indexed on keys for searching No joins, no SQL, no real queries Often a speed advantage in storage and retrieval Credit: Key- Value stores: a practical overview, Seeger 57

58 Example Uses- Key Value Stores Key value is one of the most prominent NoSQL approaches Used for a wide variety of big data problems Most popular type is MapReduce First used for indexing the Internet (Google) Example: Visa used Hadoop to process 36 terabytes of data Traditional approach: one month Key value approach: 13 minutes 58

59 Overview Column Oriented Databases Table data is stored by column NOT row Efficient for calculating metrics on a particular set of columns Efficient for updating all values in a single column Inefficient for row operations effecting multiple columns Inefficient for writing new rows Key concern is gaining efficiency in hard disk accesses for particular types of jobs Since column data is of the same type, some compression advantages can be achieved over row- oriented databases This concept is not new. It has been used since the 1970s Credit: C- Store: A Column- oriented DBMS, Stonebraker 59

60 Example Column Oriented Databases 2010 Car 0-30 mph (sec) Turning circle Horsepower Nissan Altima 2.5S ft. 175 Mini Cooper Clubman ft. 118 Smart ft. 71 Column oriented storage: Nissan Altima 2.5S, Mini Cooper Clubman, Smart 3.3, 3.8, ft., 37 ft., 30 ft. 175, 118, 72 Row oriented storage: Nissan Altima 2.5S, 3.3, 40 ft., 175 Mini Cooper Clubman, 3.8, 37 ft., 118 Smart, 5.1, 30 ft., 71 Credit: Consumer Reports New Car Preview

61 Overview - Graph Databases Implement the Associative Model of Data By definition: index free adjacency Not record based and no global index Nodes and edges Graphs, Directed Graphs, Multi- graphs, Hyper- graphs Compared to RDBMS, graph database: Are slower for applying operations to large datasets Have no join operations Excellent for application of graph algorithms Can be faster for associative data sets Map well to object oriented structures Can evolve to changing data types (there is no rigid schema) Useful for semantic web implementations (RDF) Credit: 61

62 Example Use- Graph Databases for Semantic Web The semantic web gives meaning to data by defining relationships and properties Entities and relationships are described using RDF Subject - > Predicate - > Object Ontologies are defined using RDFS/OWL Enables inferences Full use of OWL can produce NP- complete computations RDF can be naturally represented using a labeled, directed, multi- graph SPARQL enables RDF querying Relational databases can store RDF triples easily but efficient SPARQL- >SQL querying is difficult Native graph databases may provide enhanced performance 62

63 Overview - Document Oriented Database Focused on efficient management of semi- structured data Focus on self- describing structures (e.g., YAML, JSON, PDF, MS Office, or XML) Documents are like records in a RDBMS Each document may use a different schema may populate different fields (there are no empty fields) Document may be accessed through Unique keys Metadata/tagging Collections of documents / Directory structures Query languages (varying by implementation) Credit: NoSQL and Document Oriented Databases, Morin 63

64 Example Use - Document Oriented Databases for XML Native XML vs. XML- enabled Rationale: If XML is used for communication, why not store the data natively in XML? Definition: Model based on XML document structure, not the data Each record is an XML document (analogous to a row in an RDBMS) There are no requirements on the storage technique (e.g., relational vs. non- relational) Many provide collections of documents that may be arranged hierarchically like a file system Querying Languages: Xpath, XQuery (extends Xpath) Transformations for output may use XSLT Hybrid databases have been developed that support querying with SQL and XQuery Credit: XML Database Products, Bourret; XML and Databases, Bourret 64

Tackling Big Data. Michael Cooper & Peter Mell NIST Information Technology Laboratory Computer Security Division

Tackling Big Data. Michael Cooper & Peter Mell NIST Information Technology Laboratory Computer Security Division Tackling Big Data Michael Cooper & Peter Mell NIST Information Technology Laboratory Computer Security Division IT Laboratory Big Data Working Group What exactly is Big Data? What are the issues associated

More information

An Overview of Cloud & Big Data Technology and Security Implications. Tim Grance Senior Computer Scientist NIST Information Technology Laboratory

An Overview of Cloud & Big Data Technology and Security Implications. Tim Grance Senior Computer Scientist NIST Information Technology Laboratory An Overview of Cloud & Big Data Technology and Security Implications Tim Grance Senior Computer Scientist NIST Information Technology Laboratory NIST: Mission To promote U.S. innovation and industrial

More information

BIG DATA IN THE CLOUD : CHALLENGES AND OPPORTUNITIES MARY- JANE SULE & PROF. MAOZHEN LI BRUNEL UNIVERSITY, LONDON

BIG DATA IN THE CLOUD : CHALLENGES AND OPPORTUNITIES MARY- JANE SULE & PROF. MAOZHEN LI BRUNEL UNIVERSITY, LONDON BIG DATA IN THE CLOUD : CHALLENGES AND OPPORTUNITIES MARY- JANE SULE & PROF. MAOZHEN LI BRUNEL UNIVERSITY, LONDON Overview * Introduction * Multiple faces of Big Data * Challenges of Big Data * Cloud Computing

More information

Challenges for Data Driven Systems

Challenges for Data Driven Systems Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Quick History of Data Management 4000 B C Manual recording From tablets to papyrus to paper A. Payberah 2014 2

More information

So What s the Big Deal?

So What s the Big Deal? So What s the Big Deal? Presentation Agenda Introduction What is Big Data? So What is the Big Deal? Big Data Technologies Identifying Big Data Opportunities Conducting a Big Data Proof of Concept Big Data

More information

Introduction to NOSQL

Introduction to NOSQL Introduction to NOSQL Université Paris-Est Marne la Vallée, LIGM UMR CNRS 8049, France January 31, 2014 Motivations NOSQL stands for Not Only SQL Motivations Exponential growth of data set size (161Eo

More information

Big Data Buzzwords From A to Z. By Rick Whiting, CRN 4:00 PM ET Wed. Nov. 28, 2012

Big Data Buzzwords From A to Z. By Rick Whiting, CRN 4:00 PM ET Wed. Nov. 28, 2012 Big Data Buzzwords From A to Z By Rick Whiting, CRN 4:00 PM ET Wed. Nov. 28, 2012 Big Data Buzzwords Big data is one of the, well, biggest trends in IT today, and it has spawned a whole new generation

More information

A survey of big data architectures for handling massive data

A survey of big data architectures for handling massive data CSIT 6910 Independent Project A survey of big data architectures for handling massive data Jordy Domingos - jordydomingos@gmail.com Supervisor : Dr David Rossiter Content Table 1 - Introduction a - Context

More information

Advanced Data Management Technologies

Advanced Data Management Technologies ADMT 2014/15 Unit 15 J. Gamper 1/44 Advanced Data Management Technologies Unit 15 Introduction to NoSQL J. Gamper Free University of Bozen-Bolzano Faculty of Computer Science IDSE ADMT 2014/15 Unit 15

More information

Transforming the Telecoms Business using Big Data and Analytics

Transforming the Telecoms Business using Big Data and Analytics Transforming the Telecoms Business using Big Data and Analytics Event: ICT Forum for HR Professionals Venue: Meikles Hotel, Harare, Zimbabwe Date: 19 th 21 st August 2015 AFRALTI 1 Objectives Describe

More information

Study and Comparison of Elastic Cloud Databases : Myth or Reality?

Study and Comparison of Elastic Cloud Databases : Myth or Reality? Université Catholique de Louvain Ecole Polytechnique de Louvain Computer Engineering Department Study and Comparison of Elastic Cloud Databases : Myth or Reality? Promoters: Peter Van Roy Sabri Skhiri

More information

Application Development. A Paradigm Shift

Application Development. A Paradigm Shift Application Development for the Cloud: A Paradigm Shift Ramesh Rangachar Intelsat t 2012 by Intelsat. t Published by The Aerospace Corporation with permission. New 2007 Template - 1 Motivation for the

More information

Distributed Data Stores

Distributed Data Stores Distributed Data Stores 1 Distributed Persistent State MapReduce addresses distributed processing of aggregation-based queries Persistent state across a large number of machines? Distributed DBMS High

More information

NoSQL Databases. Nikos Parlavantzas

NoSQL Databases. Nikos Parlavantzas !!!! NoSQL Databases Nikos Parlavantzas Lecture overview 2 Objective! Present the main concepts necessary for understanding NoSQL databases! Provide an overview of current NoSQL technologies Outline 3!

More information

Managing Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database

Managing Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Managing Big Data with Hadoop & Vertica A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Copyright Vertica Systems, Inc. October 2009 Cloudera and Vertica

More information

Big Data, Cloud Computing, Spatial Databases Steven Hagan Vice President Server Technologies

Big Data, Cloud Computing, Spatial Databases Steven Hagan Vice President Server Technologies Big Data, Cloud Computing, Spatial Databases Steven Hagan Vice President Server Technologies Big Data: Global Digital Data Growth Growing leaps and bounds by 40+% Year over Year! 2009 =.8 Zetabytes =.08

More information

A Brief Outline on Bigdata Hadoop

A Brief Outline on Bigdata Hadoop A Brief Outline on Bigdata Hadoop Twinkle Gupta 1, Shruti Dixit 2 RGPV, Department of Computer Science and Engineering, Acropolis Institute of Technology and Research, Indore, India Abstract- Bigdata is

More information

Chukwa, Hadoop subproject, 37, 131 Cloud enabled big data, 4 Codd s 12 rules, 1 Column-oriented databases, 18, 52 Compression pattern, 83 84

Chukwa, Hadoop subproject, 37, 131 Cloud enabled big data, 4 Codd s 12 rules, 1 Column-oriented databases, 18, 52 Compression pattern, 83 84 Index A Amazon Web Services (AWS), 50, 58 Analytics engine, 21 22 Apache Kafka, 38, 131 Apache S4, 38, 131 Apache Sqoop, 37, 131 Appliance pattern, 104 105 Application architecture, big data analytics

More information

ON-LINE VIDEO ANALYTICS EMBRACING BIG DATA

ON-LINE VIDEO ANALYTICS EMBRACING BIG DATA ON-LINE VIDEO ANALYTICS EMBRACING BIG DATA David Vanderfeesten, Bell Labs Belgium ANNO 2012 YOUR DATA IS MONEY BIG MONEY! Your click stream, your activity stream, your electricity consumption, your call

More information

NoSQL for SQL Professionals William McKnight

NoSQL for SQL Professionals William McKnight NoSQL for SQL Professionals William McKnight Session Code BD03 About your Speaker, William McKnight President, McKnight Consulting Group Frequent keynote speaker and trainer internationally Consulted to

More information

Big Data Management in the Clouds. Alexandru Costan IRISA / INSA Rennes (KerData team)

Big Data Management in the Clouds. Alexandru Costan IRISA / INSA Rennes (KerData team) Big Data Management in the Clouds Alexandru Costan IRISA / INSA Rennes (KerData team) Cumulo NumBio 2015, Aussois, June 4, 2015 After this talk Realize the potential: Data vs. Big Data Understand why we

More information

Big Data Solutions. Portal Development with MongoDB and Liferay. Solutions

Big Data Solutions. Portal Development with MongoDB and Liferay. Solutions Big Data Solutions Portal Development with MongoDB and Liferay Solutions Introduction Companies have made huge investments in Business Intelligence and analytics to better understand their clients and

More information

Big Data Challenges in Bioinformatics

Big Data Challenges in Bioinformatics Big Data Challenges in Bioinformatics BARCELONA SUPERCOMPUTING CENTER COMPUTER SCIENCE DEPARTMENT Autonomic Systems and ebusiness Pla?orms Jordi Torres Jordi.Torres@bsc.es Talk outline! We talk about Petabyte?

More information

2.1.5 Storing your application s structured data in a cloud database

2.1.5 Storing your application s structured data in a cloud database 30 CHAPTER 2 Understanding cloud computing classifications Table 2.3 Basic terms and operations of Amazon S3 Terms Description Object Fundamental entity stored in S3. Each object can range in size from

More information

Why NoSQL? Your database options in the new non- relational world. 2015 IBM Cloudant 1

Why NoSQL? Your database options in the new non- relational world. 2015 IBM Cloudant 1 Why NoSQL? Your database options in the new non- relational world 2015 IBM Cloudant 1 Table of Contents New types of apps are generating new types of data... 3 A brief history on NoSQL... 3 NoSQL s roots

More information

Big Data Zurich, November 23. September 2011

Big Data Zurich, November 23. September 2011 Institute of Technology Management Big Data Projektskizze «Competence Center Automotive Intelligence» Zurich, November 11th 23. September 2011 Felix Wortmann Assistant Professor Technology Management,

More information

Big Data and Apache Hadoop s MapReduce

Big Data and Apache Hadoop s MapReduce Big Data and Apache Hadoop s MapReduce Michael Hahsler Computer Science and Engineering Southern Methodist University January 23, 2012 Michael Hahsler (SMU/CSE) Hadoop/MapReduce January 23, 2012 1 / 23

More information

Cloud Scale Distributed Data Storage. Jürmo Mehine

Cloud Scale Distributed Data Storage. Jürmo Mehine Cloud Scale Distributed Data Storage Jürmo Mehine 2014 Outline Background Relational model Database scaling Keys, values and aggregates The NoSQL landscape Non-relational data models Key-value Document-oriented

More information

Implement Hadoop jobs to extract business value from large and varied data sets

Implement Hadoop jobs to extract business value from large and varied data sets Hadoop Development for Big Data Solutions: Hands-On You Will Learn How To: Implement Hadoop jobs to extract business value from large and varied data sets Write, customize and deploy MapReduce jobs to

More information

Data Management in the Cloud -

Data Management in the Cloud - Data Management in the Cloud - current issues and research directions Patrick Valduriez Esther Pacitti DNAC Congress, Paris, nov. 2010 http://www.med-hoc-net-2010.org SOPHIA ANTIPOLIS - MÉDITERRANÉE Is

More information

Big Data Management and Security

Big Data Management and Security Big Data Management and Security Audit Concerns and Business Risks Tami Frankenfield Sr. Director, Analytics and Enterprise Data Mercury Insurance What is Big Data? Velocity + Volume + Variety = Value

More information

Big Data: Opportunities & Challenges, Myths & Truths 資 料 來 源 : 台 大 廖 世 偉 教 授 課 程 資 料

Big Data: Opportunities & Challenges, Myths & Truths 資 料 來 源 : 台 大 廖 世 偉 教 授 課 程 資 料 Big Data: Opportunities & Challenges, Myths & Truths 資 料 來 源 : 台 大 廖 世 偉 教 授 課 程 資 料 美 國 13 歲 學 生 用 Big Data 找 出 霸 淩 熱 點 Puri 架 設 網 站 Bullyvention, 藉 由 分 析 Twitter 上 找 出 提 到 跟 霸 凌 相 關 的 詞, 搭 配 地 理 位 置

More information

BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES

BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES Relational vs. Non-Relational Architecture Relational Non-Relational Rational Predictable Traditional Agile Flexible Modern 2 Agenda Big Data

More information

Hadoop Ecosystem Overview. CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook

Hadoop Ecosystem Overview. CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook Hadoop Ecosystem Overview CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook Agenda Introduce Hadoop projects to prepare you for your group work Intimate detail will be provided in future

More information

INDUS / AXIOMINE. Adopting Hadoop In the Enterprise Typical Enterprise Use Cases

INDUS / AXIOMINE. Adopting Hadoop In the Enterprise Typical Enterprise Use Cases INDUS / AXIOMINE Adopting Hadoop In the Enterprise Typical Enterprise Use Cases. Contents Executive Overview... 2 Introduction... 2 Traditional Data Processing Pipeline... 3 ETL is prevalent Large Scale

More information

Cloud Database Emergence

Cloud Database Emergence Abstract RDBMS technology is favorable in software based organizations for more than three decades. The corporate organizations had been transformed over the years with respect to adoption of information

More information

Big Data & QlikView. Democratizing Big Data Analytics. David Freriks Principal Solution Architect

Big Data & QlikView. Democratizing Big Data Analytics. David Freriks Principal Solution Architect Big Data & QlikView Democratizing Big Data Analytics David Freriks Principal Solution Architect TDWI Vancouver Agenda What really is Big Data? How do we separate hype from reality? How does that relate

More information

NoSQL in der Cloud Why? Andreas Hartmann

NoSQL in der Cloud Why? Andreas Hartmann NoSQL in der Cloud Why? Andreas Hartmann 17.04.2013 17.04.2013 2 NoSQL in der Cloud Why? Quelle: http://res.sys-con.com/story/mar12/2188748/cloudbigdata_0_0.jpg Why Cloud??? 17.04.2013 3 NoSQL in der Cloud

More information

The Future of Data Management

The Future of Data Management The Future of Data Management with Hadoop and the Enterprise Data Hub Amr Awadallah (@awadallah) Cofounder and CTO Cloudera Snapshot Founded 2008, by former employees of Employees Today ~ 800 World Class

More information

Apache Hadoop: The Big Data Refinery

Apache Hadoop: The Big Data Refinery Architecting the Future of Big Data Whitepaper Apache Hadoop: The Big Data Refinery Introduction Big data has become an extremely popular term, due to the well-documented explosion in the amount of data

More information

Understanding NoSQL on Microsoft Azure

Understanding NoSQL on Microsoft Azure David Chappell Understanding NoSQL on Microsoft Azure Sponsored by Microsoft Corporation Copyright 2014 Chappell & Associates Contents Data on Azure: The Big Picture... 3 Relational Technology: A Quick

More information

Oracle s Big Data solutions. Roger Wullschleger.

Oracle s Big Data solutions. Roger Wullschleger. <Insert Picture Here> s Big Data solutions Roger Wullschleger DBTA Workshop on Big Data, Cloud Data Management and NoSQL 10. October 2012, Stade de Suisse, Berne 1 The following is intended to outline

More information

BIG Big Data Public Private Forum

BIG Big Data Public Private Forum DATA STORAGE Martin Strohbach, AGT International (R&D) THE DATA VALUE CHAIN Value Chain Data Acquisition Data Analysis Data Curation Data Storage Data Usage Structured data Unstructured data Event processing

More information

Cloud & Big Data a perfect marriage? Patrick Valduriez

Cloud & Big Data a perfect marriage? Patrick Valduriez Cloud & Big Data a perfect marriage? Patrick Valduriez Cloud & Big Data: the hype! 2 Cloud & Big Data: the hype! 3 Behind the Hype? Every one who wants to make big money Intel, IBM, Microsoft, Oracle,

More information

Domain driven design, NoSQL and multi-model databases

Domain driven design, NoSQL and multi-model databases Domain driven design, NoSQL and multi-model databases Java Meetup New York, 10 November 2014 Max Neunhöffer www.arangodb.com Max Neunhöffer I am a mathematician Earlier life : Research in Computer Algebra

More information

Apache Hadoop FileSystem and its Usage in Facebook

Apache Hadoop FileSystem and its Usage in Facebook Apache Hadoop FileSystem and its Usage in Facebook Dhruba Borthakur Project Lead, Apache Hadoop Distributed File System dhruba@apache.org Presented at Indian Institute of Technology November, 2010 http://www.facebook.com/hadoopfs

More information

BIG DATA & ANALYTICS. Transforming the business and driving revenue through big data and analytics

BIG DATA & ANALYTICS. Transforming the business and driving revenue through big data and analytics BIG DATA & ANALYTICS Transforming the business and driving revenue through big data and analytics Collection, storage and extraction of business value from data generated from a variety of sources are

More information

Suresh Lakavath csir urdip Pune, India lsureshit@gmail.com.

Suresh Lakavath csir urdip Pune, India lsureshit@gmail.com. A Big Data Hadoop Architecture for Online Analysis. Suresh Lakavath csir urdip Pune, India lsureshit@gmail.com. Ramlal Naik L Acme Tele Power LTD Haryana, India ramlalnaik@gmail.com. Abstract Big Data

More information

Reference Architecture, Requirements, Gaps, Roles

Reference Architecture, Requirements, Gaps, Roles Reference Architecture, Requirements, Gaps, Roles The contents of this document are an excerpt from the brainstorming document M0014. The purpose is to show how a detailed Big Data Reference Architecture

More information

Big Data: Tools and Technologies in Big Data

Big Data: Tools and Technologies in Big Data Big Data: Tools and Technologies in Big Data Jaskaran Singh Student Lovely Professional University, Punjab Varun Singla Assistant Professor Lovely Professional University, Punjab ABSTRACT Big data can

More information

Comparison of the Frontier Distributed Database Caching System with NoSQL Databases

Comparison of the Frontier Distributed Database Caching System with NoSQL Databases Comparison of the Frontier Distributed Database Caching System with NoSQL Databases Dave Dykstra dwd@fnal.gov Fermilab is operated by the Fermi Research Alliance, LLC under contract No. DE-AC02-07CH11359

More information

Referential Integrity in Cloud NoSQL Databases

Referential Integrity in Cloud NoSQL Databases Referential Integrity in Cloud NoSQL Databases by Harsha Raja A thesis submitted to the Victoria University of Wellington in partial fulfilment of the requirements for the degree of Master of Engineering

More information

Big Data Analytics Platform @ Nokia

Big Data Analytics Platform @ Nokia Big Data Analytics Platform @ Nokia 1 Selecting the Right Tool for the Right Workload Yekesa Kosuru Nokia Location & Commerce Strata + Hadoop World NY - Oct 25, 2012 Agenda Big Data Analytics Platform

More information

www.objectivity.com Choosing The Right Big Data Tools For The Job A Polyglot Approach

www.objectivity.com Choosing The Right Big Data Tools For The Job A Polyglot Approach www.objectivity.com Choosing The Right Big Data Tools For The Job A Polyglot Approach Nic Caine NoSQL Matters, April 2013 Overview The Problem Current Big Data Analytics Relationship Analytics Leveraging

More information

Big Data and Industrial Internet

Big Data and Industrial Internet Big Data and Industrial Internet Keijo Heljanko Department of Computer Science and Helsinki Institute for Information Technology HIIT School of Science, Aalto University keijo.heljanko@aalto.fi 16.6-2015

More information

NoSQL systems: introduction and data models. Riccardo Torlone Università Roma Tre

NoSQL systems: introduction and data models. Riccardo Torlone Università Roma Tre NoSQL systems: introduction and data models Riccardo Torlone Università Roma Tre Why NoSQL? In the last thirty years relational databases have been the default choice for serious data storage. An architect

More information

Big Data and Data Science: Behind the Buzz Words

Big Data and Data Science: Behind the Buzz Words Big Data and Data Science: Behind the Buzz Words Peggy Brinkmann, FCAS, MAAA Actuary Milliman, Inc. April 1, 2014 Contents Big data: from hype to value Deconstructing data science Managing big data Analyzing

More information

The Future of Data Management with Hadoop and the Enterprise Data Hub

The Future of Data Management with Hadoop and the Enterprise Data Hub The Future of Data Management with Hadoop and the Enterprise Data Hub Amr Awadallah Cofounder & CTO, Cloudera, Inc. Twitter: @awadallah 1 2 Cloudera Snapshot Founded 2008, by former employees of Employees

More information

Data Modeling in the Age of Big Data

Data Modeling in the Age of Big Data Data Modeling in the Age of Big Data Pete Stiglich Pete Stiglich is a principal at Clarity Solution Group. pstiglich@clarity-us.com Abstract With big data adoption accelerating and strong interest in NoSQL

More information

Advanced Big Data Analytics with R and Hadoop

Advanced Big Data Analytics with R and Hadoop REVOLUTION ANALYTICS WHITE PAPER Advanced Big Data Analytics with R and Hadoop 'Big Data' Analytics as a Competitive Advantage Big Analytics delivers competitive advantage in two ways compared to the traditional

More information

NoSQL Evaluation. A Use Case Oriented Survey

NoSQL Evaluation. A Use Case Oriented Survey 2011 International Conference on Cloud and Service Computing NoSQL Evaluation A Use Case Oriented Survey Robin Hecht Chair of Applied Computer Science IV University ofbayreuth Bayreuth, Germany robin.hecht@uni

More information

Big Data on Cloud Computing- Security Issues

Big Data on Cloud Computing- Security Issues Big Data on Cloud Computing- Security Issues K Subashini, K Srivaishnavi UG Student, Department of CSE, University College of Engineering, Kanchipuram, Tamilnadu, India ABSTRACT: Cloud computing is now

More information

L1: Introduction to Hadoop

L1: Introduction to Hadoop L1: Introduction to Hadoop Feng Li feng.li@cufe.edu.cn School of Statistics and Mathematics Central University of Finance and Economics Revision: December 1, 2014 Today we are going to learn... 1 General

More information

Ubuntu and Hadoop: the perfect match

Ubuntu and Hadoop: the perfect match WHITE PAPER Ubuntu and Hadoop: the perfect match February 2012 Copyright Canonical 2012 www.canonical.com Executive introduction In many fields of IT, there are always stand-out technologies. This is definitely

More information

Performance Testing of Big Data Applications

Performance Testing of Big Data Applications Paper submitted for STC 2013 Performance Testing of Big Data Applications Author: Mustafa Batterywala: Performance Architect Impetus Technologies mbatterywala@impetus.co.in Shirish Bhale: Director of Engineering

More information

Big Data and Advanced Analytics Applications and Capabilities Steven Hagan, Vice President, Server Technologies

Big Data and Advanced Analytics Applications and Capabilities Steven Hagan, Vice President, Server Technologies Big Data and Advanced Analytics Applications and Capabilities Steven Hagan, Vice President, Server Technologies 1 Copyright 2011, Oracle and/or its affiliates. All rights Big Data, Advanced Analytics:

More information

Big + Fast + Safe + Simple = Lowest Technical Risk

Big + Fast + Safe + Simple = Lowest Technical Risk Big + Fast + Safe + Simple = Lowest Technical Risk The Synergy of Greenplum and Isilon Architecture in HP Environments Steffen Thuemmel (Isilon) Andreas Scherbaum (Greenplum) 1 Our problem 2 What is Big

More information

BIG DATA Alignment of Supply & Demand Nuria de Lama Representative of Atos Research &

BIG DATA Alignment of Supply & Demand Nuria de Lama Representative of Atos Research & BIG DATA Alignment of Supply & Demand Nuria de Lama Representative of Atos Research & Innovation 04-08-2011 to the EC 8 th February, Luxembourg Your Atos business Research technologists. and Innovation

More information

Infomatics. Big-Data and Hadoop Developer Training with Oracle WDP

Infomatics. Big-Data and Hadoop Developer Training with Oracle WDP Big-Data and Hadoop Developer Training with Oracle WDP What is this course about? Big Data is a collection of large and complex data sets that cannot be processed using regular database management tools

More information

International Journal of Innovative Research in Information Security (IJIRIS) ISSN: 2349-7017(O) Volume 1 Issue 3 (September 2014)

International Journal of Innovative Research in Information Security (IJIRIS) ISSN: 2349-7017(O) Volume 1 Issue 3 (September 2014) SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCE N.Alamelu Menaka * Department of Computer Applications Dr.Jabasheela Department of Computer Applications Abstract-We are in the age of big data which

More information

You Have Your Data, Now What?

You Have Your Data, Now What? You Have Your Data, Now What? Kevin Shelly, GVP, Global Public Sector Data is a Resource SLIDE: 2 Time to Value SLIDE: 3 Big Data: Volume, VARIETY, and Velocity Simple Structured Complex Structured Textual/Unstructured

More information

Getting Started Practical Input For Your Roadmap

Getting Started Practical Input For Your Roadmap Getting Started Practical Input For Your Roadmap Mike Ferguson Managing Director, Intelligent Business Strategies BA4ALL Big Data & Analytics Insight Conference Stockholm, May 2015 About Mike Ferguson

More information

NoSQL and Graph Database

NoSQL and Graph Database NoSQL and Graph Database Biswanath Dutta DRTC, Indian Statistical Institute 8th Mile Mysore Road R. V. College Post Bangalore 560059 International Conference on Big Data, Bangalore, 9-20 March 2015 Outlines

More information

!"#$%&' ( )%#*'+,'-#.//"0( !"#$"%&'()*$+()',!-+.'/', 4(5,67,!-+!"89,:*$;'0+$.<.,&0$'09,&)"/=+,!()<>'0, 3, Processing LARGE data sets

!#$%&' ( )%#*'+,'-#.//0( !#$%&'()*$+()',!-+.'/', 4(5,67,!-+!89,:*$;'0+$.<.,&0$'09,&)/=+,!()<>'0, 3, Processing LARGE data sets !"#$%&' ( Processing LARGE data sets )%#*'+,'-#.//"0( Framework for o! reliable o! scalable o! distributed computation of large data sets 4(5,67,!-+!"89,:*$;'0+$.

More information

The? Data: Introduction and Future

The? Data: Introduction and Future The? Data: Introduction and Future Husnu Sensoy Global Maksimum Data & Information Technologies Global Maksimum Data & Information Technologies The Data Company Massive Data Unstructured Data Insight Information

More information

Comparing the Hadoop Distributed File System (HDFS) with the Cassandra File System (CFS)

Comparing the Hadoop Distributed File System (HDFS) with the Cassandra File System (CFS) Comparing the Hadoop Distributed File System (HDFS) with the Cassandra File System (CFS) White Paper BY DATASTAX CORPORATION August 2013 1 Table of Contents Abstract 3 Introduction 3 Overview of HDFS 4

More information

The Big Data Ecosystem at LinkedIn Roshan Sumbaly, Jay Kreps, and Sam Shah LinkedIn

The Big Data Ecosystem at LinkedIn Roshan Sumbaly, Jay Kreps, and Sam Shah LinkedIn The Big Data Ecosystem at LinkedIn Roshan Sumbaly, Jay Kreps, and Sam Shah LinkedIn Presented by :- Ishank Kumar Aakash Patel Vishnu Dev Yadav CONTENT Abstract Introduction Related work The Ecosystem Ingress

More information

Building Scalable Big Data Infrastructure Using Open Source Software. Sam William sampd@stumbleupon.

Building Scalable Big Data Infrastructure Using Open Source Software. Sam William sampd@stumbleupon. Building Scalable Big Data Infrastructure Using Open Source Software Sam William sampd@stumbleupon. What is StumbleUpon? Help users find content they did not expect to find The best way to discover new

More information

Big Data

<Insert Picture Here> Big Data Big Data Kevin Kalmbach Principal Sales Consultant, Public Sector Engineered Systems Program Agenda What is Big Data and why it is important? What is your Big

More information

Fault Tolerance in Hadoop for Work Migration

Fault Tolerance in Hadoop for Work Migration 1 Fault Tolerance in Hadoop for Work Migration Shivaraman Janakiraman Indiana University Bloomington ABSTRACT Hadoop is a framework that runs applications on large clusters which are built on numerous

More information

How In-Memory Data Grids Can Analyze Fast-Changing Data in Real Time

How In-Memory Data Grids Can Analyze Fast-Changing Data in Real Time SCALEOUT SOFTWARE How In-Memory Data Grids Can Analyze Fast-Changing Data in Real Time by Dr. William Bain and Dr. Mikhail Sobolev, ScaleOut Software, Inc. 2012 ScaleOut Software, Inc. 12/27/2012 T wenty-first

More information

All You Wanted to Know About Big Data Projects Chida Sadayappan @schida. Jan 2014

All You Wanted to Know About Big Data Projects Chida Sadayappan @schida. Jan 2014 All You Wanted to Know About Big Data Projects Chida Sadayappan @schida Jan 2014 1 WHAT WE DISCUSS HERE AGENDA > > > > > > Need History Open Source - Hadoop BigData EcoSystem Use Cases Managing BigData

More information

Big Data: Beyond the Hype

Big Data: Beyond the Hype Big Data: Beyond the Hype Why Big Data Matters to You WHITE PAPER Big Data: Beyond the Hype Why Big Data Matters to You By DataStax Corporation October 2011 Table of Contents Introduction...4 Big Data

More information

Data Management in the Cloud

Data Management in the Cloud Data Management in the Cloud Ryan Stern stern@cs.colostate.edu : Advanced Topics in Distributed Systems Department of Computer Science Colorado State University Outline Today Microsoft Cloud SQL Server

More information

InfiniteGraph: The Distributed Graph Database

InfiniteGraph: The Distributed Graph Database A Performance and Distributed Performance Benchmark of InfiniteGraph and a Leading Open Source Graph Database Using Synthetic Data Objectivity, Inc. 640 West California Ave. Suite 240 Sunnyvale, CA 94086

More information

Big Data Management. Big Data Management. (BDM) Autumn 2013. Povl Koch September 30, 2013 29-09-2013 1

Big Data Management. Big Data Management. (BDM) Autumn 2013. Povl Koch September 30, 2013 29-09-2013 1 Big Data Management Big Data Management (BDM) Autumn 2013 Povl Koch September 30, 2013 29-09-2013 1 Overview Today s program 1. Little more practical details about this course 2. Recap from last time 3.

More information

Introduction to Analytics and Big Data - Hadoop. Rob Peglar EMC Isilon

Introduction to Analytics and Big Data - Hadoop. Rob Peglar EMC Isilon Introduction to Analytics and Big Data - Hadoop Rob Peglar EMC Isilon SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individual members may use

More information

Hadoop & its Usage at Facebook

Hadoop & its Usage at Facebook Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the Storage Developer Conference, Santa Clara September 15, 2009 Outline Introduction

More information

A Brief Introduction to Apache Tez

A Brief Introduction to Apache Tez A Brief Introduction to Apache Tez Introduction It is a fact that data is basically the new currency of the modern business world. Companies that effectively maximize the value of their data (extract value

More information

Cisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database

Cisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database Cisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database Built up on Cisco s big data common platform architecture (CPA), a

More information

New Modeling Challenges: Big Data, Hadoop, Cloud

New Modeling Challenges: Big Data, Hadoop, Cloud New Modeling Challenges: Big Data, Hadoop, Cloud Karen López @datachick www.datamodel.com Karen Lopez Love Your Data InfoAdvisors.com @datachick Senior Project Manager & Architect 1 Disclosure I m a Data

More information

NOT IN KANSAS ANY MORE

NOT IN KANSAS ANY MORE NOT IN KANSAS ANY MORE How we moved into Big Data Dan Taylor - JDSU Dan Taylor Dan Taylor: An Engineering Manager, Software Developer, data enthusiast and advocate of all things Agile. I m currently lucky

More information

Apache MRQL (incubating): Advanced Query Processing for Complex, Large-Scale Data Analysis

Apache MRQL (incubating): Advanced Query Processing for Complex, Large-Scale Data Analysis Apache MRQL (incubating): Advanced Query Processing for Complex, Large-Scale Data Analysis Leonidas Fegaras University of Texas at Arlington http://mrql.incubator.apache.org/ 04/12/2015 Outline Who am

More information

BIG DATA TOOLS. Top 10 open source technologies for Big Data

BIG DATA TOOLS. Top 10 open source technologies for Big Data BIG DATA TOOLS Top 10 open source technologies for Big Data We are in an ever expanding marketplace!!! With shorter product lifecycles, evolving customer behavior and an economy that travels at the speed

More information

Making Sense of Big Data in Insurance

Making Sense of Big Data in Insurance Making Sense of Big Data in Insurance Amir Halfon, CTO, Financial Services, MarkLogic Corporation BIG DATA?.. SLIDE: 2 The Evolution of Data Management For your application data! Application- and hardware-specific

More information

Testing Big data is one of the biggest

Testing Big data is one of the biggest Infosys Labs Briefings VOL 11 NO 1 2013 Big Data: Testing Approach to Overcome Quality Challenges By Mahesh Gudipati, Shanthi Rao, Naju D. Mohan and Naveen Kumar Gajja Validate data quality by employing

More information

Big Data Analytics - Accelerated. stream-horizon.com

Big Data Analytics - Accelerated. stream-horizon.com Big Data Analytics - Accelerated stream-horizon.com Legacy ETL platforms & conventional Data Integration approach Unable to meet latency & data throughput demands of Big Data integration challenges Based

More information

A Survey on Big Data Analytical Tools

A Survey on Big Data Analytical Tools A Survey on Big Data Analytical Tools Mr. Mahesh G Huddar Sr. Lecturer Dept. of Computer Science and Engineering Hirasugar Institute of Technology, Nidasoshi, Karnataka, India Manjula M Ramannavar Asst.

More information