Choosing The Right Big Data Tools For The Job A Polyglot Approach

Transcription

1 Choosing The Right Big Data Tools For The Job A Polyglot Approach Nic Caine NoSQL Matters, April 2013

2 Overview The Problem Current Big Data Analytics Relationship Analytics Leveraging Alternative Technologies NoSQL The Polyglot Approach

3 About Objectivity Inc. Company Objectivity, Inc. is headquartered in Sunnyvale, CA. Established in 1988 to tackle database problems that network/hierarchical/relational and file-based technologies struggle with. Objectivity has over two decades of Big Data and NoSQL experience Products Develops NoSQL platforms for managing and discovering relationships and patterns in complex data: Objectivity/DB - an object database that manages localized, centralized or distributed databases InfiniteGraph - a massively scalable graph database built on Objectivity/DB that enables organizations to find, store and exploit the relationships in their data Markets The Big Data market is projected to be around $12B in 2012, with a CAGR of 28% over the next five years. 40% per year data growth, cloud adoption, mobile usage and improved real-time analytics underpin Objectivity s growth opportunities as a Big Data analytics enabler. Customers Embedded in hundreds of enterprises, government organizations and products - millions of deployments. Financials Consistently generates increased revenues. Privately held by the employees and a few venture capital companies. Copyright Objectivity, Inc. 2012

4 The Problem Information Overload! Making sense of it all takes time and $$$ Current Big Data Analytics Does one size fit all?

5 A Typical Big Data Analytics Challenge Data Aggregation and Analytics Applications Commodity Linux Platforms and/or High Performance Computing Clusters RDBMS Column Store Data W/H Graph DB Object DB Hadoop Doc DB K-V Store Structured Semi-Structured Unstructured

6 Strictly Confidential 6 Navigating connections requires performance and scalability

7 Copyright InfiniteGraph How do I store graph data? Graph Data Your One Size Fits All Database

8 Copyright InfiniteGraph Careful! Large graphs can be difficult to manage

9 InfiniteGraph UseCases

10 Government customer #1 Ingesting more than 1 billion vertices and edges per day Currently distributing graph repository across 14 servers VMWare-based architecture designed to scale processing and storage linearly using InfiniteGraph

11 Georgetown Phase #1 - complete Initial objective, was to model and store 7 billion people objects and 3 billion objects representing other data types in InfiniteGraph 2 nd objective was to provide web-based earthview drill down interface from planet to sidewalk to see the people objects distributed around the planet in near real-time.

12 Georgetown Phase #2 Currently in development Objective is to detect and report on possible emergent planet and population events based on data streams from around the globe (Predictive analytics). InfiniteGraph will be the basis of event models that get activated when an event is emerging.

13 Leveraging NoSQL Technologies

14 Not Only SQL a group of 4 primary technologies Users choose between four different primary technologies for different purposes: Key-Value Stores Big Table Clones Document Databases Object and Graph databases (including InfiniteGraph) Many implementations trade off consistency (ACID transactions, CAP eventual consistency) for performance.

15 Key-Value Stores: Pros & Cons Strengths: Simple data model Great at scaling out horizontally Scalable Available Weaknesses: Simplistic data model Poor for complex data Unsuited for interconnected data KEY VALUE

16 Big Table Clones Pros & Cons Strengths: Data model supports semi-structured data Naturally indexed (columns) Good at scaling out horizontally Weaknesses: Complex data model Unsuited for highly interconnected data KEY Column Column Name Value D/Time

17 Document Databases Pros & Cons Strengths: Simple, powerful data model Good scalability if sharding is supported Weaknesses: KEY DOCUMENT Unsuited for interconnected data Query model limited is to keys and indexes Generally uses Map-Reduce (designed for batch operations) for larger queries

18 Object Databases Pros & Cons Strengths: Simple, powerful data model that includes inheritance and polymorphism Every object has a class (type) and a unique Object Identifier Uses Object Identifiers instead of JOIN tables to support very fast navigational operations Weaknesses: OID OBJECT Connections The query language never became a standard Supports standard object oriented languages but isn't supported by a wide range of third party tools in the way that SQL is.

19 Graph Databases Pros & Cons Strengths: Extremely fast for connected data Scales out, typically Easy to query (navigation) Simple powerful data model Weaknesses: Not all vendors support distribution or sharding Requires conceptual shift... a different way of thinking VERTEX 2 N EDGE

20 Competing Big Data Appliance Solutions

21 Incremental Improvements Aren t Enough All current big vendor solutions use the same basic architectural model In general their current solutions don t have a way to store connections between entities across disparate data sources Most analytic technology focuses on the content of the data nodes, rather than the many kinds of connections between the nodes and the data in those connections Why? Because relational solutions are bad at handling relationships. Object and Graph databases can efficiently store, manage and query the many kinds of relationships that are hidden in the data.

22 Relationship Analytics

23 Example 1 - Market Analysis The 10 companies that control a majority of U.S. consumer goods brands

24 Example 2 - Demographics Used in social network analysis, marketing, medical research etc.

25 Example 3 - Seed To Consumer Tracking?

26 Example 4 - Ad Placement Networks Smartphone Ad placement - based on the the user s profile and location data captured by opt-in applications. The location data can be stored and distilled in a key-value and column store hybrid database, such as Cassandra The locations are matched with geospatial data to deduce user interests. As Ad placement orders arrive, an application built on a graph database such as InfiniteGraph, matches groups of users with Ads: Maximizes relevance for the user. Yields maximum value for the advertiser and the placer.

27 Relationship (Connection) Analytics... Relational Database Think about the SQL query for finding all links between the two blue rows... Good luck! Table_A Table_B Table_C Table_D Table_E Table_F Table_G Relational databases aren t good at handling complex relationships!

28 Relationship (Connection) Analytics... Relational Database Table_A Table_B Table_C Table_D Table_E Table_F Table_G InfiniteGraph - The solution can be found with a few lines of code A3 G4

29 Visual Analytics

30 The Polyglot Approach

31 SUMMARY: A Polyglot Approach Works Best... LANGUAGE REPOSITORY PROBLEM ANALYTICS BI TOOLS GRAPH TOOLS VISUAL ANALYTICS

32 ...SUMMARY: A Polyglot Approach Works Best Raw Data

33 InfiniteGraph THE BIG DATA CONNECTION PLATFORM

34 Thank You! Please take a look at For Online Demos, White Papers, Free Downloads, Samples & Tutorials Stop by our stand for more information