DEMYSTIFYING IN-MEMORY DATA GRID, IN-MEMORY DATA FABRIC AND NOSQL DB PRADEEP NAIK

Size: px

Start display at page:

Download "DEMYSTIFYING IN-MEMORY DATA GRID, IN-MEMORY DATA FABRIC AND NOSQL DB PRADEEP NAIK"

Bonnie Norris
7 years ago
Views:

1 DEMYSTIFYING IN-MEMORY DATA GRID, IN-MEMORY DATA FABRIC AND NOSQL DB PRADEEP NAIK

2 SPEAKER INTRODUCTION Pradeep Naik, Principal Consultant, CTO Oﬃce, Wipro Technologies Responsible for incubating next generation Wipro 20+ years of experience in Database Management Cross-domain IT Solution Architecting and Managing database solutions in Telecom, Healthcare and Financial Industries

55 Bn) Partner to Industry Leaders Attracts the Best Talent Global

1** 100+ Active Global Clients Workforce Leader in Software & Services

Q1 results 2015 16 for Global IT Services business ** Global leader in

World Indices 5th year in a row Ranked 8th in the Best Companies for

3 ABOUT WIPRO Sustained Growth $7.6 Bn Revenue in FY (IT Services $ 7.06 Bn, IT Products 0.55 Bn) Partner to Industry Leaders Attracts the Best Talent Global Leaders Global Presence 1071* 161,789* No.1** 100+ Active Global Clients Workforce Leader in Software & Services industry category Serving clients in 100+ countries *Figures based on Q1 results for Global IT Services business ** Global leader in the Software & Services category, Member of Dow Jones Sustainability World Indices 5th year in a row Ranked 8th in the Best Companies for Leaders 2015 list in a study conducted by Chally Group Honored as a World s Most Ethical Companies by Ethisphere Institute for the Fourth Successive Year, 2015

4 IN-MEMORY TECHNOLOGY OVERVIEW, CONCEPTS AND KEY DIFFERENCES

5 WHAT BUSINESS DEMANDS? Big Data Use case High Throughput Low Latency High Throughput Data Ingestion Data Storage Data Processing Low Latency Data Access Data Analytics Data Visualization

6 TRADE OFF BETWEEN HIGH THROUGHPUT AND LOW LATENCY High Throughput Low Latency Auto scaling Eventual Consistency Localized processing Parallel Computing Stream Processing Parallel Computing Localized Processing In-memory computing

7 IN-MEMORY WORLD In-Memory Database (IMDB) NoSQL In-Memory Data Grid (IMDG) In-Memory Data Fabric (IMDF) Operational Analytical HTAP

IN-MEMORY DATABASE (IMDB) Architecture of In-Memory Database Source: Oracle Times Ten Good ANSI SQL Support Strong support for ACID transactions Lack of co-location processing Based on vertically

8 IN-MEMORY DATABASE (IMDB) Architecture of In-Memory Database Source: Oracle Times Ten Good ANSI SQL Support Strong support for ACID transactions Lack of co-location processing Based on vertically scalable Symmetrical Processing Architecture Does not support distributed computing Minimal application changes for upgrading to IMDB Cannot work directly with domain objects. Users need to perform Object-To-Relational Mapping which typically adds significant performance overhead Unit of movement is Data and not the Process

NOSQL Architecture of NoSQL A distributed data storage with in-memory option Most commonly used for high throughput requirements Read latency in the range of millisecond to seconds Low latency data

9 NOSQL Architecture of NoSQL A distributed data storage with in-memory option Most commonly used for high throughput requirements Read latency in the range of millisecond to seconds Low latency data access is achieved by caching the table/document in NoSQL Cluster memory Data is always stored in disk and can be configured to cache in memory Achieves high availability via replication mechanism Tunable consistency (Eventual and Immediate) Limitation on size of table that can be cached

10 IN-MEMORY DATA GRID (IMDG) Architecture of In-Memory Data Grid IMDG is a data structure that completely resides in memory and distributed across multiple severs Application Servers Data In-Memory Data Grid Memory from each servers K1,V1 K2,V2 K3,V3 K4,V4 Node1 Node2 Node3 Node4 Scale horizontally persistence Read-through Fault tolerance uses master-master or master-slave topology Varied variety of data structures supported to map domain objects Distributed computing - Collocate processing to cluster node where data is cached Distributed Concurrency- Supports distributed transaction locking Write-through or Write behind Database server Persistence- Supports seamless synchronous read through, write through or asynchronous write-behind to other data sources Support applications with low latency requirements

IN-MEMORY DATA FABRIC (IMDF) IMDF is a comprehensive in-memory data platform that Architecture of In-Memory Data Fabric includes data grid, clustering, compute grid, Complex Event Processing and

11 IN-MEMORY DATA FABRIC (IMDF) IMDF is a comprehensive in-memory data platform that Architecture of In-Memory Data Fabric includes data grid, clustering, compute grid, Complex Event Processing and real-time streaming Application Servers It is a superset of IMDG Data/ Task/Query Supports standard SQL for querying in-memory data including support for distributed SQL joins. Distribute computations and data processing across multiple computers in a cluster in order to gain high performance and low latency Support multiple execution paths for same events executing in parallel on one or more nodes In-Memory Data Fabric Data Grid Compute Grid Streaming Map Reduce Hadoop Accl. CEP Distributed Cluster Works on underlying concept of MPP architecture Converged platform to support multiple use cases NoSQL/RDBMS File System Messaging

TECHNOLOGY EVALUATION CRITERIA High Throughput vs. Low Latency High Availability Scalability (Vertical vs. Horizontal) Distributed disk based data storage vs.

12 TECHNOLOGY EVALUATION CRITERIA High Throughput vs. Low Latency High Availability Scalability (Vertical vs. Horizontal) Distributed disk based data storage vs. distributed in-memory storage Co-location of data processing Distributed transactional ACID support Eventual Consistency vs. Strong Consistency Application change impacts Reuse existing database technology stack vs. migrate to new databases? Platform support for in-memory computation and storage Support for flexible Data Structure

SUMMARIZING THE DIFFERENCES In-Memory Database NoSQL Strong ACID Support IMDB IMDF IMDG NoSQL Structured Data ANSI SQL Support Low latency performance Mixed workload Leverage existing database Stack

13 SUMMARIZING THE DIFFERENCES In-Memory Database NoSQL Strong ACID Support IMDB IMDF IMDG NoSQL Structured Data ANSI SQL Support Low latency performance Mixed workload Leverage existing database Stack Flexible Data Structure (document/columnar/key-value) Distributed data storage for high volume of data Data Caching for low read latency (ms) In-Memory Data Grid In-Memory Data Fabric Varied variety of data structure Distributed in-memory data store Scalable and Fault tolerant need Low latency performance (ns to µs ) High performance distributed processing Data persistence for high availability Converged platform to support multiple use cases High performance distributed parallel processing Distributed in-memory data store Co-located data processing Accelerate Hadoop ecosystem

14 IN-MEMORY SOLUTIONS KEY PLAYERS In-Memory Database In-Memory Data Fabric In-Memory Data Grid Technology Vendors NoSQL

15 CASE STUDY EASY TO ADD THE LOGO AND TEXT

SEARCH ENGINE OPTIMIZATION Problem Statement E-commerce online retail store dealing with the SEO across different browser, countries and languages are implemented usually using XML sitemap.

16 SEARCH ENGINE OPTIMIZATION Problem Statement E-commerce online retail store dealing with the SEO across different browser, countries and languages are implemented usually using XML sitemap. Few search engines such as Baidu and Yandex does not support the XML sitemap implementation of hreflang which limits use of sitemap implementation of tag to address multiple countries/ languages Does not consider the localization causing incorrect search results Yandex only supports the on-page hreflang tags

17 ECOMMERCE SEARCH ENGINE OPTIMIZATION Design Decision The data in the sitemap XML needs to be populated in the head of pages for both canonical urls and alternate URLs <link rel="alternate" hreflang="en-us" href= > <link rel="alternate" hreflang="en-mx" href= > For low latency access consider caching data from DB. The data in DB is not structured and needs to be transformed and aggregated to minimize the size of the cache. Use MapReduce framework for the data transformation and aggregation. The challenge was refreshing the cache within 4 hours to avoid having stale data.

18 ECOMMERCE SEARCH ENGINE OPTIMIZATION A. Store the data into NoSQL database such as MongoDB, run periodic map reduce jobs for transformation and aggregation before the data getting refreshed in the database. The challenge was to complete the Map Reduce within 4 hours to avoid getting the stale data B. Supplement data store with some sort of caching layer to read the data from native memory rather from the disk. The solution was to use IMDB., but then we need to upgrade the tech stack to implement the Map Reduce jobs on the underlying IMDB. C. Store the data in IMDG which can also facilitate the Map Reduce framework, so that the Map Reduce jobs can be achieved in the stipulated time

ECOMMERCE SEARCH ENGINE OPTIMIZATION Implemented solution using IMDG such as Hazelcast Two data center with 5 node each Data is stored and reduced in

19 ECOMMERCE SEARCH ENGINE OPTIMIZATION Implemented solution using IMDG such as Hazelcast Two data center with 5 node each Data is stored and reduced in the native memory using the Map Reduce feature set/api(s) Configured near cache option to fetch the data locally from the server where the URLs is served.

20 THANK YOU PRESENTER Pradeep Naik TEAM MEMBERS: Viresh Kumar, Chandra Sekar K.R

In Memory Accelerator for MongoDB

In Memory Accelerator for MongoDB Yakov Zhdanov, Director R&D GridGain Systems GridGain: In Memory Computing Leader 5 years in production 100s of customers & users Starts every 10 secs worldwide Over 15,000,000