Hybrid Solutions Combining In-Memory & SSD

Transcription

1 Hybrid Solutions Combining In-Memory & SSD Author:

2 Agenda Overview of the big data technology landscape Building a high-speed SSD-backed data store Complex & compound queries Orchestration in Big Data 2

3 Making Sense of the Exploding Big Data World SQL NoSQL SSD IMDG Key / Value Stream Processing 3

4 Let s look at some tools & technologies

5 SQL Technologies Query: ANSI 92 Semantics: CRUD Aggregation Projection Partial update Performance: 100 s/sec Consistency: Transactional Scaling: Mostly Scale-UP Availability: Disk Based 5

6 NoSQL Technologies Query: Proprietary but rich Semantics: CRUD Map/Reduce No Projection No Partial update Performance: 1000 s/sec Consistency: Eventual Scaling: Mostly Scale-Out Availability: Replication based 6

7 IMDG Technologies Query: Proprietary but rich Semantics: CRUD Aggregation API + Map/Reduce Projection Partial Update Performance: 100k/sec Consistency: Transactional Scaling: Mostly Scale-Out Availability: Replication based 7

8 Key/Value Technologies Query: Key, Value Semantics: Mostly Read No Aggregation No Projection No Partial update Performance: 1M s/sec Consistency: Atomic Scaling: Mostly Scale-Out Availability: Limited 8

9 Stream Processing Technologies Semantics: Event-driven data processing Performance: 10M s/sec Machine learning Real-time analytics Depend on external persistency for maintaining state 9

10 Stream Processing Integration - Streaming tuples Tuple stream (FIFO) Stream Producer IMDG Storm 10

11 Stream Processing Integration - Keeping state Only non-transient data in SSD State IMDG State Result Maintain state and SSD-backed Grid write results in IMDG Stream Producer Tuple stream Storm 11

12 SSD Technology is quickly shaping the Big Data Landscape Great for heavy-reads initial-loads SSD-powered in-memory products can provide great performance with increased server density Big data but also fast data Bigger capacity, better performance, cheaper than RAM 12

13 Summary Many API s - Same Data Use-case requirements across tools SSD enables greater scale at cost Need for big & fast data 13

14 Building an SSD-backed API Data-Store

15 Using IMDG & SSD to build a hybrid big-data solution Create a common API High-speed Data Store Store indexes in RAM payload in SSD Use native low API 15

16 IMDG & SSD integration Hold Indexes & cached objects in RAM Transient Data Old Data Initial Load Client New Data Client writes new data to the space Proc. Data Transient Data Proc. Data Hold Payload in SSD 16

17 Why an SSD-backed IMDG Higher server density and capacity Added resilience and increased availability Faster grid warmup, less down-time Rich query capabilities with persistency 17

18 A typical Big Data App logical architecture can look like this Front End Front-end users accessing a distributed multi-facet service Application / Service RT Analytics Service Storage Batch Processing back-end users accessing a business insights and system maintenance metrics Back End 18

19 We need a High-Speed Data Store Front End Back End Key / Value Document Graph Map / Reduce Transactional Stream based High-speed Data Store Common Data Store serving Multiple Semantics/API Disk becomes the new tape 19

20 We can use IMDG technologies to integrate all our Data Sources RT Transactional Front End Back End Data Access High-speed Data Bus (IMDG) RT Streaming Storm Speed Layer Hadoop Sync Direct Access Mongo Sync RDBMS Sync Data bus: Resilient, FT & HA Hadoop Transactional High-throughput MongoDB MySQL Batch Layer Web Storage Layer 20

21 What really goes on in the grid Proc. Data Client New Data Client New Data New data automatically synced to ES Client writes new data to the space Polling container writes to MongoDB Polling Container Notify Container Proc. Data Application Storage MongoDB Mirror DB Space mirroring Service 21

22 The familiar Lambda architecture Front End Cache Layer RT Stream Processing Back End Front-end users accessing a distributed multi-facet service Storage Batch Processing back-end users accessing a business insights and system maintenance metrics 22

23 Introduce a intermediate near real-time layer Front End Cache Layer RT Stream Processing Back End Front-end users accessing a distributed multi-facet service Near Real-time Processing back-end users accessing a business insights and system maintenance metrics Storage Batch Processing 23

24 Hybrid Architecture Hold Indexes & cached objects in RAM Transient Data Old Data Initial Load Client New Data Client writes new data to the space Proc. Data Transient Data Proc. Data Hold Payload in SSD Application Storage MongoDB Mirror DB Data Mirroring 24

25 Telco customer real-world use-case Make payment (Transactional) Fast history search High-speed Data Bus (IMDG) RT Streaming State persistency Direct Access Storm Real-time business analytics on user activity (if needed) Sync new data available to end user Hadoop Sync business analytics Downstream System Hadoop Long-term analytics and storage 25

26 What we achieved 10-fold Grid growth - 20GB to 200GB Improved initial loading times - less downtime Customer history up to 15 years - previously 3 Longer visibility on intermediate customer analytics 26

27 Summary Using an SSD-backed IMDG as data hub Creating a hybrid solution architecture Using SSD to add a near real-time layer 27

28 Using rich query semantics with data grids

29 Nested Queries & Projections Query for a Person who lives in New York: A = new SQLQuery<Person>(Person.class, address.city = New York ); Query for a Dealer which sells a Honda: B = new SQLQuery<Dealer>(Dealer.class, cars[*] = Honda ); Query for a Person with projections on first and last names: C IdsQuery<Person> idsquery = new IdsQuery<Person> (Person.class, new Long[]{id1, id2}).setprojections( firstname, lastname ); Person result[] = space.readbyids(idsquery).getresultsarray(); 29

30 Basic Aggregations Create a query that yields a results set: A SQLQuery<Employee> = new SQLQuery<Employee> (Employee.class, country=? OR country=? ); query.setparameter(1, UK ); query.setparameter(2, USA ); Perform aggregations for that result set: B Integer maxageinspace = max(space, query, age ); Integer minageinspace = min(space, query, age ); Integer combinedageinspace = sum(space, query, age ); Double averageage = average(space, query, age ); Person oldestpersoninspace = maxentry(space, query, age ); Person youngestpersoninspace = minentry(space, query, age ); 30

31 Complex & Compound Aggregations Create a query that yields a results set: A SQLQuery<Employee> = new SQLQuery<Employee> (Employee.class, country=? OR country=? ); query.setparameter(1, UK ); query.setparameter(1, USA ); B Perform group and filtering aggregations for that result set: = groupby(space, query, new GroupByAggregator().select(average("salary"), min("salary"), max( salary")).groupby("department", gender )).having(new GroupByFilter() { public boolean process(groupbyvalue group) { return group.getdouble("avg(salary)") > 18000;}})); 31

32 Fast Update & Change API Performing changes at the data-store level: A IdsQuery<Person> idsquery = new IdsQuery<Person> (Person.class, id, routing) space.change(idsquery, new ChangeSet().increment( balance.euro, 5.2D)); Performing a series changes at the data-store level: B IdsQuery<Person> idsquery = new IdsQuery<Person> (Person.class, id, routing) space.change(idsquery, new ChangeSet().increment( someintproperty, 1).set( somestringproperty, newvalue ).putinmap( somenestedproperty.somemapproperty, mykey, 2); 32

33 Orchestration in Big Data

34 The Application Deployment Lifecycle Manage Provision Monitor Install Deploy Configure 34

35 Create a Standardised Blueprint of the Application Topology Node Type: Container Node Type: Container Node Node Type: Server Type: DB Node Type: App Connected To relationship Contained In relationship 35

36 Using TOSCA & YAML to Describe the Application Topology... host: type: cloudify.nodes.libcloud.compute... ################################################################################## # Tomcat server ################################################################################## tomcat_server: type: cloudify.nodes.tomcatserver relationships: - type: cloudify.relationships.contained_in target: host ################################################################################## # MongoDB node as a backend data-store for the example Tomcat application ################################################################################## mongodb: type: cloudify.nodes.mongodb relationships: - type: cloudify.relationships.contained_in target: host... 36

37 Post-deployment Management & Monitoring 37

38 Thanks For Attending Author: