Business Intelligence in Microservice Architecture. Debarshi bol.com

Transcription

1 Business Intelligence in Microservice Architecture Debarshi bol.com

2 What can you expect? - Introduction Monolithic days Mapreduce Era Flink Era Operational Aspect

3 Who am I? Debarshi Basak Software engineer at bol.com Part of Bigdata platform team and online marketing

4 About bol.com - Leader in Dutch ecommerce Scrum employees 40+ scrum teams Young and relaxed You build it. You run it. You love it.

5 How big is big data at bol.com? - > products for sale Catalog > newsletter responses new clicks every day - 26 node cluster More than 300 jobs a month in production

6 What is Microservice?

7 What is Microservice? App Database

8 What is Microservice? Services Services App Services Services Services Services Database Services

9 Business Intelligence 101

10 Business Intelligence 101

11 Business Intelligence Analyzing data and presenting actionable items

12 Business Intelligence Analyzing data and presenting actionable items - Automated and Continuous Integration with internal and external data sources

13 Business Intelligence Analyzing data and presenting actionable items - Automated and Continuous Integration with internal and external data sources - Flexible Analytics

14 Business Intelligence ETL - Extract from source Transform the data Load into target data models

15 Business Intelligence ETL - - Extract from source Transform the data Load into target data models Business data modelling - Kimball s dimensional modeling technique OLAP cubes

16

17 Monolithic days

18 Evolution of BI Systems (at bol.com) Online systems

19 Evolution of BI Systems (at bol.com) Online systems Data hub

20 Evolution of BI Systems (at bol.com) Online systems Replication Data hub

21 Evolution of BI Systems (at bol.com) Online systems Data hub Replication Data warehouse

22 Evolution of BI Systems (at bol.com) Online systems Data hub Data warehouse ETL

23 Evolution of BI Systems (at bol.com) - Easy to implement Complexities are abstracted Data Overheads Latency

24 Evolution of BI Systems (at bol.com) Online systems

25 Evolution of BI Systems (at bol.com) Online systems Publish Message Broker

26 Evolution of BI Systems (at bol.com) Online systems Message Broker Listener Data warehouse

27 Evolution of BI Systems (at bol.com) Online systems Message Broker Listener Data warehouse

28 Evolution of BI Systems (at bol.com) - Loss of Messages and Consistency guarantees Database are kind of not made for this Complex implementation Nightmare for operations

29 Challenges in Microservice Architecture

30 Challenges in Microservice Architecture - Too many sources Can affect scalability and stability of reports BI cannot scale Extraction logic, transformation operations for each Service Joins.

31 Hadoop era

32 History of Hadoop at bol.com Operational experience in hbase, hadoop based tooling - Supplier connector - Recommendation system

33 Service Definition RPC over HTTP Message Queues Service Bulk Interfaces

34 Bulk Interfaces t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

35 Bulk Interfaces t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50 We can replay event to get the latest state of the event. This is also known as Event Sourcing Pattern. Similar key design can be found in OpenTSDB

36 Re-imagine traditional BI on hadoop Source systems Staging-Transformation Loading in reporting tooling

37 Re-imagine traditional BI on hadoop Source systems Staging-Transformation Loading in reporting tooling

38 Supplier Service Offers Services Pricing Services Data warehouse

39 From Queues Supplier Service Offers Services Pricing Services Data warehouse

40 Supplier Service Offers Services Pricing Services Data warehouse

41 Supplier Service Offers Services Pricing Services Data warehouse

42 Supplier Service Offers Services Pricing Services Data warehouse

43 Supplier Service Offers Services Pricing Services Data warehouse

44 Automation Supplier Service Integration Unit Offers Services Pricing Services Data warehouse

45 What kind of jobs we build - Aggregation jobs - - Single service, aggregate on a function key Interface concatenation - Multiple services combined on one/many functional keys.

46 Automation { "bulk_interface" : "transport_acc_public_v1_transporter_versions", "table_name" : "transporter", "primary_key" : "f:transporter.transporterid", "ctl_name" : "transporter", "service_version" : "v2", "col_map":[ { "hbase_col_name": "f:transporter.transporterid", "ora_col_name":"id", "function_name":"transporter id", "data_type" : "NUMBER" }, { "hbase_col_name": "f:transporter.transportercode", "ora_col_name":"code", "function_name":"transporter code", "data_type" : "VARCHAR2(40)" }, { "hbase_col_name": "f:transporter.transportername", "ora_col_name":"transportername", "function_name":"transporter Name", "data_type" : "VARCHAR2(40)" } ] }

47 Problem Source systems Staging-Transformation Loading in reporting tooling

48 Problem Source systems Staging-Transformation Loading in reporting tooling

49 Problem Source systems Staging-Transformation Loading in reporting tooling

50 But everything is stream. Nature of data in most of use cases is asynchronous. Clicks are asynchronous Orders are asynchronous Updates are asynchronous In fact, Batch is a bounded stream.

51 Streaming era

52 Enter Flink Low entry barrier Java/Scala functional apis. Operational expertise.

53 Emulating Stream You don t always need queues for stream Streaming HBase tables.

54 Give a starting point in stream t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

55 Give next x records t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

56 Give next x records t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

57 while(true){ Give next x records } t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

58 Offers Product Catalog

59 ProductId Offers ProductId Product Catalog Product-Offer-Join

60 Offers Product Catalog Product-Offer-Join

61 Offers Product Catalog Product-Offer-Join

62 Offers Product Catalog Product-Offer-Join

63 Offers Product Catalog Product-Offer-Join

64 Offers Product Catalog Product-Offer-Join

65 Offers Eventual Consistency ProductId Product Catalog Product-Offer-Join

66 Offers Product Catalog Sources Product-Offer-Other Sources-Join

67 Offers Product Catalog Sources Cube or star

68 Offers Product Catalog Sources Star

69 Can we automate this?

70 Can we automate this? Yes, We can.

71 Can we automate this? cube_builder.from( table("productoffer_tst_public_v1.0_sellingoffers_versions") ).on( key("f:globalid", key -> new StringBuilder(key).reverse().toString()) ).lookup( key("f:globalid"), table("financecategory_tst_public_v1_productfinancecategorycurrents") ).to( table("final_join_version"), table("reverse_index_lookup", key("f:globalid"), columns("f:offerid")), table("final_join_version1", columns("f:sellingofferdata.listprice")) ).build().execute();

72 Operational Aspect Build

73 Operational Aspect Build

74 Operational Aspect Build Docker Registry

75 Operational Aspect Build Docker Registry Deploy

76 Operational Aspect Build Docker Registry Deploy

77 Operational Aspect Build Docker Registry Deploy

78 Lessons learned - Dedicated team for hadoop Think not tools but how to solve problems Flink can be flinky Frameworks are out there Kylin Think infrastructure too

79