Apache Phoenix: Transforming HBase into a SQL database. James

Transcription

1 Apache Phenix: Transfrming HBase int a SQL database James

2 Abut me Architect at Salesfrce.cm in Big Data grup Started Phenix as internal prject ~3 years ag Open-surce n Github ~1.5 years ag Apache incubatr fr Cmpleted ~5 mnths Graduated as Tp Level Prject in May 2014 Engineer at BEA Systems XQuery-based federated query engine SQL-based cmplex event prcessing engine

3 Agenda What is Apache Phenix? Why is it s fast? Hw des it help HBase scale? Radmap Q&A Cmpleted

4 What is Apache Phenix? Cmpleted

5 What is Apache Phenix? 1. Turns HBase int a SQL database Query Engine MetaData Repsitry Cmpleted Embedded JDBC driver Only fr HBase data

6 What is Apache Phenix? 2. Fastest way t access HBase data HBase-specific push dwn Cmpiles queries Cmpleted int native HBase calls (n map-reduce) Executes scans in parallel

7 SELECT * FROM t WHERE k IN (?,?,?) Phenix Stinger (Hive Cmpleted 0.11) 0.04 sec 280 sec 7,000x faster * 110M rw table

8 What is Apache Phenix? 3. Lightweight N additinal servers required 100% Java Cmpleted Included in HDP 2.1 distributin Available in Amazn EMR Otherwise cpy Phenix jar int HBase lib directry n each RS

9 HBase Cluster Architecture

10 HBase Cluster Architecture Phenix

11 HBase Cluster Architecture Phenix Phenix

12 What is Apache Phenix? 4. Integratin-friendly Map t existing HBase table Integrate with Cmpleted Apache Pig Integrate with Apache Flume Integrate with Apache Sqp (wip)

13 What is Apache Phenix? 1. Turns HBase int a SQL database 2. Fastest way t access HBase data 3. Lightweight Cmpleted 4. Integratin-friendly

14 Why is Phenix s fast? Cmpleted

15 Why is Phenix s fast? 1. HBase Fast, but dumb (n purpse) 2. Data mdel Cmpleted Supprt fr cmpsite primary key Binary data srts naturally 3. Client-side parallelizatin 4. Push dwn Custm filters and cprcessrs

16 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld HBase Table

17 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld HBase Table Clumn Family A Clumn Family B

18 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld HBase Table Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3

19 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld Rw Key 1 HBase Table Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 KeyValue

20 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld Rw Key 1 HBase Table Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 KeyValue Rw Key 2 KeyValue KeyValue

21 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld Rw Key 1 HBase Table Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 KeyValue Rw Key 2 KeyValue KeyValue Rw Key 3 KeyValue

22 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld HBase Table HBase Table Clumn Family A Clumn Family B Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 Qualifier 1 Qualifier 2 Qualifier 3 Rw Key 1 KeyValue Rw Key 1 Value Rw Key 2 KeyValue KeyValue Rw Key 2 Value Value Rw Key 3 KeyValue Rw Key 3 Value

23 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld HBase Table HBase Table HBase Table Clumn Family A Clumn Family B Clumn Family A Clumn Family B Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 Qualifier 1 Qualifier 2 Qualifier 3 Rw Key 1 KeyValue Qualifier 1 Qualifier 2 Qualifier 3 Rw Key 1 Value Rw Key 1 Value Rw Key 2 KeyValue KeyValue Rw Key 2 Value Value Rw Key 2 Value Value Rw Key 3 KeyValue Rw Key 3 Value Rw Key 3 Value

24 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld HBase Table HBase Table HBase Table Clumn Family A Clumn Family B Clumn Family A Clumn Family B Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 Qualifier 1 Qualifier 2 Qualifier 3 Rw Key 1 KeyValue Qualifier 1 Qualifier 2 Qualifier 3 Rw Key 1 Value Rw Key 1 Value Rw Key 2 KeyValue KeyValue Rw Key 2 Value Value Rw Key 2 Value Value Rw Key 3 KeyValue Rw Key 3 Value Rw Key 3 Value Multiple Versins

25 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld Phenix Table HBase Table Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 Rw Key 1 KeyValue Rw Key 2 KeyValue KeyValue Rw Key 3 KeyValue

26 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld Phenix Table HBase Table Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 Rw Key 1 KeyValue Rw Key 2 KeyValue KeyValue Rw Key 3 KeyValue Key Value Clumns

27 Phenix Data Mdel Phenix maps HBase data mdel t the relatinal wrld Phenix Table HBase Table Clumn Family A Clumn Family B Qualifier 1 Qualifier 2 Qualifier 3 Rw Key 1 KeyValue Rw Key 2 KeyValue KeyValue Rw Key 3 KeyValue Primary Key Cnstraint Key Value Clumns

28 Example Over metrics data fr servers with a schema like this: SERVER METRICS HOST VARCHAR DATE DATE RESPONSE_TIME INTEGER GC_TIME INTEGER CPU_TIME INTEGER IO_TIME INTEGER Rw Key

29 Example Over metrics data fr servers with a schema like this: SERVER METRICS HOST DATE RESPONSE_TIME GC_TIME CPU_TIME IO_TIME VARCHAR DATE INTEGER INTEGER INTEGER INTEGER Key Values

30 Example DDL cmmand lks like this: CREATE TABLE SERVER_METRICS ( HOST VARCHAR, DATE DATE, RESPONSE_TIME INTEGER, GC_TIME INTEGER, CPU_TIME INTEGER, IO_TIME INTEGER, CONSTRAINT pk PRIMARY KEY (HOST, DATE))

31 Example With data that lks like this: SERVER METRICS HOST + DATE RESPONSE_TIME GC_TIME SF SF SF SF SF Rw Key

32 Example With data that lks like this: SERVER METRICS HOST + DATE RESPONSE_TIME GC_TIME SF SF SF SF SF Key Values

33 Phenix Push Dwn: Example SELECT hst, avg(respnse_time) FROM server_metrics WHERE date > CURRENT_DATE() 7 AND hst LIKE SF% Cmpleted GROUP BY hst

34 Phenix Push Dwn: Example SELECT hst, avg(respnse_time) FROM server_metrics WHERE date > CURRENT_DATE() 7 AND hst LIKE SF% Cmpleted GROUP BY hst

35 Phenix Push Dwn: Example SELECT hst, avg(respnse_time) FROM server_metrics WHERE date > CURRENT_DATE() 7 AND hst LIKE SF% Cmpleted GROUP BY hst

36 Phenix Push Dwn: Example SELECT hst, avg(respnse_time) FROM server_metrics WHERE date > CURRENT_DATE() 7 AND hst LIKE SF% Cmpleted GROUP BY hst

37 Phenix Push Dwn: Example SELECT hst, avg(respnse_time) FROM server_metrics WHERE date > CURRENT_DATE() 7 AND hst LIKE SF% Cmpleted GROUP BY hst

38 Phenix Push Dwn 1. Skip scan filter 2. Aggregatin 3. TpN 4. Hash Jin

39 Phenix Push Dwn: Skip scan SELECT hst, avg(respnse_time) FROM server_metrics WHERE date > CURRENT_DATE() 7 AND hst LIKE SF% GROUP BY hst

40 Phenix Push Dwn: Skip scan R 1 R 2 Cmpleted R 3 R 4

41 Phenix Push Dwn: Skip scan Client-side parallel scans R 1 scan 1 scan 2 R 2 Cmpleted R 3 scan 3 R 4

42 Phenix Push Dwn: Skip scan Server-side filter SKIP Cmpleted

43 Phenix Push Dwn: Skip scan Server-side filter INCLUDE Cmpleted

44 Phenix Push Dwn: Skip scan Server-side filter SKIP Cmpleted

45 Phenix Push Dwn: Skip scan Server-side filter INCLUDE Cmpleted

46 Phenix Push Dwn: Skip scan Server-side filter SKIP

47 Phenix Push Dwn: Skip scan Server-side filter INCLUDE

48 Phenix Push Dwn: Skip scan Server-side filter INCLUDE INCLUDE INCLUDE

49 Phenix Push Dwn: Aggregatin SELECT hst, avg(respnse_time) FROM server_metrics WHERE date > CURRENT_DATE() 7 AND hst LIKE SF% GROUP BY hst

50 Phenix Push Dwn: Aggregatin Aggregate n server-side SERVER METRICS HOST DATE KV 1 KV 2 KV 3 SF1 Jun 2 10:10: SF1 Jun 3 23:05: SF1 Jun 9 08:10: SF1 Jun 9 08:10: SF1 Jun 1 11:18: SF1 Jun 3 22:03: SF1 Jun 3 22:03: SF2 Jun 1 10:29: SF2 Jun 2 14:55: SF2 Jun 3 12:46: SF2 Jun 3 12:46: SF2 Jun 8 08:23: SF2 Jun 1 10:31: SF3 Jun 1 10:31: SF3 Jun 3 10:31: SF3 Jun 8 10:31: SF3 Jun 1 10:31: SF3 Jun 1 10:31: SF3 Jun 8 10:31: SF3 Jun 9 10:31: SF3 Jun 3 10:31: HOST SF SF SF SERVER METRICS AGGREGATE VALUES

51 Phenix Push Dwn: TpN SELECT hst, date, gc_time FROM server_metrics WHERE date > CURRENT_DATE() 7 Cmpleted AND hst LIKE SF% ORDER BY gc_time DESC LIMIT 5

52 Phenix Push Dwn: TpN Client-side parallel scans R 1 scan 1 scan 2 R 2 Cmpleted R 3 scan 3 R 4

53 Phenix Push Dwn: TpN Each regin hlds N rws R 1 scan 1 R 2 Cmpleted R 3 R 4

54 Phenix Push Dwn: TpN Each regin hlds N rws R 1 scan 2 R 2 Cmpleted R 3 R 4

55 Phenix Push Dwn: TpN Each regin hlds N rws R 1 R 2 Cmpleted R 3 scan 3 R 4

56 Phenix Push Dwn: TpN Client-side final merge srt Scan 1 Scan 2 SERVER METRICS HOST DATE GC_TIME SF3 Jun 2 10:10: SF5 Jun 3 23:05: Scan 3 SF2 Jun 1 11:18: SF2 Jun 9 08:10: SF1 Jun 3 22:03:

57 Phenix Push Dwn: TpN Secndary Index CREATE INDEX gc_time_index ON server_metrics (gc_time DESC, date DESC) INCLUDE (respnse_time) GC_TIME_INDEX GC_TIME INTEGER DATE DATE HOST VARCHAR RESPONSE_TIME INTEGER Cmpleted Rw Key

58 Phenix Push Dwn: TpN Secndary Index CREATE INDEX gc_time_index ON server_metrics (gc_time DESC, date DESC) INCLUDE (respnse_time) GC_TIME_INDEX GC_TIME INTEGER DATE DATE HOST VARCHAR RESPONSE_TIME INTEGER Cmpleted Key Value

59 Phenix Push Dwn: TpN Secndary Index Original query desn t change Phenix rewrites query t use index table All referenced clumns Cmpleted must exist in index table fr it t be cnsidered Stats cming sn!

60 Phenix Push Dwn: Hash Jin SELECT m.*, i.lcatin FROM server_metrics m JOIN hst_inf i ON m.hst = i.hst Cmpleted WHERE m.date > CURRENT_DATE() 7 AND i.lcatin = SF ORDER BY m.gc_time DESC LIMIT 5

61 Phenix Push Dwn: Hash Jin Separate LHS and RHS SELECT m.*, i.lcatin FROM server_metrics m JOIN hst_inf i ON m.hst = i.hst Cmpleted WHERE m.date > CURRENT_DATE() 7 AND i.lcatin = SF ORDER BY m.gc_time DESC LIMIT 5

62 Phenix Push Dwn: Hash Jin Separate LHS and RHS SELECT m.*, i.lcatin FROM server_metrics m JOIN hst_inf i ON m.hst = i.hst Cmpleted WHERE m.date > CURRENT_DATE() 7 AND i.lcatin = SF ORDER BY m.gc_time DESC LIMIT 5

63 Phenix Push Dwn: Hash Jin Separate LHS and RHS LHS SELECT * FROM server_metrics WHERE date > CURRENT_DATE() 7 ORDER BY gc_time DESC LIMIT 5 Cmpleted RHS SELECT lcatin FROM hst_inf WHERE lcatin = SF

64 Phenix Push Dwn: Hash Jin Execute & bradcast RHS t each RS RHS RS 1 Cmpleted RS 2

65 Phenix Push Dwn: TpN Server-side map lkup during scan LHS scan 1 R 1 RHS scan 2 scan 3 R 2 R 3 Cmpleted RHS scan 4 R 4

66 Hw des Phenix help HBase scale?

67 Hw des Phenix help HBase scale? 1. Phenix allws multiple tables t share same physical HBase table Updateable VIEW Multi-tenant TABLE + tenant-specific VIEW Supprt fr secndary indexes n VIEWs

68 Hw des Phenix help HBase scale? 2. HBase wants small # f big tables instead f large # f small tables Each regin fr each clumn family f each table cnsumes resurces

69 Phenix Shared Tables: VIEW CREATE TABLE event ( type CHAR(1), event_id BIGINT, created_date DATE, created_by VARCHAR, Cmpleted CONSTRAINT pk PRIMARY KEY (type, event_id)); CREATE VIEW web_event ( referrer VARCHAR) AS SELECT * FROM event WHERE type= w ; Includes clumns frm TABLE Cannt define PK Updateable if nly equality expressins separated by AND

70 Phenix Shared Tables: VIEW Same physical HBase table EVENT CHAT_EVENT type = c MOBILE_EVENT Cmpleted type = m PHONE_EVENT WEB_EVENT type = p type = w

71 Phenix Shared Table: MULTI_TENANT CREATE TABLE event ( tenant_id VARCHAR, First PK clumn identifies tenant ID type CHAR(1), event_id BIGINT, Cmpleted created_date DATE, created_by VARCHAR, CONSTRAINT pk PRIMARY KEY (tenant_id, type, event_id)) MULTI_TENANT=true;

72 Phenix Shared Table: MULTI_TENANT CREATE VIEW web_event ( referrer VARCHAR) AS SELECT * FROM event WHERE type= w ; Cmpleted Tenant-specific cnnectin DriverManager.cnnect( jdbc:phenix:lcalhst;tenantid=me ); CREATE VIEW my_web_event AS SELECT * FROM web_event; Tenant-specific view

73 Phenix Shared Tables: MULTI_TENANT Same physical HBase table EVENT tenant_id = aaa tenant_id = aab tenant_id = zzz

74 Phenix Shared Tables: MULTI_TENANT Same physical HBase table EVENT CHAT_EVENT type = c MOBILE_EVENT PHONE_EVENT WEB_EVENT PER tenant_id type = m type = p type = w

75 Phenix Shared Tables: MULTI_TENANT Tenant-specific cnnectin may nly see and perate n their data MetaData APIs hnr this Phenix autmatically manages scan ranges Primary key cnstraint f base table may nt be changed Indexes in separate shared table may be added t a VIEW DDL peratins restricted N ALTER f base table N DROP f clumns referenced in WHERE clause

76 Phenix Radmap Derived/nested tables (in 3.1/4.1) Lcal Indexes (in 4.1) Transactins Cmpleted Mre Jin strategies Crrelated sub-queries Cst-based query ptimizer OLAP extensins

77 Thank yu! Questins/cmments?