Agenda. ! Strengths of PostgreSQL. ! Strengths of Hadoop. ! Hadoop Community. ! Use Cases

Transcription

1 Postgres & Hadoop

2 Agenda! Strengths of PostgreSQL! Strengths of Hadoop! Hadoop Community! Use Cases

3 Best of Both World Postgres Hadoop World s most advanced open source database solution Enterprise class including MVCC, streaming replication & rich data type support (to name a few!) Robust transaction support with strong ANSI-SQL compliance Big data distributed framework Reliable, massively scalable & proven Failures handled at the application layer allowing commodity hardware

4 Strengths of PostgreSQL! Strong Data Types! Concurrency! Transactions! Security! Indexes! Connectors

5 Components of PostgreSQL! Database! Connectors JDBC ODBC Libpq! Foreign Data Wrappers

6 Strengths of Hadoop! Parallelism! Flexibility! Redundancy! Scalability

7 Components of Hadoop! HDFS! Hive! Flume! Sqoop

8 HDFS! Hadoop Distributed File System

9 Hive! Adds SQL on Hadoop! Converts SQL (HQL) to MapReduce Jobs

10 Flume! Streams data into HDFS! Distributed and Highly Available

11 Sqoop! Allows for bulk transfers of data between Hadoop and a RDBMS

12 Hadoop Community! Much more like the Linux community than the PostgreSQL community! Some competing commercial interests makes the direction unclear to some

13 Use Cases

14 Hive Metastore! All of the meta data of the Hive tables reside in a RDBMS! The default is to use Derby Limits to a single connection

15 Hive Metastore (cont.)! Use PostgreSQL for scalability and reliability! Many concurrent users

16 PostgreSQL Backups! PostgreSQL's WAL archiving and Point In Time Recovery is powerful But it requires a lot of storage! Typically used with some sort of NFS

17 PostgreSQL Backups (cont.)! Use HDFS Redundancy & Scalability

18 PostgreSQL Backups (cont.)! Archive Command archive_command = 'hadoop dfs -copyfromlocal %p /user/postgres/wal/ %f'

19 Log Files! Maintain log files for months or years! May use Syslog to consolidate multiple database logs! Turning on query logging makes the log file huge

20 Log Files (cont.)! Use Flume! Consolidates logs across databases! MapReduce allows for parallel analysis

21 Log Files (cont.)! Setup Syslog to forward messages to Flume rsyslog.conf: Configure Flume to act as a Syslog server pglogs.sources.sl.type = syslogudp pglogs.sources.sl.port = 5140 pglogs.sources.sl.host =

22 Log Files (cont.)! MapReduce jobs can quickly analyze the logs public static class MapClass extends MapReduceBase implements Mapper<StatementOffset, Text, Text, LongWritable> { private final static String STATEMENT_DELIM = "statement: "; private final static String SYSLOG_IDENT = "postgres"; private final static LongWritable one = new LongWritable(1); public void map(statementoffset key, Text value, OutputCollector<Text, LongWritable> output, Reporter reporter) throws IOException { String line = value.tostring(); }... if (line.startswith(syslog_ident) && line.contains(statement_delim)) { output.collect(getstatementtype(line), one); }

23 Transaction History! History Tables grow very rapidly! Maintaining the tables over time is a huge undertaking! Partitioning frequently used

24 Transaction History (cont.)! Use Sqoop Add a sequence to the table for fast incremental loads

25 OLAP Cubes! Can take a very long time to build! PostgreSQL will use only a single CPU! Drilling down to the details can be a very long query

26 OLAP Cubes! Use a Foreign Data Wrapper! Looks like a native table to reporting tools! Drill down takes place on Hadoop

27 OLAP Cubes (cont.)! Create a Foreign Server CREATE EXTENSION hive_fdw; CREATE SERVER hive_server FOREIGN DATA WRAPPER hive_fdw OPTIONS (address ' ', port '10000'); CREATE USER MAPPING FOR PUBLIC SERVER hive_server;

28 OLAP Cubes (cont.)! Create a Foreign Table CREATE FORIEGN TABLE order_line ( ol_w_id integer, ol_d_id integer, ol_o_id integer, ol_number integer, ol_i_id integer, ol_delivery_d timestamp, ol_amount decimal(6,2), ol_supply_w_id integer, ol_quantity decimal(2,0), ol_dist_info varchar(24) ) SERVER hive_server OPTIONS (table 'order_line');

29 OLAP Cubes (cont.)! Loading PostgreSQL aggregate tables is a simple SQL statement! Use Hive views for more complex aggregations INSERT INTO item_sale_month SELECT ol_i_id as i_id, EXTRACT(YEAR FROM ol_delivery_d) as year, EXTRACT(MONTH FROM ol_delivery_d) as month, sum(ol_amount) as amount FROM order_line GROUP BY 1, 2, 3;

30 OLAP Cubes (cont.)! Drill downs pass the processing down to Hive postgres=# explain verbose select sum(ol_amount) from order_line where ol_i_id = 34928; QUERY PLAN Aggregate (cost= rows=1 width=14) Output: sum(ol_amount) -> Foreign Scan on public.order_line (cost= rows=1000 width=14) Output: ol_w_id, ol_d_id, ol_o_id, ol_number, ol_i_id, ol_delivery_d, ol_amount, ol_supply_w_id, ol_quantity, ol_dist_info Remote SQL: SELECT * FROM order_line WHERE ((ol_i_id = 34928)) (5 rows)

31 Audit History! All database access should be audited and autonomously logged! Must be maintained for years

32 Audit History (cont.)! Use the Hadoop Foreign Data Wrapper to Flume

33 Audit History (cont.)! Create a writable foreign table (experimental) CREATE FORIEGN TABLE audit ( audit_id bigint, event_d timestamp, table varchar, action varchar, user varchar, ) SERVER hive_server OPTIONS (table 'audit', flume_port '44444');

34 Getting the Components

35 Or...! BigSQL.org

36 Demo