Products for the registry databases and preparation for the disaster recovery Naoki Kambe, JPRS <kambe@jprs.co.jp> 28 th CENTR Tech workshop, 3 Jun 2013 Copyright 2013 Japan Registry Services Co., Ltd. 1
Introduction To introduce products and preparation for Disaster Recovery To measure performance of products in lab To evaluate results for Disaster Recovery Copyright 2013 Japan Registry Services Co., Ltd. 2
Our primary and secondary sites Secondary site Primary site Properties about network: WAN (Leased line) RTT around 15ms (actual measurement) Bandwidth 100Mbps over 500km distance Copyright 2013 Japan Registry Services Co., Ltd. 3
Important things for Disaster Recovery We're considering so far: High throughput from many clients Propagation delay time in replication through databases: less than 1 sec Time to complete failover to each slave database: less than 1 min Copyright 2013 Japan Registry Services Co., Ltd. 4
Primary site Architecture supporting failover and replication Secondary site Clients Clients R/W R/O pgpool-ii Load balancing pgpool-ii / Handling failover R/W R/O R/O R/O Master 1 Synchronous Replication WAL (Write Ahead Log) 2 Asynchronous Replication 3 Copyright 2013 Japan Registry Services Co., Ltd. 5
Used products for the disaster recovery PostgreSQL for registry databases Open source SQL database Streaming replication for data replication Built-in replication feature of PostgreSQL pgpool-ii for handling failover Open source middleware for PostgreSQL Handling detection of failure on Master and failover to Copyright 2013 Japan Registry Services Co., Ltd. 6
Performance tests in lab Copyright 2013 Japan Registry Services Co., Ltd. 7
Environment in lab R/W Client (pgbench) pgpool-ii R/O Transactions loading R/O pgpool-ii R/O Master 1 WAN Emulator (Netem) delaying packet transfer 2 3 Copyright 2013 Japan Registry Services Co., Ltd. 8
Specification of lab test 4 virtual machines (KVM) 1 CPU, 8GB Memory Host's CPU: Intel Xeon 1.80GHz 4 cores Cent OS 6.3 PostgreSQL 9.2.3 pgpool-ii 3.2.3 Network between 1 and 2 RTT around 10 ms (manipulated by Netem) Bandwidth 100 Mbps Copyright 2013 Japan Registry Services Co., Ltd. 9
Throughput in TPS Measurements measured for each SQL with clients: 1 to 128, total transactions: 51,200 Propagation delay time since updating on Master until applying WAL on 3 measured for each write SQL calculated from PostgreSQL log files Failover time since a failure on master until failing over to each Copyright 2013 Japan Registry Services Co., Ltd. 10
Results of lab test Copyright 2013 Japan Registry Services Co., Ltd. 11
Throughput in TPS TPS 2000 1500 1000 SELECT UPDATE 500 0 INSERT DELETE 1 8 16 32 64 128 Number of clients Copyright 2013 Japan Registry Services Co., Ltd. 12
Thoughts on throughput result "SELECT" transactions scaled. Load balanced using pgpool-ii "UPDATE" and "INSERT" not scaled. Same behaviors as single database "DELETE" resulted in under 100 TPS. No index on table utilized using pgbench as: DELETE.. WHERE name = 'DOMAIN' to_char( :num, '00000' ); How can we improve DELETE? Over 500 TPS Copyright 2013 Japan Registry Services Co., Ltd. 13
Propagation delay time WAL Master 1 Netem 2 3 updating time Propagation delay time applying time INSERT: 14.35 ms UPDATE: 36.40 ms DELETE: 7.40 ms* less than 1 sec * too fast, packets not delayed by Netem Copyright 2013 Japan Registry Services Co., Ltd. 14
Failover handling time 1)fallen down Case1: 2) detect pgpool-ii 3) fail over 1)fallen down Case2: 2) detect pgpool-ii 3) fail over Master 1 Netem 2 3 Case1: Master => 1 Case2: 2 => 3 less than 1 min Copyright 2013 Japan Registry Services Co., Ltd. 15
Configuration of pgpool-ii Script to promote to (new) Master: Remote login to by SSH Changing configuration file of PostgreSQL Invoking "pg_ctl" to restart Example of pgpool.conf : failover_command = '/path/to/failover.sh %M %m %H %D' %M: Old Master Number %H: New Master Hostname %m: New Master Number %D: Data Dir Copyright 2013 Japan Registry Services Co., Ltd. 16
Conclusion and Future Work Satisfied: SELECT, INSERT, and UPDATE Throughput: over 500 TPS Propagation delay time: less than 1 sec Failover handling in less than 1 min Not satisfied: DELETE Throughput: under 100 TPS Propagation delay time: too fast (< 10 ms) Future Work DELETE: improving and re-measuring Copyright 2013 Japan Registry Services Co., Ltd. 17
What about sharing good ideas? pgbench How can it utilize indexes when DELETing? PostgreSQL + pgpool-ii Please share good ideas and good practices, e.g. best configuration on pgpool-ii Disaster Recovery products Do you have any other good products? How to switch over to another site Manually? Automatically? Copyright 2013 Japan Registry Services Co., Ltd. 18
Thank you Copyright 2013 Japan Registry Services Co., Ltd. 19