Java MySQL Connector & Connection Pool

Transcription

1 Java MySQL Connector & Connection Pool Features & Optimization Kenny Gryp November 4,

2 2 Please excuse me for not being a Java developer DISCLAIMER

3 What I Don t Like 3 Brussels Sprouts Taxes Calories Java( s chattiness)

4 4 MYSQL CONNECTORS CONNECTION POOLS

5 5 Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction MYSQL CONNECTORS CONNECTION POOLS

6 MySQL Connector/J & MariaDB Java Client 6 MySQL Connector/J Oracle Latest Compatible with MySQL Percona Server MariaDB

7 MySQL Connector/J & MariaDB Java Client 7 MariaDB Java Client MariaDB Fork from Drizzle Connector Latest Compatible with MySQL Percona Server MariaDB

8 MySQL Connector/J Features 8 Enterprise Plugin: Query Analyzer MySQL Fabric Integration Load Balancing Failover Replication

9 9 Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction MYSQL CONNECTORS CONNECTION POOLS

10 Creating Connection 10 Connection con =!! DriverManager.getConnection!! ( jdbc:mysql://node2/employees?!!! user=connj&password=test");! Statement stmt = con.createstatement();! String query =!! "select * from employees!!! where emp_no = 20000;";! ResultSet rs = stmt.executequery(query);!...!! MariaDB: jdbc:mariadb://node2/employees!!?user=connj&password=test"

11 Creating Connection - Tomcat w. JDBC- Pool 11 context.xml (local):! <Resource name="jdbc/test"! auth="container"! type="javax.sql.datasource"! username= jdbc-pool" password="test"! driverclassname="com.mysql.jdbc.driver"! url= jdbc:mysql://node2:3306/employees! />!! MariaDB: driverclassname="org.mariadb.jdbc.driver"!

12 Creating Connection - JDBC URL 12! jdbc:mysql://node2:3306/employees? useserverprepstmts=true&...!!

13 13 Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction MYSQL CONNECTORS CONNECTION POOLS

14 Connector/J - Creating Connection 14 SHOW VARIABLES WHERE!! Variable_name ='language' OR! SET NAMES latin1;! SET character_set_results = NULL;! SET autocommit=1;! SET sql_mode=!! 'NO_ENGINE_SUBSTITUTION,STRICT_TRAN S_TABLES';!

15 MariaDB - Creating Connection 15 set autocommit=1;! USE employees;! show variables like 'sql_mode';!

16 Creating Connection - Defaults 16 Connector/J:!!!! MariaDB Java Client:

17 Connector/J & MariaDB Java Client - Verbosity 17 Connector/J is more verbose when starting a connection Usually not a problem: connection pools are commonly used (more coming soon ) connections are reused Actually I like but not too much.

18 Optimization 18 MariaDB Java Client vs MySQL Connector/J Prepared Statements

19 Connector Performance - SELECT 1 localhost QPS ConnectorJ MariaDB localhost

20 Connector Performance - MariaDB %faster 20 20% 15% 13% Faster % 10% 5% SELECT1 LO MariaDB Connector +Speed% ConnectorJ

21 Connector Performance - MariaDB %faster 21 20% 15% 13% Faster % 10% 5% 4% 4% SELECT1 LO SELECT1 net pk range MariaDB Connector +Speed% ConnectorJ 0%

22 22 Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction MYSQL CONNECTORS CONNECTION POOLS

23 Client or Server Side Prepared Statements 23 Server side Prepared statements: reduce network traffic query is already optimized on server. Support: MariaDB Java client only supports client side Connector/J default in client side

24 Server Side Prepared Statements 24 PREPARE stmt1 FROM!! select * from employees!!! where emp_no =?;! EXECUTE # API CALL!! select * from employees!!! where emp_no = 20000;! DEALLOCATE PREPARE stmt1;

25 Connector/J: Server Side Prepared Statements 25 useserverprepstmts = false! disabled by default Mind looking at: cacheprepstmts = false! do PREPARE, EXECUTE, DEALLOCATE every time, 3 round trips? prepstmtcachesize = 25! prepstmtcachesqllimit = 256! low defaults

26 Benchmark: Prepared Statements QPS 3.342QPS 3.506QPS QPS QPS MariaDB CONN/J CONN/J SRV CONN/J SRV+Cache select * from employees_alotofindexes where first_name='moss' and birth_date > " " and gender="m" and hire_date > " "\G

27 Cracked!! 27

28 28 Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction MYSQL CONNECTORS CONNECTION POOLS

29 Connector/J Optimization + Default JDBC- Pool 29 Connection con = ds.getconnection();! con.settransactionisolation!!! (Connection.TRANSACTION_READ_COMMITTED);! con.setautocommit(false);! PreparedStatement stmt = con.preparestatement("select * from employees where emp_no =?");! stmt.setint(1, 20000);! ResultSet rs = stmt.executequery();! stmt.close();! rs.close();! con.commit();! con.close();

30 Connector/J Optimization + Default JDBC- Pool 30 SET SESSION TRANSACTION!!! ISOLATION LEVEL READ COMMITTED;!! SET autocommit=0;!! # administrator command: Prepare;!! select * from employees!!!! where emp_no = 20000;! Taxes! # administrator command: Close stmt;!! commit;

31 Connector/J Optimization 31 useconfigs=maxperformance! cacheprepstmts=true! cachecallablestmts=true! cacheserverconfiguration=true! uselocalsessionstate=true! elidesetautocommits=true! alwayssendsetisolation=false! enablequerytimeouts=false

32 Connector/J Optimization 32 uselocaltransactionstate=true commit() / rollback()

33 Connector/J Optimization - Tuned 33 JDBC URL: useconfigs=maxperformance&! useserverprepstmts=true:!! select * from employees!! where emp_no = 20000;! commit;

34 MariaDB Java Client Optimization 34 SET SESSION TRANSACTION!!! ISOLATION LEVEL READ COMMITTED;!! select * from employees!!!! where emp_no = 20000;!! COMMIT;!

35 MariaDB Java Client Optimization - Code 35 if! (! con.gettransactionisolation()!=!!!! Connection.TRANSACTION_READ_COMMITTED! )! {! con.settransactionisolation!!! (Connection.TRANSACTION_READ_COMMITTED);! }

36 MariaDB Java Client Optimization - Interceptors 36 select * from employees!! where emp_no = 20000;! COMMIT;!! Now add JDBC Interceptor: <Resource name="jdbc/test"! auth="container"! factory=! "org.apache.tomcat.jdbc.pool.datasourcefactory"! jdbcinterceptors="connectionstate"! driverclassname="org.mariadb.jdbc.driver"! url="jdbc:mysql://node2:3306/employees"/>!

37 MariaDB Java Client Optimization - Optimized! 37 select * from employees!! where emp_no = 20000;! COMMIT;

38 Benchmark - Connector Concurrency - SELECT 1 38 HikariCP- bench with JDBC Pool, 4 Threads, SELECT 1 (4,8,16,32 Pool Size) QPS MariaDB CONN/J

39 Benchmark - Connector Concurrency - TRX 39 HikariCP- bench with JDBC Pool, 4 Threads, TRX (4,8,16,32 Pool Size) QPS Conn/J MariaDB Conn/J Oprm MariaDB Oprm

40 40 MYSQL CONNECTORS CONNECTION POOLS

41 41 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

42 Java Connection Pools 42 The Usual: C3P0 Commons- DBCP (v1&v2) JDBC Pool (fork commons- DBCP) Out In The Wild: Vibur- DBCP HikariCP BoneCP

43 Java Connection Pools 43 The Usual: C3P0 Commons- DBCP (v1&v2) JDBC Pool (fork commons- DBCP) Out In The Wild: Vibur- DBCP HikariCP BoneCP

44 Connection Pool Key Points 44 Connection Management Pool Sizing Connection Testing Avoid Lingering Transactions

45 45 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

46 Connection Pool Issues 46 Coming from DBA side, I do not like chattiness

47 Connection Pool Issues 47

48 Connection Pool Issues 48

49 Connection Pool Issues 49

50 50 Because of application bugs WHY DOES IT HAVE TO DO THAT?

51 Connection Pools - Why Chattiness Examples 51 *maybe* forgot to COMMIT / ROLLBACK! wanting AUTOCOMMIT=1 but a previous TRX set it to 0 Changing TRX Isolation Level Is connection still working?

52 52 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

53 Connection Pool - Resetting Status 53 JDBC- Pool C3P0 DBCP2 HikariCP Rollback rollbackonreturn=false autocommitonclose=false rollbackonreturn =true Commit commitonreturn=false autocommitonclose=false n/a n/a always rollback() Avoid see above forceignoreunresolvedtrans actions=false see above Auto Commit Driver Driver enableautocommit OnReturn=true Driver

54 54 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

55 Connection Pool - Testing 55 Making sure the connection is still active If not, maybe reopen a connection Not recommended as DB However, applications: do not like errors do not retry gracefully

56 Connection Pool - Testing 56 If connections REALLY need to be tested do not specify test query like: SELECT 1! SELECT * FROM DUAL! Leave default, all of the connection pools use: JDBC4 isvalid();

57 Connection Pool - Testing 57 JDBC- Pool C3P0 DBCP2 HikariCP Test Before testonborrow=false testconnectiononcheckout =false Test After testonreturn=false testconnectiononcheckin =false Test While Idle testwhileidle=false idleconnectiontestperiod =0 testonborrow=false testonreturn=false testwhileidle=false n/a n/a n/a JDBC4 isvalid() default default default jdbc4connectiontest =true (default) Query validationquery (isvalid) preferredtestquery=null validationquery (isvalid) connectiontestquery =none Interval? validationinterval=30000 n/a n/a n/a

58 Connection Pool - Testing 58 JDBC: validationinterval=30s WHY? It defeats the whole purpose!

59 59 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

60 Connection Pool - Pool Sizing 60 Funnelling on Application Level, is good Smaller Number is Better +- * CPU s on DB maybe a bit more (waiting on IO ) all application servers combined Response Time vs Throughput

61 Connection Pool - Pool Sizing 61 Amount of Connections Maximum Idle Connections Minimum Idle Connections JDBC- Pool C3P0 DBCP2 HikariCP maxactive=100 maxpoolsize=15 maxtotal=8 maximumpoolsize=10 maxidle=100 maxidletime=0** maxidle=8 n/a minidle=10 minpoolsize=3 minidle=0 minimumidle=max Startup Size initialsize=10 initialpoolsize=3 initialsize=0 minimumidle

62 62 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

63 Connection Pool - Avoid Lingering Transactions 63 Application forgets to return the connection Statements that take longer than! Avoid this! Fix Application

64 Connection Pool - Avoid Lingering Transactions 64 JDBC- Pool KILL removeabandoned=false removeabandonedtimeout=60 abandonwhenpercentagefull=0 Warning suspecttimeout=0 C3P0 unreturnedconnectiontimeout=0 n/a DBCP removeabandoned=false removeabandonedtimeout=300 Only When: getnumidle() < 2 and getnumactive() > getmaxtotal() - 3) HikariCP n/a leakdetectionthreshold=0 n/a

65 65 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

66 Connection Pools - How To Look At Workload? 66 Slow Query Log tcpdump! pt-query-digest! Percona Cloud Tools

67 67 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

68 Connection Pool - Example Transaction 68 Connection con = ds.getconnection();! con.settransactionisolation!!! (Connection.TRANSACTION_READ_COMMITTED);! con.setautocommit(false);! PreparedStatement stmt = con.preparestatement("select * from employees where emp_no =?");! stmt.setint(1, 20000);! ResultSet rs = stmt.executequery();! stmt.close();! rs.close();! con.commit();! con.close();

69 For Connectors - RECAP 69 MySQL Connector/J useconfigs=maxperformance! useserverprepstmts=true! MariaDB Java Client JDBC- Pool: jdbcinterceptors="connectionstate" Other Pools: UNKNOWN

70 Connection Pool - TRX JDBC 70 select * from employees!!!! where emp_no = 20000;!! commit; 200

71 Connection Pool - TRX C3P0 71 SET SESSION TRANSACTION!! ISOLATION LEVEL READ COMMITTED;! SET autocommit=0;! select * from employees!!!! where emp_no = 20000;! commit;! SET autocommit=1;! SET SESSION TRANSACTION! 600!! ISOLATION LEVEL REPEATABLE READ;

72 Connection Pool - TRX C3P0 72 mysql> set global!! tx_isolation= READ-COMMITTED ;!! forceignoreunresolvedtransactions=true! 200

73 Connection Pool - TRX DBCP 73 SET autocommit=1;! # administrator command: Ping;! SET autocommit=0;! select * from employees!!!! where emp_no = 20000;! commit;! rollback;! SET autocommit=1;! 700

74 Connection Pool - TRX DBCP 74 testonborrow=false! rollbackonreturn=false! enableautocommitonreturn=false!! jdbcurl: uselocaltransactionstate=true! 200

75 Connection Pool - TRX HikariCP 75 SET SESSION TRANSACTION!! ISOLATION LEVEL READ COMMITTED;! SET autocommit=0;! select * from employees!!!! where emp_no = 20000;! commit;! rollback;! SET autocommit=1;! 700 SET SESSION TRANSACTION!!! ISOLATION LEVEL REPEATABLE READ;!

76 Connection Pool - TRX HikariCP 76 mysql> set global!! tx_isolation= READ-COMMITTED ;!! autocommit=false!!! --> will always do rollback!!!!! 300

77 Connection Pools 77

78 MariaDB vs. Connector/J 78

79 79 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

80 Connection Pools - Graceful Failover 80

81 Connection Pools - Graceful Failover 81 HAProxy stats socket /etc/haproxy/haproxy.cfg! global!...! stats socket /tmp/haproxy.sock level admin! Disable Node # echo "disable server database/node1" socat stdio /tmp/haproxy.sock

82 Connection Pools - Graceful Failover 82

83 Connection Pools - Graceful Failover 83 During maintenance, what do we do? KILL old connections? Wait until connections are closed? (Define lifetimes?) Ignore it?

84 Connection Pools - Graceful Failover 84 Some connection pools can close connections gracefully, when idle. For synchronous replication systems using JMX No Application Errors! Method JDBC- Pool purgeonreturn() C3P0 softresetallusers() DBCP n/a HikariCP closeidleconnections() Only closes current idle connections

85 Connection Pools - Graceful Failover 85

86 Connection Pools - Graceful Failover 86

87 Connection Pools - Graceful Failover 87

88 Connection Pools - Graceful Failover 88 0 Application Errors Completely seamless

89 89 Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features MYSQL CONNECTORS CONNECTION POOLS

90 Connector/J - Extra Features 90 Load Balancing jdbcurl: jdbc:mysql:loadbalance:// node1,node2/db? loadbalanceconnectiongroup=lb& loadbalanceenablejmx=true! loadbalancestrategy (random/bestresponsetime)! Failover ReplicationDriver (setreadonly) Combining with Connection Pools is less useful Fabric

91 Victory! 91

92 Java MySQL Connector & Connection Pool Optimization 92 j/en libraries/mariadb- java- client/ doc/jdbc- pool.html dbcp/ MYSQL CONNECTORS CONNECTION POOLS Kenny Gryp November 4,