managing planned downtime with RAC Björn Rost

Transcription

1 managing planned downtime with RAC VIP D ASM C CVU SQL UC CP OPS OUI RAC F OUI RAC FCF LBA ONS UI RAC FCF LBA ONS FAN C RAC FCF LBA ONS FAN TAF OD AC FCF LBA ONS FAN TAF CRS VIP A FCF LBA ONS FAN TAF CRS VIP GSD ODA FCF LBA ONS FAN TAF CRS VIP GSD ASM SQ LBA ONS FAN TAF CRS VIP GSD ASM CVU SQL UC F LBA ONS FAN TAF CRS VIP GSD ASM CVU RAC FCF L CRS VIP GSD ASM CVU SQL UCP OPS OUI RAC LBA ON IP GSD ASM CVU SQL UCP OPS OUI RAC ODA LBA FAN T ASM CVU SQL RAC UCP OPS OUI RAC FCF ASM LBA ON CVU SQL UCP OPS OUI RAC LBA ODA ONS TAF FAN TAF CVU SQL UCP OPS ODA OUI RAC FCF LBA ONS FAN U SQL UCP OPS OUI FCF LBA ONS FAN TAF CRS P OPS OUI RAC FCF ONS FAN TAF CRS VIP I RAC FCF LBA ONS FAN TAF CRS VIP C FCF LBA ONS FAN TAF CRS VIP ONS FAN TAF CRS VIP GSD AS S FAN TAF CRS VIP GSD RS VIP GSD ASM CV SD ASM CVU SQ VU SQL UC Björn Rost

2 Björn Rost founder, manager and DBA RAC SIG European Chair ACE Director

3 about us Software production company founded 2001 mostly J2EE logistics telecommunication media and publishing customers demand full lifecycle support hardware resale datacenter operations 3rd party software

4 project lifecycle consulting J2EE Hardware hosting specification php SW-Licenses monitoring documentation database installation patching feasibility studies benchmarking backups tuning planning design integration operation

5 TAF

6 TAF Minimize downtime! Go implement this TAF thing. Just turn it on, it is completely transparent!

7 TAF Minimize downtime! Go implement this TAF thing. Just turn it on, it is completely transparent! let me check the docs and get right back

8 TAF

9 use OCI driver TAF

10 TAF use OCI driver can do

11 TAF use OCI driver can do delay or overhead?

12 TAF use OCI driver can do delay or overhead? was expecting some cost

13 TAF use OCI driver can do delay or overhead? no DML! was expecting some cost

14 TAF use OCI driver can do delay or overhead? no DML! was expecting seriously? some cost

15 TAF use OCI driver can do delay or overhead? no DML! was expecting seriously? some cost yup, only SELECT will fail over...

16 expectation a clustered HA system should always be UP

17 the reality even with RAC implemented, there are still many (if not more) outages :(

18 limits a session can never move between nodes session creation (lb) decided on connection HA needs to be supported in Apps some of this stuff can be confusing

19 12c app continuity

20 Agenda introduction walkthrough load balancing connection pools srvctl app continuity

21 reasons to use RAC

23 reasons to use RAC You probably don t need RAC!

25 reasons to use RAC scalability & performance

26 reasons to use RAC scalability & performance high availability

27 reasons to use RAC scalability & performance high availability unplanned

28 reasons to use RAC scalability & performance high availability unplanned planned

29 RAC One node RAC without scaling across multiple nodes migration to full RAC online possible seamless crash failover

30 unplanned downtime

31 unplanned downtime hardware fault

32 unplanned downtime hardware fault servers come with redundant components disks power supplies fans components are getting better, too

33 unplanned downtime

34 unplanned downtime hardware fault

35 unplanned downtime hardware fault software crash or hang

36 unplanned downtime hardware fault software crash or hang DOS attacks / security issues

37 unplanned downtime hardware fault software crash or hang DOS attacks / security issues human error

38 planned downtime

39 planned downtime hardware upgrade (RAM, CPU,...)

40 planned downtime hardware upgrade (RAM, CPU,...) firmware upgrades

41 planned downtime hardware upgrade (RAM, CPU,...) firmware upgrades OS updates

42 planned downtime hardware upgrade (RAM, CPU,...) firmware upgrades OS updates Oracle Software patches

43 planned downtime hardware upgrade (RAM, CPU,...) firmware upgrades OS updates Oracle Software patches network re-patching

44 planned downtime hardware upgrade (RAM, CPU,...) firmware upgrades OS updates Oracle Software patches network re-patching SAN reconfiguration

45 downtime

46 failure types

47 failure types app not connected (only on demand)

48 failure types app not connected (only on demand) session open but idle/no tx

49 failure types app not connected (only on demand) session open but idle/no tx app needs to reconnect

50 failure types app not connected (only on demand) session open but idle/no tx app needs to reconnect tx in progress, SELECT only

51 failure types app not connected (only on demand) session open but idle/no tx app needs to reconnect tx in progress, SELECT only start over or display error

52 failure types app not connected (only on demand) session open but idle/no tx app needs to reconnect tx in progress, SELECT only start over or display error tx in progress, DML

53 failure types app not connected (only on demand) session open but idle/no tx app needs to reconnect tx in progress, SELECT only start over or display error tx in progress, DML rollback/replay/handle error

54 failure types app not connected (only on demand) session open but idle/no tx app needs to reconnect tx in progress, SELECT only start over or display error tx in progress, DML rollback/replay/handle error important: don t commit twice

55 failure types app not connected (only on demand) session open but idle/no tx app needs to reconnect tx in progress, SELECT only start over or display error tx in progress, DML rollback/replay/handle error important: don t commit twice (re)join of cluster node

56 failure types app not connected (only on demand) session open but idle/no tx app needs to reconnect tx in progress, SELECT only start over or display error tx in progress, DML rollback/replay/handle error important: don t commit twice (re)join of cluster node

57 maintenance rqmts

58 maintenance rqmts remove nodes from cluster without user interruption

59 maintenance rqmts remove nodes from cluster without user interruption don t break running sessions

60 maintenance rqmts remove nodes from cluster without user interruption don t break running sessions ok to kill idle sessions, let them reconnect

61 maintenance rqmts remove nodes from cluster without user interruption don t break running sessions ok to kill idle sessions, let them reconnect don t loose data/transactions/new orders

62 maintenance rqmts remove nodes from cluster without user interruption don t break running sessions ok to kill idle sessions, let them reconnect don t loose data/transactions/new orders stay up or available

63 load balancing

64 load balancing client side tnsnames.ora and/or SCAN

65 load balancing client side tnsnames.ora and/or SCAN server side on connection long goal: # of connections short goal: system load avg

66 load balancing client side tnsnames.ora and/or SCAN server side on connection long goal: # of connections short goal: system load avg runtime advisory events sent to conn. pools

67 SCAN RAC_OLTP = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = scan.db.portrix.net)(port = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = OLTP) )) oracle@rac1:~$ host scan.db.portrix.net scan.db.portrix.net has address scan.db.portrix.net has address scan.db.portrix.net has address

68 Services: OLTP batch

69 Services: OLTP: RAC1, RAC2 batch: RAC1 RAC1 RAC2

70 1 Services: OLTP: RAC1, RAC2 batch: RAC1 RAC1 RAC2

71 Services: 1 OLTP: RAC1, RAC2 batch: RAC1 RAC1 RAC2

72 Services: OLTP: RAC1, RAC2 batch: RAC1 1 RAC1 RAC2

73 2 Services: OLTP: RAC1, RAC2 batch: RAC1 1 RAC1 RAC2

74 Services: 2 OLTP: RAC1, RAC2 batch: RAC1 1 RAC1 RAC2

75 Services: OLTP: RAC1, RAC2 batch: RAC1 2 1 RAC1 RAC2

76 3 Services: OLTP: RAC1, RAC2 batch: RAC1 2 1 RAC1 RAC2

77 Services: 3 OLTP: RAC1, RAC2 batch: RAC1 2 1 RAC1 RAC2

78 Services: OLTP: RAC1, RAC2 batch: RAC RAC1 RAC2

79 Services: OLTP: RAC2 batch: RAC RAC1 RAC2

80 4 5 Services: OLTP: RAC2 batch: RAC RAC1 RAC2

81 Services: 4 5 OLTP: RAC2 batch: RAC RAC1 RAC2

86 5 4 3 RAC1 RAC2

88 Services: OLTP: RAC1, RAC2 batch: RAC RAC1 RAC2

89 app requirements reconnect regularly handle connection failures set max_sessions to the right value

90 connection pools pool will open and hold connections app loans session for tx as needed when tx is done, app returns session pool can decide which connection to lend to app

91 connection pools save resources memory connection time reconnect help load balancing abstraction layer for errors

92 UCP and FAN

93 UCP and FAN Fast Connection Failover

94 UCP and FAN Fast Connection Failover crash

95 UCP and FAN Fast Connection Failover crash planned outage

96 UCP and FAN Fast Connection Failover crash planned outage (re)join

97 UCP and FAN Fast Connection Failover crash planned outage (re)join run-time load balancing

98 UCP and FAN Fast Connection Failover crash planned outage (re)join run-time load balancing session affinity

99 UCP and FAN Fast Connection Failover crash planned outage (re)join run-time load balancing session affinity transaction affinty

100 services A service is an entity to which users connect configured with connection settings on client registered through clusterware each service has: a list of preferred and available instances load-balancing goal TAF and other parameters 12c multitenant: each PDB has it s own service

101 services default service is always active on all nodes ORA-01033: ORACLE initialization or shutdown in progress seperation might improve performance helpful in other areas of administration resource management EM monitoring grouping

102 srvctl srvctl config service -d PTXRAC -s OLTP Service name: OLTP Service is enabled Server pool: PTXRAC_OLTP Cardinality: 2 Disconnect: false Service role: PRIMARY Management policy: AUTOMATIC DTP transaction: false AQ HA notifications: false Failover type: NONE Failover method: NONE TAF failover retries: 0 TAF failover delay: 0 Connection Load Balancing Goal: LONG Runtime Load Balancing Goal: SHORT TAF policy specification: NONE Edition: Preferred instances: PTXRAC1,PTXRAC2 Available instances:

103 verify service cfg lsnrctl status listener_scan1 LSNRCTL for Solaris: Version Production on 29-SEP :35:58 Copyright (c) 1991, 2010, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))) STATUS of the LISTENER Alias LISTENER_SCAN1 Version TNSLSNR for Solaris: Version Production Start Date 30-APR :09:28 Uptime 151 days 12 hr. 26 min. 30 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora Listener Log File /u01/app/11.2.0/grid/log/diag/tnslsnr/sun1os/listener_scan1/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST= )(PORT=1521))) Services Summary... Service "BATCH.DB.PORTRIX.NET" has 1 instance(s). Instance "PTXRAC2", status READY, has 1 handler(s) for this service... Service "OLTP.DB.PORTRIX.NET" has 2 instance(s). Instance "PTXRAC1", status READY, has 1 handler(s) for this service... Instance "PTXRAC2", status READY, has 1 handler(s) for this service... Service "PTXRAC.DB.PORTRIX.NET has 2 instance(s). Instance "PTXRAC1", status READY, has 1 handler(s) for this service... Instance "PTXRAC2", status READY, has 1 handler(s) for this service... Service "PTXRACXDB.DB.PORTRIX.NET" has 2 instance(s). Instance "PTXRAC1", status READY, has 1 handler(s) for this service... Instance "PTXRAC2", status READY, has 1 handler(s) for this service... The command completed successfully

104 srvctl srvctl modify service Moves a service member from one instance to another. Additionally, this command changes which instances are to be the preferred and the available instances for a service. This command supports some online modifications to the service, such as: When there are available instances for the service, and the service configuration is modified so that a preferred or available instance is removed, the running state of the service may change unpredictably: The service is stopped and then removed on some instances according to the new service configuration. The service may be running on some instances that are being removed from the service configuration. These services will be relocated to the next free instance in the new service configuration. srvctl relocate service -d db_unique_name -s service_name {-c source_node -n target_node -i old_instance_name -t new_instance_name} [-f]

105 srvctl if service only up on one node: relocate up on muliple nodes: modify

106 shutdown srvctl stop instance -d db_unique_name {[-n node_name] [-i "instance_name_list"]} [-o stop_options] [-f] stops all services on the node (with -f ) better relocate service yourself!

107 shutdown srvctl stop instance -d db_unique_name {[-n node_name] [-i "instance_name_list"]} -o transactional refuses new connections disconnects sessions after commit/rollback

108 steps (again) relocate services away (relocate/modify) wait until sessions are done with work shutdown (transactional) perform maintenance restart services relocate services back

109 rolling upgrades available in a lot of patches two RDBMS versions running simultaneously built-in support in OPatch

110 rolling upgrades tmp]$ opatch query -is_rolling_patch Invoking OPatch Oracle Interim Patch Installer version Copyright (c) 2009, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/11.2.0/db_1 Central Inventory : /u01/app/orainventory from : /etc/orainst.loc OPatch version : OUI version : OUI location : /u01/app/oracle/product/11.2.0/db_1/oui Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/ opatch _ am.log Patch history file: /u01/app/oracle/11.2.0/db_1/cfgtoollogs/opatch/ opatch_history.txt Patch is a rolling patch: true

111 12c app continuity 2 part system transaction guard reliably determine the state of commits app continuity (replay driver) driver records and caches requests and validation information reconnects and verifies commit state replays and validates requests

112 activate app continuity driver needs replay boundaries UCP and WebLogic add these automatically beginrequest/endrequest for 3rd party apps jdbc-thin only mutable calls (seq.nextval, sysdate) does not work with default service consider memory&cpu overhead

113 review TAF Load-Balancing services UCP FAN and FCF App Continuity

114 summary set up at least one extra service possibly more make sure application reconnects regularly use UCP if possible try and use this make it part of app rqrmts patch regularly

115 und weiter? RAC SIG Wahlen laufen gerade! RAC SIG -

116 DOAG 2013 unconference: DEMO 12c RAC auf laptop, UCP und app continuity mit java app

117 Danke RAC Attack RAC SIG - b.rost@portrix.net