The Power of FAN. UKOUG 1 December 2008 Bernhard De Cock Buning

Transcription

1 The Power of FAN UKOUG 1 December 2008 Bernhard De Cock Buning

2 Who Am I? Bernhard de Cock Buning Since 3 years an Independent Consultant/Grid-it. Living in the Netherlands. Special interest area: HA/RAC/ASM/CRS/MAA 11 years working with Oracle Products. Previous job at Oracle support (The Netherlands). OCP 8i,9i,10g and OCE on RAC Founder of GRID-IT

3 Agenda Starting point What is Fast Application Notification Event Forwarding Daemon Service Side Call out Oracle Notification Service Racgimon Listeners Clients Total Overview Prereq: Have RAC knowledge 3

4 The Oracle Cluster VIP1 Node1 Database instance 1 ASM Instance 1 Oracle Clusterware Operating System public network cluster interconnect cache to cache VIP2 Node2 Database instance 2 ASM Instance 2 Oracle Clusterware Operating System cluster interconnect VIP3 Node3 Database instance 3 ASM Instance 3 Oracle Clusterware Operating System... shared storage redo logs Database files control files OCR and Voting Disk 4

5 What is Fast Application Notification? ONS EVMD A small powerful component in the Oracle Cluster where parts are installed by default. Other parts need to be configured according to business needs. ons.jar sqlplus -F 5

6 What is Fast Application Notification? A notification mechanism that Oracle RAC uses to notify other processes about cluster configuration and service-level information. 6

7 What is Fast? For cluster configuration changes, the high availability framework publishes a FAN event immediate when a change occurs regarding the state in the cluster. Instead of waiting for the application to query the database and detect a problem, applications can receive FAN events and react immediately. 7

8 What is an Application? In context of FAN, it is not just notification of an Application but notification to subscribers. Subscribers : Nodes Listeners Middle tiers sqlplus EMGC OCI JDBC 8

9 What is the type of Notification? Status change for Nodes Instances Services Up and Down events Load Balancing Advisory events. 9

10 Processes involved? The following processes are involved which can deal with events. Event Forwarding Deamon Oracle Notification Service RACGIMON DBMS_HA_ALERTS_PRVT Listener Client connections 10

11 Overview Node1 PMON Event takes place RACGIMON LSNR ONS DBMS_HA_ALERTS_PRVT EVMD Cliënt JDBC Middletier Node2 ONS.jar ONS ONS EVMD

12 Clusterware EVMD Node1: init Node2: evmd evmd evmlogger racgevt action action callout

13 EVMD [Event Forwarding Daemon] Spawns a permanent child evmlogger. Evmlogger, on demand, spawns RACGEVT children. Runs as oracle. Evmd restarted automatically on failure. evmwatch program can view/format event stream. evmshow program can view/format logs. RACGEVT scans callout directory and invokes callouts. $CRS_HOME/racg/usrco. 13

14 EVMD [Event Forwarding Daemon] usrco]$ evmwatch -A RAC: ora.rac.rac1.inst: imcheck: RAC: ora.rac.rac1.inst: imup: RAC: ora.racworkshop1.asm1.asm: imcheck: RAC: ora.racworkshop1.asm1.asm: imup: CRS ora.rac.rac1.inst is transitioning from state ONLINE to state OFFLINE on member racworkshop1 RAC: ora.rac.rac1.inst: down: RAC: ora.rac.rac1.inst: imstop:

15 EVMD [Event Forwarding Daemon] Event layout: DATABASE VERSION=1.0 service=rac database=rac instance= host=workshop1 status=down reason=user timestamp=09-nov :09:19 Script can be downloaded from website rachelp.nl. 15

16 ONS [Oracle Notification Services] Node1 PMON Event takes place RACGIMON LSNR ONS DBMS_HA_ALERTS_PRVT EVMD Cliënt JDBC Middletier Node2 ONS.jar ONS ONS EVMD

17 ONS [Oracle Notification Services] ONS part of Clusterware installation. Possible to install ONS daemon on middle tiers. Runs as Oracle. ONS receives and send events to subscribers. Commandline utility onsctl. Config file for ONS located in $ORA_CRS_HOME/opmn/conf/ons.config. Trace level 9 shows events. Logfile for ons is located in $ORA_CRS_HOME/opmn/logs. 17

18 ONS [Oracle Notification Services] Configure ONS processes using RACGONS install]$ racgons -h To add ONS daemons configuration: /u01/app/crs/bin/racgons.bin add_config hostname:port [hostname:port]... To remove ONS daemons configuration: /u01/app/crs/bin/racgons.bin remove_config hostname[:port] [hostname:port] add_config workshop1:6251 workshop2:

19 ONS [Oracle Notification Services] ONS commandline utility. conf]$ onsctl usage: /u01/app/crs/bin/onsctl start stop ping reconfig debug start stop ping debug reconfig help detailed - Start opmn only. - Stop ons daemon - Test to see if ons daemon is running - Display debug information for the ons daemon - Reload the ons configuration - Print a short syntax description (this). - Print a verbose syntax description. 19

20 ONS [Oracle Notification Services] Subscribers to ONS processes ======== ONS ======== Listeners: NAME BIND ADDRESS PORT FLAGS SOCKET Local Remote Request No listener Resources: Notifications: Received: 0, in Receive Q: 0, Processed: 0, in Process Q: 0 Pools: Message: 24/25 (1), Link: 25/25 (1), Subscription: 24/25 (1) 20

21 ONS [Oracle Notification Services] Config file for ONS processes. conf]$ cat ons.config Localport=6150 loglevel=9 useocr=on allowgroup=true usesharedinstall=true 21

22 ONS [Oracle Notification Services] Logfile event trace. 08/11/09 21:16:46 [7] Connection 3, ,6150 Message content length is 87 08/11/09 21:16:46 [7] Connection 3, ,6150 Body using 87 of 87 reuse 08/11/09 21:16:46 [8] Connection 3, ,6150 body: VERSION=1.0 service=rac instance=rac1 database=rac host=workshop1 status=up reason=user 08/11/09 21:16:46 [7] Connection 3, ,6150 connectiongetmsg 22

23 RACGIMON Node1 PMON Event takes place RACGIMON LSNR ONS DBMS_HA_ALERTS_PRVT EVMD Cliënt JDBC Middletier Node2 ONS.jar ONS ONS EVMD

24 RACGIMON [Rac instance monitor] RACGIMON spawned for every instance. RACGIMON monitors instance and ASM status. RACGIMON support FAN. RACGIMON involved when srvctl is used. srvctl start database -d RAC srvctl start asm -n workshop1 24

25 RACGIMON [Rac instance monitor] RACGIMON will send events to ONS, EVMD and executes rdbms package. sys.dbms_ha_alerts_prvt To see which events are send enable racgimon tracing. Modify racgwrap in the $ORACLE_HOME/bin directory where the racgimon is active from. # Set _USR_ORA_DEBUG to enable more tracing _USR_ORA_DEBUG=1 && export _USR_ORA_DEBUG 25

26 tracing [Rac instance monitor] racg]$ grep clsrfdbe_enqueue * imon_rac.log: :21:59.413: [ RACG][ ] [3714] [ ][ora.RAC.RAC1.inst]: clsrfdbe_enqueue: TRACING: evttypname='down' type='1' resource='ora.rac.rac1.inst' node='workshop1' time=' :21: :00 card=0' imon_rac.log: :21:59.425: [ RACG][ ] [3714] [ ][ora.RAC.RAC1.inst]: clsrfdbe_enqueue: prepare post alert 0.020s imon_rac.log: :21:59.739: [ RACG][ ] [3714] [ ][ora.RAC.RAC1.inst]: clsrfdbe_enqueue: execute post alert 0.310s imon_rac.log: :21:59.739: [ RACG][ ] [3714] [ ][ora.RAC.RAC1.inst]: clsrfdbe_enqueue: called SYS.DBMS_HA_ALERTS_PRVT.POST_HA_ALERT imon_rac.log: :21:59.739: [ RACG][ ] [3714] [ ][ora.RAC.RAC1.inst]: clsrfdbe_enqueue: posted successfully: evttypname='down' type='1' resource='ora.rac.rac1.inst' node='workshop1' time=' :21: :00' card=0 26

27 RACGIMON [Rac instance monitor] srvctl stop instance -d RAC -i RAC1 RACGIMON ONS DBMS_HA_ALERTS_PRVT EVMD 27

28 Listener Listener will subscribe to ONS. Set ORACLE_CONFIG_HOME to clusterware home to solve listener subscribe issue. srvctl setenv nodeapps -n `hostname` -T "ORACLE_CONFIG_HOME= $ORA_CRS_HOME" Use onsctl debug to see listener PMON also send load information to Listener, which can send to ONS and load info send to JDBC connections pools.

29 Listener srvctl stop instance -d RAC -i RAC1 RACGIMON PMON LSNR ONS DBMS_HA_ALERTS_PRVT EVMD

30 Overview Node1 srvctl stop instance -d RAC -i RAC1 RACGIMON PMON LSNR ONS DBMS_HA_ALERTS_PRVT EVMD Cliënt JDBC Middletier Node2 ONS.jar ONS ONS EVMD

31 JDBC Clients Fast Connection Failover provides high availability to FAN integrated clients. (JDBC,OCI,ODP.NET) JDBC FCF clients can deal with FAN events. Configure clients to enable FCF, which will automatic subcribe to ONS for FAN events. Can react on Database up/down events. Oracle provides an connection to a active service/ instance.

32 Configure JDBC CLient Configure JDBC FCF. ods.setfastconnectionfailoverenabled(tr ue); Export CLASSPATH=ojdbc5.jar;ons.jar;. Set ONSCONFIG to server ONS. onsconfig=nodes=workshop1:6251,work shop2:6251

33 Overview Node1 srvctl stop instance -d RAC -i RAC1 RACGIMON PMON LSNR ONS DBMS_HA_ALERTS_PRVT EVMD Cliënt JDBC Middletier Node2 ONS.jar ONS ONS EVMD

34 The Power A lot of functionality is already in place. Always get informed when an event occurs. Use the server side callout based on business needs. Build your own events. The power depends on what you do with it.

35 Additional Bug (ocrnodename=xxx full qualified name) White paper Client Failover Best Practices for Highly Available Oracle Databases Presentation oracle jdbc high availability load balancing best practice and road map. (Kuassi Mensah) Metalink note: Metalink note: Metalink note:

36 Questions? / 36