Beyond the Basics: Making the Most of Oracle Enterprise Manager 12c Monitoring

Transcription

1

2 Beyond the Basics: Making the Most of Oracle Enterprise Manager 12c Monitoring Ana McCollum Director of Product Management Oracle Enterprise Manager September 29, 2014

3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

4 Enterprise Monitoring 3 rd party systems Planning Monitoring Set up Managing Events/Incidents

5 Enterprise Monitoring: Basics Planning Monitoring White paper on OTN: Strategies for Scalable, Smarter Monitoring using Oracle Enterprise Manager Cloud Control 12c ( Monitoring Set up Automate, standardize monitoring using Administration Groups, Template Collections Use Incident Rule Sets to automate notification, incident management processes Managing Events/Incidents Extend monitoring using Metric Extensions Enterprise Manager Administration Guide 12c

6 Enterprise Monitoring: Beyond the Basics Planning Monitoring Set up Metric Alerts Host Status Monitoring Event Rules and Notifications Managing Events/Incidents

7 Metric Alerts Metric Alert: generated when metric crosses threshold Scenario 1: Alert after metric is above threshold for sustained period Dynamic duo: Collection schedule + No. of Consecutive Occurrences Alert generated: metric is above threshold for N consecutive collection intervals Alert cleared: metric is below thresholds for N consecutive collection intervals Example: Collection schedule: 15 minutes, No. of Consecutive Occurrences: 3 Alert when metric crosses thresholds 45 minutes (15 minutes x 3 consecutive occurrences) Usage Tips: Adjust collection schedule to appropriate frequency (more frequent for rapidly changing values, less frequent otherwise) Set No. of Occurrences to avoid premature alerts

8 Alert After Sustained Period Time before alert is triggered is calculated for you based on Collection Schedule + Number of Occurrences 2 x 5 ==== 10 minutes

9 Database Monitoring The Fine Print Database monitoring leverages DB server generated alert mechanism Database detects condition and generates alerts For metrics that rely on DB server generated alerts: DB observation period (defaults to 2 minutes) + Number of Occurrences determines when alert is generated Example: No. of occurrences=3, alert generated after 6 minutes (3 x 2 minutes) Applicable to Enterprise Manager and earlier, DB Plugin and earlier Enterprise Manager and DB Plugin New in R4 Database metrics that support user-specified thresholds cutover to Enterprise Manager metric alerting Metric semantics now the same for these subset of database metrics (Table 1) Enterprise Manager collection schedule, number of occurrences used for alerting Requires DB Plugin deployed to OMS and agent. If not deployed to agent, previous semantics used. Subset of database alerts do not require thresholds, database detects condition and sends alert to Enterprise Manager (Table 2)

10 Table1: Database Metric Groups cutover to Enterprise Manager metric system for alerting New in R4 Global Cache Statistics Instance Throughput Instance Efficiency Database Resource Usage System Response Time Per Call Tablespace Full Database Files Database Services SGA Pool Wastage Wait Bottlenecks User Block Wait Class Sessions

11 Table2: Database Metric Groups that Continue to Use Server-Generated Alerts (no user specified thresholds) Availability (Data Guard Failover) Suspended Session Recovery Area Snap Shot Too Old Database Replay Database Replay Client Stream Capture Streams Apply Streams Propagation Streams Pool Usage

12 Tablespace Metric Metric Group Tablespaces Full Tablespace Free Space (MB) Tablespace Space Used (%) Cutover to Enterprise Manager metrics system for alerting Automatically excludes monitoring of UNDO and TEMP tablespaces Usage Tip: No need to do anything if you use it to monitor non UNDO/TEMP tablespaces Two new Metric Groups added to support UNDO and TEMP monitoring New in R4 1. Tablespaces Full (Temp) Tablespace Free Space (MB) (Temp) Tablespace Space Used(%) (Temp) 2. Tablespaces Full (Undo) Tablespace Free Space (MB) (Undo) Tablespace Space Used(%) (Undo)

13 Database Monitoring: Usage Tips If using Enterprise Manager and earlier, you can control alerting using Number of Occurrences + 2 minute observation period Example: Would like alert if condition is exists for at least 10 minutes, set Number of Occurrences = 5 (2 minutes x 5 = 10 minutes) If using Enterprise Manager and DB Plugin (must be deployed to agents) (Upgrade): If relying on DB 2 minute observation period, might need to adjust collection schedule/occurrences in Enterprise Manager to control alerting. Always use Enterprise Manager to set thresholds. DO NOT use DBMS_SERVER_ALERT to set thresholds, they will NOT be used by Enterprise Manager.

14 Time-based Static Thresholds New in R4 Scenario 2: Different workloads for day vs. night, thus different thresholds needed based on time Example: metric to monitor backlog of work queues Day: End users are generating workload. Low thresholds to alert right away if backlog starts. Night: Batch processes generate workload. High thresholds are ok. Use Time-based Static Thresholds Specify different static (fixed) thresholds based on time (target time zone) Day and Night (Day: 7:00 am 7:00 pm; Night 7:00 pm 7:00 am) Weekday (Mon-Fri) and Weekend (Sat-Sun) Day and Night, over Weekday and Weekend

15 Adaptive Thresholds New in R4 Scenario 3: Alert when target deviates from normal behavior, adjust to target workload Challenging to set meaningful numeric values especially for performance metrics (e.g. Number of Transactions per second) Easier to specify threshold calculations related to normal (baseline) behavior Thresholds automatically adjust for expected workload changes Lower thresholds during weekend Higher thresholds during work week

16 Setting Adaptive Thresholds 1. Specify threshold frequency change (workload usage profiles on target) By Day and Night or By Weekdays and Weekend, etc. 2. Specify how much data to collect for baseline (moving window) Trailing Days Automatically recommended based on threshold frequency change 3. Select metrics and threshold settings

17 Two Kinds of Adaptive Threshold Settings Significance Level Used to alert on unusual values of the metric Metric statistically stable during good target performance, unusual metric values signal poor performance Thresholds based on percentiles Example: Critical threshold =.99 percentile Alert when metric value is 99 th percentile of that time period Percentage of Maximum Used to alert on peaks in workload Assumes target is sized for peak workloads, high metric value indicates workload at peak Thresholds based on user-specified percentage X maximum value (.99 percentile value) Example: Critical threshold = 150% x.99 percentile value Default: Significance Level, Warning:.95 Percentile Critical:.99 Percentile, Occurrences: 2 Usage tip: Start with defaults if you re not sure. Verify using Analyze Thresholds page.

18 Adaptive Thresholds Analyze Thresholds page helps you understand workload pattern for target, review threshold settings Higher metric values during the week Low metric values during weekend Lower thresholds during weekend Higher thresholds during weekdays (work week)

19 Database Adaptive Threshold Metrics New in R4 Database metrics that supported DB Adaptive Thresholds have been cutover to Enterprise Manager metric framework Use Enterprise Manager adaptive threshold feature Backwards compatibility for DB adaptive thresholds) New Metric Group Server Adaptive Threshold Metric containing all DB metrics that support DB adaptive thresholds New alerts from the database will go to the new metrics Usage tip: Before upgrading DB plug-in on agent, make sure open alerts on DB server adaptive metrics are cleared Update event rules to include new metrics to continue to receive notifications

20 Analyzing Metrics: Which target types consume most storage? Are you getting the right metric storage return on investment (ROI)? Setup-> Manage Cloud Control-> Repository -> Metrics New in R4 Each bubble: storage consumed by all metrics for a target type Biggest bubble: Target Type that consumes most storage More number of collections: more frequent collection schedule More number of rows: metric has many key values Clicking on bubble gives you drilldown

21 Analyzing Metrics: Drilldown of Target Type Bubble Option 1: Show By Top 25 Metrics For a specific target type, top 25 metrics that take most storage Biggest bubble is metric that uses most storage for that target type To reduce storage: Less frequent collection schedules Disable collection if metric not used Use Alerting only option

22 Analyzing Metrics: Drilldown of Target Type Bubble Option 2: Show By Top 25 Targets For a specific target type, top 25 targets that take most storage Biggest bubble represents target that consumes most storage Reduce storage for specific target: Disabled unused metrics Reduce collection frequency Use Alerting only option

23 Analyzing Metric Alerts New in R4 Am I keeping up with managing metric alerts? Setup-> Manage Cloud Control-> Repository -> Metrics Watch trend of alert backlog (blue line) Straight / downward trend: keeping up with alerts To manage alerts: Review 20 Most Comment Alerts information publisher report Review alert settings If metric alert not of interest, remove thresholds/disable metric

24 Customizable Alert Messages New in R4 Alert messages can be customized Change format/content to align with internal datacenter standards WYSIWYG: Message will be sent as is, no language translation Alert context can be embedded in the alert message

25 Correlating Metrics New in R4 All Metrics Choose Metric Option in metric chart Problem Analysis Better insight into target health by visually correlating metrics (workload, performance, etc.) within same time period Correlate different metrics for a target or against different targets

26 Repository-side Metric Extensions New in R4 Metric Extensions (ME) that query data in the repository Alert based on multiple metric alerts on the same target Alert based on multiple metric alerts on different targets Metric / Alert is associated with the target on which metric extension is deployed Guidelines: SQL Query only (no semi-colons at the end) Include TARGET_GUID as first column in the query Query on management repository views only Query will execute as MGMT_VIEW user Check instruction text for additional details and examples

27 Host Status Monitoring

28 Enhanced Host Status Monitoring New in R4 Prior to , host down detected via Agent Unreachable with specific message indicating host is down New Host Down Status Requires Partner Agent Partner Agent: Agent that, in addition to its agent duties, has also been assigned to monitor another agent and its host Automatically assigned by Enterprise Manager based on network proximity, can be overridden by EM CLI verb manage_agent_partnership Partner Agent is shown in agent homepage If host is down, partner agent detects, sends this to OMS Host 1 Agent Host 2 Partner Agent OMS

29 Host Down Scenario Status in Console Target Availability event (availability state) Agent Down Down (fatal severity) Host Down Down (fatal severity) Comments Some pages show Down: Target homepage, Availability page, All Targets. Other pages show Agent Unreachable Some pages show Down: Target homepage, Availability page, All Targets. Other pages show Agent Unreachable Monitored Target Agent Down Agent Down (warning severity) Some pages show Agent Down: Target homepage, Availability page, All Targets. Other pages show Agent Unreachable

30 If Partner Agent is not available Fallback to pre-enterprise Manager behavior Agent and Host down detection will use these semantics: OMS expects agent heartbeat every 60 seconds If agent does not provide 2 heartbeats, OMS does reverse ping to agent. If no response, also pings its host. Based on results of ping, Agent, Host and all monitored targets are marked as agent unreachable with message indicating if agent is unreachable or host is unreachable

31 Event Rule Usage Tips: sysadmin or host administrator Rule for Host target down Event rule: Target Availability event Target Type: Host Choose Down and Agent Unreachable Either: Down event (fatal severity) triggered if partner agent exists OR Agent Unreachable event (warning severity) triggered if no partner agent Event clears when host and agent are back up

32 Event Rule Usage Tips: sysadmin or host administrator Rule for Host not being monitored With partner agent, can detect scenario where agent is down, host is up Host is up, but no other aspects of host is being monitored (since agent is down) Event rule: Target Availability event Target type: Host Choose Up but unmonitored (warning severity) Event clears when agent is back up

33 Event Rule Usage Tips: target administrators (DBA, MW Admin) Rule for monitored Target Down Event rule: Target Availability event Target type: Database WLS, etc. Choose Down (fatal) Event clears when target is back up

34 Event Rules and Notifications

35 Incident Rule Sets Usage Tips Use Groups as targets for Incident Rules Targets with common notification requirements put together in a group All actions (incident management, notification) for the group should be within same incident rule set Once custom rule sets are defined, remember to disable out-of-box rule sets to avoid unwanted side effects Out-of-box rule sets meant to illustrate best practices, provide examples. Might change in future. Create-like on out-of-box rule sets and use these for your rules notifications: Leverage To vs. cc recipients To : mandatory recipient, he cannot remove himself as recipient Usage Tip: Use To recipients to enforce operational practices cc : optional recipient, he can add/remove himself as recipient

36 Event Rule Simulation New in R4 Setup Incidents Incident Rules Simulate Rules Verifies rule set up: Will the expected actions occur if the event occurs? Step 1: Search and select event on a target Or can substitute another target (useful if event hasn t happened yet on this 2 nd target) Example: generate event on Test target, simulate on Prod target Step 2: Start Simulation. Rule engine will evaluate all rule sets as if the event occurred now. Step 3: Review results: actions to be taken if the event occurred now. Actions will not actually take place. Unexpected results: No incident creation: rule owner must have Manage Target Events on target No user must have address setup, rule filters may be incorrect Refer: Troubleshooting section in Setting Up Rule Sets (Enterprise Manager 12c Administrator s guide)

37 SNMP Traps Notification method: common way to send events (not metric data) to third party management systems One type of trap for ALL event types and target types Variable oraemngeventtype indicates the event type (e.g. metric_alert, target_availability, job_status_change, etc.) Variables oraemngeventtypeattr1, oraemngeventtypeattr2, etc. contain event type specific information (e.g. metric id, metric name, etc.) Details: Appendix A Interpreting Variables of the Enterprise Manager MIB, Enterprise Manager 12c Administrator s Guide Variable oraemngeventsequenceid Unique identifier that can be used to relate events that are part of the same event lifecycle

38 Event Sequence Event sequence - sequence of related events that represent the life of a specific issue from the time it is detected and an event is raised to the time it is fixed and corresponding clear event is generated. Event Sequence #1 Event Sequence #2 Time CPU Utilization CPU Utilization CPU Utilization CPU Utilization CPU Utilization 9:00 am 5:00 pm Event sequence ID hexadecimal identifier for the sequence Usage tip: You can use Event Sequence ID to manage lifecycle of events in third party management systems by correlating events with same event sequence ID

39 SNMPv3 Traps New in R4 Notifications on events can be sent using SNMPv3 traps SNMPv3 SNMPv3 offers security Authentication: message is from a valid source Message integrity and encryption: ensures packet sent has not been tampered with Enables datacenters to comply with security best practices

40 SNMPv3 Trap Setup Before Setup: After Setup: Guided setup through the console 1. Enable (each) OMS to send traps (port, engine ID) Usage tip: Each time you add an OMS, remember to enable the OMS to send traps so sending of traps is distributed across OMSs 2. Configure SNMPv3 Notification Method Define User Security Model (USM): name, authentication protocol, privacy protocol, passwords Define SNMP receiver host and port Important: Make sure USM credentials are defined in the trap receiver 3. Test Notification Method is working Involves sending trap from Enterprise Manager. Verify trap is received in 3 rd party system. 4. Use notification method in event rules

41 Monitoring Enhancements and Best Practices 3 rd party systems Scalable, smarter monitoring Flexibility in supporting wide variety of target alerting requirements Enables IT to provide high quality service

42

43