PTC System Monitor Solution Training Patrick Kulenkamp June 2012
Agenda What is PTC System Monitor (PSM)? How does it work? Terminology PSM Configuration The PTC Integrity Implementation Drilling Down and Analyzing Data Release Timelines Questions/Discussion 2
What is PTC System Monitor (PSM)? PSM is a custom solution that can be split into two parts: The core dynatrace monitoring application. The custom Integrity Solution (Fastpack) that has been built on top of it The dynatrace Product: Centralized multi-server system monitoring Out of the Box monitoring for generic servers Out of the Box notifications for common incidents The Customized Integrity Solution: Custom hooks into the PTC Integrity product Custom notifications for core Server components Custom notifications for end user operations Custom actions for when incidents occur Powered By 3
Why dynatrace? Scalable architecture Can monitor multiple servers at the same time. Customizable & modular configuration Custom plugins/scripts. Custom dashboards, incidents, sensors. Wholistic monitoring of application, system and environment End to end transaction visability (Database, OS, JVM) Strings all data together for quick root cause analysis Always-on, low overhead and in-depth monitoring Self managing overhead Real-time and interactive data analysis Heap and Thread Dump analysis Instant PurePath analysis and comparison 4
dynatrace Components Separate monitoring server with 5 main components: Server Agent(s) Collector(s) Analysis Server (Optional) Client (Installed or Webstart) Server Stand alone dedicated application server that requires a good physical server. Real time analysis and reporting. Requires an enterprise database referred to as the Performance Warehouse. *The Performance Warehouse only stores Time Series data. Agent(s) Lightweight, compact and native object that is deployed on each server that is to be monitored and it resides within the application JVM. Uses very little memory (under 50MB) and managed CPU overhead (under ~2%). Auto upgrades. Communicates directly with the Collector. 5
dynatrace Components Collector(s) Collects all data from Agents and sends it to the server. Not lightweight but not as big as the server. Does not necessarily need a dedicated machine. Must be local to all Agents that are being monitored. Can be remote from the DT Server (WAN). Install a collector for each geographical location of servers being monitored. Analysis Server (Optional) Used to analyze proprietary dynatrace heap and thread dumps. If not analyzing these dumps this server is not needed. Customers don t need to install this. More for Tactical and Dev. Client Thick Java client that can be installed locally or deployed from the DT Server webpage on demand via Webstart. Exact same client whether its installed or deployed. Auto upgrades for minor releases. There is also a WebUI directly on the server but this has limited reporting functionality. 6
Solution Overview Simple dashboards with drill down capabilities Instant Notifications/ Actions Custom Scripts dynatrace Server dynatrace Client Collector Database Administrator Performance Warehouse Always On Monitoring Integrity Servers with embedded Agents Time Series Data only 7
Initial Configuration Server Settings Allocate at least 4G to the server memory, preferably 6G. Disable embedded collector since it is not recommended for production use. Setup SMTP settings. Verify that the separately installed collector is up and running and connected to the server. Setup the database connection (Oracle or SQL are recommended). Setup LDAP (if applicable) and user/group management. Defined appropriate permissions. Enable Continuous Session Recording and give it a quota of at *least* 20G. The more you allocate to the quote the further back in time you can analyze PurePaths. If this is not enabled all PurePaths are stored in the DT Server memory and the retention will be very low! Define hosts that will be monitored and put them in the appropriate Host Groups. Note that the stand alone Collector by default uses 2G memory. Profile Settings Rename the default System Profile so future FastPacks do not overwrite it. Define custom Incident thresholds to fit your environment For example the default Warning threshold for an Item Edit transaction is 3 sec. This may need to be increased. Configure appropriate Monitors Windows Performance Monitor Unix Monitor Oracle Monitor 8
Terminology Hierarchy Quick overview of the dynatrace Terminology Hierarchy Sensors Incidents Fast Pack Profile License Plugins Measures/ Thresholds Business Transactions Agent Mappings Tasks/ Monitors Dashboards Dashlets Dashboard Filter 9
Terminology FastPack Container made by PTC Integrity that contains the entire Integrity implementation for dynatrace. Installs like a normal plugin. Requires a dynatrace server restart. Includes the following: License System Profile Business Transactions Measures/Thresholds Incidents Sensors Tasks/Monitors Dashboards Plugins 10
Terminology Dashboards/Dashlets Dashboards contain dashlets. Dashboards are a similar concept to Viewsets in Integrity. Can be locked down view permissions. Can be opened from the Dashboard menu or from right clicking in a Dashboard and selecting Show in Dashboard. 11
Terminology Profiles/Configurations The profile contains all the Sensors, Incidents, Measure definitions/thresholds. Many Agents can map to a single Profile if they share the same thresholds. Configurations can enable and disable sensors on the fly. Example: Integrity Debugging Configuration. 12
Terminology Pure Paths Start to Finish transaction recording. Can cross multiple Agents to built a full end to end transaction Captures method calls, return/argument values Captures method execution/suspension times. SQL Query Details 13
Terminology Business Transactions Logical grouping of PurePaths Can be used as a basis for Incidents Enhances visibility into usage of the product Can filter out PurePaths based on criteria Can be used as chart measures - We have defined a number of key Out-of-the-Box Business Transactions 14
Terminology Custom Business Transactions Filter What PurePaths should be grouped into this BT Example: Invocation of Item Edit Calculations What additional calculations do you want? Example: Amount of time spent in triggers Splitting How do you want the data split or grouped? Example: user name, operation, type, etc Tips Be mindful of performance Set aggregate to first where possible Minimize calculations and splitting Always split by Agent 15
Terminology Business Transactions (Example) Example of the Item-New Business Transaction. Here you can see who created how many items and sort by Avg, Max, etc times. 16
Terminology Measures Single data point that is captured from either a PurePath or Monitor Can define Thresholds levels for each measure. The server will store measure data for each agent in the DB as Time Series data Can capture method return value, argument values, exec time, etc. Can base it off Invocation of a certain method 17
Terminology Measures - Resolution Duration Measures will get averaged out and purged over time. This is referred to as Resolution Duration High Resolution equates to minute by minute data Medium Resolution equates to hour by hour data Low Resolution equates to day by day data Can be configured to whatever resolution is desired - Settings -> dynatrace Server - > Performance Warehouse -> Storage Management 18
Terminology Incidents An event that you would like to be notified on or take action on. Define incident criteria based off one or more Measures/Business Transactions. Notifications include pop-ups in the DT client, emails, etc. Actions can include: Thread Dump Heap Dump dynatrace Session Dump Integrity API Execution Generic Command Execution (script) More... 19
Terminology Monitors/Tasks Monitors can be configured to execute on remote machines on a scheduled basis and store the resulting data in measures that can be tracked over time and used as a data point for Incidents. Tasks can be configured to execute on remote machines on a scheduled basis and execute a plugin/task. This does not capture any measure data however. 20
Terminology Sensors Define entry points in the code for dynatrace to start tracking. Define return argument values to capture. Be cautious when creating sensors as they can have a lot of overhead. Sensors require a restart of the Agent(Application Server) or a Hot Sensor Placement(HSP). Be cautious when performing a Hot Sensor Placement in Production. Recommend after hours. Never use HSP to /change/ a sensor, only add new ones. Changing a sensor requires complete rewrite of bytecode. 21
Terminology Sensor Packs Sensor Packs are a collection of Sensor Groups and Sensors. Out of the Box there are a number of Senor Packs that are enabled that collect valuable information 22
Terminology Sensor Packs/Group Configuration Some Sensor Packs and Sensor Groups can be configured Be cautious when changing these properties as they could have a large overhead For example extending the SQL capture length past 8192 could have a negative performance effect. Sensor Groups/Packs can be set as active, active and start pure paths and inactive at the Configuration level. 23
Terminology Agents/Agent Groups The actual application servers that are being monitored are referred to as Agents. These servers have the Agent component installed/embedded in them. By default Agents automatically map to the default system profile. Example Agent Config Line (mksservice.conf): mks.java.additional.30=-agentpath:"c:/progra~1/dynatrace/dtagent/agent/lib64/dtagent.dll"=name="integrity_server",server=dtserver:9998 24
Terminology Sessions Contains all timeseries and PurePath data for a specific timeframe. Automatically generated via an Incident or manually captured. Can be exported to a single file and sent into support. Stored on the dynatace server filesystem, not in the database. 25
Terminology Cockpit Starting point to open up any individual dashlet or view. By default always located on the left in every view. Can access Overview, Profiles and Sessions from it. Not Dashboards. 26
Terminology Start Center Opens every time you state the dynatrace client Alerts you of various configuration options that should be set/changed Only place to load built in dynatrace server monitoring dashboards 27
Terminology Filter/Status Line Global data source filter (Unless overridden at the dashlet level) In the example below: Integrity Overview is the Dashboard PTC Integrity Server is the System Profile From last 1 hour is the timeframe Business Transaction Item New is a custom filter for that specific data Tip: Not seeing the data you expected? Check the filter! 28
Terminology Hierarchy Quick recap of the dynatrace Terminology Hierarchy Sensors Incidents Fast Pack Profile License Plugins Measures/ Thresholds Business Transactions Agent Mappings Tasks/ Monitors Dashboards Dashlets Dashboard Filter 29
PSM Integrity Dashboards Out of the Box Dashboards The following is a list of Dashboards that come Out of the Box Dashboards can be customized, locked down, copied, etc 30
PSM Integrity Dashboards Integrity Overview Default dashboad. Displays status of all servers associated with the profile. Meant to be used as a central hub to drill into sister dashboards 31
PSM Integrity Dashboards Integrity Overview Cont d The Status Icon for each server is based on an aggregate of a set of defined Incident Rules. To see the Incident Rules simple hover over the status icon. 32
PSM Integrity Dashboards Integrity Overview Cont d To drill down from the main Integrity Overview dashboard into another one simply right click on the Agent and select Show in Dashboard and then select the dashboard that you would like to drill down into. Doing this will automatically filter the new dashboard by the Agent that you had selected. 33
PSM Integrity Dashboards Integrity Server Displays important Integrity Server data such as Memory and CPU usage Incident, Errors and Database queries are listed at the bottom. 34
PSM Integrity Dashboards Integrity Host Server Displays important Host Server data for the Integrity Server such as Over all Memory and CPU usage All data is captured from either the Windows or UNIX Monitor that must be preconfigured. 35
PSM Integrity Dashboards Integrity Usage Displays important Integrity Usage data such as Create and Edit Item times. 36
PSM Integrity Dashboards Source Usage Displays important Source Usage data such as Project Checkpoint and SubmitCP times. 37
PSM Integrity Dashboards Oracle Dashboard Displays important Oracle Metric information such as Cache Hit Ratios, System Waits, Concurrent Sessions, etc. All data is captured from the Oracle Monitor that must be configured. 38
PSM Integrity Dashboards MS SQL Server Dashboard Displays important MS SQL Metric information such as Buffer Page reads/writes, User Connections, Transactions, etc. All data is captured from the SQL Server Performance Monitor that must be configured. 39
Built in dynatrace Dashboards dynatrace built in Dashboards Can only be accessed from Start Center -> Monitoring Dashboards for monitoring DT Server, Collectors, Performance Warehouse, etc 40
Built in dynatrace Dashboards Example: dynatrace Server Health Displays important metrics pertaining to the actual dynatrace Server CPU Usage, Memory Usage, JVM Suspension Time, etc 41
Other dynatrace Views Database Queries Here you can see the top offending SQL queries in a descending list. 42
Other dynatrace Views Errors Overview Here you can all the errors that were thrown on the server whether they were handled or unhandled. Can sort by any of the columns below and drill right into the PurePaths. 43
dynatrace Tools CPU Sampling Can be ran on demand for a specific Agent or trigger via an Incident. Polls the JVM for thread stacks at an optional interval until you cancel it and stitches them all together. Great for determining where to set sensors Can be used to find problems for which the sensors are not catching - Tip: Sort by Total CPU Time and select view by Thread name (Highlighted with red box) 44
dynatrace Tools Heap Dump Analysis (Total and Selective) Total Memory Dump Similar to a Java heap dump but in dynatraces proprietary format. Expensive pause the world operation. Selective Memory Dump Only captures JVM Heap Data for which Memory Sensors are defined. We have not defined any Memory Sensors so this will generate no data. - All heap dumps are analyzed via the dynatrace Analysis Server 45
dynatrace Tools Thread Dump Similar to a normal JVM thread dump but in dynatraces proprietary format. Can be collected manually or via an Incident trigger. Not to be confused with CPU Sampling. 46
Analyzing Data Examples High Application CPU Usage This example will go through a situation where a High Application CPU Usage Incident was triggered *Sort PurePaths by CPU Sum Don t forget about the Thread dump! 47
Analyzing Data Examples High Application Memory Usage This example will go through a situation where a High Application Memory Usage Incident was triggered *dynatrace does not capture memory usage data per transaction. Sorting by Duration may give you some hints but Memory Dump is the only definitive way to determine what caused the spike. 48
Analyzing Data Examples Slow Business Transaction This example will go through a situation where a Slow Business Transaction Incident was triggered Here you can see the JDBC API is the main culprit Actual database queries that contributed to the slowness 49
Gathering Support Data Data to send in to PTC Integrity Support Exported Session dump Right click on the session and select Export. Always export PurePaths and Time Series data. Avoid removing confidential Strings as this will strip SQL, user names, etc. Any Thread or Heap Dumps if applicable (These are NOT contained in the Session Dump). dynatrace Support Package System Information -> Right click on DT Server and select Package All Included Files 50
Questions/Discussions 51