Adobe Marketing Cloud Data Workbench Monitoring Profile

Transcription

1 Adobe Marketing Cloud Data Workbench Monitoring Profile

2 Contents Data Workbench Monitoring Profile...3 Installing the Monitoring Profile...5 Workspaces for Monitoring the Data Workbench Server...8 Data Workbench Historic workspace...8 Data Workbench Profile Status workspace...11 Data Workbench Server Status workspace...13 Data Workbench Profile Dimensions and Metrics...16 Dimensions in the Data Workbench Historic profile...16 Dimensions in the Data Workbench Profile Status profile...20 Dimensions in the Data Workbench Server Status profile...22 Metrics in the Data Workbench Historical Monitoring profile...25 Last updated 10/16/2014 Data Workbench Monitoring Profile

3 Data Workbench Monitoring Profile 3 Data Workbench Monitoring Profile The data workbench Monitoring Profile allows administrators to monitor the general health of data workbench servers at defined intervals, report errors in real-time, and optimize data workbench load capacity and processing schedules. By providing support and monitoring capabilities for data workbench servers, the data workbench administrator can better optimize server load and comply with enterprise standards for data workbench analytic services, including features to Provide proactive, application-level support to manage data workbench server clusters. Increase transparency and awareness of the server state in order to optimize scheduling and process capacity. Furnish stakeholders with improved SLA reporting and compliance. The data workbench Monitoring Profile uses native data workbench functionality to analyze and report the health of data workbench servers. It includes an application-level monitoring agent and several profiles that provide feedback detailing performance, capacity, and current status. The agent is invoked periodically (default is ten minutes) on each host running a data workbench server and captures performance data for both FSU (file system) and DPU (data processing) servers within the data workbench server cluster. The captured event data is then saved to a VSL file and read into data workbench profiles in real-time to generate Monitoring datasets. These profile-based and server-based profiles populate historical and real-time reports to view within data workbench Client and data workbench Report. In addition, if the agent identifies significant system errors, it will immediately deliver an to the administrator. Note: It is required that the Sensor and Monitoring profiles be set up on a separate non-production server or an internal FSU or DPU server running on another port. The Monitoring profile requires minimal processing and I/O capabilities. alert types DOWN data workbench server unresponsive

4 Data Workbench Monitoring Profile 4 CRITICAL data workbench component not OK WARNING Profile not accurate UP/OK Upon recovery the agent generates corresponding . Why is an agent required for monitoring data workbench? The data workbench Monitoring agent parses the Detailed Status page where some values are valid only under certain server "states" or conditions, which requires an agent to accurately determine the services running and the validity of capturing information during certain periods. For example, some values make sense to harvest only under certain conditions such as making sure not to estimate performance metrics during a fast merge or other times when data is being loaded and a query is not advised. Likewise, some values must be measured between polling times to accurately identify the elapsed time for a fast input or a fast merge. In addition, in an environment where there are many servers to monitor, a distributed polling scheme requires a guaranteed polling interval provided by an installed agent. Accommodating complex, reflexive logic for an data workbench server cluster requires a monitoring agent deployed on each server.

5 Installing the Monitoring Profile 5 Installing the Monitoring Profile Directions for installing the data workbench Monitoring Profile. Installation Steps Check the Installation Notes below for additional information when installing. 1. Configure a new Sensor instance as if it will be used for tagged web page data collection. Be sure the zig.gif file is in the Sensor web server document root. Sensor can be run on the same host as the monitor profiles. (This is not an issue if using a text file for this purpose.) Note: This Sensor instance must be dedicated to receiving only traffic from the Monitoring Agents. Also, the Sensor can be configured to run on a different port if you are re-purposing a web server for this collection. 2. In the txlogd.conf file there is the default line: ContentFilterExclude image/,text/css,application/x-javascript,text/javascript For the data workbench Monitoring Profile application (or any "tagged" page implementation), the image type has to be removed in order to collect via a GIF file. The updated line is: ContentFilterExclude text/css,application/x-javascript,text/javascript 3. Copy the insight_monitor.zip/insight_monitor_agent to a temporary location. 4. Update insight_monitor_agent.cfg file for your environment. Follow the comments inside the configuration file: The Monitoring configuration file: Define where you are collecting all information and provide URL address. This needs to be a dedicated sensor and should receive no traffic except for this application. There are paths assuming there is an e: disk. You may want to change this path for your environment. Sometime when running a Transform profile, data workbench can be unresponsive. This value lets you send an alert if three times in a row the process is unresponsive. This is a way of reducing false positive alerts. This is where you set the environment and group dimensions. This may be different from host to host. This is where you can see exactly what the monitor agent is doing by viewing error logs in this path.

6 Installing the Monitoring Profile 6 This is to use the temp db internally. It may be alerted when reaching capacity. This is different than physical disk usage. 5. Copy the insight_monitor_agent folder to each DPU and FSU host running the data workbench server. The default location as indicated in the configuration file is e:\insight_monitor_agent but you can change this location. 6. Add a Windows scheduled task to invoke the agent every 10 minutes (this period is assumed in the processing rate calculations). The program is e:insight_monitor/insight_monitor_agent.exe. The argument is config-file e:\insight_monitor\insight_monitor.cfg. Start in e:\insight_monitor. The user running executing the task must have permission to read/write e:\insight_monitor and read the Win32 OLE object root\cimv2 (required to ascertain the data workbench server service start mode and to check the percentage of space on local disks) 7. Confirm that the VSL file is starting to grow as monitor records accumulate. This will take some time as the traffic volume will be extremely low in a small installation (every 10 minutes the agent sends only one hit for the host-specific data, plus one hit per processing profile). 8. Unzip insight_monitor.zip\profiles\insight Historic to a temporary location. 9. Update host name in profile.cfg, dataset\cluster.cfg, and the dataset\segment export.cfg. 10. Update the files to the data workbench profiles directory. 11. Update log server and path in dataset\log processing.cfg to the location where the Sensor VSLs are accumulating. 12. [Optionally] do the same with the profiles Insight Profile Status and Insight Server Status. In addition, the status profiles should be reprocessed nightly with a trailing two day window. Add a Windows scheduled task: The program is e:\insight_monitor\insight_reprocess.exe. The argument is --profile-path="path TO PROFILES\insight profile status" --start-days-ago=2. Leave start in blank. Add another scheduled task for "insight server status". insight_reprocess.exe requires read/write access to log processing.cfg to update the start time. 13. In addition, the status profiles should be reprocessed nightly with a trailing two day window. Add a Windows scheduled task: The program is e:\insight_monitor\insight_reprocess.exe. The argument is --profile-path="path TO PROFILES\insight profile status" --start-days-ago=2. Leave start in blank. Add another scheduled task for "insight server status". insight_reprocess.exe requires read/write access to log processing.cfg to update the start time. Confirm that each profile is reading the monitor VSLs as they accumulate. Again, this will take some time probably hours because of the extremely low volume. Installation Notes Configuring the Monitoring Profile in a licensed test environment. The test environment package is included with your implementation of data workbench, allowing you to install and configure the application. If installing on a production FSU or DPU server, then you will need to configure the server to run on a separate port. Deploying a new Sensor specifically for the Monitoring Profile. You will need to install a new instance of Sensor to the server running the Monitoring Profile. This is in addition to the production instance of Sensor. (There is no additional charge for installing Sensor on either a production or non-production server specifically for the Monitoring Profile.) Disable the monitor agent during data workbench maintenance. To avoid polluting the uptime and performance metrics, you can set service start mode to manual for the service InsightServer (Omniture Insight Server). A handy PowerShell command is set-service -name insightserver -startuptype manual. Set it back to automatic after the maintenance: set-service -name insightserver -startuptype automatic. Another option is to temporarily disable the monitor agent scheduled task. The Status profiles need a trailing window to drop old hosts and profiles as well as old host-profile mappings. However, if the amount of event data is so small that data workbench won't buffer it in, then you may need to extend the size of the window quite a bit to get it to process. The agent collects the overall and oldest as-of time from data workbench detailed status, which is reported in local host time assuming the event data log time stamps are in UTC (as in VSL files). If the event data timestamps are in a non-utc

7 Installing the Monitoring Profile 7 time zone the as-of time will be offset in the resulting Insight Profile Status profile. If all of your event data time stamps are in the same time zone you can add that offset to Insight Profile Status\metrics\as of delay minutes.metric. Two new dimensions were introduced to help the customer group their servers if they are in different states, such as production, staging, testing servers, and servers in other states. For example if you are looking for "uptime", then you look at servers only in production mode. As a result, the Group dimension is just another way of arbitrarily grouping servers for your needs. For example, in themonitoring Configuration file you can set which host your department is servicing, such as Operations, Development, or Marketing.

8 Workspaces for Monitoring the Data Workbench Server 8 Workspaces for Monitoring the Data Workbench Server To successfully identify server health and performance, you can use standard data workbench profiles to monitor the server from the installed agent using current data, or employ profiles of historic data sets to view the impact of performance changes over time. The data workbench workspaces most commonly used include: Insight Historic workspace Insight Profile Status workspace Insight Server Status workspace To select a profile, open the drop-down menu from the upper-left corner of the data workbench Client interface. Data Workbench Historic workspace Use the data workbench Historic profile to see how configuration, hardware, and other changes impact performance, stability, and server capacity over time. The Historic profile includes a profile-based Profile Performance dataset and the server-based Server Performance dataset under the Performance tab. These are the most commonly used datasets viewed for a past perspective of the data workbench server performance. In addition, you can view the Components and Processing Mode by selecting the Up Time tab. In addition, you can view the Components and Processing Mode by selecting the Up Time tab.

9 Workspaces for Monitoring the Data Workbench Server 9 For additional reference information about the dimensions used in the data workbench Historic profile, see Dimensions in the Insight Historical profile. Profile Performance workspace This dataset includes the following relevant metrics for data workbench monitoring. Fast Input MegaBytes per Minute metrics displaying heavy data input during initial log processing. Fast Merge MegaBytes per Minute metrics displaying transformation. Note: To do a real performance assessment of your profile, look at rate rather than elapsed calendar time. The rate is measured as the changed values between polling every ten minutes. Server Performance workspace This dataset monitors server metrics beyond the scope of included profiles, and includes the following relevant server metrics for data workbench monitoring. Estimated Sweep Minutes Estimated query resolution time. Poll Latency Milliseconds Indicator of how busy software is by measuring how long it takes to get through a full cycle of servicing every component.

10 Workspaces for Monitoring the Data Workbench Server 10 Components workspace This dataset is located under the Up Time tab. The Components dataset includes two aspects for component health: Communications metric Did the data workbench server process respond? All Components metric At top of Detailed Status page is a list of components the host is servicing within the data workbench server processes. If any component is in an error state then it is listed under the Components in Error table.

11 Workspaces for Monitoring the Data Workbench Server 11 Processing Mode workspace This workspace is located under the Up Time tab. This workspace lets you observe how much time is taken in fast input, fast merge, and real-time modes. This dataset provides important server load characteristics, such as identifying data load for Day of the week (for example a Fast Input Rate on Tuesday and Wednesday), Hour of Day (what percentage of the day is it in Fast Input mode?) Data Workbench Profile Status workspace The data workbench Profile Status profile provides current information about the data workbench server health based on the profile rather than server metrics or historical data.

12 Workspaces for Monitoring the Data Workbench Server 12 Data Workbench Profile Status This status profile provides the data workbench server information that is current, but not quite real-time because the agent is polled every ten minutes and reporting always includes this ten-minute latency. More precisely, the datasets generated by this profile provide the latest observation of the server from the agent, which most often has a default polling period of ten minutes. For additional reference information about the dimensions used in the data workbench Profile Status profile, see Insight Profile Status profile. This report is more for monitoring operations rather than components or specific traffic fluctuations.

13 Workspaces for Monitoring the Data Workbench Server 13 This gives us a sense of who is in what mode: If we see a high Fast Input rate for a certain profile then that profile is in Fast Input mode. If the Stalled metric is 1, then the server is stalled. If the value is 0, then the server is not stalled. Log Reading for large batch loads Data Workbench Server Status workspace The data workbench Server Status profile provides current information about data workbench server health based on the server rather than profile metrics or historical data.

14 Workspaces for Monitoring the Data Workbench Server 14 General Status Open the General Status dataset view within the data workbench Server Status profile. For additional reference information about the dimensions used in the data workbench Server Status profile, see the Dimensions in the Insight Server Status profile profile. Disk Status View current disk usage including internal usage of temp.db.

15 Workspaces for Monitoring the Data Workbench Server 15

16 Data Workbench Profile Dimensions and Metrics 16 Data Workbench Profile Dimensions and Metrics This document describes the profiles with their fields, dimensions, and metrics employed by the data workbench Monitoring Profile. To monitor data workbench servers, data is collected using a script that parses the Detailed Status while also capturing specific server information. Data workbench server information is then passed to a page tag call for the data workbench Sensor to collect and save to a VSL file. Profiles Employed by the Data Workbench Monitoring Profile These profiles provide dimensions and metrics that allow you to view server state and performance data: Dimensions in the Data Workbench Profile Status profile Dimensions in the Data Workbench Server Status profile Dimensions in the Data Workbench Historic profile Metrics in the Data Workbench Historical Monitoring profile The Status profiles allow you to see how data workbench is currently performing from an operational perspective. The Profile Status profile and the Server Status profile gather data from the Detailed Status and the data workbench servers. All gathered data is placed into the cs-uri-query field for use. The Historic profiles allow you to assess the impact of configuration and hardware changes using historical data. The historical profile may be the most useful because it allows you to assess the impact of configuration and hardware changes over time. Dimensions in the Data Workbench Historic profile The following dimensions are available for use in the data workbench Historic profile. Dimensions in the Data Workbench Historic profile The following dimensions are available for use in the data workbench Historic profile. Block This is the only countable in this configuration, it is the root for all dimensions. A block can be considered a server. It is using the x-trackingid field. Note: The block ID is a hash of the server name plus host name, so there will be approximately one block per server per profile. Ping Alert Critical Alert Down Alert Warning This is a countable dimension built off of the Block countable. Each row of data in the profile is a ping from the monitoring agent. Numeric Dimension built from the cs-uri-query(ad) value. It is built at the Ping level conditioned that cs-uri-query(a) matches "1". Numeric Dimension built from cs-uri-query(ac) value. It is built at the Ping level, conditioned that cs-uri-query(a) matches "1". Numeric Dimension built from cs-uri-query(ae) value. It is built at the Ping level conditioned that cs-uri-query(a) matches "1".

17 Data Workbench Profile Dimensions and Metrics 17 Any Profile Reprocessing As of Delay Minutes Capacity Row Percentage Capacity Size Percentage Component Check Success Components in Error Detailed Check Seconds Detailed Check Success Dimension GigaBytes Disk "x" Used Percentage Estimated Sweep Dekaseconds Numeric Dimension built from cs-uri-query(aa) value. It is built at the Ping level conditioned that cs-uri-query(a) matches "1" and cs-uriquery(k) is not empty. cs-uri-query(bi) is placed into the x-as-of-delay-minutes field and rounded to the nearest minute. It is built at the Ping level conditioned that cs-uri-query(a) matches "1". Numeric Dimension built from cs-uri-query(r) value. It is built at the Ping level conditioned that cs-uri-query(a) matches "1" and cs-uriquery(k) is not empty. Numeric Dimension built from cs-uri-query(n) value. It is built at the Ping level conditioned that cs-uri-query(a) matches "1" and cs-uriquery(k) is not empty. Simple Dimension built from cs-uri-query(v) value. It is built at the Ping level, and conditioned that cs-uri-query(a) matches "1". cs-uri-query(ao) is split by the " " delimiter and copied into the x-components-in-error field. Many to Many Dimension built from the x-components-in-error field. Built at the Ping level. Numeric Dimension built from cs-uri-query(l) value. It is built at the Ping level conditioned that cs-uri-query(k) is not empty. Simple Dimension built from the cs-uri-query(k) value. It is built at the Ping level conditioned that cs-uri-query(a) matches "1". cs-uri-query(bc) is copied into the x-dimension-gigabytes field. The x-dimension-gigabytes field is user for this Simple Dimension, conditioned on cs-uri-query(a) matching "2". These Numeric Dimensions are configured from the cs-uri-query(ah, ai, aj, ak, al) values. They are built at the Ping level and conditioned on cs-uri-query(a) matches "1" and cs-uri-query(k) is not empty. The x-estimated-sweep-dekaseconds field is used in this Numeric Dimension. This is the estimated sweep time of the servers divided by ten (reduced resolution of sweep measurement to make dimension more reasonably sized). Fast Input MegaBytes per Minute The cs-uri-query(bj) value is used for this dimension. The Last Row for a block is used as the value for the dimension. If the dataset is in Fast Input this Numeric Dimension's value will display the MB per minute at which the system is inputting data. Fast Merge MegaBytes per Minute The cs-uri-query(bk) value is used for this dimension. The Last Row for a Block is used as the value for the dimension. If the dataset is in Fast Merge This Numeric Dimension's value will display the MB per minute at which the system is merging.

18 Data Workbench Profile Dimensions and Metrics 18 Field GigaBytes The cs-uri-query(bg) value is used for this dimension. The value is divided by 1000 and rounded to the nearest whole number. This Numeric Dimension's value will display the amount of space the Fields in the dataset are using. Group Host Last Ping Load Average Simple Dimension built at the Ping level from the cs-uri-query(x) value. The cs-uri-query(b) value is used for this dimension. The Simple dimension's value is the Last Row for a Block. the x-unixtime field is copied into x-last-ping and divided by 10 to reduce the cardinality. The Numeric Dimension is built at the Block level and uses the x-last-ping field. This is a Numeric dimension using the Last Row for a given Server's cs-uri-query(i) value. It is conditioned on cs-uri-query(k) not being empty. This dimension is used to calculate the average load on the servers in the system being monitored. Log Reading Percentage the cs-uri-query(be) value is used for this numeric dimension, built at the Ping level. This dimension is used to calculate the percentage of logs being read. Memory Page File Percentage This is a Numeric dimension using cs-uri-query(o) value, built at the Ping level. It is conditioned on cs-uri-query(k) not being empty, and cs-uri-query(a) matching "1". This dimension is used to calculate the percent of page file memory usage. Memory Physical MegaBytes Total This is a Numeric dimension using the cs-uri-query(ag) value, built at the Ping level. It is conditioned on cs-uri-query(k) not being empty, and cs-uri-query(a) matching "1. Memory Physical Percentage This is a Numeric dimension using the cs-uri-query(ag) value, built at the Ping level. It is conditioned on cs-uri-query(k) not being empty, and cs-uri-query(a) matching "1. This dimension is used to calculate the percent of physical memory usage of each Server.

19 Data Workbench Profile Dimensions and Metrics 19 Memory Query Percentage This is a Numeric dimension using the cs-uri-query(s) value at the Ping level. It is conditioned on cs-uri-query(k) not being empty and cs-uri-query(a) matching "1. This dimension is used to calculate the percent of query memory usage of each Server. Network Connections This is a Numeric dimension using the cs-uri-query(q) value built at the Ping level. It is conditioned on cs-uri-query(k) not being empty and cs-uri-query(a) matching "1. This is used to show the number of network connections there are for a given server. Output Rows Ping Type ID Poll Latency Centiseconds cs-uri-query(bh) value is divided by and copied into the x-output-rows field to reduce the size of the dimension. X-output-rows is used in a Numeric Dimension built at the Ping level, conditioned that cs-uri-query(a) matches "2". Simple Dimension built using the cs-uri-query(a) value at the Ping level. This shows what kind of Ping was recorded. The cs-uri-query(m) value is divided by 10 to reduce dimension size, and copied into the x-poll-latency-centiseconds field. This is a Numeric dimension built at the Ping level, conditioned that cs-uri-query(k) is not empty, and cs-uri-query(a) matches "1"/ This dimension is used to calculate the poll latency. Processing Mode ID The cs-uri-query(bb) value is used for this Simple Dimension, built at the Ping level. It is conditioned that cs-uri-query(bb) is not empty, and that cs-uri-query(a) matches "2" Processing Mode ID allows one to see what mode of processing the system is in (Fast Input, Fast Merge, Real Time). Note: This dimension is hidden then re-exposed with friendly values in the client-side dimension Processing Mode. Profile Quick Check Seconds Quick Check Success Real Time Processing Percentage Source Furthest Behind The cs-uri-query(ba) value is used for this Simple Dimension, it is built at the Ping level. This dimension displays the name(s) of the profile(s) currently being monitored. The cs-uri-query(h) value is used for this Numeric Dimension. It is built at the Ping level conditioned that cs-uri-query(a) matches "1". This is a Simple dimension built from the cs-uri-query(g) value built at the Ping level. It is used to calculate the quick check metric. Numeric Dimension using the cs-uri-query(t) value built at the Ping level. It is conditioned that cs-uri-query(a) matches "1". Simple Dimension built from the cs-uri-query(bl) value built at the Ping level. This dimension displays the when the last contact with a data source occurred.

20 Data Workbench Profile Dimensions and Metrics 20 Temp DB Space Percentage Numeric Dimension built using the cs-uri-query(an) value, built at the Ping level. It is conditioned that cs-uri-query(k) is not empty, and cs-uri-query(a) matches "1". It is used to calculate the percentage of used Temp DB space on a given server. Transformation Percentage the cs-uri-query(bf) value is used for this numeric dimension. It is built at the Ping level. This dimension is used to calculate the percentage of complete data transformation. Data Workbench Version Data Workbench Version Major The cs-uri-query(ab) value is used for this Simple Dimension. It is built at the Ping level and conditioned that cs-uri-query(ab) is not empty, and cs-uri-query(a) matches "1". This displays the version(s) of data workbench server running on each server. The cs-uri-query(ab) value is split and the major release value is copied into the x-insight-version-major field. It is a Simple Dimension built at the Ping level and conditioned that x-insight-version-major is not empty, and cs-uri-query(a) matches "1". Dimensions in the Data Workbench Profile Status profile The following dimensions are available for use in the data workbench Profile Status profile. Block As of Delay Minutes Environment Fast Input MegaBytes per Minute This is the only countable in this configuration and exists as the root for all dimensions. A block can be considered a server. It is using the x-trackingid field. The block ID is a hash of the server name plus host name, so there will be approximately one block per server per profile. cs-uri-query(bi) is placed into the x-as-of-delay-minutes field and rounded to the nearest minute. As of Delay Minutes is a numeric dimension that takes the Last Row from x-as-of-delay-minutes for a block. The cs-uri-query(c) value is used for the Environment ID. The Last Row for a Block is used as the value for the dimension. This Simple Dimension will display the Environment in which your Servers are running (provided it is configured properly). This can be set in the insight_monitor_agent.cfg file The cs-uri-query(bj) value is used for this dimension. The Last Row for a block is used as the value for the dimension. If the dataset is in Fast Input this Numeric Dimension's value will display the MB per minute at which the system is inputting data. Fast Merge MegaBytes per Minute The cs-uri-query(bk) value is used for this dimension. The Last Row for a Block is used as the value for the dimension. If the dataset is in Fast Merge This Numeric Dimension's value will display the MB per minute at which the system is merging.

21 Data Workbench Profile Dimensions and Metrics 21 Field GigaBytes The cs-uri-query(bg) value is used for this dimension. The value is divided by 1000 and rounded to the nearest whole number. This Numeric Dimension's value will display the amount of space the Fields in the dataset are using. Host Last Ping The cs-uri-query(b) value is used for this dimension. The Simple dimension's value is the Last Row for a Block. x-last-ping is x-unixtime divide by 10 (to accommodate Numeric dimensions size constraints). Last Ping is the Last Row for a given Block, and it represents the last time the monitoring agent logged the system health. Log Reading Percentage the cs-uri-query(be) value is used for this numeric dimension. It is the Last Row for a given Block. This dimension is used to calculate the percentage of logs being read. Processing Mode ID The cs-uri-query(bb) value is used for this Simple Dimension. It the Last Row for a given Block. Processing Mode ID allows one to see what mode of processing the system is in (Fast Input, Fast Merge, Real Time). Note: This dimension is hidden then re-exposed with friendly values in the client-side dimension Processing Mode. Processing Stalled The x-processing-stalled field is created through various conditions to indicate whether the profile is currently running or not. It is a simple dimension. Note: This dimension works best when there are a large number of input logs to fairly distribute amongst the DPUs. If there is, for example, only one large file loaded per day, then data workbench can appear to "stall" for an hour or more resulting in a false positive reading from this dimension. Profile Source Furthest Behind Transformation Percentage The cs-uri-query(ba) value is used for this Simple Dimension. This dimension displays the name(s) of the profiles currently being monitored. The last row of cs-uri-query(bl) is copied into the x-source-furthest-behind field. The Simple Dimension uses the Last Row for a given Block. This dimension displays the when the last contact with a data source occurred. the cs-uri-query(bf) value is used for this numeric dimension. It is the Last Row for a given Block. This dimension is used to calculate the percentage of complete data transformation. Time Dimensions Hour, Day, Week, Month, Hour of Day, and Day of Week are all derived from the x-timestamp field.

22 Data Workbench Profile Dimensions and Metrics 22 Group Metrics As Of Delay Minutes As Of Delay Seconds Blocks Fast Input MB per Minute Fast Merge MB per Minute Field GigaBytes Last Ping Age Last Ping Time Log Reading Processing Stalled Transformation Grouping word that gives you another way to filter the resulting dataset. Set in the insight_monitor_agent.cfg file. The following lists the metrics included in the data workbench Profile Monitoring Profile and how they are derived. This metric is the sum of the As Of Delay Minutes for each Block, then divided by the Blocks metric. This metric is the sum of the As Of Delay Seconds for each Block, and divided by the total number of Blocks. (As of Delay Seconds Dimension not configured out of the box) Sum one for each Block. The sum of Fast Input MegaBytes per Minute for each Block divided by the number of Blocks when Fast Input MegaBytes per Minute is greater than zero. The sum of Fast Merge MegaBytes per Minute for each Block divided by the number of Blocks when Fast Merge MegaBytes per Minute is greater than zero. The sum of Field Gigabytes for each Block divided by Blocks metric. The As Of Time minus Last Ping Time. The sum of Last Ping for each Block divided by Blocks, then multiplied by 10. Where Log Reading Percentage is greater than zero, Log Reading is the sum of Log Reading Percentage for each Block, divided by Blocks metric, all of which is divided by 10. The sum of the Processing Stalled Dimension for each Block, divided by the Blocks metric. The sum of Transformation Percentage for each Block divided by the Blocks metric, when Transformation Percentage is greater than zero, all divided by 10. Dimensions in the Data Workbench Server Status profile The following dimensions are available for use in the data workbench Server Status profile. Server Agent Version Any Profile Reprocessing Capacity Row Percentage Capacity Size Percentage Common Name Built from the x-trackingid field, this countable dimension represents the Servers currently running data workbench. The cs-uri-query(af) value is used for this Simple Dimension. It is the Last Nonblank value for a Server. This displays the build date and time for the version(s) of the monitoring agent running. The cs-uri-query(aa) field is used for this Numeric Dimension, it is the value of the Last Row for a given Server, conditional on cs-uri-query(k) is not empty. This dimension is used to indicate if any profiles are Reprocessing. The cs-uri-query(r) field is used for this Numeric Dimension, it is the value of the Last Row for a given Server, conditional on cs-uri-query(k) is not empty. The cs-uri-query(n) field is used for this Numeric Dimension, it is the value of the Last Row for a given Server, conditional on cs-uri-query(k) is not empty. The sc-ur-query(am) field is used for this Simple Dimension, it is the value of the Last Nonblank value for a given Server. It displays the Common Name of the servers being monitored.

23 Data Workbench Profile Dimensions and Metrics 23 Component Check Success Components in Error Environment The cs-uri-query(v) field is used for this Simple Dimension, it is the value of the Last Row for a given Server. This dimension checks on the components of the server to verify they are properly functioning. A Crossrows transformation takes the Last Row value of the cs-uri-query(ao) and copies it into the x-components-in-error field. This Many to Many dimension displays any components in error on servers being monitored. The cs-uri-query(c) value is used for the Environment ID. The Last Row for a Block is used as the value for the dimension. This Simple Dimension will display the Environment in which your Servers are running (provided it is configured properly). Note: This dimension is set in insight_monitor_agent.cfg. Estimated Sweep Dekaseconds The x-estimated-sweep-dekaseconds field is used in this Numeric Dimension. This is the estimated sweep time of the servers divided by ten (reduced resolution of sweep measurement to make dimension more reasonably sized). Host Last Ping Load Average The cs-uri-query(b) value is used for this dimension. The Simple dimension's value is the Last Row for a Block. x-last-ping is x-unixtime divide by 10 (to accommodate Numeric dimensions size constraints). Last Ping is the Last Row for a given Block, and it represents the last time the monitoring agent logged the system health. This is a Numeric dimension using the Last Row for a given Server's cs-uri-query(i) value. It is conditioned on cs-uri-query(k) not being empty. This dimension is used to calculate the average load on the servers in the system being monitored. Memory Page File Percentage This is a Numeric dimension using the Last Row for a given Server's cs-uri-query(o) value. It is conditioned on cs-uri-query(k) not being empty. This dimension is used to calculate the percent of page file memory usage. Memory Physical MegaBytes Total This is a Numeric dimension using the Last Row for a given Server's cs-uri-query(ag) value. It is conditioned on cs-uri-query(k) not being empty.

24 Data Workbench Profile Dimensions and Metrics 24 Memory Physical Percentage This is a Numeric dimension using the Last Row for a given Server's cs-uri-query(ag) value. It is conditioned on cs-uri-query(k) not being empty. This dimension is used to calculate the percent of physical memory usage of each Server. Memory Query Percentage This is a Numeric dimension using the Last Row for a given Server's cs-uri-query(s) value. It is conditioned on cs-uri-query(k) not being empty. This dimension is used to calculate the percent of query memory usage of each Server. Network Connections This is a Numeric dimension using the Last Row for a given Server's cs-uri-query(q) value. It is conditioned on cs-uri-query(k) not being empty. This is used to show the number of network connections there are for a given server. Poll Latency Centiseconds This dimension is used to calculate the poll latency. The cs-uri-query(m) value is divided by 10 to reduce dimension size, and copied into the x-poll-latency-centiseconds field. This is a Numeric dimension which takes the Last Row for a given server. Quick Check Success Temp DB Space Percentage This is a Simple dimension built from the cs-uri-query(g) value of the Last Row for a given Server. It is used to calculate the quick check metric. The last row of the cs-uri-query(an) value is copied into the x-temp-db-space-percentage field. This is a Numeric Dimension that is used to calculate the percentage of used Temp DB space on a given server. Insight Version Group The cs-uri-query(ab) value is used for this Simple Dimension. It is the Last Nonblank value for a Server. This displays the version(s) of data workbench server running on each server. Grouping word that gives you another way to filter the resulting dataset. Note: This dimension is set in insight_monitor_agent.cfg. Metrics Capacity Overall Capacity Row The following lists the metrics included in the data workbench Profile Monitoring Profile and how they are derived. This is the Capacity Size metric times two plus the Capacity Row metric divided by 3. This is the sum of the Capacity Row Percentage for each Server divided by Servers metric.

25 Data Workbench Profile Dimensions and Metrics 25 Capacity Size Component Check Disk "x" Estimated Sweep Minutes Last Ping Time Load Memory Page File Memory Physical Memory Query Network Connections Poll Latency Milliseconds Quick Check Servers Temp DB This is the sum of the Capacity Size Percentage for each Server divided by the Servers metric. This is the number of Servers where Component Check Success equals one, divided by the Server where Service is DPU or FSU, all multiplied by 100 (to make it a percentage). The Disk metrics are calculated by taking the sum of their Disk Used Percentage for each Server, divided by the Servers metric. This is the sum of the Estimated Sweep Dekaseconds for each Server, divided by the Servers metric where Estimated Sweep Dekaseconds is greater than zero, all divided by 6. The sum of Last Ping for each Block divided by Blocks, then multiplied by 10. This is the sum of the Load Average for each Server, divided by the Servers metric. This is the sum of the Memory Page File Percentage for each Server, divided by the Servers metric. This is the sum of the Memory Physical Percentage for each Server, divided by the Servers metric. This is the sum of the Memory Query Percentage for each Server, divided by the Servers metric. This is the sum of the Network Connections for each Server divided by the Servers metric. This is the sum of the Poll Latency Centiseconds for each Server, divided by the Servers metric, all of which is multiplied by 10. This is the number of Servers where Quick Check Success equals one, divided by the Servers metric, all of which is multiplied by 100. This is the sum of one for each Server, or the total number of monitored Servers. This is the sum of the Temp DB Space Percentage for each Server, divided by the Servers metric. Metrics in the Data Workbench Historical Monitoring profile The following lists the metrics included in the data workbench Historical Monitoring Profile and how they are derived. Alert Criticals Alert Downs Alert Warnings All Components As Of Delay Minutes The sum of the Alert Critical dimension for each Ping. The sum of the Alert Down dimension for each Ping. The sum of the Alert Warning dimension for each Ping. The count of Pings where Component Check Success equals "1" divided by the Pings metric multiplied by 100. This metric is the sum of the As Of Delay Minutes for each Ping, then divided by the Pings metric.

26 Data Workbench Profile Dimensions and Metrics 26 Blocks Block All Capacity Overall Capacity Row Capacity Size Communications Detailed Check Seconds Dimension GigaBytes Disk "x" Estimated Sweep Minutes Fast Input MB per Minute Fast Input Mode Fast Merge MegaBytes per Minute Fast Merge Mode Field GigaBytes Load Log Reading Memory Page Memory Physical Memory Query Memory Total GB Network Connections Pings x Capacity Overall Poll Latency Milliseconds The sum of one for each Block. All Blocks. The Capacity Size Metric times 2 plus the Capacity Row metric, divided by 3. The sum of the Capacity Row Percentage dimension for each Ping divided by the Pings metric. The sum of the Capacity Size Percentage dimension for each Ping, divided by the Pings metric. The number of Pings where Quick Check Success matches "1", divided by the Pings metric. The sum of the Detailed Check Seconds dimension for each Ping where the ping type is "server", divided by the Pings metric. The sum of Dimension Gigabytes for each Ping, divided by the Pings metric. The Disk metrics are calculated by taking the sum of their Disk Used Percentage for each Ping, divided by the Pings metric. This is the sum of the Estimated Sweep Dekaseconds for each Ping, divided by the Pings metric where Estimated Sweep Dekaseconds is greater than zero, all divided by 6. The sum of Fast Input MegaBytes per Minute for each Ping divided by the number of Pings when Fast Input MegaBytes per Minute is greater than zero. Pings where Processing Mode dimension is equal to "Fast input" divided by Pings. The sum of Fast Merge Megabytes per Minute for each Ping, divided by Pings metric. Pings where Processing Mode equals "fast merge" divided by the Pings metric. The sum of Field Gigabytes dimension for each Ping divided by Pings metric. The sum of the Load Average dimension for each Ping, divided by the Pings metric. The sum of Log Reading Processing dimension for each Ping, divided by the Pings metric, all divided by 10. The sum of Memory Page File Percentage for each Ping, divided by the Pings metric. The sum of the Memory Physical Percentage dimension for each Ping, divided by the Pings metric. The sum of the Memory Query Percentage for each Ping, divided by the Pings metric. The sum of Memory Physical MegaBytes Total dimension for each Ping, divided by the Pings metric. This is the sum of the Network Connections for each Ping divided by the Pings metric. The Pings metric multiplied by the Capacity Overall metric. The sum of the Poll Latency Centiseconds dimension for each Ping, divided by the Pings metric, all multiplied by 10.

27 Data Workbench Profile Dimensions and Metrics 27 Query Running Quick Check Seconds Output Rows Real Time Mode Reprocessing Mode Stalled Temp DB Transformation The sum of one for each Ping where Estimated Sweep Dekaseconds is greater than "0", divided by the Pings metric where Ping Type equals "server". The sum of Quick Check Seconds for each Ping where Ping Type is equal to "server", divided by the Pings metric. The sum of Output Rows dimension for each ping divided by the Pings metric, multiplied by The number of Pings where Processing Mode dimension equals "real time", divided by the Pings metric, all multiplied by minus the number of Pings where Processing Mode equals "real time" divided by the Pings metric, multiplied by 100. The sum of the Processing Stalled dimension in the Insight Profile Status profile. The sum of Temp DB Space Percentage for each Ping, divided by the Pings metric. The sum of Transformation Percentage for each Ping divided by the Pings metric all divided by 10.