An Introduction to HIPPO V4 - the Operational Monitoring and Profiling Solution for the Informatica PowerCenter platform.

Transcription

1 An Introduction to HIPPO V4 - the Operational Monitoring and Profiling Solution for the Informatica PowerCenter platform. A Functional overview. ` Runner-Up 2012 MVP Competition Copyright Assertive Software Ltd All rights reserved. Page 1

2 Introduction!... 3 The Complexity Challenge.!... 3 HIPPO s Unique Perspective!... 3 HIPPO s Three-Step Approach;!... 4 Monitor: the HIPPO Active Notification Centre (HANC)!... 4 Diagnose and Remediate: the HIPPO Console!... 4 HIPPO s unique approach!... 5 HIPPO s Key Features!... 6 HANC - HIPPO s Active Notification Centre!... 6 HANC s Alert Levels and Scope!... 6 HANC s Alert mechanisms! HIPPO s Dashboard!... 6 HIPPO s Filtering options! Resources! Vital Signs! Activity Monitor! Search! Analysis! Workflow! Task! Capacity Planning! Timeline! Summary! Appendix! HANC Notification Rules Summary! Published May, Copyright Assertive Software Ltd All rights reserved. Page 2

3 Introduc)on We have created HIPPO because you need a holistic view of your PowerCenter estate, not simply at an individual Mapping or Folder level, but also at a business level interpreting the activity, incidents, trends and capacity in your PowerCenter environments. HIPPO is an active monitoring, diagnostic and analysis tool that gives you an unrivaled understanding of your Informatica estate and the Data Integration processes that it supports to enable you to improve performance at every level. The$Complexity$Challenge. An organization s Data Integration environment is a significant investment that provides a key operational asset for the Business. However the environment is a complex one: CPU and Memory resources, the Operating System, Network, Database, ETL and many other processes interoperate to integrate operational, financial and business intelligence data in ever larger volumes and in ever more demanding timeframes. Considerable effort is devoted to optimizing these data integration processes but success is limited by the lack of a big picture; how much Informatica capacity is needed? What benchmark level of performance can be expected? How do I factor in other parallel data integration processes when I estimate if SLAs will be met? How can variations in demand be accommodated? Why does this Workflow fail intermittently for the same reason? Over the past two years we have spent many hours meeting with Informatica Administrators, Developers and Support staff from the smallest to the largest Informatica customers to find out what they needed to know to make their jobs easier. We identified five challenges that HIPPO needed to solve. They were; 1. It is extremely difficult to get visibility across the entire Informatica environment and there is little operational history available 2. Users need to connect to multiple tools to understand what is going on 3. Users need to do lots of manual work to identify trends in performance, resource availability and capacity in my Informatica environment 4. Informatica tools are too focused on individual repositories - Users need the big picture and then to focus on what has failed or is performing differently from normal irrespective of which repository it comes from 5. Too much time is spent log trawling! HIPPO V4 has been designed to meet these challenges by combining an operational data warehouse with real time monitoring and alerting linked to a real-time visualisation of your entire Informatica environment. HIPPO s$unique$perspec=ve Starting with Informatica, HIPPO monitors and profiles everything from an individual transformation to an entire project. The Informatica estate aggregated as a service across your Grid or multi-node environment. Trends in performance by sessions, workflows, projects or nodes are available over any timescale and HIPPO profiles non-informatica processes as well, allowing you to identify their share of CPU and Memory resource. Copyright Assertive Software Ltd All rights reserved. Page 3

4 HIPPO s/three3step/approach; Monitor:$the$HIPPO$Ac=ve$No=fica=on$Centre$(HANC) HANC, the HIPPO active monitoring solution, constantly monitors the activity and resource usage within your Informatica estate alerting you to over thirty different kinds of issues which include Service, Workflow and Session failure, abnormal patterns in CPU and Memory usage at the Project, Workflow or Session level and breaches to SLAs and changing trends in your Data Integration KPIs. At its heart lies a powerful combination of information collection and interpretation technologies that are key to the proactive monitoring of your PowerCenter estate and Data Integration applications and services. Between them, these technologies facilitate business activity monitoring of the Data Integration processes critical to your organization. Diagnose$and$Remediate:$the$HIPPO$Console HIPPO is unique because it analyses your environment from the bottom-up; CPU, Memory, Data Movement and other key metrics are monitored and analyzed at the individual Transformation, Session and Workflow level. This enables Developers to get the information that they need to diagnose performance bottlenecks, Testers to identify if performance thresholds have been met and Support Analysts to be alerted to incidents in the production environment. HIPPO then aggregates this operational information into a higher-level analysis. Firstly HIPPO presents this information at a Folder, Domain, Repository, Grid, Integration Service or Estate level to enable Informatica Administrators and Capacity Planners to understand the trends in usage and performance in an intuitive way. Secondly, HIPPO does something completely unique; it allows you to take a business view by logically grouping Informatica and non-informatica processes together to provide the Project-level understanding of incidents, trends and performance that is vital to project managers and stakeholders. Copyright Assertive Software Ltd All rights reserved. Page 4

5 HIPPO s$unique$approach Using a rapidly deployed, three-tier architecture with a light footprint, HIPPO delivers a powerful enterprise wide monitoring and management capability for your entire Informatica PowerCenter estate. HIPPO is entirely Grid-compliant and can simultaneously monitor any number of Grids, Domains, Repositories, Integration Services and Hosts. HIPPO's first tier, the instrumentation and collection tier, consists of software agents called Collectors, which are located on each of the nodes to be monitored in your environment. HIPPO Topology BI & Reporting Tools Infa Repository Browser BI Server Task & Workflow Metadata Alerts Dashboard & Resources Search & Analysis Fact Engine OS Profiler HANC HIPPO Repository Activity Monitor OS Metrics Infa Task Metrics Logfile Analysis Facts & Dimensions HANC Engine HIPPO Collector Repository Monitor Server HIPPO Server Informatica Node The Collectors report the activity on each node to the second tier, HIPPO s Analysis and Consolidation tier, where HIPPO correlates infrastructure metrics with Informatica activity and the performance values are consolidated and stored in the HIPPO Repository. This tier includes HIPPO s Active Notification Centre (HANC), the Fact Engine, the OS Profiler and the Repository Monitor. HANC then combines information from the HIPPO Repository with user-defined business logic that allows meaningful alerts to be routed through and other forms of integration with your in-house monitoring systems to the most appropriate support resource, providing your operations teams with the facts to take corrective action fast. The third tier in HIPPO's architecture is the data visualization tier. This is HIPPO s webdriven user interface which we call the HIPPO Console. The Console provides an intuitive graphical interface that supports real-time, color-coded visualisations at both logical and technical levels. Interactive features, such as HIPPO s TIMELINE, allow Users to direct their analysis activities naturally to widen and narrow their perspective as required. Copyright Assertive Software Ltd All rights reserved. Page 5

6 HIPPO s$key$features 1. HANC$L$HIPPO s$ac=ve$no=fica=on$centre HANC is HIPPO s active alerting & notification module. HANC enables you to define over thirty kinds of monitoring rules that will trigger Alerts to be raised in a wide variety of circumstances. These rules are split into three categories; Exceptions, Deviations and SLAs and examples include sessions exceeding normal elapsed times, Service outages and resource thresholds being exceeded by individual sessions or workflows and a wide variety of other circumstances. See the Appendix at the end of this document for details. HANC s Alert Levels and Scope HANC allows you to define the scope of a rule; at Folder, Workflow or Session Level, for example if you define a Cache Spill to Disk alert at Folder level then any session within that Folder which spills a cache to disk will trigger HANC to raise an alert. HANC s Alert mechanisms HIPPO relays the alert in three ways; firstly direct to the HIPPO console where you can click on a Review button to go straight to a detailed analysis of the problematic execution of that session or workflow. Secondly by to your account or to a group account. Lastly, HIPPO also integrates directly with your in-house monitoring applications. 2.$HIPPO s$dashboard$ HIPPO s Dashboard presents the key metrics for your environment; Elapsed Time by Folder breaks down where Informatica is spending its time in your environment, Dashboard Metrics breaks down activity in a row-based split by aggregating Source, LookUp and Target rows across your environment, Failures vs. Success vs. Rejections; summarizes session activity across your environment by the number of successful and failed executions. Memory % used by Project analyses memory consumption by project over your chosen date & filter range to give you an overall perspective on memory usage by project. CPU % used by Project analyses CPU consumption by project to give a perspective on which Projects are responsible for CPU usage in your environment. Data Movement by Project analyses the volume of data read from Sources and written to Targets across your environment and you can drill down, for instance by selecting a Node filter, and so on. The final chart is the SLA Trending chart, which presents the number of SLAs achieved vs. the number missed for each sample point over the date & time and filter criteria that you have set. If the filter type chosen was Project then this provides a high-level indicator of the health of that project in terms of SLA performance. HIPPO s Filtering options All of HIPPO s filtering options are available on the dashboard screen which means that you can select a particular date and time range to be applied to the information displayed or select a filer, for example, NODE, DOMAIN, REPOSITORY, HOST, GRID and Integration SERVICE. You can also filter by PROJECT which is a logical collection of Informatica and non-informatica processes which is defined by you using the HIPPO Console. Copyright Assertive Software Ltd All rights reserved. Page 6

7 3.$Resources$ HIPPO analyses Resource usage by four categories: CPU, MEMORY, DATA MOVEMENT and FILE DIRECTORIES. Each category contains up to nine charts: CPU analyses CPU usage by Project and by Node across the Date and Filter criteria that you have set. HIPPO s analysis is fully grid compatible. The first five charts provide the overall CPU resource available in your environment and then breaks down its usage by Project and by Node. HIPPO aggregates all of your CPU resource together grouping Grid and Non- Grid nodes together to give an overall perspective of Informatica as a single service no matter how distributed it is or if the workflows and sessions are distributed across your Grid resources. In this feature s remaining charts, HIPPO focuses on the top consumers of CPU resource in your environment by firstly listing the top ten consumers of resource. This chart will include Informatica and non-informatica processes to enable you to understand where your valuable CPU resource is being consumed. You can drill down from these charts into an in-depth analysis of each workflow and session listed. MEMORY analyses memory resource usage in a set of eight charts focusing on Memory availability, usage and top consumers. In this function HIPPO also presents metrics on Cache by Project and Memory free by Node to give you a guide to load balancing in your environment and how successfully it is being achieved. DATA MOVEMENT provides nine charts which analyze data movement by volume of Source & Target rows, project and by Node. Again you can apply Date & Time and a variety of other filters to focus on a particular project or Node. This function in HIPPO is often used as a recharge mechanism to enable projects to be recharged for use of a shared Informatica central service, sometimes referred to as a Hub. The fourth category in the Resources option is FILE DIRECTORIES which is there to help HIPPO Users meet the challenge of high disk space demand often relating to large log files or unplanned cache spill to disk activity. 4.$Vital$Signs$ Vital Signs is designed to provide an overall view of the health of the services in your Informatica environment. As well as providing first line support to developers and support analysts, Vital Signs can also be displayed plasma screen style on an Informatica Administrator s workstation to constantly monitor your environment in real-time. Copyright Assertive Software Ltd All rights reserved. Page 7

8 There are three options: each provides a color-coded status for each of the Services in your environment; Integration, Domain and Repository Services are displayed in green for active and in red if the service is unavailable. 5.$Ac=vity$Monitor HIPPO s Activity Monitor Is designed to present a view of all workflow and session activity from across your entire Informatica estate. There are two views; Live View offers a real-time perspective on all of the workflows and sessions that are running across your entire estate. There is no need to connect to individual Integration services or Repositories: instead HIPPO handles all of that for you to enable a 360-degree perspective on active processes. Activity Monitor s Historic View groups all of the workflows and sessions, which have executed across your environment. The Date & Time and other Filter options are now available which means that you select activity from a particular date & time range or an individual Project, Node, Repository, Domain, Grid or Integration Service. Any combination of the above filter criteria can be applied to the Historic View. You can drill down into the details of individual Sessions and Workflows from anywhere within the Activity Monitor to access all of the salient details from the session execution including the reason for failure. This focus on the key information means that the support analyst need not trawl the log file to diagnose the reason for failure but instead use HIPPO to go direct to the issue which has triggered the failure. Copyright Assertive Software Ltd All rights reserved. Page 8

9 6.$Search Search at Folder, Workflow and Session level: HIPPO enables you to search by a name, or part of a name with or without a date and time of execution. Search by Resource Usage: HIPPO enables you to search for individual executions of workflows and sessions which have exceeded a CPU threshold (in CPU seconds), a Memory threshold (in MBs), individual cache size (in MBs), Cache Spilled to Disk or Time to First Row (in seconds). Search for Executed SQL enables a User to search by a fully resolved SQL statement. Lastly the SQL override Search facility enables us to identify all sessions which have used this feature which has a significant effect on Informatica s generation of automatic memory settings. 7.$Analysis This is HIPPO s Tuning workshop to enable you to focus your remediation efforts on those workflows and sessions most likely to result in performance improvements. There are four sub-functions within the Analysis feature: Folder Analysis, Top Workflows, Top Sessions and Trending. The Folder Analysis feature enables you to focus on an individual folder to identify top Workflows by elapsed time, Data movement and concurrency metrics. The Top Workflows examines the highest consumers of CPU, Memory, Data Movement and elapsed time in your environment. All filters apply so you can focus on a particular Project for your analysis. Drill down directly from here to HIPPO s Workflow$feature. The Top Sessions examines the Sessions which are the highest consumers of CPU, Memory, Data Movement and elapsed time in your environment. As above, all filters apply so you can focus on a particular Project or Repository for your analysis. Drill down directly from here to HIPPO s Task$feature for an in-depth analysis of an individual Session. The Trending feature analyses Node-level load metrics by any of HIPPO s filter criteria. 8.$Workflow An example chart from Analysis / Top Sessions showing Sessions by Elapsed Time In the Workflow option HIPPO presents a detailed examination of an individual Workflow performance via three options: Execution, Workflow Trends and History to gain a full perspective of a single execution of a Workflow in the context of its execution trends. Copyright Assertive Software Ltd All rights reserved. Page 9

10 9.$Task Task presents a detailed examination of an individual Task in five menu categories: Transformations, Data Movement, Resources, Task Trends and Historical Analysis. This is where HIPPO analyzes performance at the lowest level to enable you to identify performance bottlenecks and incorrect session configuration settings. (Detail from Task / Data Movement showing the commit point profile over the execution period of an individual Session) 10.$Capacity$Planning Capacity Planner groups all of HIPPO s trending charts into a single place to enable Capacity Planners to understand trends in Data Movement, CPU and Memory usage by Project and by Node. Trend all the way back to when HIPPO was installed. (detail from Capacity Planner showing Memory usage by Project) 11.$Timeline Timeline feature offers a unique perspective on how the resources available in your environment are being shared between and consumed by the workflows and sessions that are active at any point in time. Use the bar at the bottom of the screen to move along the timeline and then zoom in on a particular point in time. Copyright Assertive Software Ltd All rights reserved. Page 10

11 Summary HIPPO s scope is wide. As the chart below illustrates, HIPPO collates data from across the Informatica and Infrastructure technology stack.and it is the ability to correlate this data at a granular level that is key to HIPPO s ability to monitor and profile your entire Informatica estate and the Infrastructure on which depends upon and interacts. 10% 12% 21% 30% 27% Repository Integration Services HOST / OS Storage Network HIPPO is designed to alert you to issues in your environment, to help you to diagnose the underlying cause of those issues and to ensure that the corrective steps that you take are the right ones. However HIPPO goes further: HIPPO allows you to take a business focused overview of your Data Integration estate itself and of the Projects that it supports to ensure your Data Integration environment is optimised to meet business demand now and in the future. Installation Process and Platform Support HIPPO can be rapidly installed using HIPPO s wizard-driven installation process to enable you to begin monitoring and profiling your environment immediately. HIPPO supports version to version 9.x of the informatica PowerCenter product. HIPPO can be installed on any of the range of platforms supported by version 9.x of PowerCenter. These platforms include Windows, Linux and Unix. HIPPO is shipped with its own built-in Apache Web Server and PostgreSQL database but can also be installed using your own Apache installation (Version 2.2 and above) and Oracle (Version 11g and above). HIPPO requires two gigabytes of disk space for initial installation. About Assertive Software Ltd Assertive was founded in the UK in 2010 by a team of Data Integration and Infrastructure specialists to bring new and innovative software products to the Data Integration market. Since our inception we have been shortlisted for a number of Industry awards including the nomination of HIPPO as a finalist in the 2012 Informatica Innovation Awards and being awarded the runner-up prize in the 2012 Most Valuable Partner competition run by the Informatica Marketplace division. Find out more about HIPPO and Assertive Software at our website where you will find a wealth of material including a library of short videos which demonstrate the use of each of HIPPO s key features. Copyright Assertive Software Ltd All rights reserved. Page 11

12 Appendix HANC Notification Rules Summary Exceptions Deviations SLA / KPI Service Goes Down Zero Records Read/Written/In Look Up Cache SLA on W'flow start time Rejected Records Session/Workflow elapsed time SLA on W'flow end time Session/Workflow Failure Source Rows /Target Rows Rows Processed Waiting Session warning Lookup Cache # of Rows Elapsed Time Continuous Throughput Rule Lookup Cache build time Read rate Read Throughput Lookup Cache size Write rate Write Throughput Integration Service CPU Integration Service Memory Process CPU Process Memory Specific directory > threshold value Workflow/Session CPU Threshold Session Cache Spills to Disk Workflow/Session Memory Threshold Aggregator Cache # of Rows Aggregator Cache build time Aggregator Cache size Sorter Cache # of Rows Sorter Cache build time Sorter Cache size Joiner Cache # of Rows Joiner Cache build time Joiner Cache size Time to First Row Read Throughput Write Throughput End of Appendix. Copyright Assertive Software Ltd All rights reserved. Page 12