IBM. Netcool /Proviso 4.3-X. Tivoli. NetFlow IPFIX Technology Pack User s Guide. Document Revision R2E1

Transcription

1 Tivoli Netcool /Proviso 4.3-X IBM Document Revision R2E1 NetFlow IPFIX Technology Pack User s Guide

2 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information about the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user s responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY U.S.A. For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: IBM World Trade Asia Corporation Licensing 2-31 Roppongi 3-chome, Minato-ku Tokyo 106, Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement might not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM Corporation 2Z4A/ Burnet Road Austin, TX U.S.A. Such information may be available, subject to appropriate terms and conditions, including in some cases payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products.

3 Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. All statements regarding IBM s future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating systems. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating system for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces. If you are viewing this information in softcopy form, the photographs and color illustrations might not appear. Trademarks AIX, DB2, developerworks, eserver, i5/os, IBM, the IBM logo, IBMLink, iseries, Lotus, Netcool, Netcool/Proviso, OMEGAMON, Passport Advantage, pseries, Rational, Redbooks, OS/390, OS/400, Tivoli, the Tivoli logo, Tivoli Enterprise Console, WebSphere, and zseries are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Intel is a registered trademark of Intel Corporation or its subsidiaries in the United States, other countries, or both. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, and service names may be trademarks or service marks of others. Copyright IBM Corp US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

4

5 Contents Preface ix Audience ix Organization ix Chapter 1: Introduction Overview How is the Data Collected? Inventory Collection How is the Data Processed and Loaded into the Database? CME Operation How is the Data Reported? Chapter 2:Supported Reports Overview Reporter Sets Reporter Set Tree Reports Throughput Summary Group Throughput Summary Resource TopN ToS Summary Group Traffic Summary Group Traffic Summary Resource Chapter 3: Devices and Services Overview Summary of Device Technology Technology Pack Overview Components and Data Flow Aurora Netflow Collector Configuration File Input Files Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X v

6 Contents Chapter 4: Element and Sub-Element Properties Overview NetFlow IPFIX Device Element Properties NetFlow IPFIX Device Sub-Element Properties Element.STATE NF_Application NF_DestIPAddress NF_DestIPAsNumber NF_DestMask NF_DestNet NF_DestPort NF_DstAsNum NF_InputIFIndex NF_NetworkName NF_NextHop NF_OutputIFIndex NF_Protocol NF_Router NF_Site NF_SourceIPAddress NF_SourceIPAsNumber NF_SourceMask NF_SourceNet NF_SourcePort NF_SrcAsNum NF_TCPFlags NF_Topic NF_Tos STATE Chapter 5: Sub-Element Grouping Rules Overview NOC Reporting Grouping Rules Sub-Element Collect Grouping Rules vi Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

7 Contents Chapter 6: Defined Resources Overview NetFlow IPFIX Technology Pack Elements NetFlow IPFIX Technology Pack Sub-Elements Tables of Sub-Element Types Networks NetApp ASNum NetMatrix NetHost HostNet NetToS Chapter 7: Bulk Collection Formulas Overview Data Collection Formulas and Metrics Formula Reference Bytes (throughput) Bytes In (throughput) Bytes In Bytes Out (throughput) Bytes Out Bytes BytesTotal Flows Packets (throughput) Packets In (throughput) Packets In Packets Out (throughput) Packets Out Packets PacketsTotal Appendix A: Configuration Overview Before You Begin Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X vii

8 Contents Configure the Technology Pack Other Tasks and Considerations Verify Resources Appendix B: UBA Parameters Reference (Netcool/Proviso 443 and Higher) UBA Parameters Reference Custom Parameters Enabling Host reporting Appendix C: Template File Reference (Pre-443 Only) Pack-Specific UBA Parameters Appendix D: Aurora Netflow Collector Installation and Configuration Overview Deployment Considerations Topology Duplication Scalability Aurora Installation Aurora Configuration Configuration By Means of Aurora Script Aurora Configuration and Load Balancing Additional Copyright Information viii Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

9 Preface This manual describes the Netcool/Proviso NetFlow IPFIX Technology Pack, focusing on the NetFlow IPFIX environment. Specifically, the manual describes the reports for displaying information about the devices and technologies that operate in the NetFlow IPFIX environment. The manual also describes the data collection model, properties, resources, collection formulas, and metrics that the NetFlow IPFIX Technology Pack provides to create the associated reporter sets. Audience The audiences for this manual are the network administration engineers at IBM customer sites who will install, configure, and use the NetFlow IPFIX Technology Pack as part of their Netcool/Proviso installation. IBM Professional Services engineers may also find this manual useful. To install and use the NetFlow IPFIX Technology Pack, you should have a working knowledge of the following subjects: Netcool/Proviso DataMart TCP/IP networks Telecom network management Administration of the operating system The audiences should also be familiar with the specific technology that the NetFlow IPFIX Technology Pack deals with, in this case NetFlow IPFIX. Organization This guide is organized as follows: Chapter 1, Introduction Provides a general introduction to technology packs. Chapter 2, Supported Reports Provides information about the reports that the NetFlow IPFIX Technology Pack provides for each device operating in the NetFlow IPFIX environment. Chapter 3, Devices and Services Describes the devices and services that the NetFlow IPFIX Technology Pack supports. Chapter 4, Element and Sub-Element Properties Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X ix

10 Preface Describes the properties that the NetFlow IPFIX Technology Pack discovers during the inventory process. Chapter 5, Sub-Element Grouping Rules Describes the sub-element collection grouping rules and the NOC reporting grouping rules that the NetFlow IPFIX Technology Pack defines. Chapter 6, Defined Resources Provides information about the elements and sub-elements that the technology pack defines in the Proviso database. Chapter 7, Bulk Collection Formulas Provides information about the Bulk collection formulas that the NetFlow IPFIX Technology Pack provides for devices operating in the NetFlow IPFIX environment that generate and store data in Bulk input files. Specifically, the chapter discusses the metrics on which the Bulk collection formulas operate. Appendix A, Configuration Describes how to configure the NetFlow IPFIX Technology Pack. Appendix B, UBA Parameters Reference (Netcool/Proviso 443 and Higher) Provides UNIX reference pages for each of the pack-specific UBA parameters supplied with the XSD file for the NetFlow IPFIX Technology Pack. Appendix C, Template File Reference (Pre-443 Only) Explains the edits (using a text editor) that need to be made to pack-specific UBA parameters delivered in the pack s template file. Appendix D, Aurora Netflow Collector Installation and Configuration Provides instructions for installing, configuring, and running the Aurora Netflow Collector. The Netcool/Proviso Product Suite Netcool/Proviso is made up of the following components: Netcool/Proviso DataMart is a set of management, configuration and troubleshooting GUIs that the Netcool/Proviso System Administrator uses to define policies and configuration, as well as verify and troubleshoot operations. Netcool/Proviso DataLoad provides flexible, distributed data collection and data import of SNMP and non-snmp data to a centralized database. Netcool/Proviso DataChannel aggregates the data collected through Netcool/Proviso DataLoad for use by the Netcool/Proviso DataView reporting functions. It also processes on-line calculations and detects realtime threshold violations. Netcool/Proviso DataView is a reliable application server for on-demand, web-based network reports. Netcool/Proviso Technology Packs extend the Netcool/Proviso system with service-ready reports for network operations, business development, and customer viewing. The following figure shows the different Netcool/Proviso modules. x Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

11 Preface Figure 1: Netcool/Proviso Modules DataLoad collects network data. DataChannel computes aggregations and stores data in DataMart. DataMart provides data management and applications. DataView produces and manages reports. Netcool/Proviso documentations consists of the following: release notes configuration recommendations user guides technical notes online help The documentation is available for viewing and downloading on the infocenter at: Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X xi

12 Preface xii Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

13 Chapter 1: Introduction This chapter discusses the following topics: Topic Page Overview 1 How is the Data Collected? 1 How is the Data Processed and Loaded into the Database? 2 How is the Data Reported? 3 Overview Netcool/Proviso Technology Packs are individually installed modules that contain discovery, grouping, collection, and reporting instructions created by Technology Pack developers or Professional Services personnel for a specific network technology or network equipment provider, such as NetFlow IPFIX. Once installed, a Technology Pack programs Netcool/Proviso to inventory and group specific network resources, perform mathematical calculations on raw or aggregated data streams collected from these resources, and produce key performance monitoring metrics that are stored on the Netcool/Proviso database. Reporter sets designed for each Technology Pack query the database for this information and display the results on a web portal in the form of specialized report tables, graphs, and charts. This chapter provides an overview of how Technology Packs work with Netcool/Proviso to collect, process, load, and report data from a network environment. How is the Data Collected? Inventory After a Netcool/Proviso Technology Pack has been installed, an inventory process must be run so that the Technology Pack s target network resources can be identified and modeled within Netcool/Proviso as subelements. Sub-elements must be created before data from any network resource can be collected, processed, and stored in the database. How sub-elements are discovered depends on whether the target resources are SNMP resources or non-snmp resources: For SNMP network resources, a Technology Pack provides a discovery formula. Discovery formulas are used to evaluate network resources, determine which resources become sub-elements, and set values for items defined as properties in the formula. Once all the sub-elements have been discovered and created in the database, the inventory process organizes the sub-elements according to the grouping rules provided by a Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 1

14 Chapter 1: Introduction Technology Pack. The grouping rules use the properties and values defined within inventory formulas or adaptor design files to filter the sub-elements into their appropriate collection or reporting groups. For non-snmp network resources, a Technology Pack provides a Bulk Adaptor Design File. Bulk Adaptor Design Files define how data fields that reside in files generated by network resources are used to create subelements and set values for items defined as properties. During the database synchronization phase of the inventory process, the list of discovered sub-elements is compared to what exists in the database. Newly discovered sub-elements are assigned a resource identifier (RID) and timestamp, and the database uses both items to create a new entry for the sub-element to reflect the known state of the network inventory. Collection Following the completion of the inventory process, Netcool/Proviso is ready to collect performance data for a Technology Pack s target network resources. A Technology Pack provides Netcool/Proviso with collection formulas that instruct a DataLoad collector residing in a DataChannel to collect specific types of performance data against a particular sub-element. The types of collection formulas applied to the data depend on whether the sub-element is an SNMP resource or non-snmp resource: For SNMP network resources, a Technology Pack provides SNMP collection formulas. SNMP collection formulas instruct the SNMP Collector to take the data gathered during a specified collection interval, perform some mathematical operation on the data, and output the result as a performance metric. For non-snmp network resources, a Technology Pack provides Bulk collection formulas. A Bulk collection formula, unlike an SNMP collection formula, has no executable instructions. A Bulk collection formula consists of a metric name that is mapped by the Bulk Adaptor Design File to a statistic that resides in the input file generated by a network resource. Either type of collection formula outputs a metric and corresponding unique metric identifier (MID) that are passed along the DataChannel to the Complex Metric Engine for processing. How is the Data Processed and Loaded into the Database? The Complex Metric Engine (CME) is a component in the DataChannel that performs calculations on data gathered by the SNMP or Bulk collectors deployed in a network. These calculations include the following: Pre-defined formulas that are provided by a Technology Pack User-defined formulas created using the CME Formula API Time aggregations for sub-elements In addition to performing calculations on the data stream in the DataChannel, the CME also buffers and sorts metric records according to their RIDs and MIDs to optimize how the data and metrics are stored in the database. 2 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

15 Chapter 1: Introduction CME Operation The CME is designed to work with data that is gathered over the space of one hour. Within that hour, there are several distinct phases that characterize the operation of the CME: Loading Phase Building Phase Processing Phase Output Phase Loading Phase During the beginning of each hour, the CME creates a new network configuration model based on the inventory structure that exists in the database at that time. This model is used as a snapshot of the network environment and serves as the basis for all metric processing for the entire duration of the hour. At the end of the hour, the CME polls the database for any changes that have been made to the inventory and creates a new network configuration model for the next hour. As the CME reloads its configuration hourly, any metrics produced by sub-elements following the last polling period are rejected by the CME until the beginning of the next hour. Building Phase Once the current configuration model has been built, the CME creates a table of sub-elements and metrics that are expected for the current hour. The CME uses the RIDs and MIDs to build the table and determine which metrics should be arriving from the collectors. The table also specifies how resources are related, and determines if there are any CME formulas that must be applied to a sub-element s metrics once the data is gathered. Processing Phase Whenever new data arrives at the CME, it is evaluated and stored in the appropriate table location, along with any Resource Aggregation information. Once the input and processing dependencies for a metric in the table have been met, the CME processes the metrics and stores the data until the end of the hour. Output Phase At the end of the current hour, the CME outputs everything in memory to the Hourly and Daily Loaders. The data sent to the database loaders includes the sorted data for the current hour, and resource and group aggregations for each of the processing periods up to the current time. The Hourly Loader computes group and resource aggregations, while the Daily Loader creates metric statistics and inserts the data into the database. How is the Data Reported? The data collected, processed, and stored in the database by Netcool/Proviso is organized and output for customers using reporter sets that are designed by developers and Professional Services personnel for a specific Technology Pack. Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 3

16 Chapter 1: Introduction A Technology Pack reporter set is a related group of reports that provide performance information on a specific set of devices or resources that exist in a network. Each report consists of a series of queries that retrieve related sub-elements and their corresponding metrics from the database using the RIDs and MIDs assigned during the inventory and collection processes. The retrieved results are then organized within the report and are displayed on a web portal in the form of tables, graphs, and charts. A Technology Pack provides a variety of reports and charts, including: Resource Summary Reports (RSTs) (sometimes referred to as Resource Summary Tables) aggregate data across time for an individual sub-element and define the statistics that are relevant for a measure of its network performance. Group Summary Reports (GSTs) (sometimes referred to as Group Summary Tables) aggregate data across both time and sub-elements, and define the statistics that are relevant for a measure of network performance for a group of resources. Detail Charts (DCs) display raw data for a metric that has been aggregated for an individual sub-element over a particular period of time. Dashboards display a series of abbreviated tables and charts that provide an overall summary of the metrics collected for the target network resource supported by the Technology Pack. Timeseries charts present time series information, where the x axis is time and the y axis is the data value. TopN reports list the N highest values during a specified time range for a specific metric. Resource Distribution charts (often referred to as pie charts) show the distribution of resources across specific ranges. Typically, a legend for the Resource Distribution chart appears in the report. Resource Over Threshold Tables (RTTs) display a list of resources that have violated their threshold for a specific metric. Typically, users navigate to an RTT from a Group Summary Table (GST), using the threshold overflow cell navigation link. Ratio charts show the relationship of a single metric among different resources or a single resource among different metrics. 4 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

17 Chapter 2: Supported Reports This chapter discusses the following topics: Topic Page Overview 5 Reporter Sets 5 Overview The NetFlow IPFIX Technology Pack supplies a set of reports suitable for displaying information about the devices and activity associated with the NetFlow IPFIX environment. Reports contain metrics that are generated by the formulas that this technology pack provides. Metric names are the same as the names of the formulas that generate them. For information about a metric that is listed for a particular report, see the description of the associated formula in the collection formula chapter of this guide. This chapter includes information to help you navigate to a particular report on the DataView portal. This navigation path is the same as the path where the report has been deployed through the DataMart Resource Editor. Note, however, that some reports are not explicitly deployed on the portal navigation path. You can display such a report by "drilling down" to it from other reports. For information about understanding report types, creating reports, configuring reports, viewing and working with reports, and deploying reports, see the Netcool/Proviso DataView User s Guide. This Technology Pack User s Guide assumes an understanding of the report-related topics discussed in the Netcool/Proviso DataView User s Guide. Reporter Sets Reporter Set Tree This technology pack provides the reporter sets listed below. The list presents the reporter sets as they appear in the DataView Navigator tree structure: AP Cisco NetFlow Pack Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 5

18 Chapter 2: Supported Reports Reports This technology pack provides the following reports: Throughput Summary Group Throughput Summary Resource TopN ToS Summary Group Traffic Summary Group Traffic Summary Resource The reports in this technology pack are described on the following pages. 6 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

19 Chapter 2: Supported Reports Throughput Summary Group This group summary table displays data for throughput for sub-elements, aggregated across time and subelements: bytes in and out, packets in and out, and total bytes and packets. Reporter Set AP Cisco NetFlow Pack Tables and Charts Tables: Charts: Group Summary Table (1 of 2 implemented) Group Time Series Chart (2 of 2 implemented) Metrics Used in Report Bytes (throughput) Bytes In (throughput) Bytes Out (throughput) Packets (throughput) Packets In (throughput) Packets Out (throughput) Navigation Path Navigation path on the DataView portal: NOC Reporting > Cisco NetFlow > AS Matrix NOC Reporting > Cisco NetFlow > Network Applications NOC Reporting > Cisco NetFlow > Network Matrix NOC Reporting > Cisco NetFlow > Network to Host NOC Reporting > Cisco NetFlow > Networks NOC Reporting > Cisco NetFlow > Network ToS NOC Reporting > Cisco NetFlow > ToS Network NOC Reporting > Cisco NetFlow > Host to Network Drills Down To Throughput Summary Group Throughput Summary Resource Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 7

20 Chapter 2: Supported Reports Drilled Down From Throughput Summary Group 8 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

21 Chapter 2: Supported Reports Throughput Summary Resource This resource summary table displays aggregated data for throughput for sub-elements: bytes in and out, packets in and out, and total bytes and packets. Reporter Set AP Cisco NetFlow Pack Tables and Charts Charts: GroupTimeSeries (6 implemented) Metrics Used in Report Bytes (throughput) Bytes In (throughput) Bytes Out (throughput) Packets (throughput) Packets In (throughput) Packets Out (throughput) Navigation Path This report is not deployed on the portal navigation path during the initial configuration of the technology pack. To display this report, drill down to it from another report or deploy it with the DataMart Resource Editor. Drills Down To Drilled Down From Throughput Summary Group Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 9

22 Chapter 2: Supported Reports TopN ToS Summary Group This report summarizes the N highest values for total bytes and packets during a specified time range. Reporter Set AP Cisco NetFlow Pack Tables and Charts Tables: Charts: TopN (2 of 9 implemented) TopN (2 of 9 implemented) Metrics Used in Report Bytes Packets Navigation Path Navigation path on the DataView portal: NOC Reporting > Cisco NetFlow > ToS Network NOC Reporting > Cisco NetFlow > Network ToS Drills Down To TopN ToS Summary Group Traffic Summary Resource Drilled Down From TopN ToS Summary Group 10 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

23 Chapter 2: Supported Reports Traffic Summary Group This group summary table displays aggregated traffic data: Bytes in the inbound and outbound directions, and total bytes Packets in the inbound and outbound directions, and total packets Reporter Set AP Cisco NetFlow Pack Tables and Charts Tables: Charts: Group Summary Table (1 of 2 implemented) Group Time Series Chart (2 of 2 implemented) Metrics Used in Report Bytes In Bytes Out Bytes Packets In Packets Out Packets Navigation Path Navigation path on the DataView portal: NOC Reporting > Cisco NetFlow > Networks NOC Reporting > Cisco NetFlow > AS Matrix NOC Reporting > Cisco NetFlow > Network Applications NOC Reporting > Cisco NetFlow > Network Matrix NOC Reporting > Cisco NetFlow > Network to Host NOC Reporting > Cisco NetFlow > Network ToS NOC Reporting > Cisco NetFlow > ToS Network NOC Reporting > Cisco NetFlow > Host to Network Drills Down To Traffic Summary Group Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 11

24 Chapter 2: Supported Reports Traffic Summary Resource Drilled Down From Traffic Summary Group 12 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

25 Chapter 2: Supported Reports Traffic Summary Resource This resource summary table displays aggregated traffic data for sub-elements, as a time series chart: Bytes in the inbound and outbound directions, and total bytes Packets in the inbound and outbound directions, and total packets Reporter Set AP Cisco NetFlow Pack Tables and Charts Charts: GroupTimeSeries (6 implemented) Metrics Used in Report Bytes In Bytes Out Bytes Packets In Packets Out Packets Navigation Path This report is not deployed on the portal navigation path during the initial configuration of the technology pack. To display this report, drill down to it from another report or deploy it with the DataMart Resource Editor. Drills Down To Drilled Down From TopN ToS Summary Group Traffic Summary Group Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 13

26 Chapter 2: Supported Reports NOTES 14 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

27 Chapter 3: Devices and Services This chapter discusses the following topics: Topic Page Overview 15 Summary of Device Technology 15 Technology Pack Overview 16 Components and Data Flow 16 Configuration File 17 Input Files 17 Overview This chapter provides a summary of NetFlow IPFIX and how this technology pack uses NetFlow datagrams to monitor and provide performance reports about NetFlow-enabled routers. Note: The NetFlow IPFIX Technology Pack operates with routers running NetFlow, versions 1, 5 through 9, and IETF IPFIX. Summary of Device Technology NetFlow IPFIX is a proprietary network accounting protocol. Cisco routers and compatible devices export details of routed packets in a stream of UDP/TCP/SCTP datagrams to a NetFlow collector. These datagrams contain detailed data that can be used to provide a comprehensive understanding of network behavior. NetFlow was developed by Cisco as an agent that runs on its routers and measures traffic flows traversing a router s interfaces. The flow information is then sent in NetFlow records, carried in UDP datagrams, to a collector to process and make the information available to users. The set of information collected and the format of the records has evolved over time and been adopted by other vendors. The IP Flow Informaton exchange (IETF IPFIX) working group has adopted it as the standard for carrying traffic flow information. A flow record includes the following information: source and destination IP address, source and destination subnet, source and destination port, AS of previous hop and AS of destination address, IP header protocol field value, ToS (DSCP, IP Prec), a timestamp, and the number of packets and octets. It is also extendable and can contain the MPLS Label stack, and more. Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 15

28 Chapter 3: Devices and Services Technology Pack Overview This technology pack supports NetFlow and IPFIX flow measurement technology. It obtains NetFlow datagrams directly from NetFlow-enabled routers and uses this information to provide performance reports about network- and application-layer IP traffic on these devices. Components and Data Flow The major technology pack components are: Aurora Netflow Collector the technology pack adapter that captures NetFlow datagrams and exports this information to CSV files. The Aurora Netflow Collector captures NetFlow datagrams, performs data enrichment, aggregations, and publishes the results as CSV files. CiscoNetFlowAdaptor.js the technology pack s UBA design file that obtains the CSV files and imports the data into Netcool/Proviso. Aurora Netflow Collector The Aurora NetFlow system has been extended to interoperate with Netcool/Proviso. The key issues for using Aurora with Proviso are: Routers can be configured to send NetFlow/IPFIX information over UDP, TCP or SCTP directly to the Aurora server. The receive port can be configured (the default is UDP/2055). Router interfaces can be explicitly ignored, for instance to avoid duplicate flow reporting. The size of the receive buffer can be configured (the default is50mb, i.e., a burst of 1 M flows can be accommodated). The Aurora release for Proviso is configured by default to write a CSV file for every full hour. The file is written once after no more NetFlow information is expected to affect this hour. The database is configured such that daily, monthly and yearly information is not maintained. If a flow spreads over two hours, its volume will be correctly split between the two hours. Aurora can be configured with user-defined aggregations. An aggregation or topic is called aspect in Aurora. The default configuration for Proviso contains pre-configured user-defined aspects. For more information about aspects, see Appendix D, Aurora Netflow Collector Installation and Configuration. Aurora for Proviso is written in C and compiles under Linux, Solaris and AIX. The aurorad executable is controlled by means of the shell script aurora. Aurora processes run under system user pvuser in group staff (on Solaris) and user aurora in group aurora on Linux. The configuration can be modified by means of the aurora script and by editing the configuration files directly. The aurora script and configuration files are described in more detail in Appendix D, Aurora Netflow Collector Installation and Configuration. An accurate system time is important for running Aurora. The system time must be identical to the system time of the exporting routers. The following provides a high-level description of the data flow involved: 1. At regular intervals, NetFlow-enabled routers export datagrams to the Aurora engine. 2. The Aurora engine aggregates data according to the site configuration and creates CSV files in the data directory. 3. The CiscoNetFlowAdaptor.js adaptor file polls the data directory on a regular basis and processes the CSV files using an inline inventory. 16 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

29 Chapter 3: Devices and Services The NetFlow datagrams come in a variety of versions. Version 9 is the latest version and is the basis for the IETF standard called IPFix. IPFix, Version 5 is the most common today and contains the following fields: Source IP address Destination IP address Source TCP/UDP port Destination TCP/UDP port IP protocol Type of Service (Tos) byte Input physical interface index Output physical interface index TCP flags (cumulative OR) Next hop router IP Packet count Byte count Start of flow timestamp End of flow timestamp Source AS number Destination AS number Source subnet mask Destination subnet mask The first seven fields are used as the primary key to identify unique flows. Configuration File Information about this technology pack must be added to the DataChannel s configuration file (dc.cfg). To facilitate this process, a template configuration file is included with this pack when it is installed. The template file is named cisco_netflow_sample_dc.cfg and contains pack-specific parameters that must be added to dc.cfg. Information about using this file is located in Appendix A, Configuration and Appendix C, Template File Reference (Pre-443 Only). Input Files The Aurora Netflow Collector performs data enrichment, aggregates the data according to a mapping file, and creates CSV files in its data directory. The CSV files are named using the following syntax: The exact syntax is: <yyyy>-<mm>-<dd>_<hh>-<mm>.csv The structure of the CSV files are defined in the /opt/aurora/var/sites/proviso/etc/report_site.ash file. The following is a sample record. Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 17

30 Chapter 3: Devices and Services site aspect select NetHost octets $keys$ = site aspect get keys loop i 1 { site aspect select NetHost octets $bytes_sent$ = site aspect printf $bytes_recv$ = site aspect printf site aspect select NetHost packets $packets_sent$ = site aspect printf $packets_recv$ = site aspect printf site aspect printf "Source=Aurora,Site=proviso,Topic=NetHost,Timestamp=$date$,SE,ip_router =%flow:exporter_ip%,domain_src=%flow:domain_src#name%,ip_dst=%flow:ip_dst# name%,application=%flow:application#name%,ue,packetsout=$packets_sent$, } site aspect select HostNet octets $keys$ = site aspect get keys loop i 1 { site aspect select HostNet octets $bytes_sent$ = site aspect printf $bytes_recv$ = site aspect printf site aspect select HostNet packets $packets_sent$ = site aspect printf $packets_recv$ = site aspect printf site aspect printf "Source=Aurora,Site=proviso,Topic=HostNet,Timestamp=$date$,SE,ip_router =%flow:exporter_ip%,ip_src=%flow:ip_src#name%,domain_dst=%flow:domain_dst# name%,application=%flow:application#name%,ue,packetsout=$packets_sent$, } SE fields correspond to Proviso Sub-Element properties. UE fields correspond to Proviso metrics values. Since a fixed field structure is not used, each record is read in as a single field. Javascript string processing functions are used to process the record into its constituent parts. 18 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

31 Chapter 4: Element and Sub-Element Properties This chapter discusses the following topics: Topic Page Overview 19 NetFlow IPFIX Device Element Properties 19 NetFlow IPFIX Device Sub-Element Properties 19 Overview A Netcool/Proviso property is an attribute or characteristic of an element or sub-element. Typically, a Netcool/Proviso Technology Pack uses an inventory process to collect information generated by the devices, interfaces, and services operating in the technology environment it supports. A technology pack uses Netcool/Proviso properties for some or all of the following purposes: To organize information in reports To display values in reports To construct names such as sub-element names, sub-element instance names, and sub-element labels The Netcool/Proviso properties associated with sub-elements appear in the Properties list (Sub-Element view) of the Netcool/Proviso DataMart Resource Editor. See the Netcool/Proviso DataMart Configuration Guide for more information on the Resource Editor. NetFlow IPFIX Device Element Properties The NetFlow IPFIX Technology Pack creates an element for each source router. The element name is the router IP address. NetFlow IPFIX Device Sub-Element Properties The following pages describe the sub-element properties used in this technology pack. Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 19

32 Chapter 4: Element and Sub-Element Properties Element.STATE Determines whether the element is active, and controls the collection for all sub-elements attached to the element. The value must be on in order for collection and discovery to occur. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Sub-Element Collect Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 20

33 Chapter 4: Element and Sub-Element Properties NF_Application The layer-4 application name. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 21

34 Chapter 4: Element and Sub-Element Properties NF_DestIPAddress The destination IP address of the flow. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 22

35 Chapter 4: Element and Sub-Element Properties NF_DestIPAsNumber The destination IP as an integer. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 23

36 Chapter 4: Element and Sub-Element Properties NF_DestMask The destination routes mask bits. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 24

37 Chapter 4: Element and Sub-Element Properties NF_DestNet The destination mask applied to the destination IP. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 25

38 Chapter 4: Element and Sub-Element Properties NF_DestPort The destination port of the flow. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 26

39 Chapter 4: Element and Sub-Element Properties NF_DstAsNum This property is used to store the Destination ASNumber. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 27

40 Chapter 4: Element and Sub-Element Properties NF_InputIFIndex The input interface index number. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 28

41 Chapter 4: Element and Sub-Element Properties NF_NetworkName The network name. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 29

42 Chapter 4: Element and Sub-Element Properties NF_NextHop The next hop routers IP address. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 30

43 Chapter 4: Element and Sub-Element Properties NF_OutputIFIndex The output interface index number. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 31

44 Chapter 4: Element and Sub-Element Properties NF_Protocol The protocol name. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 32

45 Chapter 4: Element and Sub-Element Properties NF_Router The IP address of the router that the NetFlow was exported from. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 33

46 Chapter 4: Element and Sub-Element Properties NF_Site The Aurora site profile that is being used. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 34

47 Chapter 4: Element and Sub-Element Properties NF_SourceIPAddress The source IP address of the flow. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 35

48 Chapter 4: Element and Sub-Element Properties NF_SourceIPAsNumber The source IP as an integer. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 36

49 Chapter 4: Element and Sub-Element Properties NF_SourceMask The source routes mask bits. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 37

50 Chapter 4: Element and Sub-Element Properties NF_SourceNet The source mask applied to the source IP. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 38

51 Chapter 4: Element and Sub-Element Properties NF_SourcePort The source port of the flow. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 39

52 Chapter 4: Element and Sub-Element Properties NF_SrcAsNum This property is used to store the Source ASNumber. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 40

53 Chapter 4: Element and Sub-Element Properties NF_TCPFlags The cumulative OR of the TCP flags. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 41

54 Chapter 4: Element and Sub-Element Properties NF_Topic The name of the aggregation that the SMDR describes. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Sub-Element Collect Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 42

55 Chapter 4: Element and Sub-Element Properties NF_Tos The value of the ToS byte. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 43

56 Chapter 4: Element and Sub-Element Properties STATE Determines whether the sub-element is active, and controls the collection for one sub-element. The value must be on in order for collection and discovery to occur. Captured in Discovery Formulas Used in CME Formulas Used in Reports Used in Grouping Rules NOC Reporting Sub-Element Collect Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 44

57 Chapter 5: Sub-Element Grouping Rules This chapter discusses the following topics: Topic Page Overview 45 NOC Reporting Grouping Rules 46 Sub-Element Collect Grouping Rules 49 Overview Typically, a Netcool/Proviso Technology Pack defines sub-element grouping rules. Grouping rules organize and filter networking data into a structure that makes it easier to collect data on thousands or millions of resources. Sub-element grouping rules organize data in tree structures. Each folder (branch) in the tree is associated with a rule or condition. If a sub-element property satisfies the rule or condition associated with a folder in the grouping tree, the sub-element is placed in the folder. The sub-element is then tested against the rule or condition in the next folder below. If the sub-element passes that test, it is tested against the rule or condition in the next folder, and so on. The NetFlow IPFIX Technology Pack uses some of its Netcool/Proviso properties to define sub-element grouping rules. Grouping rule names appear in the Name field (Rule Sub-Element view) of the Netcool/Proviso DataMart Rule Editor. For information on how to use the Netcool/Proviso DataMart Rule Editor, see the Netcool/Proviso DataMart Configuration Guide. The Rule Editor contains two grouping rule trees the NOC reporting tree and the sub-element collect tree. The following sections describe the filtering rules and conditions in these trees. Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 45

58 Chapter 5: Sub-Element Grouping Rules NOC Reporting Grouping Rules The NOC reporting tree provides the framework that allows users to navigate to reports. The following illustration is a portion of the NOC Reporting tree as it appears in the DataMart Rule Editor. The shaded branches represent the NetFlow IPFIX portion of the tree: The branches are described as follows: 46 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

59 Chapter 5: Sub-Element Grouping Rules Branch Name Cisco NetFlow NetApp %(NF_NetworkName) %(NF_Protocol) %(NF_Application) Networks %(NF_NetworkName) AS Matrix Source AS%(NF_SrcAsNum) Destination AS%(NF_DstAsNum) Network Matrix Source %(NF_SourceNet) Destination %(NF_DestNet) %(NF_Application) Type: Static. Purpose: Organization only (contains all NetFlow IPFIX sub-elements). %(NF_Topic) IS NOT NULL Collects any resources that have the NF_Topic property. Type: Static Traffic summary group. %(NF_Topic) = 'NetApp' Collects resources whose NF_Topic property is equal to NetApp. Type: Rule. Purpose: Groups NetApp sub-elements according to name. Type: Rule. Purpose: Groups NetApp sub-elements according to protocol. Type: Rule. Purpose: Groups NetApp sub-elements according to application. Type: Static. Purpose: Filters by condition: %(NF_Topic) = 'Network' Uses the NF_Topic property to group Network sub-elements. Type: Rule. Purpose: Groups Network sub-elements according to name. Type: Static. Purpose: Filters by condition: %(NF_Topic) = 'AsNum' Uses the NF_Topic property to group AsNum sub-elements. Type: Rule. Purpose: Groups collected AsNum sub-elements according to originating source. Type: Rule. Purpose: Groups collected AsNum sub-elements according to destination source. Type: static %(NF_Topic) = 'NetMatrix' Uses the NF_Topic property to group NetMatrix sub-elements. Type: rule Purpose: Groups collected NetMatrix sub-elements according to originating source. Type: rule Purpose: Groups collected NetMatrix sub-elements according to destination source. Type: Rule. Purpose: Groups NetMatrix sub-elements according to application. Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 47

60 Chapter 5: Sub-Element Grouping Rules Branch Name Network to Host Source %(NF_SourceNet) Destination %(NF_DestIPAddress) %(NF_Application) Host to Network Source %(NF_SourceIPAddress) Destination %(NF_DestNet) %(NF_Application) Network ToS %(NF_NetworkName) %(NF_ToS) ToS Network %(NF_ToS) %(NF_NetworkName) Type: Static Traffic summary group. %(NF_Topic) = 'NetHost' Collects resources whose NF_Topic property is equal to NetHost. Type: rule Purpose: Groups collected NetHost sub-elements according to originating source. Type: rule Purpose: Groups collected NetHost sub-elements according to destination source. Type: Rule. Purpose: Groups collected NetHost sub-elements according to application. Type: Static Traffic summary group. %(NF_Topic) = 'HostNet' Collects resources whose NF_Topic property is equal to HostNet. Type: rule Purpose: Groups collected HostNet sub-elements according to originating source. Type: rule Purpose: Groups collected HostNet sub-elements according to destination source. Type: Rule. Purpose: Groups collected HostNet sub-elements according to application. Type: Static. Purpose: Filters by condition: %(NF_Topic) = 'NetToS' Uses the NF_Topic property to group NetToS sub-elements. Type: Rule. Purpose: Groups NetToS sub-elements according to name. Type: Rule. Purpose: Groups NetToS sub-elements according to protocol. Type: Static. Purpose: Filters by condition: %(NF_Topic) = 'NetToS' Uses the NF_Topic property to group NetToS sub-elements. Type: Rule. Purpose: Groups NetToS sub-elements according to protocol. Type: Rule. Purpose: Groups NetToS sub-elements according to name. 48 Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X

61 Chapter 5: Sub-Element Grouping Rules Sub-Element Collect Grouping Rules The sub-element collection tree organizes the requests that collect metrics from sub-elements. The following illustration is a portion of the Sub-Element Collect tree as it appears in the DataMart Rule Editor. The shaded branches represent the NetFlow IPFIX portion of the tree: The branches are described as follows: Branch Name Cisco NetFlow Type: Static. Purpose: Organization only (contains all Cisco NetFlow sub-elements). %(NF_Topic) IS NOT NULL Collects any resources that have the NF_Topic property. Netcool/Proviso NetFlow IPFIX Technology Pack User s Guide, Version 4.3-X 49