TIBCO FTL Monitoring. Software Release 5.0 May Two-Second Advantage

Transcription

1 TIBCO FTL Monitoring Software Release 5.0 May 2016 Two-Second Advantage

2 2 Important Information SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE. USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE LICENSE FILE(S) OF THE SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME. This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written authorization of TIBCO Software Inc. TIBCO, Two-Second Advantage, TIB, Information Bus, eftl and Rendezvous are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. Enterprise Java Beans (EJB), Java Platform Enterprise Edition (Java EE), Java 2 Platform Enterprise Edition (J2EE), and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle Corporation in the U.S. and other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only. THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THE SAME TIME. SEE THE README FILE FOR THE AVAILABILITY OF THIS SOFTWARE VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM. THIS DOCUMENT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME. THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES. Copyright TIBCO Software Inc. ALL RIGHTS RESERVED. TIBCO Software Inc. Confidential Information

3 3 Contents About this Product...4 TIBCO Documentation and Support Services...5 Client Monitoring...7 Realm Server Monitoring... 7 Data Flow of Monitoring Metrics...8 The Monitoring Endpoint Subscribing to the Monitoring Stream...10 Structure of Monitoring Data Messages...11 Fields of Monitoring Data Messages Catalogs of Metrics...16 Catalog of Client Metrics...16 Catalog of Persistence Metrics...18 Catalog of eftl Metrics...18 Prometheus Adapter (tibpromgateway) Command Line Reference...21 Docker Files for Prometheus and Grafana...25 Prometheus Labels...26 Security of Monitoring Data...27 Monitoring Data: Timing and Interruptions...28 Estimating the Volume of Metric Data...30

4 4 About this Product TIBCO is proud to announce the latest release of TIBCO FTL software. This release is the latest in a long history of TIBCO products that leverage the power of Information Bus technology to enable truly event-driven IT environments. To find out more about how TIBCO FTL software and other TIBCO products are powered by TIB technology, please visit us at

5 5 TIBCO Documentation and Support Services Documentation for this and other TIBCO products is available on the TIBCO Documentation site. This site is updated more frequently than any documentation that might be included with the product. To ensure that you are accessing the latest available help topics, please visit: Product-Specific Documentation Documentation for TIBCO products is not bundled with the software. Instead, it is available on the TIBCO Documentation site. To access the documentation web page for this product from a local software installation, open the following file: TIBCO_HOME/release_notes/TIB_ftl_5.0.0_docinfo.html TIBCO_HOME is the top-level directory in which TIBCO products are installed. On Windows platforms, the default TIBCO_HOME is C:\tibco. On UNIX platforms, the default TIBCO_HOME is /opt/tibco. The following documents for this product can be found on the TIBCO Documentation site. TIBCO FTL Documentation Set TIBCO FTL Concepts Everyone reads this booklet first, for an intuitive introduction to the fundamental concepts of FTL software. TIBCO FTL Development Application developers and architects read this manual to understand concepts relevant in any supported programming language. TIBCO FTL API Reference Application developers use this HTML documentation to learn the details of the FTL API in specific programming languages. TIBCO FTL Administration Administrators read this manual to learn how to use the realm server and its interfaces, and how to define a realm. Developers can also benefit from understanding FTL software from an administrator s perspective. TIBCO FTL Monitoring Administrators read this manual to learn about monitoring and metrics. Developers read this manual to learn how an application can subscribe to the stream of monitoring data. TIBCO FTL Installation Read this manual before installing or uninstalling the product. TIBCO FTL Glossary The glossary contains brief definitions of key terms used in all other parts of the documentation set. TIBCO FTL Release Notes Read the release notes for a list of new and changed features. This document also contains lists of known issues and closed issues for this release. TIBCO eftl Documentation Set TIBCO eftl software is documented separately. Administrators use the FTL realm server GUI to configure and monitor the eftl service. For information about these GUI pages, see the documentation set for TIBCO eftl software. How to Contact TIBCO Support For comments or problems with this manual or the software it addresses, contact TIBCO Support: For an overview of TIBCO Support, and information about getting started with TIBCO Support, visit this site:

6 6 If you already have a valid maintenance or support contract, visit this site: Entry to this site requires a user name and password. If you do not have a user name, you can request one. How to Join TIBCOmmunity TIBCOmmunity is an online destination for TIBCO customers, partners, and resident experts. It is a place to share and access the collective experience of the TIBCO community. TIBCOmmunity offers forums, blogs, and access to a variety of resources. To register, go to the following web address:

7 7 Client Monitoring The TIBCO FTL client library monitors each client process, sampling several values that reveal operating behavior and performance. The client library sends these monitoring samples to the realm server, which publishes them, and displays the most recent values for administrators. Scope The TIBCO FTL library gathers monitoring data from the following processes: Application program processes TIBCO FTL component processes, such as bridge processes, persistence store servers, and TIBCO eftl servers The realm server process Sampling Throughout the realm, every client process samples data at a regular interval. The realm property Client Monitoring Sample Interval specifies that interval. Zero is a special value, instructing all clients to disable the monitoring feature. Each client sends all its pending sample data to the realm server in the next client heartbeat message. The first batch of data becomes available with the first heartbeat after the first sample interval. If the heartbeat interval is large, then the realm server receives a batch of accumulated samples in each heartbeat). Realm Server Monitoring The TIBCO FTL realm server also monitors its own operation, sampling several values that measure operating behavior and performance. The realm server publishes these metrics, and displays the most recent values for administrators.

8 8 Data Flow of Monitoring Metrics The following diagram illustrates the flow of FTL monitoring data. Explanation of the Diagram Metric data begins in client processes. The TIBCO FTL client library monitors the metrics relevant to each client, and includes monitoring data values in the heartbeat data that it sends to the TIBCO FTL realm server. The realm server receives the monitoring data from its client processes. When the realm server receives monitoring data from any of its clients, the realm server immediately publishes that data in an FTL message. The stream of those messages is called the monitoring stream. Any FTL application can subscribe to the built-in monitoring endpoint to receive the monitoring stream. You can develop custom monitoring applications that subscribe to the monitoring stream and use the metrics data. TIBCO FTL software includes an executable application, tibpromgateway, that subscribes to the monitoring stream and makes the monitoring data available to Prometheus. Prometheus is an open-source systems monitoring and alerting toolkit. You can access the data directly from Prometheus, or using other tools that manipulate and display the data, for example, Grafana. For more information about this data pathway, see Prometheus, and the topics that follow it. The realm server retains only the most recent value of each metric from each client. These values are available from the realm server through its web API and browser GUI. Key to Shape and Colors in the Diagram Rectangle Executable program process Ellipse FTL application

9 9 Small Circle (inside another shape) Monitoring endpoint Rounded Rectangle API, GUI, or HTTP request Blue Component of TIBCO FTL software Red Your FTL application Green Third-party tool Change in Functionality In releases prior to 5.0, the realm server retained a set of historical monitoring data. Starting in Release 5.0, the realm server immediately publishes the data and discards all but the most recent sample in each data series. External tools can store and process the data. This arrangement reduces memory usage in the realm server, and makes the data available to analytical tools in a more timely and useful manner.

10 10 The Monitoring Endpoint The stream of monitoring metric data is available to any client program through the monitoring endpoint. The realm server publishes a stream of messages that contain monitoring metrics data. Application programs can subscribe to this stream on the monitoring endpoint. The name of this endpoint is available as a constant value in the TIBCO FTL API library. Subscribing to the Monitoring Stream To receive monitoring data messages, application programs can subscribe to the built-in monitoring endpoint. An application can use a content matcher to narrow the message stream. Procedure 1. Optional. Compose a content matcher. To receive the entire stream, omit any content matcher. To receive a sub-stream, construct an appropriate content matcher. For example, an application could subscribe to all monitoring messages, or to a sub-stream such as the following: Only event messages. Only client metric messages with a specific client label. Only messages that report a specific client status. Only messages about a specific host computer. See Fields of Monitoring Data Messages. 2. Create a subscriber object. a) Specify the API constant that denotes the monitoring endpoint. b) Optional. Specify the content matcher from step Ensure that the application program includes the other tasks required in any subscribing program. For example, the program must define appropriate callbacks, create an event queue, add the subscriber to the event queue, and start a dispatch loop. See "Structuring Programs" in TIBCO FTL Development.

11 11 Structure of Monitoring Data Messages The realm server publishes a stream of monitoring data messages. Each message can contain a group of samples, that is, data values. Each monitoring message also contains fields that identify and describe those samples. Types of Monitoring Messages A monitoring message can contain metric samples from a client, metric samples from the realm server, or data about an event. The msg_type field distinguishes among these three types of message. Descriptive Fields Many of the top level fields of a monitoring message contain descriptive information about the samples or event. For example, they describe the application program, client process, and the sampling time. This descriptive information applies to all of the data samples in the message. For example, all the samples in the message come from the same client, and report data from the same sampling interval. Samples A monitoring message can contain data samples of multiple metrics. The metrics field contains an array of data sample submessages. Data sample submessages can include the following fields: Context The object that this sample measures. For example, the context value could be the human-readable name of a specific endpoint. Type of the metric The type of counter, that is, the kind of measurement. For example, type 11 denotes the bytes sent metric, which counts the number of bytes sent outbound through a transport. For descriptions of the metric types, see Catalogs of Metrics. Value The numeric value of the counter at the end of the sampling interval. ID The unique internal identifier of the counter. This ID unambiguously identifies the specific counter, even though the context and type together are usually sufficient to describe the counter.

12 12 Fields of Monitoring Data Messages The format of monitoring data messages includes many fields that are common to all monitoring messages, and some fields that are present only in certain types of monitoring messages. The following tables describe the fields and their values. Top Level Fields Field Name Data Type Description msg_type long The type of a monitoring message indicates the variable portion of its contents. For a table of possible values, see Message Type Values. Each monitoring message type number is defined as a constant value in the TIBCO FTL API library. client_label string When connecting to the realm server, a client program can supply a label that denotes that particular client. Unlike a client ID, which denotes a specific client process, this label remains unchanged even when the client restarts as a new process. You can use this label to identify clients by their role within your enterprise. client_status string This string denotes the status of the client. For details, see the Status field in "Client Status Reference" in TIBCO FTL Administration. metrics message array Each message in this array contains one data sample of one metric. For a table of possible fields of these sample messages, see Data Sample Submessage. This field contains a value only if the msg_type is client metric or server metric. event_type long If the msg_type is event, this value indicates the type of event. For a table of possible values, see Event Type Values. Each event type number is defined as a constant value in the TIBCO FTL API library. timestamp long The sampling time or event time (in microseconds after the epoch) for the metrics in this message. application string The application name of the client. This field contains a value only if the msg_type is client metric.

13 13 Field Name Data Type Description application_instance string The application instance name of the client. This field contains a value only if the msg_type is client metric. client_id string The client ID of the client process. The realm server assigns each client process a unique client ID. This field contains a value only if the msg_type is client metric. server_id string The UUID of the realm server. This field contains a value only if the msg_type is server metric. pid long The process ID of the client process, as assigned by the operating system of the client's host computer. This field contains a value only if the msg_type is client metric. ftl_user string The client supplied this value for the username property in its realm connect call. effective_user string The user name that started the client process, according its host computer's operating system. The client process inherits permissions from this user. version string The version of the TIBCO FTL client library that the client or realm server process uses. host string The host name of the computer where the client process is running. ip string For a client process, this IP address denotes the network interface through which the client communicates with the realm server. For a realm server, this IP address denotes the primary interface of the host computer. server_uptime long Time (in seconds) that the realm server process has been running. This field contains a value only if the msg_type is server metric. Message Type Values These values occur in the msg_type field of monitoring messages. Each monitoring message type number is defined as a constant value in the TIBCO FTL API library.

14 14 Value Description 1 Client Metric 2 Server Metric 3 Event The message contains client metrics in an array of data sample submessages. The message contains realm server metrics in an array of data sample submessages. The message contains information about an event, such as a new client connection or a state change. The value of event_type field denotes the event. Data Sample Submessage Each sample submessage contains all of these fields. Field Name Data Type Description context string This human-readable string identifies the context of the metric within the client application. For example, if the metric measures some aspect of a specific endpoint, transport, or queue, then the context is the endpoint name, transport name, or queue name. type long This number identifies the type of metric. For a table of possible metrics, see Catalog of Client Metrics. Each type number is defined as a constant value in the TIBCO FTL API library. value long This field contains the data value of the metric. id long This internal ID number uniquely identifies the context of the metric. Event Type Values If the msg_type is event, these values can occur in the event_type field to indicate the type of event. Value Description 1 Connected A client process has connected to the realm server. 2 Disconnected A client process has disconnected from the realm server.

15 15 Value Description 3 Status Change The status of a client process has changed. The client_status field indicates the new status.

16 16 Catalogs of Metrics The catalogs that follow present the available monitoring metrics. ID A unique ID denotes each type of counter. When subscribing to the monitoring stream, content matchers can refer to these ID values using API constants. Metric Type A human-readable string represents each type of counter. These strings appear in Prometheus user interfaces. Description This column explains the meaning of each type counter and the quantity it measures. Tracking This column describes the object that each counter tracks. For example, a counter could collect data about the operation of an endpoint, a transport, an event queue, a client process, or a persistence store. Catalog of Client Metrics These tables describe metrics that you can use to assess the operation of any client process. Standard Client Metrics TIBCO FTL software gathers these metrics for processes, event queues, transports, and endpoints. Unless otherwise specified, a metric reports a counter's value at the end of each sample interval, and the counter resets to zero for the next interval. ID Metric Type Description Tracking 1 Messages Sent Outbound data messages through an endpoint in the client process. Per endpoint 2 Messages Received Inbound data messages through an endpoint in the client process. Per endpoint 12 Bytes Received Inbound data bytes on a transport in the client process. Per transport 11 Bytes Sent Outbound data bytes on a transport in the client process. Per transport 31 Data Lost Inbound data loss events on a transport in the client process. Per transport 32 Format Unavailable It is not possible to measure the magnitude of the events in bytes. Messages with an unrecognized format carried on a transport in the client process. Per transport 41 Queue Backlog Maximum number of messages in an event queue in the client process during the sample interval. 42 Queue Discards Inbound messages discarded by a queue in the client process. Per queue Per queue

17 17 ID Metric Type Description Tracking 51 Dynamic Formats Number of distinct dynamic formats registered within the client. Creating a message with a dynamic format can increment this counter. Receiving an inbound message with a dynamic format can increment this counter. This counter can decrease as the client library automatically unregisters unused formats. Per client 67 Store Mismatch Messages 70 Resident Memory Set Size 71 Peak Resident Memory Set Size 72 Virtual Memory Size Number of message flows that result from store mismatch. Any non-zero value indicates a store mismatch misconfiguration. Store mismatch is a misconfiguration in which a direct path transport connects two endpoints that are associated with two different persistence stores. Current size in kilobytes of an application's working memory set. Maximum size in kilobytes, since the process started, of an application's working memory set. Current size in kilobytes of an application's virtual memory set. Per endpoint Per client Per client Per client 73 User CPU Time Total user CPU time in microseconds, that is, time executing the process' object code. 74 System CPU Time Total system CPU time in microseconds, that is, time executing operating system calls from the process. Per client Since the process started Per client Since the process started Bytes sent and received need not correlate exactly with messages sent and received on corresponding endpoints. For example, filtering and optimizations can decouple these metrics. Client Metrics for Multicast and RUDP Transports TIBCO FTL software gathers additional metrics for each transport that uses UDP-based protocols, such as multicast and RUDP transports. For all metrics in the following table, tracking is per transport. Unless otherwise specified, a metric reports a counter's value at the end of each sample interval. These metrics are cumulative since the transport established its bus.

18 18 ID Metric Type Description 61 Packets Sent Outbound data packets on a multicast or RUDP transport. This counter includes all outbound data packets, including original data packets and retransmitted data packets. 62 Packets Received Inbound data packets on a multicast or RUDP transport. This counter includes all inbound data packets, including original data packets and retransmitted data packets. 63 Packets Retransmitted Outbound packets retransmitted on a multicast or RUDP transport. If the transport retransmits a packet several times, it increments this counter for each retransmission. 64 Packets Missed Number of missed inbound packets on a multicast or RUDP transport. 65 Packets Lost Outbound 66 Packets Lost Inbound Number of outbound packets discarded on a multicast or RUDP transport. Number of inbound packets deemed lost on a multicast or RUDP transport. The transport accounts a packet as lost after its reliability constraints expire. Catalog of Persistence Metrics This table describes metrics that you can use to assess the operation of persistence stores. Each persistence server is itself a client process, and tracks client metrics; see Client Metrics. The sampling interval for persistence metrics is fixed at 2 seconds. (Unlike the sampling interval for client metrics, you cannot change this interval.) Store Metrics ID Metric Type Description Tracking 1000 Message Count Number of messages in a store at the end of the sample interval. Per store 1001 Message Size Storage (in bytes) that messages occupy in a store at the end of the sample interval Durable Count Number of durables in a store at the end of the sample interval. Per store Per store Catalog of eftl Metrics This table describes metrics that you can use to assess the operation of eftl channels. Each TIBCO eftl server is itself a client process, and tracks client metrics: for example, metrics for the server's endpoints, and metrics for the transports that implement those endpoints. See Client Metrics. The sampling interval for eftl metrics is fixed at 2 seconds. (Unlike the sampling interval for client metrics, you cannot change this interval.)

19 19 eftl Metrics ID Metric Type Description Tracking 4000 Connection Count 4001 Subscription Count 4002 Inbound eftl Message Count 4003 Outbound eftl Message Count 4010 Discarded Message Count 4011 Dataloss Message Count 4012 Connection Pending Count 4013 Connection Suspended Count Number of eftl client connections on a channel at the end of the sample interval. Open subscriptions from eftl clients on a channel at the end of the sample interval. Messages inbound from eftl clients on a channel. Messages outbound to eftl clients on a channel. Data loss from publishers: the number of messages that the server discarded on a channel. Discards can occur because a publisher sent a message that exceeded the maximum message size. Data loss to subscribers: the number of messages that the server discarded on a channel. Discards can occur because a subscriber s queue exceeded the maximum outbound backlog. Number of new eftl client connections on a channel that are not yet fully initialized at the end of the sample interval. Number of eftl client connections disconnected by a temporary network outage that have not yet automatically reconnected nor fully disconnected at the end of the sample interval. Per channel Per channel Per channel Per channel Per channel Per channel Per channel Per channel

20 20 Prometheus You can use a variety of other third-party tools, such as Prometheus and Grafana, to store, analyze, and visualize monitoring data. Prometheus is an open source toolkit for systems monitoring and alerting. Grafana is an open source application for visualizing large-scale measurement data. You can run Prometheus and Grafana processes in Docker containers. TIBCO FTL software includes sample Docker files to configure and build Docker images for this purpose. Grafana dashboards in the sample are tailored to display FTL monitoring data. (See Docker Files for Prometheus and Grafana.) Pushgateway and Adapter Prometheus scrapes data from monitoring sources into its data repository. Prometheus cannot receive data directly from FTL applications or the realm server. Instead, Prometheus provides an intermediate component, called Pushgateway, that receives data and caches it so that Prometheus can scrape it at a regular interval. Pushgateway is not an FTL application, so it cannot subscribe to the monitoring message stream. TIBCO FTL software includes an adapter application, called tibpromgateway, that subscribes to the monitoring stream and posts the data to Prometheus Pushgateway. To understand the context within which these processes cooperate, see the diagram in Data Flow of Monitoring Metrics. For technical details, see Monitoring Data: Timing and Interruptions. Grafana Dashboard Example You can use Grafana to display monitoring data stored in a Prometheus server. Grafana can display dashboards, such as the following example, in a web browser.

21 21 Adapter (tibpromgateway) Command Line Reference TIBCO FTL Prometheus adapter (tibpromgateway) acts as an adapter between the realm server and a Prometheus Pushgateway process. Location The tibpromgateway executable file is located in the bin directory of the product installation tree.

22 22 Help Parameter Arguments Description --help -h Display a help message describing the command line parameters and options. Communications Parameter Arguments Description --push-gateway -g --realmserver -rs --secondaryrealmserver -s IP:port URL URL Required. The adapter application sends its data output to the Prometheus Pushgateway at this location. Supply an IP address and port number, separated by a colon character (:). Required. URL of the realm server. The adapter application receives both its realm definition and monitoring data from the realm server. The adapter application contacts the realm server at this location. Supply the location of the primary realm server or a satellite server (see "Connect Port" in TIBCO FTL Administration). Optional. URL of the backup realm server. If the regular realm server is unavailable, the adapter application contacts the backup realm server at this location to receive its realm definition. Supply the location of the backup realm server (see "Connect Port" in TIBCO FTL Administration).

23 23 Data Parameter Arguments Description --exclude-timestamp -x Optional. When present, the adapter application removes the client's sampling interval time stamp from the data samples before sending it to the Prometheus Pushgateway. When the clocks of the clients' host computers are not approximately synchronized, then Prometheus queries that refer to the time stamp field can yield unexpected results. Excluding the time stamp causes the Prometheus Pushgateway to supply a new time stamp value, representing the time that the adapter posted the data sample to the Pushgateway process. (This posting time is necessarily later than the time the client collected the monitoring sample.) --source-name name Optional. Supply a human-readable string that denotes the source of the monitoring data. The adapter process tags data with this string. This name becomes the value in the Prometheus exported_job label. You can use this value in Prometheus queries to distinguish data from different sources, for example, primary and satellite realm servers. When absent, the default value is the URL of the ream server, that is, the value of the --realmserver command line parameter. --max-age -a age Optional. When the adapter stops receiving monitoring data about a client, the adapter waits for this time (in seconds) before instructing the Pushgateway to reclaim the memory that held data about that client. When absent, the default value is 1800 seconds (30 minutes).

24 24 JAAS Security Parameter Arguments Description --password-file -pf path Optional. (Required for JAAS authentication.) When present, the adapter application reads a user name and password from the file at path, and authenticates itself to the realm server using those credentials. For details, see "Password File" in TIBCO FTL Admistration. TLS Security Parameter Arguments Description --trust-file path Optional. (Required for TLS communication with a secure realm server.) When present, the adapter application reads a trust file from path, and uses that trust data in communications with the secure realm server. See "Running a Secure Realm Server" in TIBCO FTL Admistration. --trust-everyone Optional. The adapter application trusts any realm server without verifying trust in the server's certificate. Do not use this parameter except for convenience in development and testing. It is not secure. Tracing and Logging Parameter Arguments Description --trace -t level Optional. When present, the adapter application outputs trace messages to stderr. You may specify any of the standard log level strings (see Tuning the Log Level in TIBCO FTL Development). When absent, the default trace level is info.

25 25 Parameter Arguments Description --logfile logfile_prefix Optional. --max-log-size size Optional. When present, the adapter application logs to a rolling set of log files instead of the console. The logfile_prefix argument may denote a path. All of the directories in the path must already exist. For more information about rotating log files, see Log Output Targets in TIBCO FTL Development. When absent, the adapter application sends log output to the console, ignoring the parameters -- max-log-size and --max-logs. Limits the maximum size, in bytes, of log files. The value must be greater than 100 kilobytes (102400). The default value is 2 megabytes (2*1024*1024). --max-logs Optional. Limits the maximum number of rolling log files. The default is 50. Docker Files for Prometheus and Grafana You can build sample Docker images to quickly start working with Prometheus and Grafana. Disclaimer For your convenience, TIBCO distributes sample Docker files that build images that include Prometheus and Grafana dashboards configured for TIBCO FTL monitoring data. However, the TIBCO FTL distribution does not include these third-party components, and TIBCO does not provide support for any of these third-party components. Prerequisites for Building the Docker Files TIBCO FTL software must be installed using a Linux 64-bit installation package. Docker must already be installed on the host computer. The Docker build process must have access the Internet to download Prometheus and Grafana. Location The Docker files are located in the samples/docker directory of the product installation tree. The file README.txt contains instructions for building and using the images. HTTP Ports When using the Docker images, you can connect to the browser interfaces using these default HTTP ports.

26 26 Process Default HTTP Port Prometheus Pushgateway 9091 Prometheus Server 9090 Grafana Server 3000 Grafana To access the Grafana server, supply Grafana's default login credentials: user name admin, password admin. Prometheus Labels Prometheus identifies each datum with a collection of labels, each of which has the form key=value. The key and value of a label can originate at any stage of the monitoring data pipeline: client, realm server, tibpromgateway adapter, or Pushgateway. A label that originates in either a client or a realm server is a straightforward transformation of a field in the monitoring message that carried the datum (see Fields of Monitoring Data Messages). However, not every field of the monitoring message becomes a label. The following table presents additional label keys, which originate either in the tibpromgateway adapter or in the Pushgateway. Key exported_job job instance Description The realm server that published the datum. See the --source-name parameter in tibpromgateway Command Line Reference. The intermediary source that scraped the datum. The value is the job_name in the Prometheus scrape configuration. The scraped target address. The default value is the address of the Pushgateway, in the form host:port.

27 27 Security of Monitoring Data TIBCO FTL components and clients can secure monitoring data using TLS. If your enterprise uses secure realm servers, then monitoring data is secure as it travels from client to realm server, and then to subscribers. However, TIBCO makes no assertions about the security of third-party components, such as Prometheus or Grafana. Realm Server A secure realm server uses a secure TCP transport to communicate with its clients. TLS secures the heartbeat messages that carry monitoring data from clients to the realm server. A secure realm server can secure the monitoring message stream using TLS. See "Secure Realm Servers" in TIBCO FTL Administration. Adapter The adapter (tibpromgateway) supports JAAS and TLS to secure its client connection to a secure realm server. Administrators must supply appropriate command line arguments when starting the adapter. Prometheus The data paths that link the tibpromgateway adapter, Pushgateway, the Prometheus server, and Grafana are not secure.

28 28 Monitoring Data: Timing and Interruptions Monitoring data flows through several segments on its path from a client to Prometheus. Timing interactions along this path can affect the data. Some segments of this path retain data across interruptions, while others do not retain data. Best Practice for Intervals Default values for the monitoring interval, heartbeat interval, and scraping interval are a good choice for many applications. If your enterprise needs a different set of values, follow these guidelines for best results: Set the client monitoring sample interval and the client-server heartbeat interval to identical values. Set the Prometheus scrape interval to approximately half of the client-server heartbeat interval. Remember that the monitoring sample interval for persistence servers and TIBCO eftl servers is fixed at 2 seconds. If it is critical to your enterprise that Prometheus capture all monitoring data from these processes, set the Prometheus scrape interval to 1 second. If your enterprise can accept coarser granularity, discarding some of the data samples from these processes, then use a larger scrape interval. Client Most monitoring data begins in a client process. Each client gathers data and transfers it to the realm server in its regular heartbeat messages. For the most complete results, ensure that the client monitoring sample interval is identical to the client-server heartbeat interval. (You can configure these intervals as global realm properties of the realm server. However, the monitoring sample interval for persistence servers and TIBCO eftl servers is fixed.) If the heartbeat interval is longer than the sample interval, the client accumulates data from several sample intervals, and sends the accumulated samples to the realm server in the next heartbeat. If the client cannot communicate with the realm server, the client retains its monitoring data until it successfully reconnects to the server, then it transfers the accumulated samples. Memory availability could limit the amount of data that a client can accumulate. When a client stops or restarts, it does not preserve monitoring data it has not yet sent to the realm server. Realm Server The realm server receives data from clients, and also generates monitoring data that measures its own operations. The realm server publishes a stream of monitoring data messages on the built-in monitoring endpoint. As each monitoring sample set arrives from a client, the realm server immediately publishes it on the monitoring stream. The realm server also stores the new data samples, overwriting previous data samples for that client. The realm server publishes each sample in a separate message. The monitoring message stream is available only to current subscribers. Subscribers cannot receive any messages that the realm server sends while a subscribing process is not running, or while the communications link is interrupted. Monitoring data in those messages is no longer available to the subscriber. You cannot associate a persistence store with the monitoring endpoint.

29 29 Adapter The adapter (tibpromgateway) subscribes to the monitoring message stream. As each message arrives, the adapter immediately posts the data to the Prometheus Pushgateway and then discards the message. The adapter cannot receive any messages that the realm server sends while the adapter is not running, or while the communications link from the realm server is interrupted. Monitoring data in those messages is no longer available to the adapter. If the adapter process stops, or if communications from the realm server are interrupted, then it cannot receive any messages that the realm server sent in the interim. Monitoring data in those messages is lost. Pushgateway The adapter transfers data to the Prometheus Pushgateway. The Pushgateway behaves as a singlerecord last value cache. Upon receiving a monitoring sample, it stores the inbound data for scraping. For each client and metric, Pushgateway retains only the most recent data sample. Monitoring data is lost if the Pushgateway process stops or communications from the adapter are interrupted. Prometheus Server Prometheus scrapes data from the Pushgateway at regular intervals, and stores it for analysis and notification. If the scrape interval is too long, the Pushgateway could receive data from several sampling intervals before the next scrape. Each sample posted to Pushgateway overwrites the previous sample for the same client and metric, so Prometheus scrapes only the newest sample for each metric, missing older samples. For the most complete data sets, ensure that the Prometheus scrape interval is approximately half of the realm server's client-server heartbeat interval. In this way, Prometheus scrapes about twice as fast as the monitoring messages arrive, so it scrapes each sample at least once before the next sample overwrites it. Set the parameter scrape_interval in the Prometheus configuration file.

30 30 Estimating the Volume of Metric Data You can estimate the size of client monitoring data to gauge its impact on network bandwidth and other resources. The number of counters depends on the number of application processes, and on the number of endpoints, transports, and event queues in each application process. Procedure 1. For each application instance definition, determine the number of counters per client process of that application instance. Combine information from the application instance definitions with information from the following table. Client Monitoring Resource Usage 11 counters per client process 3 counters per endpoint 2 counters per event queue 3 counters per transport (other than multicast transports) 9 counters per multicast transport (6 specific to multicast, in addition to the 3 counters common to all transports) 2. For each application instance definition, multiply the number of counters per client process (as determined in the previous step) by the expected number of application process instances. 3. Sum that number of counters over all the application instance definitions to obtain the enterprise total number of data samples generated in each sample interval. When estimating bandwidth requirements, remember that each data sample travels between multiple components. Depending on the network architecture, the contents of a specific data sample may traverse the network several times, possibly across the same network segments or links. See Data Flow of Monitoring Metrics. 4. Multiply the enterprise total by the number of samples you plan to retain in your monitoring data repository (such as Prometheus). This product estimates the total volume of metric data.