Published on WSO2 Inc (http://wso2.com) Home > Stories > Monitor Your Key Performance Indicators using WSO2 Business Activity Monitor Monitor Your Key Performance Indicators using WSO2 Business Activity Monitor By kasunw.wso2.com Created 2013-06-26 07:07 Applies To WSO2 BAM Version 2.0.0 and above Table of Content Introduction [1] BAM architecture [2] Use case [3] KPIs for this use case [4] Collect information [5] BAM data-agent (Java API) [6] Non Java Data-agent [7] REST API [8] BAM data receiver [9] Cassandra explorer [10] Data Analysis [11] Writing a hive script for analyzing captured data [12] Visualization [13] Introduction Key performance indicators are essential for your organization to measure the performance or success in terms of progress of its goals. Also It is crucial for you to choose the right KPIs with better understanding of what is important to the organization to measure and improve the business productivity of it. WSO2 Business activity monitor has an extremely flexible framework that allows to model your own key performance indicators to suit your monitoring requirement. In this article I am going to explain how you can build your own KPI using WSO2 Business Activity Monitor. BAM architecture Following I have shown the BAM architecture which help us to achieve this greater flexibility.
Data that needs to be monitored goes through the above modules and the data flow is as follows. The system that you are going to monitor have the BAM data agents and these agents will capture the required data and send it to BAM. In addition to the data agents users can send data to BAM through the REST API. Data receiver in BAM receive this data and store them in Cassandra data store. Analyzer engine will run the analytics written in hive language according to the monitoring requirement, the summarize data can be persist into RDBMS data store. In the presentation layer summarized data will fetch from RDBMS and show it in the dashboard/reports. Use case I am going to illustrate the capability of defining KPIs by using an use-case. Let's think You have hosted a news website and You need to monitor the traffic comes to your website and analyze it to understand how you can enhance your website and marketing strategies to increase the profit. KPIs for this use case No of unique visitors per day How many visitors does it take for your website to achieve its goals? Your profit will proportional to the no of visitors to your website. If this shows some growth day by day, which means you are making a progress. Most popular news category This category will depends on several factors (time, incidents, gossip, events etc.). Hence analyzing your traffic and identifying the current in demand news categories are essential for you to promote your contents more in those sources and increase the profits. Number of peoples from different locations This will helpful for identifying the regions that have low no of visitors and take some actions to improve the traffic from these areas. Now I am going to illustrate step by step on how to use BAM to define above KPIs. Collect information
Website traffic information can be captured and published into BAM using following ways. Using Java API Using client generated by thrift IDL Using REST API BAM data-agent (Java API) BAM provides high performance, low latency, load-balancing (between multiple receivers ), non-blocking and multi-threaded API for sending large amount of business events over various transport (TCP,Http) using Apache Thrift. This API has been provided as a Java SDK and you can use it easily for publishing captured data from your Java based system to BAM for analysis. In addition to that these data-agents are compatible with WSO2 CEP that can be used for real time analysis. Custom data-agent Following I have written a simple asynchronous data-agent for publishing web traffic information to BAM. This will help you to understand the agent API usage. For more information about writing a data publisher for BAM, you can refer this article [1] import org.apache.log4j.logger; import org.wso2.carbon.databridge.agent.thrift.agent; import org.wso2.carbon.databridge.agent.thrift.asyncdatapublisher; import org.wso2.carbon.databridge.agent.thrift.conf.agentconfiguration; import org.wso2.carbon.databridge.agent.thrift.exception.agentexception; import org.wso2.carbon.databridge.commons.event; public class DataAgent { private static Logger logger = Logger.getLogger(DataAgent.class); public static final String ONLINE_NEWS_STATS_DATA_STREAM = "online_news_stats" public static final String VERSION = "1.0.0"; public static final String DATA = "Kasun,Colombo,Sri Lanka vs Australia 3rd ODI,Sports,136643358 "Amal,Kaluthara,Businessman killed in Expressway accident,acci "Kamal,Colombo,Navy intelligence investigating boat escape,mil "Kalum,Mathara,No leadership change - JVP,Politics,1366433520\ "Nuwan,Galle,Marginal improvement at this year?s O/L exam resu "Sampath,Mathara,Marginal improvement at this year?s O/L exam "Chamath,Mathara,Attempts to replace Minister Sirisena: UNP,Po "Prabath,Colombo,WikiLeaks; LTTE could have threatened Karunan "Amila,Kandy,Private bus owners to launch one-day strike,trans "Budhdhika,Colombo,ICC introduces new 'No ball' playing condit "Rangana,Nuwara Eliya,Annual horse racing event in Nuwara Eliy "Gamini,Colombo,O/L 2012 results released,education,1366434987 "Tharindu,Colombo,O/L best results,education,1366334087\n" + "Janaka,Kaluthara,O/L best results,education,1366839787\n" + "Anuranga,Jaffna,Sri Lanka vs Australia 3rd ODI,Sports,1366336 "Kasun,Colombo,Marginal improvement at this year?s O/L exam re "Ranga,Kandy,Private bus owners to launch one-day strike,trans "Denis,Kaluthara,ICC introduces new 'No ball' playing conditio "Harsha,Galle,No leadership change - JVP,Politics,1366533333\n public static void main(string[] args) {
AgentConfiguration agentconfiguration = new AgentConfiguration(); System.setProperty("javax.net.ssl.trustStore", "/opt/wso2bam-2.2.0/reposit System.setProperty("javax.net.ssl.trustStorePassword", "wso2carbon"); Agent agent = new Agent(agentConfiguration); //Using Asynchronous data publisher AsyncDataPublisher asyncdatapublisher = new AsyncDataPublisher("tcp://127. String streamdefinition = "{" + " 'name':'" + ONLINE_NEWS_STATS_DATA_STREAM + "'," + " 'version':'" + VERSION + "'," + " 'nickname': 'News stats'," + " 'description': 'Stats of the news web site'," + " 'metadata':[" + " {'name':'publisherip','type':'string'" + " ]," + " 'payloaddata':[" + " {'name':'username','type':'string'," + " {'name':'region','type':'string'," + " {'name':'news','type':'string'," + " {'name':'tag','type':'string'," + " {'name':'timestamp','type':'long'" + " ]" + ""; asyncdatapublisher.addstreamdefinition(streamdefinition, ONLINE_NEWS_STATS publishevents(asyncdatapublisher); private static void publishevents(asyncdatapublisher datapublisher) { String[] datarow = DATA.split("\n"); for (String row : datarow) { String[] data = row.split(","); Object[] payload = new Object[]{data[0], data[1], data[2], data[3], Long.parseLong(data[4]); Event event = eventobject(null, new Object[]{"10.100.3.173", payload) try { datapublisher.publish(online_news_stats_data_stream, VERSION, even catch (AgentException e) { logger.error("failed to publish event", e); private static Event eventobject(object[] correlationdata, Object[] metadata, Event event = new Event(); event.setcorrelationdata(correlationdata); event.setmetadata(metadata); event.setpayloaddata(payloaddata); return event; Non Java Data-agent Thrift IDL can be used to generate thrift clients from different languages to publish data. In this case asynchronous data publishing implementations will be to developed by the developers the way similar to
how it has been implemented in Java API. Languages are supported by thrift C++ C# Cocoa D Delphi Erlang Haskell Java OCaml Perl PHP Python Ruby Smalltalk REST API You can capture the website traffic information from your system and use BAM REST API for publishing those data to BAM via Http transport. BAM data receiver Data send from above data agents will receive by the data receiver which use thrift and internal optimization techniques to achieve very high throughput. Those received data will directly write to Cassandra which is high performance and scalable big data storage. It persists events into a column family with name equal to the stream name. Also BAM data receiver can be used share the events with WSO2 CEP for real time analysis. Cassandra explorer It provides graphical user interface for viewing the data in a Cassandra column family. Now let's see the data published from our sample client. Go to Home? Manage? Cassandra Explorer? Connect to Cluster Type the connection details as below. You can see all the keyspaces and column families are listed in Cassandra explorer ui.
Writing This CREATE RDBMS <datasource> From Once Visualization After Also Here Number Popular Visitors References Kasun Hide Analytics Intermediate footer settimeout(function(){var Company Customers Partners Products Middleware Overview Application Carbon Complex Dashboard Data Enterprise Governance Identity IoT Machine Message Microservices Process New Storage Development Developer Cloud Private Managed Service Public API Use Technology IT Business --> Events Webinars Workshops WSO2Con Conferences Resources Library Articles Analyst Case On-Demand Presentations White Documentation Videos Training Support Subscribe Legal Privacy Report 2016 Source Links: [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] http://docs.wso2.org/wiki/display/bam220/gadget+generation+tool Challenges http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#introduction http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#use http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#kpis http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#collect http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#non http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#rest http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#bam div#footer Server Cloud http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#cassandra http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#data http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#writing http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#visualization http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events-bamcep http://wso2.org/library/articles/2012/09/using-different-reporting-frameworks-wso2-business-activity-monitor http://jaggeryjs.org/ Cases Manager you Analytics Services I Studies Papers have Cloud the Advantage Private _bizo_data_partner_id WSO2 URL: Overview Calendar Weranga Distribution Events Reports var Level News Server & System of Center a Learner Tutorials can problem analyze Broker Process Rules Challenges Cloud EXTERNAL run Training Hive Mobility Service Store Studio hive Visitors 2.3.0 Server Factory Platform used Webinars Registry the rw_ext_id= storage Tools Agreement { Framework Category Server PaaS Inc. background-color: Processor script Overview Login newsletter onwards Gunathilake, WSO2 gadget above with and Server Bus Per Manager can to TABLE summarize this Day for analyze script Jaggery '';var generation you be for page a=document.createelement("script");var analyzing configured Java IF = can Senior hive pkbaseurl "7067";(function() the <name>wso2bam_datasource</name> framework[3] data transparent; NOT use read tool above Software according captured EXISTS from data for KPIs. = generating (("https:" master-datasources.xml from Engineer, Generation OnlineNewsStats to div#ifooter the { your == and KPIs, document.location.protocol)? s deploying = document.getelementsbytagname("script")[0]; { gadget you background-color: hive Inc. storage need b=document.getelementsbytagname("scrip script gadgets in generation (key WSO2BAM/repository/conf/datasource and case Java visualize for architecture data-agent API Analysis information STRING, this summarize a Data-agent receiver hive dashboard. can tool use explorer #000000; script your be (Java case "https://connect.wso2.co visualize username configured kpis. for it, API) <description>th font-family: then analyzing You Since your the can from STRING,re those summarize captured find KPIs. Arial; master inform v Author Then go to online_news_stats in the EVENT_KS. You can view the data by going to the column family. After successfully publish the data we can do the analysis. Data Analysis You can run data analytics on captured data by using the BAM analytics framework which is powered by Apache Hadoop for scaling out data processing operations on a large number of data processing nodes, in order to handle large data volumes. WSO2 BAM provides an easy way to do the map/reduce operation on Hadoop by using Apache Hive that provides a simple way to write query and managing large datasets residing in distributed storage using a SQL-like language called HiveQL. KPIs Following I have list down the KPIs that we are going to analyze. No of unique visitors per day Most popular news category No of peoples from different regions