opennms reporting generation tool

Transcription

1 opennms reporting generation tool Juan Pedro Escalona DevOps Southampton, UK

2 Juan Pedro Escalona DevOps / Systems Administrator with over 6 years experience administering different OS, network systems and anything that is thrown to my desk. At the present, I work in QA Automation & Performance group, providing tools to automate processes. Monitoring lover, background in several Open Source monitoring solutions. Six months ago, these words changed me: - Hi Juan Pedro. Have you met opennms? Let me introduce you... Manuel Villarejo

3 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Results Q&A

4 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Results Q&A

5 Introduction Over 1000 servers in production Exact copy of production environment for test Releases every 2 weeks Load Test is mandatory An average of 10 Load Test every week

6 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Results Q&A

7 Load Test Processes Launch Load Test Prepare Load Test Report Load Test

8 Load Test Processes Prepare Load Test: Set initial configuration in order to replicate production status (ie. Refresh DBs)

9 Load Test Processes Prepare Load Test: Set initial configuration in order to replicate production status (ie. Refresh DBs) Launch Load Test: Most of load generators and monitors are compatible only with Windows OS Webload & Neoload + 25 Load Generators (Windows / Linux)

10 Load Test Processes Prepare Load Test: Set initial configuration in order to replicate production status (ie. Refresh DBs) Launch Load Test: Most of load generators and monitors are compatible only with Windows OS Webload & Neoload + 25 Load Generators (Windows / Linux) Report Load Test: opennms: Main key metrics of servers and graphs Webload / Neoload Report

11 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Results Q&A

12 Automating Processes Progress Prepare Load Test: It's complex but for the time being we use scripts to do it Orchestration: check dependencies + call scripts

13 Automating Processes Progress Prepare Load Test: It's complex but for the time being we use scripts to do it Orchestration: check dependencies + call scripts Launch Load Test: Generate load templates based on load profiles Run/stop and control status of Load Generator Consoles

14 Automating Processes Progress Prepare Load Test: It's complex but for the time being we use scripts to do it Orchestration: check dependencies + call scripts Launch Load Test: Generate load templates based on load profiles Run/stop and control status of Load Generator Consoles Report Load Test: Schedule report generation or trigger when load test has finished Aggregate different servers with different type of graphs in same report Provide a single view to check status of all nodes we're testing

15

16 Automating Processes Progress > Report At the beginning: Time: minutes No health key metrics of server Site Catalyst Webload Report

17 Automating Processes Progress > Report Spreadsheet reports: Time: From 1 hour to days OpenNMS: health statistics Screenshot + Cut + Paste (1 node) Manually

18 Automating Processes Progress > Report

19 Automating Processes Progress > Report KSC Reports: Time: 30 minutes to several hours Join different servers + Graphs Save as HTML Manually

20 Automating Processes Progress > Report

21 Automating Processes Progress > Report Report generation is not automatic Not possible join several KSC Reports in once KSC Report configuration is not easy Metrics resolution over the time (RRD)

22 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Improvements & Results Q&A

23 Reporting Tool in Load Test Processes Key features Support multiple environments (a.k.a. multiple OpenNMS instances) Easy configuration building new Server Reports ( hosts + graphs) Easy report generation: few clicks + few minutes Liveview: lets you aggregate different Server Reports in a same view just on click

24 Reporting Tool in Load Test Processes Architecture and components: Python OpenNMS Client module: Retrieve info from opennms Contributed by Jason Viloria Memcache: Cache all info obtained from opennms Django Application: Provide UI interface Automate report generation (Celery)

25 Reporting Tool in Load Test Processes

26 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Improvements & Results Q&A

27 Python opennms library Retrieve the list of available nodes in OpenNMS instances: Assets: Node label, node id, categories Node Resources: JMX, SNMP Graphs Node Level: CPU, Memory Interface Level: Net Interface Bandwidth, Disk IO Caching

28 Python opennms library Get a node by label

29 Python opennms library Get a node by id

30 Python opennms library Get node IP Interfaces

31 Python opennms library Get node graphs

32 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Improvements & Results Q&A

33 Demo

34 Demo > Report Archives

35 Demo > Add new node

36 Demo > Add new graph

37 Demo > Add new Server Report

38 Demo > New Report

39 Demo > New Recurrent Report

40 DEMO > Review report

41 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Improvements & Results Q&A

42 Liveview

43 Agenda Introduction Load Test Processes Automating processes progress Reporting Tool in Load test processes Python opennms Library Demo Liveview Improvements & Results Q&A

44 Improvements & Results

45 Improvements & Results Time Spent per Stage 7 Overall Reporting Time Report Generation OpenNMS graphs Analysis

46 Improvements & Results Load Test Executions Load Tests Capacity Jan Feb Mar Apr May Jun Jul Aug Sep Oct Executions x 3.5 since January: st January 2013 (1 Big Event): 30 executions August 2013 (2 nd Big Event): 105 executions Results Analysis: 2 X analysis time 1/3 overall report time

47 Improvements & Results At the beginning No server key metrics Spreadsheets Key metrics (1 node) Manual process Much time KSC Reports Key metrics (all) Save time Still manually Reporting Tool Key metrics (all) Save huge time Automatic

48 We did it

49 Thanks! Questions?