GRNET NOC In-house management tools

Transcription

1 GRNET NOC In-house management tools TF-NOC George Kargiotakis Andreas Polyrakis Leonidas Poulopoulos

2 TF-NOC 2

3 Automation Time PEBKACs Share Why??? Where there an API there is a way No API??? Try to create one TF-NOC 3

4 Tools MuPy DCmon Optical Network Mon Servermon TF-NOC 4

5 MUPY TF-NOC 5

6 Munin: POWERFUL Deployed in GRNET Server monitoring No AAI scheme No info aggregation No groupings Why? BUT No saved queries Show all disk IOs graphs ( Show disk graphs from gud and knuth TF-NOC 6

7 What? Parser BeautifulSoup lib Wrapper Python Django Saved queries Open source Demo: TF-NOC 7

8 DATACENTER POWER MONITORING TF-NOC 8

9 Datacenter Power-monitoring Datacenter Ministry of Education Appliance Collects data Graphs Weekly FTP CSV TF-NOC 9

10 Why? Appliance web server is slooooowwww Live queries cause load No combination of metrics No PUE calculation No public web interface But Provides data in html table/csv Every 5 mins TF-NOC 10

11 How? HTML results parser Store nodes/metrics in db Store data in rrd files Combine data to calculate PUE Provide web interface Beta internal Source code soon available TF-NOC 11

12 Sneak Peek-Dashboard TF-NOC 12

13 Sneak Peek-Nodes TF-NOC 13

14 Sneak Peek-Graphs TF-NOC 14

15 OPTICAL NETWORK MONITORING TF-NOC 15

16 Optical Network Monitoring Alcatel 1626LM, 1696MS 3 months ago: Alcatel NMS (PM module) No attenuation (manual) No triggers Fiber Cuts Manual filters(alarm Surveillance) Manual triggers TIME TIME TIME TIME TF-NOC 16

17 Veeeryyy old UI No graphing Alcatel tools TF-NOC 17

18 Optical NetAdmin 3 months ago TF-NOC 18

19 Harvest Element info Performance data Topology map Graphs Alerting Visual, Mail Open data Web Requirements TF-NOC 19

20 Element Graphs Topology Attenuation Attenuation Diff Fiber Cuts Element Info Architecture Optical Network IOO client NMS GENOS IOO API Django RRD Memcached Bootstrap UI RRD Analyser TF-NOC 20

21 Optical Mon (Optical Network Mon) IOO Client: Source code soon open Open access Frequent feature updates TF-NOC 21

22 On RRD files Anomaly detection Attenuation (diff of counters) Iterate measurements Pattern matching If att==0 for x minutes then fiber_cut If att_diff >a for x minutes then anomaly More simple yet smart checks Show visual alerts Send mail to optnetadmins TF-NOC 22

23 Anomaly detection improvements RRD Holt-Winters Time Series Forecasting TF-NOC 23

24 Optical NetAdmin now TF-NOC 24

25 SERVERMON TF-NOC 25

26 Sneak Peek TF-NOC 26

27 Sneak Peek TF-NOC 27

28 Sneak Peek TF-NOC 28

29 Sneak Peek TF-NOC 29

30 Source Code TF-NOC 30

31 Thank you Questions? TF-NOC 31