http://www.grnet.gr GRNET NOC In-house management tools TF-NOC George Kargiotakis (kargig@noc.grnet.gr) Andreas Polyrakis (apolyr@noc.grnet.gr) Leonidas Poulopoulos (leopoul@noc.grnet.gr)
TF-NOC 2
Automation Time PEBKACs Share Why??? Where there an API there is a way No API??? Try to create one TF-NOC 3
Tools MuPy DCmon Optical Network Mon Servermon TF-NOC 4
MUPY TF-NOC 5
Munin: POWERFUL Deployed in GRNET Server monitoring No AAI scheme No info aggregation No groupings Why? BUT No saved queries Show all disk IOs graphs (http://munin.ping.uio.no/) Show disk graphs from gud and knuth TF-NOC 6
What? Parser BeautifulSoup lib Wrapper Python Django Saved queries Open source https://code.grnet.gr/projects/mupy Demo: http://mupy-demo.grnet.gr/ TF-NOC 7
DATACENTER POWER MONITORING TF-NOC 8
Datacenter Power-monitoring Datacenter Ministry of Education Appliance Collects data Graphs Weekly FTP CSV TF-NOC 9
Why? Appliance web server is slooooowwww Live queries cause load No combination of metrics No PUE calculation No public web interface But Provides data in html table/csv Every 5 mins TF-NOC 10
How? HTML results parser Store nodes/metrics in db Store data in rrd files Combine data to calculate PUE Provide web interface Beta internal Source code soon available TF-NOC 11
Sneak Peek-Dashboard TF-NOC 12
Sneak Peek-Nodes TF-NOC 13
Sneak Peek-Graphs TF-NOC 14
OPTICAL NETWORK MONITORING TF-NOC 15
Optical Network Monitoring Alcatel 1626LM, 1696MS 3 months ago: Alcatel NMS (PM module) No attenuation (manual) No triggers Fiber Cuts Manual filters(alarm Surveillance) Manual triggers TIME TIME TIME TIME TF-NOC 16
Veeeryyy old UI No graphing Alcatel tools TF-NOC 17
Optical NetAdmin 3 months ago TF-NOC 18
Harvest Element info Performance data Topology map Graphs Alerting Visual, Mail Open data Web Requirements TF-NOC 19
Element Graphs Topology Attenuation Attenuation Diff Fiber Cuts Element Info Architecture Optical Network IOO client NMS GENOS IOO API Django RRD Memcached Bootstrap UI RRD Analyser TF-NOC 20
Optical Mon http://mon.grnet.gr (Optical Network Mon) IOO Client: http://code.grnet.gr Source code soon open Open access Frequent feature updates TF-NOC 21
On RRD files Anomaly detection Attenuation (diff of counters) Iterate measurements Pattern matching If att==0 for x minutes then fiber_cut If att_diff >a for x minutes then anomaly More simple yet smart checks Show visual alerts Send mail to optnetadmins TF-NOC 22
Anomaly detection improvements RRD Holt-Winters Time Series Forecasting TF-NOC 23
Optical NetAdmin now TF-NOC 24
SERVERMON TF-NOC 25
Sneak Peek TF-NOC 26
Sneak Peek TF-NOC 27
Sneak Peek TF-NOC 28
Sneak Peek TF-NOC 29
https://code.grnet.gr/ Source Code TF-NOC 30
Thank you Questions? https://twitter.com/grnetnoc https://www.facebook.com/noc.grnet.gr TF-NOC 31