Monitoring Open Source pour Java avec JmxTrans, Graphite et Nagios

Monitoring Open Source pour Java avec JmxTrans, Graphite et Nagios 27 au 29 mars 2013

Speaker @cyrilleleclerc CTO Architect Cyrille Le Clerc Fort Traffic Open Source Cloud, DevOps

Speaker @hgomez Architecte CI Henri Gomez Intégration Continue Open Source DevOps

Vous? Dev? Qa? Ops? Expérience monitoring?

Agenda 27 au 29 mars 2013

Agenda L application Le monitoring à réaliser Instrumenter une webapp Java avec JMX Monitorer Grapher avec Graphite Alerter avec Seyren (et Nagios) Monitoring & Continuous Delivery Déploiement de Graphite Conclusion

Votre mission, si toutefois vous l acceptez... e-commerce SOS Cocktails http://www.flickr.com/photos/23791504@n08/2607814349 Mettre en place le monitoring technique et métier

Démo SOS Cocktail

Les indicateurs à monitorer Pourquoi Monitorer? Quoi? 27 au 29 mars 2013

Les indicateurs à monitorer

Les Indicateurs à monitorer Operating System -> Sysload JVM -> GC duration Tomcat -> activesessions (active visitors) Application -> sales revenue & items

Open Source Monitoring for Java (4) ecommerce Web Site Dashboard - wiki Graphite Email (1) (2) (3) Seyren (5) Nagios

JMX & Métriques 27 au 29 mars 2013

Pourquoi JMX? JMX vs. pages web vs. logs vs.... Simple et sécurisé Exposable vers pages web et les logs Des alternatives à JMX

JMX vs. logs vs. pages web vs.... Logs de monitoring Parser des Go de fichiers texte? Détail par invocation vs. moyenne par minute Pages web Historisation? Applications en clusters? Sécurité?

JMX avec Spring <beans...>... <context:mbean- export/>... </beans> @ManagedResource("cocktail:name=ShoppingCartController,type=...") class ShoppingCartController { final AtomicInteger revenueincentscounter = new AtomicInteger(); @ManagedAttribute public int getrevenueincentscounter() { return revenueincentscounter.get(); } void purchase(...){... revenueincentscounter.addandget(priceincents); }

JMX

JMX & JSP Powered by JMX! Human readable et script friendly Attention sécurité! $ curl http://demo- cocktail.jmxtrans.cloudbees.net/5adruwr5/jmx- sales- metrics.txt.jsp Epoch SalesRevenueInCentsCounter SalesItemsCounter SalesOrdersCounter 1363217376 4500 7 2 http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp curl http://demo- cocktail.jmxtrans.cloudbees.net/5adruwr5/... { "Epoch":1363217653, "SalesRevenueInCentsCounter":4050, "SalesItemsCounter":7, "SalesOrdersCounter":2 }

JMX & JSP /5ADrUWr5/jmx-sales-metrics.jsp Powered by JMX! Human readable et script friendly Attention sécurité! $ curl http://demo- cocktail.jmxtrans.cloudbees.net/5adruwr5/jmx- sales- metrics.txt.jsp Epoch SalesRevenueInCentsCounter SalesItemsCounter SalesOrdersCounter 1363217376 4500 7 2 http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp curl http://demo- cocktail.jmxtrans.cloudbees.net/5adruwr5/... { "Epoch":1363217653, "SalesRevenueInCentsCounter":4050, "SalesItemsCounter":7, "SalesOrdersCounter":2 }

JMX & JSP Powered by JMX!

Alternatives à JMX

Alternatives à JMX Servo class ShoppingCartController { @Monitor(name="revenueInCents", type=counter) final AtomicInteger revenueincentscounter = new AtomicInteger(); void purchase(...){... revenueincentscounter.addandget(priceincents); }

Alternatives à JMX Metrics class ShoppingCartController { Servo final Counter revenueincentscounter = Metrics.newCounter( ShoppingCartController, "revenue- in- cents"); void purchase(...){... revenueincentscounter.inc(priceincents); } class ShoppingCartController { @Monitor(name="revenueInCents", type=counter) final AtomicInteger revenueincentscounter = new AtomicInteger(); void purchase(...){... revenueincentscounter.addandget(priceincents); }

JMX OS: Sysload JVM: Garbage Collector Tomcat: activesessions Application: Web Site Revenue

jmxtrans & embedded-jmxtrans 27 au 29 mars 2013

jmxtrans - standalone ecommerce Tomcat logistics Tomcat Graphite accounting Tomcat... jmxtrans - standalone

embedded-jmxtrans ecommerce Tomcat Graphite logistics Tomcat Librato accounting Tomcat... batch embedded-jmxtrans

standalone vs. embedded standalone embedded Packaging Modèle daemon externe, raccordement manuel des applications & des indicateurs agent embarqué, applications & indicateurs auto-raccordés / autonomes Curseur OPS - ----------- DEV OPS ---------- -- DEV Use case applis peu modifiables, middlewares, curseur OPS cloud, batch, curseur DEV/QA

jmxtrans & embedded-jmxtrans SOS Cocktail Graphite embedded

embedded-jmxtrans configuration

embedded-jmxtrans configuration <dependency> <groupid>org.jmxtrans.embedded</groupid> <artifactid>embedded- jmxtrans</artifactid> <version>1.0.5</version> </dependency>

embedded-jmxtrans configuration <dependency> <groupid>org.jmxtrans.embedded</groupid> <artifactid>embedded- jmxtrans</artifactid> <version>1.0.5</version> </dependency> <beans... xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded" xsi:schemalocation="... http://www.jmxtrans.org/schema/embedded... > <jmxtrans:jmxtrans> <jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>... </jmxtrans:jmxtrans> </beans>

embedded-jmxtrans configuration <dependency> <groupid>org.jmxtrans.embedded</groupid> <artifactid>embedded- jmxtrans</artifactid> <version>1.0.5</version> </dependency> <beans... { xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded" xsi:schemalocation="... http://www.jmxtrans.org/schema/embedded... > { "queries": [ { "objectname": "cocktail:name=shoppingcartcontroller", "resultalias": "sales", "attributes": ["SalesRevenueInCentsCounter"] }, ], "outputwriters": [ "@class": "org.jmxtrans.embedded.output.graphitewriter", "settings": { "host": "${graphite.host:localhost}", "port": "${graphite.port:2003}," <jmxtrans:jmxtrans> "enabled": "${graphite.enabled:true}" <jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration> }... } </jmxtrans:jmxtrans> ] </beans> }

jmxtrans OS: Sysload JVM: Garbage Collector Tomcat: activesessions Application: Web Site Revenue

Graphite 27 au 29 mars 2013

Graphite BD de Time Series et solution de Graphe Utilisé par des grands du Web Similaires Simplicité et Self Service

Ecriture / Alimentation Création automatique des métriques Protocoles socket plain text & sérialisation Python Pickle echo "devoxx.happycounter 1.2 1364338989" nc localhost 2003 Précision et durée de stockage par configuration [sales_1min_for_15days_5min_for_1year] pattern = ^sales\. retentions = 60s:30d,300s:365d 1.7MB

Lecture / Consommation Composer de graphe & URL API Tout le monde peut créer ses graphes Fonctions Riches sum, scale, derivative, timeshift, deviation, filter,... Exports variés : png, svg, csv, json, raw text

Graphes et monitoring 27 au 29 mars 2013

Demo Graphite sur le Cloud http://hostedgraphite.com/ 1 email: admin@jmxtrans.org password: devoxx2013 Graphite as a Service Pas de tracas d installation addons: Tasseo No-OPS 2 3

Les métriques Gauge Ever Increasing Counter Exemples: activerequests, datasource.activeconnection,... Information exploitable immédiatement Exemples: requestscount, revenue,... Conversion nécessaire en par minute

Des graphes et des formules? Ever Increasing Counter Revenue per Second

Des graphes et des formules derivative()? Ever Increasing Counter Revenue per Second

Des graphes et des formules? Per Second -> Per Hour Revenue per Hour

Des graphes et des formules summarize()? Per Second -> Per Hour Revenue per Hour

Des graphes et des formules? 2 servers Total Revenue per Hour

Des graphes et des formules sumseries()? 2 servers Total Revenue per Hour

Des graphes et des formules? server restart Ignore reset to zero

Des graphes et des formules nonnegativederivative()? server restart Ignore reset to zero

Des graphes et des formules Compare to last week?

Des graphes et des formules Compare to last week timeshift()?

Des graphes et des formules OS: Sysload JVM: Garbage Collector Tomcat: activesessions Application: Web Site Revenue

Dashoards Wiki dashboard.prod.md # Sales <table> <tr> <td> <img src="http://graphite.prod.mycompany/graphite/render/? width=400&height=300&title=revenue&xformat=%25a%20%25d %25p&vtitle=USD&lineWidth=3&from=- 5days&target=alias(scale(summarize(sumS eries(nonnegativederivative(edu.servers.*.sales.revenueincentscounter)) %2C%221h%22)%2C0.01)%2C%22Revenue%20per%20Hour%22)..."> </td> <td> <img src="http://graphite.prod.mycompany/graphite/render/?..."> </td> <tr> </table> Wiki https://github.com/jmxtrans/embedded-jmxtrans-samples/wiki/dashboard-prod

Dashoards by Email Envoyer les graphes par email

L alerting Intégrer Graphite à de l alerting URL Api Seyren Nagios 27 au 29 mars 2013

Intégrer Graphite à l alerting Utiliser l URL API Format rawtext ou JSON Pattern /render?from=- 11minutes&until=- 1minutes&format=raw&target=**** Exemple $ curl "http://graphite.exemple.com/render?from=- 11minutes&until=- 1minutes&format=raw& target=keeplastvalue(servers.cloudbees.jvm.os.systemloadaverage)" my- metric,1363225680,1363226340,60 0.03,0.01,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0

Alerting - Seyren Alerting Dashboard for Graphite Open Source Java.war + MongoDB Alertes email, PagerDuty,...

Alerting - Seyren http://seyren.jmxtrans.cloudbees.net/ OS: Sysload JVM: Garbage Collector Tomcat: activesessions Application: Web Site Revenue

Nagios Infrastructure Monitoring Open Source Standard de facto en entreprise check_graphite plugin Pierre-Yves Ritschard Jason Dixon / obfuscurity

Monitoring & Continuous Delivery 27 au 29 mars 2013

Monitoring & Continuous Delivery Livrer le monitoring avec l application Les indicateurs Les dashboards Builder en continue les configs par environnement Require: topologie des environnements intégrées au build Déploiement sur un repo (Maven,...) Déployer en automatique le monitoring

Monitoring & Continuous Delivery src/main/graphite/dashboard.template.md StringTemplate Cocktail App Dashboards - ${environment} - ${version} # Sales <table> <tr> <td> <img src="${graphite.baseurl}/render/? width=400&height=300&title=revenue&xformat=%25a%20%25d %25p&vtitle=USD&lineWidth=3&from=- 5days&target=alias(scale(summarize(sumS eries(nonnegativederivative(edu.servers.*.sales.revenueincentscounter)) %2C%221h%22)%2C0.01)%2C%22Revenue%20per%20Hour%22)..."> </td> <td> <img src="${graphite.baseurl}/render/?..."> </td> <tr> </table> ${variable} Dashboard template

Monitoring & Continuous Delivery dashboard.template.md PROD-env-topology.json... QA-env-topology.json pom.xml <gmaven-plugin> dashboard.prod.md... dashboard.qa.md dashboard-generator.groovy repository https://github.com/jmxtrans/embedded-jmxtrans-samples/wiki/monitoring-continuous-deployment Monitoring dashboards Continuous Integration

Déploiement de Graphite 27 au 29 mars 2013

Déploiement de Graphite

Cadeau bonus! 27 au 29 mars 2013

Promo Code http://hostedgraphite.com/ 60 days trial with DEVOXX promo code (2) Enter Promo Code DEVOXX (1) Signup help: cleclerc@cloudbees.com (3) Promo Code DEVOXX

Conclusion 27 au 29 mars 2013

Conclusion Monitoring technique et métier Monitoring intégrée au Continuous Delivery de l appli L approche self-service transforme le monitoring Solutions Open Source disponibles Monitoring no longer sucks in Java! http://lusislog.blogspot.fr/2011/06/why-monitoring-sucks.html

http://demo-cocktail.jmxtrans.cloudbees.net/ http://jmxtrans.org/