Continuous Monitoring of Software Services: Design and Application of the Kieker Framework

Transcription

1 Continuous Monitoring of Software Services: Design and Application of the Kieker André van Hoorn 1,3, Matthias Rohr 1,2, Wilhelm Hasselbring 1,3, Jan Waller 3, Jens Ehlers 3, Sören Frey 3, and Dennis Kieselhorst 4 1 Graduate School TrustSoft, Univ. Oldenburg, D Oldenburg, 2 BTC AG, D Oldenburg, 3 Software Engineering Group, Univ. Kiel, D Kiel, 4 EWE TEL GmbH, D Oldenburg January 14, 2010 (Kiel) Certification

2 Agenda v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 2/ 18

3 Kieker Monitoring Continuous monitoring of software runtime behavior <<component>> Kieker.Tpmon IKiekerMonitoring Probe <<component>> Monitoring Probe <<component>> TpmonController IKiekerMonitoring LogWriter <<component>> Monitoring Log Writer Flexible framework architecture Software runtime behavior measures (Distributed) tracing Loop iterations...(custom probes/records/...) <<storage>> Monitoring Log <<abstract>> Monitoring Record * 1 <<component>> Kieker.Tpan <<component>> Monitoring Record Consumer IKiekerMonitoring RecordConsumer <<component>> TpanInstance IKiekerMonitoring LogReader <<component>> Monitoring Log Reader Common (for logging & analysis) and extensible monitoring record model Analysis/: offline and online Low, linear overhead Evaluated in industry case studies A. van Hoorn, M. Rohr, W. Hasselbring, J. Waller, J. Ehlers, S. Frey, and D. Kieselhorst. Continuous monitoring of software services: Design and application of the Kieker framework. TR-0921, Dept. of Computer Science, Univ. Kiel, Germany, Nov v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 3/ 18

4 Kieker Monitoring (cont d) Communication among framework components (life cycle of monitoring records): 1.2: logmonitoringrecord(record) : TpmonController : TpanInstance * 1 1 * 1 1 : MonitoringProbeImpl : MonitoringRecordConsumerImpl 2: writemonitoringrecord(record) * : MonitoringLogWriterImpl 1 : MonitoringLogReaderImpl 1 4.2: consumemonitoringrecord(record) 1 * 1 1 3: tostringarray() 1.1: record = getinstance(...) 4.1: record = initfromstringarray(...) * * * : MonitoringRecordImpl s/analysis models, e.g., Sequence diagrams (Hierarchical) dependency graphs call trees Markov Chains Application examples in our research trace-based performance eval. [RvHG + 08, RvHH + 10] fault localization/anomaly detection based on timing behavior [MRvHH09] runtime adaptation/reconfiguration [vhrgh09, Mat09], visualization of s/w runtime behavior [RvHM + 08] application-level intrusion detection [GSR + 08] v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 4/ 18

5 Example Instrumentation (AspectJ) Instrumentation of an operation by ( ) public static void searchbook ( ) {.. } Response Time Probe public class MyRTProbe implements IKiekerMonitoringProbe { static final TpmonController CTR=TpmonController. getinstance ( ) ; value=" execution (@MyRTProbe. (.. ) ) " ) public Object probe ( ProceedingJoinPoint j ) throws Throwable { MyRTMonitoringRecord record = new MyRTMonitoringRecord ( ) ; record. component = j. getsignature ( ). getdeclaringtypename ( ) ; record. service = j. getsignature ( ). getname ( ) ; } Object r e t v a l ; long t i n = CTR. gettime ( ) ; try { r e t v a l = j. proceed ( ) ; } catch ( Exception e ) { throw e ; } finally { record. r t = CTR. gettime ( ) t i n ; CTR. logmonitoringrecord ( record ) ; } return r e t v a l ; delegate to intercepted operation create & init record measure response time log record v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 5/ 18

6 Example Monitoring Record Consumer Response Time Monitor: public class RTMonitor implements IKiekerRecordConsumer { private final long r t S l o ; public RTMonitor (long r t S l o ) { this. r t S l o = r t S l o ; } public String [ ] getrecordtypesubscriptionlist ( ) { return new String [ ] { MyRTMonitoringRecord. class. getname ( ) } ; } public void consumemonitoringrecord ( AbstractKiekerMonitoringRecord r ) { MyRTMonitoringRecord rtrec = ( MyRTMonitoringRecord ) r ; if ( rtrec. r t > this. r t S l o ) { / SLO v i o l a t i o n! / } } handler for incoming records public boolean execute ( ) { return true ; } public void terminate ( ) { } spawn thread here } for asynchr. analysis record type subscription list "analysis"; here: detection of SLO violations of individual response time observations v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 6/ 18

7 Logging & Reconstructing Information :Bookstore :CRM :Catalog searchbook(...) 0;0 getbook(...) 1;1 getoffers(...) 2;1 getbook(...) 3;2 Legend: = call message = return message = trace i;j = execution with eoi i and ess j Execution order index (eoi) i: i-th started execution in a trace Execution stack size (ess) j: execution started at stack depth j Execution Record and Representations v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 7/ 18

8 Reconstructed UML Sequence Diagram Examples $ DispatcherServlet CatalogBean CatalogService doget(httpservletrequest, HttpServletResponse) process(httpservletrequest, HttpServletResponse) viewcategory() getproductlistbycategory(string) getcategory(string) v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 8/ 18

9 Reconstructed Call Tree Examples (cont d) Call Tree: '61' occurrences $ DispatcherServlet.doPos(.) DispatcherServlet.proces(.) AccountBean.signon() DispatcherServlet.doPos(.) AccountService.getAccoun(.) Cat.getProductListByCategor(.) DispatcherServlet.proces(.) v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 9/ 18

10 Reconstructed Calling Dependency Graph Examples (cont d) $ com.ibatis.jpetstore.web.dispatcherservlet com.ibatis.jpetstore.presentation.catalogbean com.ibatis.jpetstore.presentation.cartbean com.ibatis.jpetstore.presentation.accountbean c com.ibatis.jpetstore.service.catalogservice com.ibatis.jpetstore.service.accountservice v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 10/ 18

11 Hierarchical dependency graph [MRvHH09] Examples (cont d) $ Virtual Machine tier [ 41472/ ,03 25,90% ] Virtual Machine scooter [ 818/2176 0,07 23,43% ] Virtual Machine puck [ 1447/2943 0,03 24,48% ] Deployment Context Level Component Level Virtual Machine klotz org.apache.struts.action.actionservlet [ 41827/ ,190 7,81% ] presentation.orderbean presentation.cartbean presentation.catalogbean [ 1454/3917 0,094 7,18% ] [ 1107/2170 0,087 5,99% ] [ 18138/ ,048 6,88% ] 981 service.hessian.client.orderservice [ 484/981 0,057 6,19% ] 981 Component Level Operation Level presentation.accountbean [ 494/1088 0,062 6,16% ] 1088 service.hessian.client.accountservice [ 523/1088 0,033 6,35% ] service.hessian.client.catalogservice getproduct(string) getproductlistbycategory(string) getitemlistbyproduct(string) getcategory(string) [ 3629/9167 0,046 0,281 2,27% ] [ 6365/ ,161 0,006 3,14% ] [ 9167/9167 0,995 0,406 4,43% ] [ 6309/ ,216 0,037 3,04% ] getproductlistbycategory(string,int,int) getitemlistbyproduct(string,int,int) [ 6402/ ,161 0,106 2,82% ] [ 9167/9167 0,995 0,488 4,69% ] v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 11/ 18

12 Reconstructed Markov Chain [RvHM + 08] Examples (cont d) C,$,ActionServlet.doGet(...) 0.5 C,ActionServlet.doGet(...), CatalogBean.viewItem(...) 1.0 C,CatalogBean.viewItem(...), CatalogService.getItem(...) 1.0 R,CatalogService.getItem(...), CatalogBean.viewItem(...) 1.0 R,CatalogBean.viewItem(...), ActionServlet.doGet(...) C,ActionServlet.doGet(...), CartBean.addItemToCart(...) C,CartBean.addItemToCart(...), CatalogService.getItem(...) R,CatalogService.getItem(...), CartBean.addItemToCart(...) R,CartBean.addItemToCart(...), ActionServlet.doGet(...) R,ActionServlet.doGet(...),$ v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 12/ 18

13 Case Studies Integrated Kieker.Tpan in production systems of two companies Goals: Evaluate applicability/robustness in real systems Qualitative overhead evaluation 1 Photo finishing company, non-distributed setting 1 week application-level trace and performance data Instrumented subset of services Servlet and Spring probes 2 Telecommunication provider, distributed setting (in progress) Distributed application-level trace and performance data Servlet, Spring, and CXF/SOAP probes Results: No perceivable runtime overhead observed v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 13/ 18

14 Telecommunication Provider Case Study Case Studies (cont d) instrumented subsystem <<device>> Front End Server (on site) <<device>> Client <<device>> 0..1 H/W LB * * <<execution environment>> Web Server/ Java EE Container <<artifact>> PortalServiceA <<artifact>> PortalServiceB <<artifact>> PortalServiceC <<device>> Front End Server (off site) <<device>> Application Server <<execution environment>> Java EE Container <<artifact>> BusinessServices Capturing complete workload on FE0/1, BE0/1 Distributed tracing across presentation/business tier 8 concurrent Kieker.Tpmon instances (FS logging) Servlet, Spring, and CXF/SOAP probes 2 2 K M N <<device>> DB Cluster <<device>> Back End System <<device>> 3rd Party System v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 14/ 18

15 PhD Topic: SLAstic Architecture-Based for Increased Resource Efficiency of S/W Systems [vhrgh09, vh09, vhhr09, Stö09] instrumentation instrumented, runtime reconfigurable s/w system... measurements reconfiguration (execution) SLAstic. RECONFIG SLAstic system model (initialization) initilization SLAstic. MONITORING Model Updater Model Manager SLAstic system model (runtime) online adaptation control SLAstic.CONTROL Analysis Evaluator Workload Forecaster Predictor Adaptation Planner v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 15/ 18

16 PhD Topic: SLAstic Architecture-Based for Increased Resource Efficiency of S/W Systems [vhrgh09, vh09, vhhr09, Stö09] instrumentation public String viewcategory() {... instrumented, runtime reconfigurable s/w system } Reconfiguration Plans: 1..* Service response times: measurements reconfiguration (execution) SLAstic. RECONFIG SLAstic system model (initialization) SLA specification: SLA { Obligations { SLO { type=rtquantiles, serviceid=13, value={ (0.90,185ms), (0.95,195ms) (0.99,197ms)} }... } } PCM model Reconfiguration model... initilization SLAstic. MONITORING Model Updater Model Manager SLAstic system model (runtime) online adaptation control updated w/ response times etc. SLAstic.CONTROL Analysis Evaluator Workload Forecaster Predictor Adaptation Planner SLOViolationEvent v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 15/ 18

17 Quantitative Goals Quantitative evaluation of: 1 caused by Kieker.Tpmon components (Scenario 1) 2 Scalability w/ respect to monitoring traces (Scenario 2) v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 16/ 18

18 Results of Scen. 1 (Component ) Quantitative Response time (microseconds) % CI 1. No instr. 2. Empty probe 3. Data collection 4. Writing to disk A A + B A + B + C A + B + C + D Measurement configuration B C D Σ Mean Median Mean execution time (microseconds) D Table: Mean and median values (in microseconds) of B, C, and D as results of the experiment on overhead of Kieker components on a single monitored operation v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 17/ C B

19 Results of Scenario 2 (Recursion) Quantitative Response time (microseconds) % CI Recursion depth v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 18/ 18

20 I Imran Asad Gul, Nils Sommer, Matthias Rohr, André van Hoorn, and Wilhelm Hasselbring. of control flow traces in software applications for intrusion detection. In Proceedings of the 12th IEEE International Multitopic Conference (IEEE INMIC 2008), pages IEEE, December Jasminka Matevska. Architekturbasierte erreichbarkeitsoptimierte Rekonfiguration komponentenbasierter Softwaresysteme zur Laufzeit. PhD thesis, Department of Computer Science, University of Oldenburg, Oldenburg, Germany, July Nina S. Marwede, Matthias Rohr, André van Hoorn, and Wilhelm Hasselbring. Automatic failure diagnosis support in distributed large-scale software systems based on timing behavior anomaly correlation. In Andreas Winter, Rudolf Ferenc, and Jens Knodel, editors, Proceedings of the 13th European Conference on Software Maintenance and Reengineering (CSMR 2009), pages IEEE Computer Society, March Matthias Rohr, André van Hoorn, Simon Giesecke, Jasminka Matevska, Wilhelm Hasselbring, and Sergej Alekseev. -context sensitive performance profiling for enterprise software applications. In Samuel Kounev, Ian Gorton, and Kai Sachs, editors, - Metrics, Models and Benchmarks: Proceedings of the SPEC International Workshop 2008 (SIPEW 08), volume 5119 of Lecture Notes in Computer Science, pages , Heidelberg, June Springer. Matthias Rohr, André van Hoorn, Wilhelm Hasselbring, Marco Lübcke, and Sergej Alekseev. Workload-intensity-sensitive timing behavior analysis for distributed multi-user software systems. In 1st Joint WOSP/SIPEW International Conference on Engineering (WOSP/SIPEW 2010). ACM, January Accepted for publication. Matthias Rohr, André van Hoorn, Jasminka Matevska, Nils Sommer, Lena Stöver, Simon Giesecke, and Wilhelm Hasselbring. Kieker: Continuous monitoring and on demand visualization of Java software behavior. In Claus Pahl, editor, Proceedings of the IASTED International Conference on Software Engineering 2008 (SE 2008), pages 80 85, Anaheim, CA, USA, February ACTA Press. v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 19/ 18

21 II Lena Stöver. Ein Laufzeit-Analyse- zur Unterstützung architekturbasierter, dynamischer Adaption von Software-Systemen ( An online analysis framework supporting architecture-based runtime adaptation of software systems, in German), November Diploma Thesis, University of Oldenburg. André van Hoorn. Adaptive capacity management for the resource-efficient operation of component-based software systems. In Felix C. Freiling, Irene Eusgeld, and Ralf Reussner, editors, Proceedings of the 2008 Dependability Metrics Research Workshop, Technical Report TR , pages Department of Computer Science, University of Mannheim, Germany, May André van Hoorn, Wilhelm Hasselbring, and Matthias Rohr. Engineering and continuously operating self-adaptive software systems: Required design decisions. In Gregor Engels, Ralf Reussner, Christof Momm, and Stefan Sauer, editors, Proceedings of the 1st Workshop des GI- Arbeitskreises Langlebige Softwaresysteme (L2S2): Design for Future - Langlebige Softwaresysteme, volume 537 of CEUR Workshop Proceedings, pages 52 63, November André van Hoorn, Matthias Rohr, Asad Gul, and Wilhelm Hasselbring. An adaptation framework enabling resource-efficient operation of software systems. In Nenad Medvidovic and Tetsuo Tamai, editors, Proceedings of the 2nd Warm-Up Workshop for ACM/IEEE ICSE 2010 (WUP 09), pages ACM, April André van Hoorn, Matthias Rohr, Wilhelm Hasselbring, Jan Waller, Jens Ehlers, Sören Frey, and Dennis Kieselhorst. Continuous monitoring of software services: Design and application of the Kieker framework. Technical Report TR-0921, Department of Computer Science, University of Kiel, Germany, November v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 20/ 18

22 Information and Representation Logging & Reconstructing Information (cont d) <<abstract>> AbstractKiekerMonitoringRecord KiekerExecutionRecord +component +operation +host +traceid +sessionid +eoi +ess +tin +tout +compareto() Execution traces: executions {ordered} 1..* 1 +sender 1 1 +receiver Execution +traceid +tomessage() Message +timestamp +iscallmessage 1 1 +compareto() 1..* Straightforward derivation from monitoring log Message traces: Derived from execution traces Transformed into analysis models/visualizations Message 1 1+traceId messages {ordered} 1 v. Hoorn et al. Continuous Monitoring of Software Services: Design and Application of the Kieker 21/ 18