Mobile Analytics. mit Elasticsearch und Kibana. Dominik Helleberg

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Mobile Analytics. mit Elasticsearch und Kibana. Dominik Helleberg"

Transcription

1 Mobile Analytics mit Elasticsearch und Kibana Dominik Helleberg

2 Speaker Dominik Helleberg Mobile Development Android / Embedded Tools

3 Mobile Analytics

4 Warum? Server Software Runtime Client (Browser) OS Backend Hardware Server Client (Browser) Software Runtime OS Client (Browser) Hardware

5 Warum? Server Server Software Software Runtime Runtime OS Hardware OS Hardware

6 Warum? Client Software Runtime OS Hardware Backend Client Software Runtime OS Hardware Client Software Runtime OS Hardware

7 Was? Crash Reporting Logging Analytics

8 Crash Reporting E/AndroidRuntime( 1188): Caused by: java.lang.runtimeexception: CRASH E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.c.b(unknown Source) E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.selectriver.onstart(unknown Source) E/AndroidRuntime( 1188): at android.app.instrumentation.callactivityonstart(instrumentati E/AndroidRuntime( 1188): at android.app.activity.performstart(activity.java:5018) E/AndroidRuntime( 1188): at android.app.activitythread.performlaunchactivity(activitythre E/AndroidRuntime( 1188): more

9 Crash Reporting

10 Crash Reporting Pro-aktive Benachrichtigung bei Crashes Nachvollziehbare Stacktraces Device / App und Umgebungs-Infos Optional: Nachricht vom Benutzer Statistische Analysen

11 Crash Reporting Nachvollziehbare Stacktraces E/AndroidRuntime( 1188): Caused by: java.lang.runtimeexception: CRASH E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.c.b(unknown Source) E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.selectriver.onstart(unknown Source) E/AndroidRuntime( 1188): at android.app.instrumentation.callactivityonstart(instrumentati E/AndroidRuntime( 1188): at android.app.activity.performstart(activity.java:5018) E/AndroidRuntime( 1188): at android.app.activitythread.performlaunchactivity(activitythre E/AndroidRuntime( 1188): more Exception Type: SIGSEGV Exception Codes: SEGV_ACCERR at 0x13babeb8 Crashed Thread: 8 Thread 8 Crashed: 0 libobjc.a.dylib 0x f5bbd0 0x196f CoreFoundation 0x x QuartzCore 0x cad8 0x QuartzCore 0x aec 0x libobjc.a.dylib 0x f x196f CoreFoundation 0x d074 0x Foundation 0x acffc 0x CFNetwork 0x ec9aac 0x184dac

12 Crash Reporting Crash Analytics Mobile App Crash reporter dsym mapping DISTRIBUTION PLATTFORM Logging reporter Analytics reporter Crash reporter

13 Crash Reporting App-Stores 3rd Party Custom inhouse

14 Crash Reporting App Stores

15 Crash Reporting App Stores

16 Crash Reporting App Stores

17 Crash Reporting App Stores No unified interface Internal betas Internal apps Limited to App-Store functionality

18 Crash Reporting 3 rd Party Solutions

19 Crash Reporting 3 rd Party Solutions Internal apps (if there s internet access) Pricing Privacy

20 Crash Reporting inhouse Crash Analytics Mobile App Crash reporter dsym mapping DISTRIBUTION PLATTFORM Crash reporter

21 Crash Reporting inhouse Analytics + Alarming Crash Analytics Storage Receiver Crash reporter dsym mapping

22 Crash Reporting inhouse

23 Crash Reporting inhouse

24 Crash Reporting inhouse Limited Analytics (Trends, queries, search etc.) User Interface Fixed data-structures Custom stack

25 Crash Reporting with ELK inhouse 2.0 ELK?

26 Crash Reporting with ELK inhouse 2.0 E lasticsearch L ogstash Kibana

27 Crash Reporting with ELK inhouse 2.0 L ogstash

28 Crash Reporting with ELK inhouse 2.0 L ogstash Logstash is a data pipeline that helps you process logs and other event data from a variety of systems. With 165 plugins and counting, Logstash can connect to a variety of sources and stream data at scale to a central analytics system.

29 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input Plugins Filter Plugins Output Plugins

30 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input Plugins (~40) Filter Plugins Output Plugins elasticsearch eventlog file http jdbc kafka jmx log4j tcp twitter syslog rabbitmq redis udp zeromq

31 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input Plugins Filter Plugins (~40) Output Plugins aggregate anonymize collate csv checksum drop dns geoip grok extractnumbers mutate metrics ruby uuid throttle urldecode translate

32 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input Plugins Filter Plugins Output Plugins (~50) elasticsearch csv cloudwatch datadog file g_c_s hipchat http irc jira kafka mongodb nagios pipe s3 redmine redis sqs tcp rabbitmq syslog stdout zabbix

33 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input HTTP (json) Filter de- obfuscate (custom) Output elastic search dsym mapping

34 Crash Reporting with ELK inhouse 2.0 Elasticsearch Document / Data Store (JSON) RESTful API Uses Lucene for indexing Optimized for search queries Easy to distribute and scale Auto-Schema Easy to start with

35 Crash Reporting with ELK inhouse 2.0 Elasticsearch Terms: Index Type Document Field Mapping

36 Crash Reporting with ELK inhouse 2.0 Index: Elasticsearch logstash- %{platform}- %{PACKAGE_NAME}- %{+YYYY.MM.dd} Type crash- report Document { "APP_VERSION_NAME": "1.3.2", "AVAILABLE_MEM_SIZE": , "USER_APP_START_DATE": " T13:37: :00", "CUSTOM_DATA": {}, "BRAND": "samsung... }

37 Crash Reporting with ELK inhouse 2.0 Kibana Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps. Kibana makes it easy to understand large volumes of data. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

38 Crash Reporting with ELK inhouse 2.0 Kibana

39 Crash Reporting with ELK inhouse 2.0 Crash Analytics Kibana elastic search Logstash Crash Reporter ACRA (Android) QuincyKit (ios) dsym mapping

40 Crash Reporting with ELK inhouse 2.0

41 Crash Reporting with ELK inhouse 2.0 Crash Analytics Kibana elastic search Logstash Crash Reporter ACRA (Android) QuincyKit (ios) Kibana Plugins (soon!) Watcher dsym mapping

42 Logging with ELK Kibana Remote Logging elastic search Logstash Logging Library

43 Analytics A bit more complicated then crash reporting and logging

44 Analytics

45 Analytics

46 Analytics

47 Analytics

48 Analytics

49 Analytics

50 Analytics with ELK Analytics Kibana elastic search Logstash Analytics Library (userid, Session Tracking, PageViews) Kibana- Plugins (not yet!) Carrier GeoIP UserProfiles Device Infos

51 ELK -> Getting started Hosting: elastic.co google-compute qbox Define index strategy Check your Schema Changes are hard.raw vs. analyzed fields

52 Summary ELK Stack seems like a good fit for mobile analytics. Kibana PlugIns will help with missing Features. Logstash PlugIns and Client Side Library needed in order to improve Analytics capabilities. It needs a bit of time and work but it looks promising.

53 Danke! Danke!

Using elasticsearch, logstash and kibana to create realtime dashboards

Using elasticsearch, logstash and kibana to create realtime dashboards Using elasticsearch, logstash and kibana to create realtime dashboards Alexander Reelsen @spinscale alexander.reelsen@elasticsearch.com Agenda The need, complexity and pain of logging Logstash basics Usage

More information

Logging on a Shoestring Budget

Logging on a Shoestring Budget UNIVERSITY OF NEBRASKA AT OMAHA Logging on a Shoestring Budget James Harr jharr@unomaha.edu Agenda The Tools ElasticSearch Logstash Kibana redis Composing a Log System Q&A, Conclusions, Lessons Learned

More information

Finding the needle in the haystack with ELK

Finding the needle in the haystack with ELK Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts S by Christophe@Vandeplas.com Whoami S Working for the Belgian Government my own company S Incident

More information

Log management with Graylog2 Lennart Koopmann, FrOSCon 2012. Mittwoch, 29. August 12

Log management with Graylog2 Lennart Koopmann, FrOSCon 2012. Mittwoch, 29. August 12 Log management with Graylog2 Lennart Koopmann, FrOSCon 2012 About me 24 years old, Software Engineer at XING AG Hamburg, Germany @_lennart Graylog2 Free and open source log management system Started in

More information

Wie man aus langweiligen Logdateien Gold gewinnen kann

Wie man aus langweiligen Logdateien Gold gewinnen kann 1 Wie man aus langweiligen Logdateien Gold gewinnen kann About me 2 Klaus Bild Senior System Architect IBM Connections/Sametime/TDI Monitoring/Log Management Infrastructure (Cloud, Docker ) Blog: http://kbild.ch

More information

Log management with Logstash and Elasticsearch. Matteo Dessalvi

Log management with Logstash and Elasticsearch. Matteo Dessalvi Log management with Logstash and Elasticsearch Matteo Dessalvi HEPiX 2013 Outline Centralized logging. Logstash: what you can do with it. Logstash + Redis + Elasticsearch. Grok filtering. Elasticsearch

More information

Andrew Moore Amsterdam 2015

Andrew Moore Amsterdam 2015 Andrew Moore Amsterdam 2015 Agenda Why log How to log Audit plugins Log analysis Demos Logs [timestamp]: [some useful data] Why log? Error Log Binary Log Slow Log General Log Why log? Why log? Why log?

More information

Powering Monitoring Analytics with ELK stack

Powering Monitoring Analytics with ELK stack Powering Monitoring Analytics with ELK stack Abdelkader Lahmadi, Frédéric Beck INRIA Nancy Grand Est, University of Lorraine, France 2015 (compiled on: June 23, 2015) References online Tutorials Elasticsearch

More information

Real-time Data Analytics mit Elasticsearch. Bernhard Pflugfelder inovex GmbH

Real-time Data Analytics mit Elasticsearch. Bernhard Pflugfelder inovex GmbH Real-time Data Analytics mit Elasticsearch Bernhard Pflugfelder inovex GmbH Bernhard Pflugfelder Big Data Engineer @ inovex Fields of interest: search analytics big data bi Working with: Lucene Solr Elasticsearch

More information

Information Retrieval Elasticsearch

Information Retrieval Elasticsearch Information Retrieval Elasticsearch IR Information retrieval (IR) is the activity of obtaining information resources relevant to an information need from a collection of information resources. Searches

More information

Using Logstash and Elasticsearch analytics capabilities as a BI tool

Using Logstash and Elasticsearch analytics capabilities as a BI tool Using Logstash and Elasticsearch analytics capabilities as a BI tool Pashalis Korosoglou, Pavlos Daoglou, Stefanos Laskaridis, Dimitris Daskopoulos Aristotle University of Thessaloniki, IT Center Outline

More information

Processing millions of logs with Logstash

Processing millions of logs with Logstash and integrating with Elasticsearch, Hadoop and Cassandra November 21, 2014 About me My name is Valentin Fischer-Mitoiu and I work for the University of Vienna. More specificaly in a group called Domainis

More information

Log Analysis with the ELK Stack (Elasticsearch, Logstash and Kibana) Gary Smith, Pacific Northwest National Laboratory

Log Analysis with the ELK Stack (Elasticsearch, Logstash and Kibana) Gary Smith, Pacific Northwest National Laboratory Log Analysis with the ELK Stack (Elasticsearch, Logstash and Kibana) Gary Smith, Pacific Northwest National Laboratory A Little Context! The Five Golden Principles of Security! Know your system! Principle

More information

Data Discovery and Systems Diagnostics with the ELK stack. Rittman Mead - BI Forum 2015, Brighton. Robin Moffatt, Principal Consultant Rittman Mead

Data Discovery and Systems Diagnostics with the ELK stack. Rittman Mead - BI Forum 2015, Brighton. Robin Moffatt, Principal Consultant Rittman Mead Data Discovery and Systems Diagnostics with the ELK stack Rittman Mead - BI Forum 2015, Brighton Robin Moffatt, Principal Consultant Rittman Mead T : +44 (0) 1273 911 268 (UK) About Me Principal Consultant

More information

Using NXLog with Elasticsearch and Kibana. Using NXLog with Elasticsearch and Kibana

Using NXLog with Elasticsearch and Kibana. Using NXLog with Elasticsearch and Kibana Using NXLog with Elasticsearch and Kibana i Using NXLog with Elasticsearch and Kibana Using NXLog with Elasticsearch and Kibana ii Contents 1 Setting up Elasticsearch and Kibana 1 1.1 Installing Elasticsearch................................................

More information

Efficient Management of System Logs using a Cloud Radoslav Bodó, Daniel Kouřil CESNET. ISGC 2013, March 2013

Efficient Management of System Logs using a Cloud Radoslav Bodó, Daniel Kouřil CESNET. ISGC 2013, March 2013 Efficient Management of System Logs using a Cloud Radoslav Bodó, Daniel Kouřil CESNET ISGC 2013, March 2013 Agenda Introduction Collecting logs Log Processing Advanced analysis Resume Introduction Status

More information

Developing an Application Tracing Utility for Mule ESB Application on EL (Elastic Search, Log stash) Stack Using AOP

Developing an Application Tracing Utility for Mule ESB Application on EL (Elastic Search, Log stash) Stack Using AOP Developing an Application Tracing Utility for Mule ESB Application on EL (Elastic Search, Log stash) Stack Using AOP Mohan Bandaru, Amarendra Kothalanka, Vikram Uppala Student, Department of Computer Science

More information

Federated SQL on Hadoop and Beyond: Leveraging Apache Geode to Build a Poor Man's SAP HANA. by Christian Tzolov @christzolov

Federated SQL on Hadoop and Beyond: Leveraging Apache Geode to Build a Poor Man's SAP HANA. by Christian Tzolov @christzolov Federated SQL on Hadoop and Beyond: Leveraging Apache Geode to Build a Poor Man's SAP HANA by Christian Tzolov @christzolov Whoami Christian Tzolov Technical Architect at Pivotal, BigData, Hadoop, SpringXD,

More information

Analyzing large flow data sets using. visualization tools. modern open-source data search and. FloCon 2014. Max Putas

Analyzing large flow data sets using. visualization tools. modern open-source data search and. FloCon 2014. Max Putas Analyzing large flow data sets using modern open-source data search and visualization tools FloCon 2014 Max Putas About me Operations Engineer - DevOps BS, MS, and CAS in Telecommunications Work/research

More information

LOG- UND EVENTMANAGEMENT

LOG- UND EVENTMANAGEMENT LOG- UND EVENTMANAGEMENT OSMC 2013 23.10.2013 BERND ERK NETWAYS GMBH AGENDA Kurzvorstellung Einführung Architektur Installation Routing und Filterung von Events Interfaces & API Integration in Nagios und

More information

Elasticsearch on Cisco Unified Computing System: Optimizing your UCS infrastructure for Elasticsearch s analytics software stack

Elasticsearch on Cisco Unified Computing System: Optimizing your UCS infrastructure for Elasticsearch s analytics software stack Elasticsearch on Cisco Unified Computing System: Optimizing your UCS infrastructure for Elasticsearch s analytics software stack HIGHLIGHTS Real-Time Results Elasticsearch on Cisco UCS enables a deeper

More information

Bernd Ahlers Michael Friedrich. Log Monitoring Simplified Get the best out of Graylog2 & Icinga 2

Bernd Ahlers Michael Friedrich. Log Monitoring Simplified Get the best out of Graylog2 & Icinga 2 Bernd Ahlers Michael Friedrich Log Monitoring Simplified Get the best out of Graylog2 & Icinga 2 BEFORE WE START Agenda AGENDA Introduction Tools Log History Logs & Monitoring Demo The Future Resources

More information

LOG- UND EVENTMANAGEMENT MIT LOGSTASH UND GRAPHITE

LOG- UND EVENTMANAGEMENT MIT LOGSTASH UND GRAPHITE LOG- UND EVENTMANAGEMENT MIT LOGSTASH UND GRAPHITE LINUXTAG 2014 08.05.2014 BERND ERK NETWAYS GMBH AGENDA Kurzvorstellung Einführung Architektur Installation Routing und Filterung von Events Interfaces

More information

Log managing at PIC. A. Bruno Rodríguez Rodríguez. Port d informació científica Campus UAB, Bellaterra Barcelona. December 3, 2013

Log managing at PIC. A. Bruno Rodríguez Rodríguez. Port d informació científica Campus UAB, Bellaterra Barcelona. December 3, 2013 Log managing at PIC A. Bruno Rodríguez Rodríguez Port d informació científica Campus UAB, Bellaterra Barcelona December 3, 2013 Bruno Rodríguez (PIC) Log managing at PIC December 3, 2013 1 / 21 What will

More information

FUJITSU Software ServerView Cloud Monitoring Manager V1 Introduction

FUJITSU Software ServerView Cloud Monitoring Manager V1 Introduction FUJITSU Software ServerView Cloud Monitoring Manager V1 Introduction November 2015 Fujitsu Limited Product Overview 1 Why a Monitoring & Logging OpenStack Service? OpenStack systems are large, complex

More information

A New Approach to Network Visibility at UBC. Presented by the Network Management Centre and Wireless Infrastructure Teams

A New Approach to Network Visibility at UBC. Presented by the Network Management Centre and Wireless Infrastructure Teams A New Approach to Network Visibility at UBC Presented by the Network Management Centre and Wireless Infrastructure Teams Agenda Business Drivers Technical Overview Network Packet Broker Tool Network Monitoring

More information

Why should you look at your logs? Why ELK (Elasticsearch, Logstash, and Kibana)?

Why should you look at your logs? Why ELK (Elasticsearch, Logstash, and Kibana)? Authors Introduction This guide is designed to help developers, DevOps engineers, and operations teams that run and manage applications on top of AWS to effectively analyze their log data to get visibility

More information

the missing log collector Treasure Data, Inc. Muga Nishizawa

the missing log collector Treasure Data, Inc. Muga Nishizawa the missing log collector Treasure Data, Inc. Muga Nishizawa Muga Nishizawa (@muga_nishizawa) Chief Software Architect, Treasure Data Treasure Data Overview Founded to deliver big data analytics in days

More information

CI Pipeline with Docker 2015-02-27

CI Pipeline with Docker 2015-02-27 CI Pipeline with Docker 2015-02-27 Juho Mäkinen, Technical Operations, Unity Technologies Finland http://www.juhonkoti.net http://github.com/garo Overview 1. Scale on how we use Docker 2. Overview on the

More information

Graylog2 Lennart Koopmann, OSDC 2014. @_lennart / www.graylog2.org

Graylog2 Lennart Koopmann, OSDC 2014. @_lennart / www.graylog2.org Graylog2 Lennart Koopmann, OSDC 2014 @_lennart / www.graylog2.org About me 25 years old Living in Hamburg, Germany @_lennart on Twitter Co-Founder of TORCH - The Graylog2 company. Graylog2 history Started

More information

Introduction to IBM Worklight Mobile Platform

Introduction to IBM Worklight Mobile Platform Introduction to IBM Worklight Mobile Platform The Worklight Mobile Platform The Worklight Mobile Platform is an open, complete and advanced mobile application platform for HTML5, hybrid and native apps.

More information

Comparative Analysis of Open-Source Log Management Solutions for Security Monitoring and Network Forensics

Comparative Analysis of Open-Source Log Management Solutions for Security Monitoring and Network Forensics Comparative Analysis of Open-Source Log Management Solutions for Security Monitoring and Network Forensics Risto Vaarandi, Paweł Niziski NATO Cooperative Cyber Defence Centre of Excellence, Tallinn, Estonia

More information

Log management with Graylog2 Lennart Koopmann, Kieker Days 2012. Mittwoch, 5. Dezember 12

Log management with Graylog2 Lennart Koopmann, Kieker Days 2012. Mittwoch, 5. Dezember 12 Log management with Graylog2 Lennart Koopmann, Kieker Days 2012 About me 24 years old, Software Engineer at XING AG Hamburg, Germany @_lennart Graylog2 Free and open source log management system Started

More information

Reliable log data transfer

Reliable log data transfer OWASP Switzerland Chapter December 2015 Reliable log data transfer About (r)syslog, logstash, and log data signing A field report pascal.buchbinder@adnovum.ch Agenda Why we need log data transfer Syslog

More information

Efficient Management of System Logs using a Cloud

Efficient Management of System Logs using a Cloud , CESNET z.s.p.o.,zikova 4, 160 00 Praha 6, Czech Republic and University of West Bohemia,Univerzitní 8, 306 14 Pilsen, Czech Republic E-mail: bodik@civ.zcu.cz Daniel Kouřil, CESNET z.s.p.o.,zikova 4,

More information

Log infrastructure & Zabbix. logging tools integration

Log infrastructure & Zabbix. logging tools integration Log infrastructure & Zabbix logging tools integration About me Me Linux System Architect @ ICTRA from Belgium (...) IT : Linux & SysAdmin work, Security, ICTRA ICT for Rail for Transport Mobility Security

More information

Modern Web development and operations practices. Grig Gheorghiu VP Tech Operations Nasty Gal Inc. @griggheo

Modern Web development and operations practices. Grig Gheorghiu VP Tech Operations Nasty Gal Inc. @griggheo Modern Web development and operations practices Grig Gheorghiu VP Tech Operations Nasty Gal Inc. @griggheo Modern Web stack Aim for horizontal scalability! Ruby/Python front-end servers (Sinatra/Padrino,

More information

April 8th - 10th, 2014 LUG14 LUG14. Lustre Log Analyzer. Kalpak Shah. DataDirect Networks. ddn.com. 2014 DataDirect Networks. All Rights Reserved.

April 8th - 10th, 2014 LUG14 LUG14. Lustre Log Analyzer. Kalpak Shah. DataDirect Networks. ddn.com. 2014 DataDirect Networks. All Rights Reserved. April 8th - 10th, 2014 LUG14 LUG14 Lustre Log Analyzer Kalpak Shah DataDirect Networks Lustre Log Analysis Requirements Need scripts to parse Lustre debug logs Only way to effectively use the logs for

More information

Log Management with Open-Source Tools. Risto Vaarandi SEB Estonia

Log Management with Open-Source Tools. Risto Vaarandi SEB Estonia Log Management with Open-Source Tools Risto Vaarandi SEB Estonia Outline Why use open source tools for log management? Widely used logging protocols and recently introduced new standards Open-source syslog

More information

ITP 342 Mobile App Development. APIs

ITP 342 Mobile App Development. APIs ITP 342 Mobile App Development APIs API Application Programming Interface (API) A specification intended to be used as an interface by software components to communicate with each other An API is usually

More information

Blackboard Open Source Monitoring

Blackboard Open Source Monitoring Blackboard Open Source Monitoring By Greg Lloyd Submitted to the Faculty of the School of Information Technology in Partial Fulfillment of the Requirements for the Degree of Bachelor of Science in Information

More information

Enterprise Application Monitoring with

Enterprise Application Monitoring with Enterprise Application Monitoring with 11/10/2007 Presented by James Peel james.peel@altinity.com / www.altinity.com 1 Who am I? James Peel - james.peel@altinity.com Job: Managing Director of Altinity

More information

Creating Big Data Applications with Spring XD

Creating Big Data Applications with Spring XD Creating Big Data Applications with Spring XD Thomas Darimont @thomasdarimont THE FASTEST PATH TO NEW BUSINESS VALUE Journey Introduction Concepts Applications Outlook 3 Unless otherwise indicated, these

More information

WHITE PAPER Redefining Monitoring for Today s Modern IT Infrastructures

WHITE PAPER Redefining Monitoring for Today s Modern IT Infrastructures WHITE PAPER Redefining Monitoring for Today s Modern IT Infrastructures Modern technologies in Zenoss Service Dynamics v5 enable IT organizations to scale out monitoring and scale back costs, avoid service

More information

Introduction. Background

Introduction. Background Introduction Bro is an open-source network security monitor which inspects network traffic looking for suspicious activity. The Bro framework provides an extensible scripting language that allows an analysis

More information

TDAQ Analytics Dashboard

TDAQ Analytics Dashboard 14 October 2010 ATL-DAQ-SLIDE-2010-397 TDAQ Analytics Dashboard A real time analytics web application Outline Messages in the ATLAS TDAQ infrastructure Importance of analysis A dashboard approach Architecture

More information

WORKSHOP Log Management with NetEye 3.5

WORKSHOP Log Management with NetEye 3.5 WORKSHOP Log Management with NetEye 3.5 Program 2015 by Thomas Forrer LogAnalysis with Logstash & Kibana Log Management in NetEye Configuration of Log sources / Agents Log acquisition and configuration

More information

STREAM ANALYTIX. Industry s only Multi-Engine Streaming Analytics Platform

STREAM ANALYTIX. Industry s only Multi-Engine Streaming Analytics Platform STREAM ANALYTIX Industry s only Multi-Engine Streaming Analytics Platform One Platform for All Create real-time streaming data analytics applications in minutes with a powerful visual editor Get a wide

More information

Geo-Identification of Web Users through Logs using ELK Stack

Geo-Identification of Web Users through Logs using ELK Stack Geo-Identification of Web Users through Logs using ELK Stack Tarun Prakash Ms. Misha Kakkar Kritika Patel Amity University, Uttar Pradesh Assistant Professor, Amity University, Uttar Pradesh Amity University,

More information

ni.com Remote Connectivity with LabVIEW

ni.com Remote Connectivity with LabVIEW Remote Connectivity with LabVIEW What Is Remote Connectivity? Local Monitoring 3 Remote Mobile Access 4 What Is Remote Connectivity Two machines talking to one another Client Server PC PC Consumes Data

More information

Open Source Business Service/Process Management

Open Source Business Service/Process Management Open Source Business Service/Process Management Rodrigue Chakode Libre Software Meeting, Brussels 2013-07-11 1 Agenda My other life is Open Source Background Why Business Service Management (BSM) Deal

More information

Building a logging pipeline with Open Source tools. Iñigo Ortiz de Urbina Cazenave

Building a logging pipeline with Open Source tools. Iñigo Ortiz de Urbina Cazenave Building a logging pipeline with Open Source tools Iñigo Ortiz de Urbina Cazenave NLUUG Utrecht - Netherlands 28 May 2015 whoami; 2 Iñigo Ortiz de Urbina Cazenave Systems Engineer whoami; groups; 3 Iñigo

More information

ntopng: Realtime Network Traffic View

ntopng: Realtime Network Traffic View ntopng: Realtime Network Traffic View Luca Deri 3/28/14 1 ntop in 1998 In 1998, the original ntop has been created. Available for Unix and Windows under GPL. Contrary to many tools available

More information

Proactive and Reactive Monitoring

Proactive and Reactive Monitoring Proactive and Reactive Monitoring Serg Mescheryakov, Doctor of Science, Professor Dmitry Shchemelinin, Philosophy Doctor RingCentral Inc., San Mateo, CA, USA RingCentral IP Telecommunication Company 2

More information

Improve performance and availability of Banking Portal with HADOOP

Improve performance and availability of Banking Portal with HADOOP Improve performance and availability of Banking Portal with HADOOP Our client is a leading U.S. company providing information management services in Finance Investment, and Banking. This company has a

More information

Datasheet FUJITSU Software ServerView Cloud Monitoring Manager V1.0

Datasheet FUJITSU Software ServerView Cloud Monitoring Manager V1.0 Datasheet FUJITSU Software ServerView Cloud Monitoring Manager V1.0 Datasheet FUJITSU Software ServerView Cloud Monitoring Manager V1.0 A Monitoring Cloud Service for Enterprise OpenStack Systems Cloud

More information

The Experiment on the Effectiveness of ALICE Online-Offline Process Monitoring

The Experiment on the Effectiveness of ALICE Online-Offline Process Monitoring The Experiment on the Effectiveness of ALICE Online-Offline Process Monitoring Advisor: Dr. Phond Punchongharn Vasco Chibante Barroso Khanasin Yamnual King Mongkut s University of Technology Thonburi Faculty

More information

Centralized logging system based on WebSockets protocol

Centralized logging system based on WebSockets protocol Centralized logging system based on WebSockets protocol Radomír Sohlich sohlich@fai.utb.cz Jakub Janoštík janostik@fai.utb.cz František Špaček spacek@fai.utb.cz Abstract: The era of distributed systems

More information

Scalable Architecture on Amazon AWS Cloud

Scalable Architecture on Amazon AWS Cloud Scalable Architecture on Amazon AWS Cloud Kalpak Shah Founder & CEO, Clogeny Technologies kalpak@clogeny.com 1 * http://www.rightscale.com/products/cloud-computing-uses/scalable-website.php 2 Architect

More information

Sisense. Product Highlights. www.sisense.com

Sisense. Product Highlights. www.sisense.com Sisense Product Highlights Introduction Sisense is a business intelligence solution that simplifies analytics for complex data by offering an end-to-end platform that lets users easily prepare and analyze

More information

Elasticsearch, Logstash, and Kibana (ELK)

Elasticsearch, Logstash, and Kibana (ELK) Elasticsearch, Logstash, and Kibana (ELK) Dwight Beaver dsbeaver@cert.org Sean Hutchison shutchison@cert.org January 2015 2014 Carnegie Mellon University This material is based upon work funded and supported

More information

Security Data Analytics Platform

Security Data Analytics Platform Security Data Analytics Platform Figure 1 - Global Search Dashboard "The Data Analytics Platform has revolutionized the way we handle data from our Security monitoring infrastructure to our developers

More information

PULSE APPCONNECT. A Micro VPN That Allows Specific Applications on Mobile Devices to Independently Leverage the Connect Secure Gateway.

PULSE APPCONNECT. A Micro VPN That Allows Specific Applications on Mobile Devices to Independently Leverage the Connect Secure Gateway. PULSE APPCONNECT A Micro VPN That Allows Specific Applications on Mobile Devices to Independently Leverage the Connect Secure Gateway Published Date July 2015 Contents Executive Summary 3 Introduction

More information

Realtime Web @HuffingtonPost

Realtime Web @HuffingtonPost Realtime Web @HuffingtonPost Websockets, SockJS and RabbitMQ Adam Denenberg VP Engineering @denen adam.denenberg@huffingtonpost.com 1 Huffington Post 500 MM PVs/week 12 MM UVs/week 200MM+ Comments, 2MM

More information

Lecture 2-ter. 2. A communication example Managing a HTTP v1.0 connection. G.Bianchi, G.Neglia, V.Mancuso

Lecture 2-ter. 2. A communication example Managing a HTTP v1.0 connection. G.Bianchi, G.Neglia, V.Mancuso Lecture 2-ter. 2 A communication example Managing a HTTP v1.0 connection Managing a HTTP request User digits URL and press return (or clicks ). What happens (HTTP 1.0): 1. Browser opens a TCP transport

More information

Big Data Pipeline and Analytics Platform

Big Data Pipeline and Analytics Platform Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Source Software Sudhir Tonse (@stonse) Danny Yuan (@g9yuayon) Netflix is a log generating company that also happens to stream movies

More information

RingBase. Software Requirements Specification. Feb 11, 2014. Chandra Krintz CS 189A. Wednesday 6PM. Date Feb 11, 2013 Mentor Colin Kelley

RingBase. Software Requirements Specification. Feb 11, 2014. Chandra Krintz CS 189A. Wednesday 6PM. Date Feb 11, 2013 Mentor Colin Kelley RingBase Software Requirements Specification Feb 11, 2014 Group Name: RingBase Instructor Course Lab Section Teaching Assistant Chandra Krintz CS 189A Wednesday 6PM Geoffrey Douglas Date Feb 11, 2013 Mentor

More information

BIRT ihub 3. 2013 Actuate Customer Days. Wow that looks good! Jeff Morris & Mark Gamble

BIRT ihub 3. 2013 Actuate Customer Days. Wow that looks good! Jeff Morris & Mark Gamble BIRT ihub 3 Wow that looks good! Jeff Morris & Mark Gamble SF Nov7 - UK Nov12 - DE Nov13 - FR Nov14 - SG Nov19 - JP Nov22 - NY Dec4 2013 Actuate Customer Days Actuate BIRT ihub 3 Focus Areas Simplified,

More information

ZABBIX. An Enterprise-Class Open Source Distributed Monitoring Solution. Takanori Suzuki MIRACLE LINUX CORPORATION October 22, 2009

ZABBIX. An Enterprise-Class Open Source Distributed Monitoring Solution. Takanori Suzuki MIRACLE LINUX CORPORATION October 22, 2009 ZABBIX An Enterprise-Class Open Source Distributed Monitoring Solution Takanori Suzuki MIRACLE LINUX CORPORATION October 22, 2009 Outline Biography ZABBIX New release ZABBIX 1.8 Activity in Japan Biography

More information

Data Driven Success. Comparing Log Analytics Tools: Flowerfire s Sawmill vs. Google Analytics (GA)

Data Driven Success. Comparing Log Analytics Tools: Flowerfire s Sawmill vs. Google Analytics (GA) Data Driven Success Comparing Log Analytics Tools: Flowerfire s Sawmill vs. Google Analytics (GA) In business, data is everything. Regardless of the products or services you sell or the systems you support,

More information

Who did what, when, where and how MySQL Audit Logging. Jeremy Glick & Andrew Moore 20/10/14

Who did what, when, where and how MySQL Audit Logging. Jeremy Glick & Andrew Moore 20/10/14 Who did what, when, where and how MySQL Audit Logging Jeremy Glick & Andrew Moore 20/10/14 Intro 2 Hello! Intro 3 Jeremy Glick MySQL DBA Head honcho of Chicago MySQL meetup 13 years industry experience

More information

Monitoring and data warehousing tools Initial version

Monitoring and data warehousing tools Initial version Ref. Ares(2016)529011-01/02/2016 Developing Data-Intensive Cloud Applications with Iterative Quality Enhancements Monitoring and data warehousing tools Initial version Deliverable 4.1 Deliverable: D4.1

More information

Last time. Today. IaaS Providers. Amazon Web Services, overview

Last time. Today. IaaS Providers. Amazon Web Services, overview Last time General overview, motivation, expected outcomes, other formalities, etc. Please register for course Online (if possible), or talk to Yvonne@CS Course evaluation forgotten Please assign one volunteer

More information

DevOps Best Practices for Mobile Apps. Sanjeev Sharma IBM Software Group

DevOps Best Practices for Mobile Apps. Sanjeev Sharma IBM Software Group DevOps Best Practices for Mobile Apps Sanjeev Sharma IBM Software Group Me 18 year in the software industry 15+ years he has been a solution architect with IBM Areas of work: o DevOps o Enterprise Architecture

More information

Load and Performance Load Testing. RadView Software October 2015 www.radview.com

Load and Performance Load Testing. RadView Software October 2015 www.radview.com Load and Performance Load Testing RadView Software October 2015 www.radview.com Contents Introduction... 3 Key Components and Architecture... 4 Creating Load Tests... 5 Mobile Load Testing... 9 Test Execution...

More information

Logentries Insights: The State of Log Management & Analytics for AWS

Logentries Insights: The State of Log Management & Analytics for AWS Logentries Insights: The State of Log Management & Analytics for AWS Trevor Parsons Ph.D Co-founder & Chief Scientist Logentries 1 1. Introduction The Log Management industry was traditionally driven by

More information

Architecting Open source solutions on Azure. Nicholas Dritsas Senior Director, Microsoft Singapore

Architecting Open source solutions on Azure. Nicholas Dritsas Senior Director, Microsoft Singapore Learn. Connect. Explore. Architecting Open source solutions on Azure Nicholas Dritsas Senior Director, Microsoft Singapore Agenda Developing OSS Apps on Azure Customer case with OSS Apps Hadoop on Azure

More information

Performance Testing of Big Data Applications

Performance Testing of Big Data Applications Paper submitted for STC 2013 Performance Testing of Big Data Applications Author: Mustafa Batterywala: Performance Architect Impetus Technologies mbatterywala@impetus.co.in Shirish Bhale: Director of Engineering

More information

Building a Simple Mobile optimized Web App/Site Using the jquery Mobile Framework

Building a Simple Mobile optimized Web App/Site Using the jquery Mobile Framework Building a Simple Mobile optimized Web App/Site Using the jquery Mobile Framework pinboard.in tag http://pinboard.in/u:jasonclark/t:amigos-jquery-mobile/ Agenda Learn what a mobile framework is. Understand

More information

4/7/16. Logging for IR. Planning Central Logging

4/7/16. Logging for IR. Planning Central Logging Logging for IR Planning Central Logging 1 Objectives Failures of distributed logging Pros and cons of centralized logging Centralized logging considerations Possible solutions A Sample solution Distributed

More information

SIP Protocol as a Communication Bus to Control Embedded Devices

SIP Protocol as a Communication Bus to Control Embedded Devices 229 SIP Protocol as a Communication Bus to Control Embedded Devices Ramunas DZINDZALIETA Institute of Mathematics and Informatics Akademijos str. 4, Vilnius Lithuania ramunas.dzindzalieta@gmail.com Abstract.

More information

syslog-ng: nyers adatból Big Data

syslog-ng: nyers adatból Big Data syslog-ng: nyers adatból Big Data 2015. vday, Budapest Czanik Péter / Balabit About me Peter Czanik from Hungary Community manager at BalaBit: syslog-ng upstream Doing syslog-ng packaging, support, advocating

More information

Monitoring Linux and Windows Logs with Graylog Collector. Bernd Ahlers Graylog, Inc.

Monitoring Linux and Windows Logs with Graylog Collector. Bernd Ahlers Graylog, Inc. Monitoring Linux and Windows Logs with Graylog Collector Bernd Ahlers Graylog, Inc. Structured Logging & Introduction to Graylog Collector Bernd Ahlers Graylog, Inc. Introduction: Graylog Open source log

More information

FireEye App for Splunk Enterprise

FireEye App for Splunk Enterprise FireEye App for Splunk Enterprise FireEye App for Splunk Enterprise Documentation Version 1.1 Table of Contents Welcome 3 Supported FireEye Event Formats 3 Original Build Environment 3 Possible Dashboard

More information

Wisdom from Crowds of Machines

Wisdom from Crowds of Machines Wisdom from Crowds of Machines Analytics and Big Data Summit September 19, 2013 Chetan Conikee Irfan Ahmad About Us CloudPhysics' mission is to discover the underlying principles that govern systems behavior

More information

automated acceptance testing of mobile apps

automated acceptance testing of mobile apps automated acceptance testing of mobile apps Karl Krukow, CTO, LessPainful Goto Aarhus, 2012 karl@lesspainful.com, @karlkrukow 1 Agenda Automated testing for mobile desirable properties for an acceptance

More information

A Brief. Introduction. of MG-SOFT s SNMP Network Management Products. Document Version 1.3, published in June, 2008

A Brief. Introduction. of MG-SOFT s SNMP Network Management Products. Document Version 1.3, published in June, 2008 A Brief Introduction of MG-SOFT s SNMP Network Management Products Document Version 1.3, published in June, 2008 MG-SOFT s SNMP Products Overview SNMP Management Products MIB Browser Pro. for Windows and

More information

ITP 140 Mobile Technologies. Mobile Topics

ITP 140 Mobile Technologies. Mobile Topics ITP 140 Mobile Technologies Mobile Topics Topics Analytics APIs RESTful Facebook Twitter Google Cloud Web Hosting 2 Reach We need users! The number of users who try our apps Retention The number of users

More information

IBM MobileFirst Hands-on Labs environment with Linux on z Systems and z/os

IBM MobileFirst Hands-on Labs environment with Linux on z Systems and z/os IBM MobileFirst Hands-on Labs environment with Linux on z Systems and z/os Session Nr: 16772 + 16773 Wilhelm Mild Executive IT Architect wilhelm.mild@de.ibm.com Erich Amrehn Distinguished Engineer amrehn@de.ibm.com

More information

Mobile App Analytics Features Overview

Mobile App Analytics Features Overview Mobile App Analytics Features Overview Opentracker s app reporting is bursting at the seams and will probably cover all your needs. An added advantage of our new technology however, is flexibility, which

More information

CARRIOTS TECHNICAL PRESENTATION

CARRIOTS TECHNICAL PRESENTATION CARRIOTS TECHNICAL PRESENTATION Alvaro Everlet, CTO alvaro.everlet@carriots.com @aeverlet Oct 2013 CARRIOTS TECHNICAL PRESENTATION 1. WHAT IS CARRIOTS 2. BUILDING AN IOT PROJECT 3. DEVICES 4. PLATFORM

More information

Log Management with Open-Source Tools. Risto Vaarandi rvaarandi 4T Y4H00 D0T C0M

Log Management with Open-Source Tools. Risto Vaarandi rvaarandi 4T Y4H00 D0T C0M Log Management with Open-Source Tools Risto Vaarandi rvaarandi 4T Y4H00 D0T C0M Outline Why do we need log collection and management? Why use open source tools? Widely used logging protocols and recently

More information

Amazon Web Services Primer. William Strickland COP 6938 Fall 2012 University of Central Florida

Amazon Web Services Primer. William Strickland COP 6938 Fall 2012 University of Central Florida Amazon Web Services Primer William Strickland COP 6938 Fall 2012 University of Central Florida AWS Overview Amazon Web Services (AWS) is a collection of varying remote computing provided by Amazon.com.

More information

Introduction to Streaming Telemetry IX Forum 9. Carlos Campana ccampana@cisco.com December 2015

Introduction to Streaming Telemetry IX Forum 9. Carlos Campana ccampana@cisco.com December 2015 Introduction to Streaming Telemetry Carlos Campana ccampana@cisco.com December 2015 What has changed? New Capabilities New Requirements Speed and scale SDN and centralized control Faster traffic engineering

More information

Intel IT s Cloud Journey. Speaker: [speaker name], Intel IT

Intel IT s Cloud Journey. Speaker: [speaker name], Intel IT Intel IT s Cloud Journey Speaker: [speaker name], Intel IT Accelerating The Corporate IT Journey Cloud enables ubiquitous access to resources and applications, and workload flexibility Cloud IaaS Infrastructure

More information

Monitis Project Proposals for AUA. September 2014, Yerevan, Armenia

Monitis Project Proposals for AUA. September 2014, Yerevan, Armenia Monitis Project Proposals for AUA September 2014, Yerevan, Armenia Distributed Log Collecting and Analysing Platform Project Specifications Category: Big Data and NoSQL Software Requirements: Apache Hadoop

More information

Pulsar Realtime Analytics At Scale. Tony Ng April 14, 2015

Pulsar Realtime Analytics At Scale. Tony Ng April 14, 2015 Pulsar Realtime Analytics At Scale Tony Ng April 14, 2015 Big Data Trends Bigger data volumes More data sources DBs, logs, behavioral & business event streams, sensors Faster analysis Next day to hours

More information

Lecture 17: Mobile Computing Platforms: Android. Mythili Vutukuru CS 653 Spring 2014 March 24, Monday

Lecture 17: Mobile Computing Platforms: Android. Mythili Vutukuru CS 653 Spring 2014 March 24, Monday Lecture 17: Mobile Computing Platforms: Android Mythili Vutukuru CS 653 Spring 2014 March 24, Monday Mobile applications vs. traditional applications Traditional model of computing: an OS (Linux / Windows),

More information

Maintaining Non-Stop Services with Multi Layer Monitoring

Maintaining Non-Stop Services with Multi Layer Monitoring Maintaining Non-Stop Services with Multi Layer Monitoring Lahav Savir System Architect and CEO of Emind Systems lahavs@emindsys.com www.emindsys.com The approach Non-stop applications can t leave on their

More information

Integration of IT-DB Monitoring tools into IT General Notification Infrastructure

Integration of IT-DB Monitoring tools into IT General Notification Infrastructure Integration of IT-DB Monitoring tools into IT General Notification Infrastructure August 2014 Author: Binathi Bingi Supervisor: David Collados Polidura CERN openlab Summer Student Report 2014 1 Project

More information