Mobile Analytics. mit Elasticsearch und Kibana. Dominik Helleberg

Size: px

Start display at page:

Download "Mobile Analytics. mit Elasticsearch und Kibana. Dominik Helleberg"

Monica Little
8 years ago
Views:

1 Mobile Analytics mit Elasticsearch und Kibana Dominik Helleberg

2 Speaker Dominik Helleberg Mobile Development Android / Embedded Tools

3 Mobile Analytics

4 Warum? Server Software Runtime Client (Browser) OS Backend Hardware Server Client (Browser) Software Runtime OS Client (Browser) Hardware

5 Warum? Server Server Software Software Runtime Runtime OS Hardware OS Hardware

6 Warum? Client Software Runtime OS Hardware Backend Client Software Runtime OS Hardware Client Software Runtime OS Hardware

7 Was? Crash Reporting Logging Analytics

8 Crash Reporting E/AndroidRuntime( 1188): Caused by: java.lang.runtimeexception: CRASH E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.c.b(unknown Source) E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.selectriver.onstart(unknown Source) E/AndroidRuntime( 1188): at android.app.instrumentation.callactivityonstart(instrumentati E/AndroidRuntime( 1188): at android.app.activity.performstart(activity.java:5018) E/AndroidRuntime( 1188): at android.app.activitythread.performlaunchactivity(activitythre E/AndroidRuntime( 1188): more

onstart(unknown Source) E/AndroidRuntime( 1188): at android.app.instrumentation.

9 Crash Reporting

10 Crash Reporting Pro-aktive Benachrichtigung bei Crashes Nachvollziehbare Stacktraces Device / App und Umgebungs-Infos Optional: Nachricht vom Benutzer Statistische Analysen

11 Crash Reporting Nachvollziehbare Stacktraces E/AndroidRuntime( 1188): Caused by: java.lang.runtimeexception: CRASH E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.c.b(unknown Source) E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.selectriver.onstart(unknown Source) E/AndroidRuntime( 1188): at android.app.instrumentation.callactivityonstart(instrumentati E/AndroidRuntime( 1188): at android.app.activity.performstart(activity.java:5018) E/AndroidRuntime( 1188): at android.app.activitythread.performlaunchactivity(activitythre E/AndroidRuntime( 1188): more Exception Type: SIGSEGV Exception Codes: SEGV_ACCERR at 0x13babeb8 Crashed Thread: 8 Thread 8 Crashed: 0 libobjc.a.dylib 0x f5bbd0 0x196f CoreFoundation 0x x QuartzCore 0x cad8 0x QuartzCore 0x aec 0x libobjc.a.dylib 0x f x196f CoreFoundation 0x d074 0x Foundation 0x acffc 0x CFNetwork 0x ec9aac 0x184dac

java:5018) E/AndroidRuntime( 1188): at android.app.activitythread.performlaunchactivity(activitythre E/AndroidRuntime( 1188):.

12 Crash Reporting Crash Analytics Mobile App Crash reporter dsym mapping DISTRIBUTION PLATTFORM Logging reporter Analytics reporter Crash reporter

13 Crash Reporting App-Stores 3rd Party Custom inhouse

14 Crash Reporting App Stores

15 Crash Reporting App Stores

16 Crash Reporting App Stores

17 Crash Reporting App Stores No unified interface Internal betas Internal apps Limited to App-Store functionality

18 Crash Reporting 3 rd Party Solutions

19 Crash Reporting 3 rd Party Solutions Internal apps (if there s internet access) Pricing Privacy

20 Crash Reporting inhouse Crash Analytics Mobile App Crash reporter dsym mapping DISTRIBUTION PLATTFORM Crash reporter

21 Crash Reporting inhouse Analytics + Alarming Crash Analytics Storage Receiver Crash reporter dsym mapping

22 Crash Reporting inhouse

23 Crash Reporting inhouse

24 Crash Reporting inhouse Limited Analytics (Trends, queries, search etc.) User Interface Fixed data-structures Custom stack

25 Crash Reporting with ELK inhouse 2.0 ELK?

26 Crash Reporting with ELK inhouse 2.0 E lasticsearch L ogstash Kibana

27 Crash Reporting with ELK inhouse 2.0 L ogstash

28 Crash Reporting with ELK inhouse 2.0 L ogstash Logstash is a data pipeline that helps you process logs and other event data from a variety of systems. With 165 plugins and counting, Logstash can connect to a variety of sources and stream data at scale to a central analytics system.

29 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input Plugins Filter Plugins Output Plugins

30 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input Plugins (~40) Filter Plugins Output Plugins elasticsearch eventlog file http jdbc kafka jmx log4j tcp twitter syslog rabbitmq redis udp zeromq

31 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input Plugins Filter Plugins (~40) Output Plugins aggregate anonymize collate csv checksum drop dns geoip grok extractnumbers mutate metrics ruby uuid throttle urldecode translate

32 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input Plugins Filter Plugins Output Plugins (~50) elasticsearch csv cloudwatch datadog file g_c_s hipchat http irc jira kafka mongodb nagios pipe s3 redmine redis sqs tcp rabbitmq syslog stdout zabbix

33 Crash Reporting with ELK inhouse 2.0 L Logstash ogstash Input HTTP (json) Filter de- obfuscate (custom) Output elastic search dsym mapping

34 Crash Reporting with ELK inhouse 2.0 Elasticsearch Document / Data Store (JSON) RESTful API Uses Lucene for indexing Optimized for search queries Easy to distribute and scale Auto-Schema Easy to start with

35 Crash Reporting with ELK inhouse 2.0 Elasticsearch Terms: Index Type Document Field Mapping

36 Crash Reporting with ELK inhouse 2.0 Index: Elasticsearch logstash- %{platform}- %{PACKAGE_NAME}- %{+YYYY.MM.dd} Type crash- report Document { "APP_VERSION_NAME": "1.3.2", "AVAILABLE_MEM_SIZE": , "USER_APP_START_DATE": " T13:37: :00", "CUSTOM_DATA": {}, "BRAND": "samsung... }

37 Crash Reporting with ELK inhouse 2.0 Kibana Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps. Kibana makes it easy to understand large volumes of data. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

38 Crash Reporting with ELK inhouse 2.0 Kibana

39 Crash Reporting with ELK inhouse 2.0 Crash Analytics Kibana elastic search Logstash Crash Reporter ACRA (Android) QuincyKit (ios) dsym mapping

40 Crash Reporting with ELK inhouse 2.0

41 Crash Reporting with ELK inhouse 2.0 Crash Analytics Kibana elastic search Logstash Crash Reporter ACRA (Android) QuincyKit (ios) Kibana Plugins (soon!) Watcher dsym mapping

42 Logging with ELK Kibana Remote Logging elastic search Logstash Logging Library

43 Analytics A bit more complicated then crash reporting and logging

44 Analytics

45 Analytics

46 Analytics

47 Analytics

48 Analytics

49 Analytics

50 Analytics with ELK Analytics Kibana elastic search Logstash Analytics Library (userid, Session Tracking, PageViews) Kibana- Plugins (not yet!) Carrier GeoIP UserProfiles Device Infos

51 ELK -> Getting started Hosting: elastic.co google-compute qbox Define index strategy Check your Schema Changes are hard.raw vs. analyzed fields

52 Summary ELK Stack seems like a good fit for mobile analytics. Kibana PlugIns will help with missing Features. Logstash PlugIns and Client Side Library needed in order to improve Analytics capabilities. It needs a bit of time and work but it looks promising.

53 Danke! Danke!

Using elasticsearch, logstash and kibana to create realtime dashboards

Using elasticsearch, logstash and kibana to create realtime dashboards Alexander Reelsen @spinscale alexander.reelsen@elasticsearch.com Agenda The need, complexity and pain of logging Logstash basics Usage