Beyond SplunkWeb. Rory Blake. Senior Consultant, Professional Services, Splunk

Transcription

1 Copyright 2015 Splunk Inc. Beyond SplunkWeb Rory Blake Senior Consultant, Professional Services, Splunk

2 Disclaimer During the course of this presentafon, we may make forward looking statements regarding future events or the expected performance of the company. We caufon you that such statements reflect our current expectafons and esfmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentafon are being made as of the Fme and date of its live presentafon. If reviewed aper its live presentafon, this presentafon may not contain current or accurate informafon. We do not assume any obligafon to update any forward looking statements we may make. In addifon, any informafon about our roadmap outlines our general product direcfon and is subject to change at any Fme without nofce. It is for informafonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligafon either to develop the features or funcfonality described or to include any such feature or funcfonality in a future release. 2

3 IntroducFon Copyright 2015 Splunk Inc.

4 Agenda Introducing Alcatel- Lucent and the Velocix CDN Splunk Professional Services The Big Data Problem The Big Data SoluFon Velocix ReporFng API Velocix ReporFng User Interface Splunk Powered ReporFng PlaWorm 4

5 TesFmonial Our customers are very posifve about our Splunk Embedded analyfcs... Splunk Embedded enables us to deliver reporfng that is richer and more versafle than anything a provider can achieve by bolfng a third- party solufon onto exisfng management tools. This value proposifon gives Velocix CDN a powerful compeffve advantage. Velocix CDN, Alcatel- Lucent 5

6 Customer Profile 6

7 subscribers from the network edge. Delivery from inside the network also reduces the transit and peering expenses that go along with carrying over-the-top (OTT) video and distributing popular software packages. opportunities A service provider can use an on-net CDN to deliver QoE that can t be matched by OTT providers. Unlike third-party CDN services that deliver content from off the network, an on-net CDN avoids points of congestion by delivering content from caches close to consumers. By minimizing the distance that content travels over the network, it promotes faster and more reliable streaming. This assures quality What is a CDN? of service (QoS) and allows the service provider to manage and preserve content quality throughout delivery a crucial requirement for monetizing content and building brand equity. Figure 1. A scalable on-net CDN architecture efficiently delivers content from multiple sources Distributed delivery caches reduce backbone traffic and improve QoE Head-end (content) Subscriber Edge router Delivery cache Core network Aggregation router Interconnection with other service providers or third-party CDNs Access network Move content insertion points closer to subscribers Internet gateway On/off-net boundary Internet (content) VELOCIX CONTENT DELIVERY NETWORK ALCATEL-LUCENT BROCHURE 3 7

8 Velocix CDN Market Strong momentum, driven by TV Everywhere rollouts in NA and UK Field-proven by leading telco and cable TV innovators Endorsed by major content providers (HBO, Starz, Epix, BBC, Sky,...) Capable of supporting any network type: HFC, DSL/GPON and wireless Full ecosystem of strategic partners Global customer base, connecting over 30M subscribers to date 8

9 Velocix CDN Global References Time Warner Cable USA Telus - Canada Sasktel - Canada Major MSO Canada Deployment Trial Liberty Global InternaFonal TV Everywhere with per stream encrypfon and mulf- tenant capabilifes TV Everywhere with per stream encrypfon and mulf- tenant capabilifes MulF- screen content delivery Major MSO #3 USA 1st mulf- country CDN and cloud DVR for mulf- screen Fme shiped TV Windstream USA Major Telco Spain Mediaroom mulfscreen solufon with Quickplay overlay Major MSO #1 USA 1st mulfscreen solufon using encrypted stream to HLS devices Major MSO USA #1 Major MSO Mexico Major MSO #2 USA Major MSO USA #2 Major MSO #1 Mexico Major MSO #2 Mexico Major MSO ArgenFna Talk Talk UK Member of Canvas with YouView and CDN wholesale services Major Telco ArgenFna Live and on- demand content delivery Major Telco Tunisia Oi Brazil MediaRoom mulfscreen evolufon Major Telco South America Major Telco Middle East DistribuFon of NPVR content to classic STB using ATIS C2 SK Broadband South Korea WIND Italy wholesale CDN for media distribufon Major Telco Portugal Cloud DVR for mulfscreen Fme shiped TV 9

10 Splunk Professional Services Faster Time to Delivery Best PracFce Guidance Use Case Development Architecture Guidance Advanced CustomizaFon TroubleshooFng And more.. 10

11 Copyright 2015 Splunk Inc. The Big Data Problem

12 90 minute movie 2 different devices With just 3 bit rates for each device AdapFve Streaming The Numbers 2MB of Log Data 13,500 Content files Small fragment sizes = massive amount of individual files - requiring careful management on the origin. Smooth Manifest audio 750kbps 1500kbps 3500kbps Edge Delivery VxOA In this example, we re generafng two megabytes of log data for a single viewing of the movie. HLS Manifest 750kbps 1500kbps 3500kbps 12

13 Copyright 2015 Splunk Inc. The Big Data SoluFon

14 Advanced ReporFng Features Alerts on threshold crossing events Scalable plaworm for adapfve streaming workloads Export opfons + API access to reporfng metrics Scheduled reports All- new user interface (Console2) Built- in and custom reports CDN ReporFng and Enhanced AnalyFcs packs Log data opfmised for reporfng in several dimensions Reports run in seconds for both near- real- Fme and historical MulF- year retenfon of historical reporfng data API based architecture 14

15 OpFmising the end to end ALU solufon Commonly designed applications and logs means all components are speaking the same language Speed Efficiency Insight Value Common Log Format for all applications and Systems. Simplifies data normalisation Allows monitoring and reporting across all systems and applications Alcatel-Lucent Reporting Engine Logs Logs Logs Indexing all raw data allows any custom report to be created from source data. Alcatel-Lucent Delivery Applications Delivery Application Logs Reporting Application UI created report Custom customer report 15

16 Very High Volume Logs Seeing the wood from the trees Smooth Streaming = 30 logs per minute Concurrent Streams Multiple Servers Data Repetition Only one field change between log entries. Near Real Time Reporting Gives Customers: Insight Network Monitoring ExtracFng value from logs #Fields: s- dns date time x- duration c- ip c- port c- vx- zone c- vx- gloc cs- method cs- uri cs- version cs(user- Agent) cs(referer) cs(cookie) cs(range) sc- status s- cachestatus sc- bytes sc- stream- bytes sc- dscp s- ip s- vx- rate s- vx- rate- status x- vx- serial rs- stream- bytes rs- bytes cs- vx- token sc- vx- download- rate x- protohash #Software: Velocix PCD #Start- Date: :30:00 cp1.zzz49d1.cdn :30: external g GET testsite.zzz49s1.pub/static_file.txt 1.1 "NagiosChecker" CACHE_MEM_HIT WP: cp1.zzz49d1.cdn :30: external g GET testsite.zzz49s1.pub/static_file.txt 1.1 "NagiosChecker" CACHE_MEM_HIT WP: cp1.zzz49d1.cdn :30: external g GET testsite.zzz49s1.pub/static_file.txt 1.1 "NagiosChecker" CACHE_MEM_HIT WP: cp1.zzz49d1.cdn :30: a00:81c0:4000:3141::320: external g.gb GET "NagiosChecker" CACHE_MEM_HIT a00:81c0:4000:3151::500: WP:0b cp1.zzz49d1.cdn :30: a00:81c0:4000:3141::320: external g.gb GET "NagiosChecker" CACHE_MEM_HIT a00:81c0:4000:3151::500: WP:0b cp1.zzz49d1.cdn :30: a00:81c0:4000:3141::320: external g.gb GET "NagiosChecker" CACHE_MEM_HIT a00:81c0:4000:3151::500: WP:0b cp1.zzz49d1.cdn :31: external g GET download.zzz49.pub/bt/f5eca038739d55e031c2a4ebdd934a3494b6219c/data 1.0 "Wget/ Red Hat modified" BT:f5eca038739d55e031c2a4ebdd934a3494b6219c cp1.zzz49d1.cdn :31: a00:81c0:4000:3141::320: external g.gb GET "Wget/ Red Hat modified" a00:81c0:4000:3151::500: BT:f5eca038739d55e031c2a4ebdd934a3494b6219c cp1.zzz49d1.cdn :34: external g GET sslsite.zzz49s1.pub/static_file.txt 1.1 "NagiosChecker" CACHE_MEM_HIT WP: cp1.zzz49d1.cdn :34: external g GET 16

17 Advanced ReporFng Principles HTTP adapfve streaming leads to a big data problem dealt with by using a big data approach powered by Splunk. Efficient Log Data Storage Nothing stripped from incoming log data Distributed across a horizontally scalable plaworm Replicated across sites for high availability Optimised For Reporting Key metrics about network traffic, errors, caching efficiency and content popularity are extracted and calculated as logs arrive Daily summaries generated for long term trend analysis Custom ReporFng Just because log data might not be opfmised in a certain way shouldn t mean that you can t report on it. Bespoke reporfng allows jobs to be scheduled, run in the background and send a nofficafon when they are done. Scheduled Run Done 17

18 Velocix CDN ReporFng API Copyright 2015 Splunk Inc.

19 Ready Made & Custom ReporFng SoluFon CDN Owner Velocix Reporting Engine User Interface Customer ReporFng System API based reporfng allows customers to integrate with exisfng systems API 19

20 ReporFng Architecture CDN Logs Files Velocix Console CDN Admins Indexed Raw Data Daily Summary Index Accelerated Data Store REST API Velocix ReporFng API Custom Apps 20

21 ReporFng is API Driven { "_links": { " { "name": "jobs", "profile": " "title": "Report Job Collection" }, "self": { "href": " "profile": " "title": "Results for the Traffic analysis report" } }, "columns": [ [ 28/07/2014", 04/08/2014", 11/08/2014", 18/08/2014", 25/08/2014" ], [ " ", ", ", ", " ] ], "fields": [ "Date", "Bytes delivered (GB)" ] } 21

22 Velocix ReporFng User Interface Copyright 2015 Splunk Inc.

23 UI Technologies 23

24 VisualisaFons Line graph Area graph Column chart Pie chart Bar chart Geo map CSV export 24

25 VisualisaFons Line graph Area graph Column chart Pie chart Bar chart Geo map CSV export 25

26 VisualisaFons Line graph Area graph Column chart Pie chart Bar chart Geo map CSV export 26

27 VisualisaFons Line graph Area graph Column chart Pie chart Bar chart Geo map CSV export 27

28 VisualisaFons Line graph Area graph Column chart Pie chart Bar chart Geo map CSV export 28

29 VisualisaFons Line graph Area graph Column chart Pie chart Bar chart Geo map CSV export 29

30 VisualisaFons Line graph Area graph Column chart Pie chart Bar chart Geo map CSV export 30

31 Click To Zoom Click to Zoom allows the ability for the customer to instantly examine a portion of the reporting data in more detail. Simply click and drag over the range of data for which you require a more detailed view. The selected area will then expand to fill the visualisation area. Reset Zoom at the top right of the graph will allow the zoom to be reverted to its original visualisation when clicked. 31

32 Splunk Powered ReporFng PlaWorm Copyright 2015 Splunk Inc.

33 ReporFng System Components Delivery Tier Reporting Tier Delivery Appliance Indexers Search Head Cluster Master ExisFng Delivery Tier Forwards log data to indexers Distributes log data between all indexers Ingests Data Replicates to Other Indexers Accelerates Data For searching Searches on reporfng data as directed by Search Head Receives search requests from Console Orchestrates searching for report data across all Indexers Collates search results and returns to Search Head Coordinates ReplicaFon Asisgns Primary Searchable data copies Runs Scheduled Searches 33

34 High Level Architecture Load balanced data is spread across the indexers. Indexer clustering provides high availability and resilience. AcFve passive cluster master for failover Reporting Tier Delivery Tier Cluster Master Indexer Search Head Utility Server (Console) Site1 34

35 Service Node Hardware Reference PlaWorms Base Deployment 2X Cluster Master Search Head Indexer RHEL-7 (Red Hat Enterprise Linux 7) Storage Controller x0 RHEL-7 (Red Hat Enterprise Linux 7) Storage Controller x4 RHEL-7 (Red Hat Enterprise Linux 7) Storage Controller x12 35

36 Service Node Hardware Reference PlaWorms Scaled Deployment Indexer Indexer RHEL-7 (Red Hat Enterprise Linux 7) Storage Controller x12 RHEL-7 (Red Hat Enterprise Linux 7) Storage Controller x12 36

37 ReporFng Data Indexing & RetenFon Raw log data stored in main index Allows for flexible Fme resolufon in reports Storage space consumed is directly related to quanfty of delivery traffic and streaming technology used Delivery appliance Log data Daily summaries are compiled on a daily basis Generated aper 2 days to allow for late- arriving data from Delivery Appliances Searchable aper 3 days to allow for generafon Configurable retenfon policy main vs summary index Main Index Accelerated Data Summary Index 37

38 ReporFng Data Phase 1 Main Index Scheduled search results Summary Index Header Based Indexed Field Extractions Regenerated Hourly Using Tstats Scheduled searches Daily Aggregated Data Index Time Field Extractions 50% More Storage Accelerated Data Generated at Index Time No Acceleration Lag Refreshed Hourly Earliest - Last 7 days Latest 3 Hours Ago Based On tstats Search Statistics ready for Post Processing Filtering Wildcard Lookups Refreshed Daily Long Term Trending Separate Retention Policy Configurable capacity vs. Main Index 38

39 High Performance ReporFng Phase 1 Summary Search Results Scheduled Search Results - 1year - 7days - 3days now 39

40 ReporFng Data Indexing Phase 2 Main Index Search Time Field Extractions Accelerated Data Models This is generated continuously, as new logs arrive, maintained alongside the main index Summary Index Daily aggregated data No Indexed Extractions 50% Less Storage Simplified Searches Lookups and translations accelerated Superfluous information is removed Refreshed Daily Long Term Trending Separate Retention Policy Configurable capacity vs. Main Index Summary Index results are also data modelled to provide consistent transllations for the reporting API. 40

41 High Performance ReporFng Phase 2 Accelerated Data Model derived from Summary Search Results - 1year - 7days - 3days now 41

42 Summary Accelerated project delivery ReporFng capabilifes exceeded expectafons Search complexity abstracted away from users by API wrapper Stable, scalable and self monitoring Further InformaFon: Eric Henderson Senior Product Manager lucent.com 42

43 THANK YOU