Stream Deployments in the Real World: Enhance Opera?onal Intelligence Across Applica?on Delivery, IT Ops, Security, and More

Transcription

1 Copyright 2015 Splunk Inc. Stream Deployments in the Real World: Enhance Opera?onal Intelligence Across Applica?on Delivery, IT Ops, Security, and More Stela Udovicic Sr. Product Marke?ng Manager Clayton Ching Sr. Product Manager Mike Dickey Sr. Engineering Director

2 Disclaimer During the course of this presenta?on, we may make forward looking statements regarding future events or the expected performance of the company. We cau?on you that such statements reflect our current expecta?ons and es?mates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presenta?on are being made as of the?me and date of its live presenta?on. If reviewed ater its live presenta?on, this presenta?on may not contain current or accurate informa?on. We do not assume any obliga?on to update any forward looking statements we may make. In addi?on, any informa?on about our roadmap outlines our general product direc?on and is subject to change at any?me without no?ce. It is for informa?onal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obliga?on either to develop the features or func?onality described or to include any such feature or func?onality in a future release. 2

3 Personal Introduc?on Stela Udovicic, Sr. Product Marke?ng Manager - Responsible for IT Opera?ons/Applica?ons Delivery, Stream, Strategic Partners - Over 15 years of experience with variety of data, networking and storage technologies Clayton Ching, Sr. Product Manager - Responsible for Splunk App for Stream strategic direc?on and roadmap - 20 years in enterprise sotware management Mike Dickey, Sr. Engineering Director - Responsible for Apps Architecture and Performance - Founder of Cloudmeter, the startup company where Stream originated from 3

4 Agenda Introduc?on: Market Challenges and Splunk Solu?on Customer Success Real- World Deployment and Architecture How to Manage Splunk App for Stream in Your Environment Performance Metrics Summary 4

5 Introduc?on: Market Challenges and Splunk Solu?on

6 Market Challenges Lack of Applica,on Visibility Impacts Customer Experience Limited Cloud Insights Long MTTR Hurts the Business 6

7 Splunk App for Stream: Real-,me Applica,ons Intelligence Real-,me Insights into Applica,on Performance and Customer Experience Visibility into Cloud Services Quickly Deploys and Filters Streaming Network Data to Maximize Business Impact 7

8 Customer Success

9 Cross- Tier Visibility Helps Break the Silos Stream and Splunk help us understand issues at the high level and if exec team wants to see the details we can drill down easily. Kris Laxdal, IT Manager & Security Analyst IT Opera,ons/Applica,ons Delivery High execu?ve level view with contextual drill- down ability Easy access and visibility into produc?on MySQL environment helps app developers troubleshoot issues and roll out releases quicker Improved collabora?on between teams: IT opera?ons, QA (pre- produc?on tes?ng),security and development Improved customer response?mes due to real-?me visibility into app issues Security Key Customer Benefits Correla?on against indicators of compromise helps inves?gate and mi?gate APTs, poten?al data exfiltra?on & other risks

10 Applica?ons Visibility for Beger Customer Experience The Splunk App for Stream helps us get real-?me insight into the opera?onal performance of applica?ons, as well as the health of our claim- processing workflows. IT PlaSorms Opera,ons Manager Medical Claims Processing Company Key Customer Benefits Visibility into web applica?ons for interac?ons across frontend, middle-?er and database servers help resolve issues quicker Business process insight to help understand customer experience and claims volume Match applica?ons and infrastructure to business demands Improved applica?ons performance beger customer experience

11 Applica?ons Visibility Drives Beger Digital Asset Management With Splunk and Stream, we have this rich data plasorm that is bridging all the different data silos. Our MTTR went from days to minutes while the granularity and insight improved. We went from having very ligle visibility into opera?onal and security issues to full insight. Systems Engineer, Major Media Company Key Customer Benefits IT Opera,ons: improved opera?onal insight into digital asset management and streamlined lengthy processes Applica,on Delivery (DevOps): faster app releases due to visibility into app performance Real-?me insight into database queries and latencies Cross- correla?on with system- level performance and user access Security: Visibility into user behavior throughout en?re asset management system helps protect digital assets

12 Real- World Deployment and Architecture

13 Quick Time to Value Easy- to- Deploy SoWware Solu,on Runs on any commodity hardware Passive Data Collec,on Without applica5on overhead Low- cost Deployment With flexible resource u5liza5on 13

14 Dedicated Server Deployment End Users Internet Firewall TAP or SPAN Servers Splunk Indexers Search head Windows or Linux Forwarder Splunk_TA_Stream 14

15 End Point Deployment End Users Internet Firewall Physical or Virtual Servers Universal Forwarder Splunk_TA_stream Splunk Indexers On premises Cloud Search head 15

16 Packets Network Interface (eth1) Stream Forwarder Architecture Flows Request/ Response Request/ Response Threads Decryp,on Decryp,on Protocol Decoder (Deep Packet Inspec,on) Protocol Decoder (Deep Packet Inspec,on) Events Events Packets Network Interface (ethn) Request/ Response Decryp,on Protocol Decoder (Deep Packet Inspec,on) Events 16

17 How to Manage Splunk App for Stream In Your Environment?

18 Managing Your Streams 1. Manage your data collec?on 2. Analyze the volumes 3. Control the data volume 4. What if? 18

19 Distributed Forwarder Management Logical Group(s) Maximize Control Dynamic Data Collec,on Adapt to Your Business Needs - hgp - Diameter - sip - udp - tcp - dns - pop3 - smtp - Oracle - MySql - hgp - udp - tcp - dns - hgp - tcp - pop3 - smtp - hgp - udp - tcp - Oracle - MySql Security Database - hgp - udp - tcp ecommerce - hgp - udp - tcp = Splunk App for Stream

20 How Much Data? 20

21 Select Fields Control Data Collec?on Specify Filtering I only want to collect certain Applica,on Errors HTTP with status=404 (File Not Found) 21

22 Control Data with Aggregates Summarize Many Events to One Logically Combine Data Results Oriented Repor,ng 22

23 Applica?ons Insights with Aggrega?on Dashboard Results Oriented Dashboards Be_er Insights Effec,ve Data Management 23

24 Packets Network Interface (eth1) Stream Forwarder Architecture Flows Request/ Response Request/ Response Threads Decryp,on Decryp,on Protocol Decoder (Deep Packet Inspec,on) Protocol Decoder (Deep Packet Inspec,on) Events Events Packets Network Interface (ethn) Request/ Response Decryp,on Protocol Decoder (Deep Packet Inspec,on) Events 24

25 Tailor Data Collec?on to Your Monitoring Needs What if I could calculate the amount of data before it gets to the Splunk Index? 25

26 What If 26

27 Demo Splunk App for Stream

28 What s Up with Splunk App for Stream? Target GA 4Q 2015 GA 11/2014 GA 03/2015 Stream 6.0/6.1 New Protocols Filtering Ephemeral Streams GA 06/2015 Stream 6.2 Windows Stats only Custom Content Extrac?on New Protocols Stream 6.3 Distributed Forwarder Management New Protocols What s coming 28

29 Performance Metrics

30 Incremental Improvements 40% LESS CPU LOWER 70% IS LESS BETTER MEM VS 6.2 Test Assump,ons HTTP traffic (100K response size) was used to generate the graphs. Every request has its own TCP connec?on *Note: Data in this presenta5on recorded on September 3, Please refer to docs for up to date performance results. 30

31 Scaling Beyond 1 Gbps 3 cores Experimental memory allocator 3-4 Gbps limit for current 6.3 release Test Assump,ons HTTP traffic (100K response size) was used to generate the graphs. Every request has its own TCP connec?on 290 MB *Note: Data in this presenta5on recorded on September 3, Please refer to docs for up to date performance results. 31

32 Performance Test Environment 10 Gbps NetOp,cs Packet Broker 2.5 Gbps 2.5 Gbps 2.5 Gbps 2.5 Gbps NIC1 NIC2 NIC3 NIC4 Ixia PS- 100 w/ BreakingPoint Streamfwd (16 cores) 32

33 10 Gbps Performance Results 15 cores 16 cores, 4 GB mem, four 10 GB NICs < $5k 4 GB Test Assump,ons HTTP traffic (100K response size) was used to generate the graphs. Every request has its own TCP connec?on Packet broker used to LB traffic across NICs *Note: Data in this presenta5on recorded on September 3, Please refer to docs for up to date performance results. 33

34 Performance Summary Splunk App for Stream uses libpcap, which tops out at about 3-4 Gbps per NIC Packet broker spreads 10 Gbps traffic across four NICs Handle more traffic using mul?ple servers 2 For 20 Gbps, 4 for 40 Gbps, and 10 for 100 Gbps Use a packet broker to load balance the traffic across your servers Future Work Removing libpcap limita?ons Addi?onal reduc?ons in CPU & memory Tes?ng Splunk App for Stream with more traffic pagerns 34

35 Summary

36 Stream: See Everything. Now! Get Real-,me Applica,ons Intelligence Gain Visibility into Cloud Services Reduce MTTR to Maximize Business Impact 36

37 Next Steps Download and try Splunk App for Stream for free! Agend Sierra- Cedar session Thursday, September 24, 2015 Breakout 18 Agend Royal Caribbean Cruise Line session Wednesday, September 23, 2015 Breakout 12 Check out CanDeal case- study: Streamlining IT and Security hgp:// success- stories/ splunk- at- candeal.pdf Chat with Stream experts in our IT opera?ons booth 37

38 Ques?ons? 38

39 THANK YOU