Real World Big Data Architecture - Splunk, Hadoop, RDBMS

Transcription

1 Copyright 2015 Splunk Inc. Real World Big Data Architecture - Splunk, Hadoop, RDBMS Raanan Dagan, Big Data Specialist, Splunk

2 Disclaimer During the course of this presentagon, we may make forward looking statements regarding future events or the expected performance of the company. We caugon you that such statements reflect our current expectagons and esgmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentagon are being made as of the Gme and date of its live presentagon. If reviewed arer its live presentagon, this presentagon may not contain current or accurate informagon. We do not assume any obligagon to update any forward looking statements we may make. In addigon, any informagon about our roadmap outlines our general product direcgon and is subject to change at any Gme without nogce. It is for informagonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligagon either to develop the features or funcgonality described or to include any such feature or funcgonality in a future release. 2

3 About Your Presenter Raanan Dagan - Sr. SE, Big Data specialist In Splunk for 3.5 Years Before Splunk worked for Cloudera and Oracle 3

4 Agenda Splunk Big Data Architecture AlternaGve Open Source Approach Real- World Customer Architecture Customer Roadmap Architecture End- to- end DemonstraGon (Splunk, Hadoop, RDBMS) 4

5 Big Data Technologies Rela(onal Database Structured NoSQL Semi- Structured Hadoop Semi- Structured Splunk Schema at Write Schema at Read Schema at Read Schema at Read SQL ETL Key- Value, Column, Document & Other Stores MapReduce HDFS Storage Search Real- Time Indexing RDBMS Oracle, MySQL, IBM DB2, Teradata Cassandra, Accumulo, MongoDB 5 MapReduce Distributed File System Time- Series, Unstructured, Heterogenous

6 Big Data Are you ge\ng the most out of your Big Data investments? 6

7 Big Data in Goverment How can you accelerate your Gme to value? 7

8 Splunk Big Data Copyright 2015 Splunk Inc.

9 Splunk Big Data Technologies DB Connect Hunk Splunk Schema at Write Schema at Read Schema at Read Schema at Read SQL ETL Key- Value, Column, Document & Other Stores MapReduce HDFS Storage Search Real- Time Indexing RDBMS Oracle, MySQL, IBM DB2, Teradata Cassandra, Accumulo, MongoDB 9 MapReduce Distributed File System Time- Series, Unstructured, Heterogenous

10 Splunk Scalability Enterprise- class Availability and Scale " AutomaGc load balancing linearly scales indexing " Distributed search and MapReduce linearly scales search and reporgng Offload search load to Splunk Search Heads Auto load- balanced forwarding to Splunk Indexers Send data from thousands of servers using any combinagon of Splunk forwarders 10

11 Splunk Real- Time AnalyGcs Data Monitor Input TCP/UDP Input Scripted Input Parsing Queue Parsing Pipeline Source, event typing Character set normalizagon Line breaking Timestamp idengficagon Regex transforms Index Queue Real- Gme Buffer Indexing Pipeline Raw data Index Files Real- Gme Search Process Splunk Index 11

12 Hunk - AnalyGcs Plaeorm for Hadoop Full- featured, Integrated Product Explore Analyze Visualize Dashboard s Share Insights for Everyone Works with What You Have Today Hadoop Clusters Hadoop Client Libraries NoSQL, EMR, S3 Buckets 12

13 Hunk Unique Features Virtual Index Enables seamless use of the Splunk technology stack on data wherever it rests NaGvely handles MapReduce Schema- on- the- fly Structure applied at search Gme No briile schema AutomaGcally find paierns and trends Flexibility and Fast Time to Value InteracGve search Preview results while MapReduce jobs run Drag- and- drop analygcs Security: Access Control, Pass Through Authen(ca(on, Kerberos 13

14 Hunk Provides Self- Service AnalyGcs for Hadoop Explore Analyze Visualize Dashboards Share Hadoop Storage 14

15 What About Structured Data? Customer profile Product azributes Employee details Pricing and Rate plans Asset info 15

16 Use cases for structured data in Splunk Index machine data from databases, such as logs or sales records Enrich machine data with high- level data, such as customer records Update structured databases with Splunk info, such as risk scores InteracGvely browse structured and unstructured data from Splunk reports

17 Machine Data Delivers Real- Gme Insights Media server logs (machine data) Phone Number IP Address Track ID Mar 01 19:18:50:000 aaa2 radiusd[12548]:[id local1.info] INFO RADOP(13) acct start for from recorded OK.! :18:50: GET /sync/addtolibrary/ "Mozilla/5.0 (iphone; CPU iphone OS 5_0_1 like Mac OS X) AppleWebKit/ (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/ " ! Mar 01 19:18:50:163 aaa2 radiusd[12548]:[id local1.info] INFO RADOP(13) acct stop for @splunktel.com from recorded OK.! 17

18 Structured Data Contains Business Context Media server logs (machine data) Phone number IP address Track ID Mar 01 19:18:50:000 aaa2 radiusd[12548]:[id local1.info] INFO RADOP(13) acct start for from recorded OK.! :18:50: GET /sync/addtolibrary/ "Mozilla/5.0 (iphone; CPU iphone OS 5_0_1 like Mac OS X) AppleWebKit/ (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/ " ! Mar 01 19:18:50:163 aaa2 radiusd[12548]:[id local1.info] INFO RADOP(13) acct stop for @splunktel.com from recorded OK.! Track ID ArGst Title Format ID Run Gme Maroon 5 Moves like Jagger MP3 4:30 Customer, product databases Phone # Subscriber ID Subscriber ID First Name Last Name Age State Customer Score Jim Morrison 25 CA 93 18

19 Splunk DB Connect Reliable, scalable, real- (me integra(on between Splunk and tradi(onal rela(onal databases " Enrich search results with addigonal business context " Easily import data into Splunk for deeper analysis " Integrate mulgple DBs concurrently " Simple set- up, non- evasive and secure Database lookup Oracle database Java Bridge Server ConnecGon pooling JDBC MicrosoR SQL server Database query Other databases 19

20 Copyright 2015 Splunk Inc. Customer Open Source AlternaGve

21 Hadoop Ecosystem OpGons

22 Hadoop Advantage / Disadvantage Advantage Cheap Storage Batch Distributed Processing Disadvantage Requires Coding for most AnalyGcs No VisualizaGon Tools No OOTB Apps / SoluGons 22

23 Copyright 2015 Splunk Inc. Real- World Customer Architecture

24 Summary Architecture 3 instances Splunk / Hunk / DB Connect Search Heads Real Time Data - 25 Indexers Historical data (VIX) 60 Hortonworks nodes Enrichment data (lookup) - MySQL DB 2000 Forwarders

25 Splunk Deployment Architecture indexer Web server 2,000 forwarders ~2TB per day indexer 25 indexers 3 search head ~250 Users ~30 Concurrent Users Web server forwarder 25

26 Hadoop Architecture WebLogic app server ~30 Flume Agents ~60 Data Nodes ~1.2 PB of storage ~2 years data retengon data node WebLogic app server data node data node data node 26

27 Splunk + Hunk = All the Data indexer Web server indexer app server Real Time AnalyGcs Alerts Apps Batch Compliment Splunk AnalyGcs Historical searches data node data node data node 27

28 DB Connect Architecture Install DB Connect on a Search Head Use DB Connect for Lookup Several Lookups coming from two different MySQL Databases Lookup Enrich log data with business insight Search Head DB- 1 DB- 2 MySQL JDBC Driver 28

29 DB - Architecture Performance Impact Command Connec(on Architecture Indexing Inputs - dbmon- tail ** Recommended Inputs dbmon- dump Outputs Not Indexing Medium number of connecgons (Small amount of data - only delta) Small amount of connecgons (Lots of data per connecgon) Lots of DB ConnecGons (Small amount of data) DB to Index (connecgon pooling) DB to Index (connecgon pooling) Search Head to DB (connecgon pooling) Search DBXQuery Lots of DB ConnecGons DB to Search Head Lookups ** Selected this op(on Lots of DB ConnecGons DB to Search Head

30 Summary Architecture 3 instances Splunk / Hunk / DB Connect Search Heads Real Time Data - 25 Indexers Historical data (VIX) 60 Hortonworks nodes Enrichment data (lookup) - MySQL DB 2000 Forwarders

31 Copyright 2015 Splunk Inc. Customer Roadmap Architecture

32 Archive Splunk Data to HDFS or AWS S3 WARM Hadoop Clusters COLD FROZEN Drive Down TCO by Archiving Historical Data to Commodity Hardware

33 Unified Search Intelligently Search Across Real- Time and Historical Data using the Same Splunk Interface Real- Time Data Historical Data in Hadoop

34 Roadmap Architecture Unified Search 3 instances Splunk / Hunk / DB Connect Search Heads Real Time Data - 25 Indexers Archive Historical data (VIX) 60 Hortonworks nodes Enrichment data (lookup) - MySQL DB 2000 Forwarders

35 Copyright 2015 Splunk Inc. Customer Chosen Architecture Demo

36 Support Our Military Kids Take our Survey! Splunk will Donate $10 Dollars to our Military Kids Plus a bonus if we hit 350 number of completed surveys onsite. Text Splunk to

37 Thank You Copyright 2015 Splunk Inc.