Mobile Big Data AnalyEcs

Transcription

1 Copyright 2014 Splunk Inc. Mobile Big Data AnalyEcs Marc Courtemanche, Sr. Director Alain Brunet, Sr. Lead Developer Vantrix CorporaEon

2 Disclaimer During the course of this presentaeon, we may make forward- looking statements regarding future events or the expected performance of the company. We caueon you that such statements reflect our current expectaeons and esemates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentaeon are being made as of the Eme and date of its live presentaeon. If reviewed arer its live presentaeon, this presentaeon may not contain current or accurate informaeon. We do not assume any obligaeon to update any forward- looking statements we may make. In addieon, any informaeon about our roadmap outlines our general product direceon and is subject to change at any Eme without noece. It is for informaeonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaeon either to develop the features or funceonality described or to include any such feature or funceonality in a future release. 2

3 Agenda! Vantrix IntroducEon! The AnalyEcs Opportunity! Vantrix Big Data AnalyEcs PlaYorm! Mobile Data AnalyEcs 3

4 About the Speakers Marc Courtemanche 7+ years at Vantrix Senior Architect, Engineering University Research à Industry 15 years ago High- performance compueng, networking, media processing and delivery Alain Brunet 8+ years at Vantrix Senior Lead Developer, Engineering 20+ years experience as a developer Server- side environments, Java ecosystem 4

5 ! Established 2004 About Vantrix! Media processing experts transcoding, streaming, caching, quality! 40+ patent families; leading media university research relaeonship! 65+ operator deployments globally A History of InnovaEon Voice & Video codecs Mobile codecs 2B+ phones 2004 MMS media transcoding 2011 Mobile video opemizaeon 2014 MulEScreen Video Deployed Highest density HD/4K Open Source HEVC Client OpenStack enabled 5

6 Our Product Lines Bandwidth Op2mizer Mul2screen Video PlaAorm OpEmize delivery and Quality of Experience of Over The Top services across Mobile Networks High Density Transcoding and OpEmized Delivery of Video across all Devices and Networks Mobile Message Op2mizer Assure Quality of Experience and interoperability of Mobile MulEmedia Messaging 6

7 Comparable CompeEtor SoluEon Live transcoder The Vantrix Difference Unmatched Flexibility Software-Defined Architecture VoD transcoder Live transcoder VoD transcoder VoD Storage Packager / Streamer Admin server DRM VoD storage Policy Packager Analy2cs Admin 7

8 The AnalyEcs Opportunity

9 The AnalyEcs Opportunity KPI Dashboard Analy2cs Vantrix Gateway Average Operator 25 Million records / day 9

10 KPIs Answer WHAT? Bitrates Loca2on Video resolu2on Video length Date and Time Video TSopology ession System Quality Video dimensions Video frame rates Delivery protocol Media Types Data Volume Device Session Length Media Codec Video stall 2me Web & Video sites Media Size Session Volumes Media Container

11 AnalyEcs Explores WHY? Bitrates Loca2on Video resolu2on Video length Date and Time Video TSopology ession System Quality Video dimensions Video frame rates Delivery protocol Media Types Data Volume Device Session Length Media Codec Video stall 2me Web & Video sites Media Size Session Volumes Media Container Helping our customers plan their business strategies

12 Typical Vantrix Customer! 100+ opemizaeon nodes! 10 Million+ subscribers! Intercept all HTTP transaceons and opemize HTML, images and video! Log all opemized sessions! 25 Million video session record daily! 0GB daily = 2TB monthly! 90+ fields per record! Variable log format, unstructured Extract Business Insights from available data 12

13 Do We have Big Data?! What is Big Data anyways? Wikipedia definieon: so large it cannot be processed by tradieonal data management tools Data is in TB and even PB Samples: ê Facebook adding 1 PB daily / over 400 PB total (most probably) ê LHC (Large Hadron Collider) with 500 HB (hexabytes) daily ê Vantrix typical operator: 2TB of session logs monthly! Challenges Slow to process, hard to scale Limited hard drive capacity Machine reliability Backups Parallel processing The Challenges Apply to Vantrix Data!

14 Product Management Requirements to Engineering Scale to Manage lots and lots of data Manage 1yr of data 80 GB or 25M Records/Day 30TB or 10 Bil Recs/year TL <1hr for a day of data Queries average < 30 secs High Performance Flexible and Simple Repor2ng UI Easily explore data in new ways via a simple UI Future proof design for new use cases Use Cases Required Highly Flexible Data Structure Low Hardware Footprint Support solu2on margin targets Fit within ½ Telco Rack Low Cost of Goods Oh.. and be ready to go commercial in 90 days 14

15 Vantrix AnalyEcs PlaYorm Architecture & Components

16 Vantrix Media PlaYorm (VMP) SoRware Defined Virtual Appliance ApplicaEon Layer Transcode Fragment Package Encrypt Virtualized Media Processing ApplicaEon Layer VirtualizaEon, OrchestraEon Layer Hardware Resources Private Cloud Management Layer (OpenStack) Hardware IntegraEon COTS hardware, blades or dedicated high- density appliance Graphics/Compute AcceleraEon Storage enhancements 16

17 Vantrix Big Data AnalyEcs PlaYorm Analy2cs Applica2on Data ExploraEon and ReporEng Big Data Processing Distributed Filesystem and Processing Virtualiza2on, Orchestra2on SymKloud Cluster (16RU) 144 CPU (Hadoop) Nodes, 69 TB SSD Storage Private Cloud Management Layer Deployment and OrchestraEon High Density CompuEng Cluster 17

18 Hadoop! Open source framework enabling Big Data processing through distributed storage and parallel processing on clusters having thousands of nodes HDFS (storage) MapReduce (processing) New paradigm! Origin Created by Doug Cuvng in the context of the Nutch project (open source web crawler and indexer) Named arer his son s toy elephant! Distributed file system (HDFS) + Distributed processing (MapReduce)! Becoming the de- facto playorm for Big Data processing! Rich framework base! Surprisingly simple design

19 Hadoop s Simple Architecture Client Master Node (namenode) Secondary namenode Data Node Data Node Data Node 19

20 Hunk - Integrated AnalyEcs PlaYorm Full- featured, Integrated Product Insights for Everyone Explore Analyze Visualize Dashboard s Share Works with What You Have Today Hadoop Client Libraries Hadoop Clusters Streaming Resource Libraries NoSQL and Other Data Stores 20

21 ! Run NaEvely in Hadoop: Use Hadoop MapReduce! Mixed Mode: Allows for data Preview Hunk Features! Auto deploy SplunkD to DataNodes: On the fly Indexing! Access Control: Allows for many users / many Hadoop directories / support Kerberos! Schema On the Fly 21

22 Hadoop and Hunk Distributed Processing Hadoop Cluster Hunk Hunk Hunk Hunk HDFS + Hunk Hunk Hunk Hunk MapReduce Log Files Transparent UI migration from Splunk to Hunk 22

23 Data ExtracEon, Transform and Load (ETL) QoE Tool Log Files Log Streams Ceilometer BI JSON- REST Batch Stream Schema JSON HADOOP MapReduce HADOOP/HDFS MongoDB

24 Accelerate ETL with MapReduce Logs Hadoop Logs Logs Extract Extract Extract Transform (map- reduce) Load Load Load HDFS Storage Logs! Breaks down session data into 90+ fields! Highly efficient distributed processing / full cluster contribueon! Seamlessly process compressed files (bzip, gzip,..)! Transformed files are also stored compressed

25 Big Data ReporEng and AnalyEcs ReporEng AnalyEcs Repor2ng Hive Splunk Hunk HADOOP MapReduce JSON Schema Hadoop Connector HDFS MongoDB 25

26 Vantrix AnalyEcs PlaYorm Features and Performance

27 Hunk Hadoop- Integrated Advanced AnalyEcs! Out of the box worked on our variable record structure! Immediately idenefied the key record fields! AutomaEcally created new fields (e.g. URL tags)! AutomaEcally indexed and counted field value occurrences! Proved invaluable for idenefying and exploring use cases 27

28 Example of the Exploratory Power No query scripeng, no pre- determined searches: point and click exploraeon 28

29 Point and Click ExploraEon of Complex Query Results Rapidly explore and uncover the story behind the story 29

30 Performance Small Scale System! Transform and Load Data compressed locally and loaded on Hadoop cluster ETL process - > 2GB/second ê just in Eme data loading ê efficient use of HDFS storage ê quick recovery from corrupeon! Hunk Query Full scan for a day (80GB) in seconds Full scan for a monthly (2TB) in couple of minutes Analy2cs Applica2on Big Data Processing Virtualiza2on, Orchestra2on SymKloud Cluster (4RU) 28 CPU Nodes, 14 TB SSD Storage 30

31 Mobile Data AnalyEcs

32 32 The End Result

33 Live Event Impact The Key Olympic Hockey games drove significant peaks 33

34 Video Site CharacterisEcs YouTube dominates the consumpeon of low bitrate encoded video and higher encoding rates. A number of sites clearly focus on a narrow set of encoding rates such as Instagram dominaeng the 1.2Mbps bracket 34

35 35 Mobile Network Performance

36 Video Session Quality Generally, subscribers receive a good video session quality; however, some video sites are problemaec. 36

37 Session Quality DistribuEon by Encoding Rate Off Peak Peak On average 10% of sessions below 1Mbps (majority) experience stalling. Comparing off- peak vs peak 37

38 Video Stall Times HD HQ Low Quality Stall Eme for HD Videos twice that of HQ 38

39 User Engagement / Abandonment Watch raeo declines for videos encoded beyond 1.6Mbps 39

40 Session Quality of RelaEonship Engagement & Video Subscribers abandon video less than halfway through when video session quality is bad. 40

41 THANK YOU