Big Data Advanced Analytics for Game Monetization. Kimberly Chulis

Transcription

1 Big Data Advanced Analytics for Game Monetization Kimberly Chulis CEO Core Analytics, LLC

2 Core Analytics / Game Loyalty Bay area and Chicago based digital advanced analytics firm Big Data / NoSQL Advanced Analytics Solutions Focuses on optimization and measurement of video, socialandmobilegamesand games, in game advertising and gamification analytics Micro segmentation, dashboard reporting, data integration, predictive and multivariate analytics

3 Describe size of Gaming Industry and Exponential Growth 100 Video Games ($Billions) ($Billions) Computer Games Social Games Source: Colin Sebastian for RW Baird While these numbers indicate that the current largest share of market belongs to the video game industry, an important preferential shift and player demographic change is occurring that is expected to result in a partial decline in popularity of core video games and a rapid decline in the console variety.

4 Describe size of Gaming Industry and Exponential Growth Social games Offered at a fraction of the price through social networks, and are played on various mobile platforms 54% female, Mobile Payments and PayPal for Purchases Across Platforms and Devices Traditional video and MMO games Predominantly male (18 34) Pay cash / credit cards / Solo or within limited interactive environment Mobile games Represent a fundamental change in the gaming landscape Expanding Who, How and Why gamers are playing

5 Describe size of Gaming Industry and Exponential Growth Stark differences emerging in game revenue patterns across devices ios games generating 85% of in game revenues over Android and other platforms Expected decline in console games (related hardware and apparatus sales) and web based games becoming the go to platforms This Trend will be parallel to continued growth in video games (played on computers) and to explosive growth in mobile games Conclusion Big opportunities for Analytics Providers

6 Big Data and Social and Online Gaming

7 NoSQL TECHNOLOGY

8 Technology

9 Technology Database technology shift Shift from relational to NoSQL New solutions introduced to handle data on a very large scale Methods of extending the capacity of legacy systems were introduced Sharding Denormalization Distributed caching

10 Technology Sharding is the practice of data partitioning across diverse servers, which requires knowledge of the server location of data and is limited by the fact that you can't perform joins across shards. You must maintain schemas for each server Denormalization is another method that involves grouping and indexing redundant data and often results in latency and issues with maintaining concurrency in relational database systems Distributed caching which caches recent data in memory, is useful when data is needed. The application (web, game, social network, search engine, andso on) first checks a distributed caching system, such as memcached, for the needed data instead of going back to the relational database

11 Advantage NoSQL to RDMS RDMS Won t scale after a certain point anymore. System Cost Application Response Time Database scales out add more web servers

12 Technology Warehousing Options Commercial NoSQLoptions options that integrate with Hadoop and other open source tools and greatly extend these capabilities with analytics, text mining, in application processing, map reduce functions, and graphing options IBM InfoSpere BigInsights Cloudera (Yearly Support Fee for the Enterprise Edition)

13 NoSQL Software Types Document Document & Key Value Document (JSON) Graph hdtb Database Key value Key value & Hierarchical & Document Key value Multi level

14 NoSQL Software Source: Wikipedia

15 NoSQL Software Source: Wikipedia

16 Export Social Media Gaming Data from MySQL into Hadoop JSON Parses and store results into table MySQL in the Cloud Others like Forums / Blogs, HDFS Sqoop Client

17 Twitter stored in MySQL

18 Facebook stored in MySQL

19 Import Process MySQL Table Metadata Launch Sqoop Client MAP MAP MAP Generated Record Container Class Uses HDFS HDFS HDFS MapReduce Job

20 Migrate from MySQL to Hadoop Using Sqoop to migration fro MySQL to Hadoop Results: Retrieve the Results from Hadoop

21 Hadoop HDFS FORMAT CSV: ID, Search term, User name, Created date,,., Mention, Language

22 Real time Analytics Storm and Hadoop Hadoop run all functions all at once, but high latency Storm run incremental functions all quickly distributed dand fault tolerant l real time computation system Guarantees data will be processed Horizontally scalable Storm Concepts (see also next slide) Tuple: Named list of values Streams: Unbounded sequence of tuples Spouts: Sourceofstreams of (i.e. Twitter) Bolts: Bringing the tuples together (functions, filters, joins, talking to databases) Topology: How is everything linked together (graph of computations)

23 Storm Concepts Stream Tuple Tuple Tuple Spouts Tuple Tuple Tuple (Multiple Streams) Tuple Tuple Tuple Tuple Tuple Tuple Bolt Tuple Tuple Tuple Tuple Tuple Tuple Spout Topology Bolt Bolt Bolt Spout Bolt Bolt

24 Storm and Hadoop Integration Example: very large amount of data that needs to be parsed from different sources at real time. Implement Hadoop Create a Map/Reduce Job, but. Long time to be completed as it takes the slowest Reducer to be completed This means unknown cycle completion times

25 Storm and Hadoop Integration Another complexity could be We have a number of processes updating our feeds, and would like to control which will update the database? We should ask ourselves: is there a system that allows us to deploy the Queue / Worker system without inheriting thecomplexity? Yes it s called Storm

26 Storm Cluster Zookeeper Supervisor Master Node (similar to Hadoop Job Tracker) Nimbus Zookeeper Supervisor Supervisor Run Worker Processes Zookeeper Supervisor Zookeeper Zookeeper is used for Cluster coordination Supervisor

27 Storm and Hadoop Integration Storm allows us to write real time topologies We don t have to worry about Scalability Fail over IPC Real time Analytics!

28 Technology Filtering Hadoop Freely licensed software framework developed by the Apache Software Foundation Scale from a single computer up to thousands computers Hadoop Distributed File System (HDFS) Designed for storing very large files Streaming data access patterns Running on clusters of commodity hardware

29 Technology Filtering Hadoop Source: Apache Hadoop

30 Technology Filtering Hadoop Functional layer Modeling and development Storage and data management Sub project MapReduce, Pig, Mahout HDFS, Hbase, Cassandra Dt Data warehousing, summarization, Hive, Scoop query Data collection, aggregation, analysis Chukwa, Flume Metadata, table and schema management Cluster management,,job scheduling, workflow Data serialization Hcatalog Zookeeper, Oozie, Ambari Avro

31 Technology Filtering Hadoop Vendor Amazon Web Services Datameer EMC Greenplum Hstreaming MapR Pentaho Zettaset Vendor Cloudera Datastax Hortonworks IBM Outerthought Platform Computing

32 Technology Filtering MapReduce MapReduce is a programming framework popularized by Google and used to simplify data processing across massive data sets MapReduce1 Classic Framework MapReduce2 YARN (Yet Another Resource Negotiator) Very clusters in the region of 4000 or higher

33 Technology Filtering MapReduce MapReduce Map 1 Start Map 2 Reduce End Map N

34 Technology Filtering MapReduce1 Source: Hadoop The Definitive Guide

35 Technology Filtering MapReduce2 Source: Apache Hadoop

36 Technology Open Source Analytics There are various open source applications available Mahout R

37 R Open source R has been integrated to run massively parallel statistical processes directly in Hadoop nodes Functionality Used for statistical computing and graphics Linear and nonlinear modeling Classical statistical tests Time series analysis Classification Clustering Well design publication quality plots Runs on Unix and similarplatforms platforms, Windows and MacOS

38 Mahout Open source analytics tool on top of systems, such as Hadoop s MapReduce paradigm Apache Mahout is a machine learning engine that provides classification, clustering, and collaborative filtering

39 Technology Analytics Gap Commercial Analytics Packages like SPSS and SAS Raw data Processed dt data

40 Video and Social Games PRACTICAL EXAMPLES

41 Game Analytics Vendors Kontagent, Flurry, Mixpanel, Totango, Claritics, Google analytics Fewer vendors focusing on computer and MMO games No single Analytics Provider appears to focus on dli delivering i cross game platform ltf analytics lti

42 Game Analytics Metrics of social games Daily active users (DAU) Monthly active users (MAU) DAU / MAY ratio Engagement Measurers time spent playing a game K factor Infection rate of viral game growth as the core and casual player base expands Average revenue per user (ARPU) Lifetime Value (LTF) Capture s a player s value to the game based on in game purchases and other monetization related behaviors, player influence on virality, and net game envangelism

43 Game Monetization

44 Video Game Subscription Imagine a large MMO Video Game World of Warcraft Star Wars the Old Republic

45 Video Game Subscription i Model Pay to play Free to play Freemium Players must pay a monthly subscription fee Usually involves an upfront software cost but no additional payments Allows players to access game content and play for free but offers options to pay for additional content and access

46 Video Game Play to play Game analytics is focused on understanding who the most valuable players areandand how they play Propensity modeling Identify those players withthe the highest propensity to do one of the following Continue a subscription Return to play a game after a subscription i pause Encourage new players to subscribe Become skilled and persuasive guild leaders

47 Video Game Data elements include traditional game time dashboard key performance indicators (KPIs KPI Time to complete levels Avatar selection Gender of avatar Game related tweet Language KPI Solo vs interactive behaviors Interaction style indicators Game strategy behavior variables Socialnet network activityit

48 Video Game Micro segmentation Involves segmenting a player base to understand distinct segment preferences and behaviors to guide targeted game design, localization that reflects preferences of regional segments, and appealing targeted extension packages and additional content design

49 Social Game Ad and Virtual Good Early adaptors of Monetization Big Data Technology Cloud computing solutions Data mining applications Player analytics Allows social game studios to understand in realtime why users are abandoning a game and identify other players at risk of leaving the game so they can develop player retention strategies before those players quit.

50 Social Game Ad and Virtual Good Monetization

51 Where the Game Industry is Going

52 BrandMeter & Gaming Data PRACTICAL IMPLEMENTATION

53 Process flow Raw Social Media Comment Files, Web log data, External Files API and Data Integration Processes BrandMeter Data Warehouse BrandMeter Segmentation, Predictive Modeling Algorithms Analytic Data Mart Text Mining and Derived Variable, Data Cleansing and Manipulation BrandMeter Processes BrandMeter Social Media Data Mart BrandMeter Dashboard Reporting

54 Cloud Architecture Dashboard Reporting Analytics Segmentation Datamart BrandMeter Demographics Personality Mood Loyalty Database Datamart MySQL Geographic Other NoSQL Social Media Web Mobile

55 Social Media Data source: Facebook/Twitter 1. Search request to Facebook and Twitter p / g p 2. The app receive the results in a JSON format

56 Social Media Data source: Facebook/Twitter 3. The app calls a JSON library that parses the data 4. For each JSON record the data will be split up in several field like (Twitter) From_User Created_At Mention (actually message content) 5. Stores the record into MySQL DB

57 Social Media Data Source: Social Media BrandMeter Datamart JSON Parses and store results into table NoSQL MySQL Integration

58 Screen shots of the Dashboard / NoSQL / MySQL Data / Analytics Jact Media LLC.

59 Gaming data provided by Jact Media LLC LIVE DEMO BRANDMETER DASHBOARD