Technology for an Analytics-Driven World

Transcription

1 Technology for an Analytics-Driven World Nagui Halim, IBM Fellow IBM Software Group and IBM Research 1

2 Business are evolving rapidly ushering in a new era of computing Information from Everywhere Radical Flexibility Extreme Scalability Volume Velocity Variety 2 12terabytes of Tweets created daily 5million trade events per second 100 s video feeds from surveillance cameras

3 New analytic applications require a big data platform Advanced Analytic Applications Integrate and manage the full variety, velocity and volume of data Apply advanced analytics to information in its native form Big Data Platform Process and analyze any type of data Accelerators Visualize all available data for ad-hoc analysis Development environment for building new analytic applications Workload optimization and scheduling Security and Governance 3

4 IBM Big Data Platform Analytic Applications BI / Reporting Exploration / Visualization Functional App Industry App Predictive Analytics Content BI / Analytics Reporting IBM Big Data Platform Cost-effectively analyze petabytes of structured and unstructured information Hadoop System 4

5 IBM Big Data Platform Analytic Applications BI / Reporting Exploration / Visualization Functional App Industry App Predictive Analytics Content BI / Analytics Reporting IBM Big Data Platform Hadoop System Stream Computing Analyze streaming data and large data bursts for realtime insights 5

6 IBM Big Data Platform Analytic Applications BI / Reporting Exploration / Visualization Functional App Industry App Predictive Analytics Content BI / Analytics Reporting IBM Big Data Platform Hadoop System Stream Computing Data Warehouse Deliver deep insight with advanced in-database analytics and operational analytics 6

7 IBM Big Data Platform Analytic Applications BI / Reporting Exploration / Visualization Functional App Industry App Predictive Analytics Content BI / Analytics Reporting IBM Big Data Platform Hadoop System Stream Computing Data Warehouse Govern data quality and manage information lifecycle Information Integration & Governance 7

8 IBM Big Data Platform Gather, extract and explore data using spreadsheet metaphor BI / Reporting Exploration / Visualization Visualization & Discovery Analytic Applications Functional App Industry App Predictive Analytics IBM Big Data Platform Application Development Content BI / Analytics Reporting Systems Management Speed time to value with analytic and application accelerators Accelerators Hadoop System Stream Computing Data Warehouse Information Integration & Governance 8 Cloud Mobile Security

9 Progressing the IBM Big Data Platform: recent announcements BI / Reporting Analytic Applications Exploration / Visualization Functional App Industry App Predictive Analytics IBM Big Data Platform Content BI / Analytics Reporti ng Accelerators Text analytics tool-kit Temperature monitoring Geospatial accelerator Visualization & Discovery Application Development Systems Management Enterprise Robustness Hadoop System Accelerators Stream Computing Data Warehouse Adaptive MapReduce Cluster and workload management Enhanced user and network security Information Integration & Governance Integration HDFS connector Balanced optimization and connectivity 9

10 IBM s Ecosystem Cognos Real Time Monitoring SOA Web Service Cognos Mashups Applications Spreadsheets Fin Planning InfoSphere Warehouse Data Marts DB2 Info Server Analytic Models IBM Confidential CDRs Pre-processed Data InfoSphere Streams ERP,CRM and Other Data Sources InfoSphere BigInsights 2011 IBM Corporation

11 Big data creates new possibilities for optimized outcomes and competitive differentiation T-Mobile Dublin City Council Brocade IBM Business Partner Improving Results Network analysis to improve client experience 1.7 billion daily events Support users with real-time reports New Approaches Public transport optimization Analyzes 50 bus location updates per second Monitor 1000 buses across 150 routes daily Strategic Advantage Network security intrusion detection Sub-millisecond analysis and response No impact on network performance 11

12 Big data applies across all industries Financial Services Fraud detection 360 View of the Customer Energy and Utilities Smart Meter Analytics Asset Management Transportation Logistics optimization Traffic congestion Digital Media Real-time ad targeting Attribution Analysis Health & Life Sciences Medical Record Analytics Disease Surveillance Retail Omni-channel Marketing Real-time promotions Telecommunications Customer Profile Monetization Network Analytics & Optimization Law Enforcement Multimodal surveillance Cyber security detection 12

13 IBM InfoSphere Streams as Enabler Streaming analytic applications Multiple input streams Advanced streaming analytics Eclipse based IDE Define sources, apply operators, define intermediary and final output sinks User defined operators in Java or C++ Optimizing compiler automates deployment and connections Extremely low latency No limits on cluster size Source Adapters Operator Repository InfoSphere Streams Studio (IDE for Streams) Automated, Optimized Deploy and Management (Scheduler) Sink Adapters 2011 IBM Corporation

14 Big Data for Traffic Monitoring 14

15 Real Time Traffic Monitoring

16 Real Time Traffic Information

17 Real Time Traffic Information

18 ITS Application Flow-graph (125k GPS/second) 5min Aggr. KML Color map Inter-quartiles (IQ) weekends GPS source Time of day, day of week month of year GeoMatch Clean Sort Adapt IQ DSP ISO 8601 to timestamp Filter taxis and invalid values GpsTrack Travel times IQ Title Query Join Title Current

19 Big Data for the Telco Industry 19

20 New opportunities continue to emerge for Telco players due to hypergrowth in wireless demands Mobile Internet Smart Phone and Mobile Entertainment applications will drive >10 to 30x mobile traffic in next 6 years Wireless industry will need to transform existing voice-oriented network to content- oriented network M2M / Smart Planet M2M communication has become an multi-billion fast growing market, and will continue to grow 4x in 5 years Emerging Smart Grid, Public Security, Telematics 2.0 will drive broadband M2M growth Enterprise Wireless 70% of mobile traffic will happen in-building Femtocell / picocell covering wireless in-door will grow >10x in the next 5 years The volume of 4G femtocell / picocell will drive down the cost impacting wireless in enterprise Source: ABI research 2011 IBM Corporation

21 Growth in wireless traffic presents new challenges to Telco companies Effective customer retention Contextual awareness Better asset utilization Growing Fraud Smarter mktg campaigns Newer Govt Regulations 2011 IBM Corporation

22 New Intelligence is helping wireless carriers sell more services, retain customers, and operate in a low-cost, highly efficient, agile environment GAIN INSIGHT FROM THE INFORMATION EXPLOSION MOVE FROM REACTION TO PREDICTION 1 2 REAL TIME MEDIATION WAREHOUSE CONSOLIDATION MASTER DATA MANAGEMENT REAL-TIME ANALYTICS PROCESS PERFORMANCE METRICS OPTIMIZED PERFORMANCE CHURN PREDICTION KEY PERFORMANCE PREDICTORS BEHAVIORAL & SOCIAL NETWORK ANALYTICS CASHFLOW ANALYTICS CUSTOMER EXPERIENCE MANAGEMENT 4 ENGAGE THE ENTIRE VALUE CHAIN CONTENT PARTNERS NETWORK PARTNERS RETAIL PARTNERS DISTRIBUTORS DEVICE PARTNERS NEW INTELLIGENCE CREATE VALUE FASTER REAL-TIME CAMPAIGNS CAMPAIGN ANALYTICS MARKET BASKET ANALYSIS REAL TIME FRAUD IBM Corporation

23 Customers Experiences... A telco implementing a solution to access and analyze call, internet usage and texting detail records (xdrs) in real-time. 91% reduction in time to merge data 92% reduction in time to load data (from 95 minutes to 8 minutes) 93% reduction in storage requirements 85% reduction in servers used (80 blades to 12 blades) A telco requiring a solution to analyze up to 25M messages per second. At these volumes, in-motion analysis is the only option. Even at these volumes, Streams provided near linear scalability Streams handled at least an order of magnitude more events per second on the same hardware than competitors. (Telco s Chief Architect) A government customer required only 1.5 FTE Streams administrators for netflow analysis and video & image analysis across ~15 geographically dispersed data centers hosts each 2011 IBM Corporation

24 Analysis of Call Detail Records for Customer Retention Telcom Switch Call Detail Records + = InfoSphere Streams Mediation with Churn and Social Analytics Process CDRs in 1 min vs 12 hours 112 x86 cores vs 384 P5 cores Deduplication in Streams reduces Warehouse work Simultaneous summaries and analysis Network Equipment Providers Mediation and Analytics applications 2011 IBM Corporation

25 Mediation & Revenue Assurance Performance at IDEA 1.01 Billion CDRs in 2 hours for all circles running Telcordia IN average rate of 140K per second 2 HS22 blade dual CPU quad core servers 8 cores each, 2.5 GHz, 64 GB memory (total 16 cores) Avg CPU utilization: 75% Avg. memory utilization ~6GB Before 42 P6 cores 68M CDR/hr After 16 x86 cores 505M CDR/hr Change 62% fewer! 740% more! 98 Million subscribers Tier 1 operator in India Operate in 22 circles 2011 IBM Corporation

26 Real Time Marketing at Southeast Asian Telco The Pain: Business flexibility & responsiveness Insight Information Data active active Business value prescriptive prescriptive 100M CDRs per day from SMS from 25M subscribers Used to send bills to customers The Answer: InfoSphere Streams to create thousands of concurrent realtime marketing promotions A moment s insight is sometimes worth a life s experience. Oliver Wendell Holmes 2011 IBM Corporation

27 Social Media Analysis 2011 IBM Corporation

28 IBM analyzed over 1B social media posts to determine the reaction to Disney trailers aired during the Super Bowl How many people are talking about the film? Do they intend to actually see the film? Did the Super Bowl trailers have any impact? Who are they? What is their demographic profile Are they highly influential? Are they avid movie-goers? Are they comic book fans? What is their reaction? Did they like the trailer? What elements (plot, characters, etc.) had the best reaction? What elements (plot, characters, etc.) had the worst reaction? Why did they feel this way? 28 How does this compare to the competition? Compared to other trailers aired at the same time? Compared to other films releasing at the same time?

29 Conversations were collected in real-time providing to-theminute insight over a one month period Jan 1 Monitoring Period Feb 5th Golden Globes NFC Championship Super Bowl 5pm 6pm 7pm 8pm 9pm 10pm 11pm Data Set 1.1B tweets 5.7M blog and forum posts 3.5M relevant messages 97K referencing The Avengers 18K referencing John Carter Information extracted Buzz and sentiment Gender, Location and Occupation Avid movie-goers, comic book fans Intent to see specific films Specific attributes of the film/trailer 29

30 The data set included over 1.1B tweets and 5.7M blog & board posts, from which 3.5M relevant conversations were identified DATA SOURCES TWITTER -10% Direct Feed -Targeted search via PowerTrack Hundreds of Thousands of Message Boards Sample Board list Hundreds of Thousands of Blogs Sample Blog list fanforum.com uwants.com reddit.com/r/askreddit/ facebook.com city-data.com gaiaonline.com sherdog.net kaskus.us pinoyexchange.com investorshub.advfn.com babycenter.com boards.ie imdb.com tripadvisor.com reddit.com weightwatchers.com thebump.com ar15.com comicbookresources.com 30 baktan.wordpress.com sherwinlaranga.com/winieville ksipnistere.blogspot.com ualrtoday.wordpress.com six03.posterous.com travelpod.com doesgrey.wordpress.com chnp101.wordpress.com installblogs.com/blog bleacherreport.com blogs.forbes.com/network/rss/ merchantyellowpages.net/wordpress/ archiveofmytweets.wordpress.com halamovie.com indohr.blogspot.com bnotizie.com rockingappuse.wordpress.com americanbankingnews.com

31 Questions that Drove the Investigation Was the Super Bowl campaign effective? Should Disney adjust creative and messaging? Should the campaign be tailored around a specific segment or demographic? Should the advertising campaign be adjusted to deal with emerging threats? 31

32 Trailers airing on the Super Bowl resulted in times the daily buzz for John Carter and The Avengers January February Super Bowl generated roughly 15x more buzz than the daily average 6x more buzz than commercials run during the NFC Championship Over 6,000 individual conversations on Feb 5th Super Bowl generated roughly 20x more buzz than the daily average Over 36,000 individual conversations on Feb 5 th, 11,000 in February 6th 32 January February

33 The Avengers generated over 5 times more buzz than John Carter The Avengers The Avengers Buzz Volume John Carter The Avengers John Carter Buzz Volume John Carter 33 Feb 5 th (EST) Feb 6 th (EST)

34 Messages indicating intent to actually see the film in theaters were extracted from general conversations to judge a trailers impact Buzz Volume Buzz Volume # Feb 5 th (EST) % Intent as % of Buzz Intent as % of Buzz % of conversations indicating a desire to see John Carter dropped rapidly after the Super Bowl trailer aired, indicating the content and message of the trailer received a poor reaction % of conversations indicating a desire to see The Avengers rose with the general level of conversations indicating the trailer was effective in driving purchasing behavior # % 34 Feb 5 th (EST)

35 Of all film trailers aired during the Super Bowl, The Avengers was the clear winner in terms of social media reaction Clear spikes indicate the time each trailer aired The Avengers generated more than double the level of buzz as the next highest competitor, Act of Valor Trailers that aired during the pre-game generated a small fraction of the buzz compared to those that aired during the game Buzz by Volume Feb 5 th (EST) 35

36 Each new trailer resulted in a domination of film-related share of voice Share of Voice of Trailers Aired During Super Bowl 6pm to 10pm EST The Avengers The Dictator John Carter The Lorax Act of Valor Battleship Feb 5th 36

37 Sentiment was overwhelmingly positive for The Avengers and mixed for John Carter during the Super Bowl Sentiment by Volume Net Sentiment % Of messages indicating a clear positive or negative sentiment (i.e. This film looks incredible or This film looks terrible), The Avengers maintained a level of between % positive reactions Feb 5th Sentiment by Volume Net Sentiment % The Super Bowl trailer for John Carter generated a substantial amount of negative reactions, pushing the net sentiment level as low as 13% immediately after the trailer aired 37 Feb 5th

38 Similar titles or titles releasing the same time as The Avengers generally had a positive reaction, but at a much smaller scale 3, % 6, % % Sentiment Volume Net Sentiment% 0% Sentiment Volume -100 Net Sentiment% 0% Sentiment Volume -50 Net Sentiment% -10% Ghost Rider received negative reactions, but low volumes due to the trailer airing during pre-game which skew the figures

39 John Carter received a much more negative reaction than its competitors 7, % % Sentiment Volume Net Sentiment% Sentiment Volume Net Sentiment% % % John Carter received the most negative reactions of any film trailer airing during the Super Bowl 39

40 SPSS and Streams: Offline Model Building with Online Scoring 40

41 IBM SPSS Modeler Model Building and Scoring 41

42 InfoSphere Streams + SPSS Product Integration Architecture IBM InfoSphere Streams S SPSS Scoring Operator S R P SPSS Repository Operator SPSS Publish Operator R P In-memory In-memory SPSS Modeler Scoring Stream Change Notification File System IBM SPSS Modeler Solution Publisher IBM SPSS Collaboration & Deployment Services Model Refresh Repository 42

43 In the new era of computing, outperformers will leverage the Volume, Velocity and Variety of information for better business outcomes Broadest enterprise big data platform Strongest business partner ecosystem Leader in each era of computing Volume Velocity Variety Scale from terabytes to petabytes of structured and unstructured data Analyze streaming data and large volume bursts in milliseconds Embrace heterogeneous data types, schemas and data sources 43

44 44