Apache S4: A Distributed Stream Computing Platform

Transcription

1 Apache S4: A Distributed Stream Computing Platform Presented at Stanford Infolab Nov 4, (migrating from S4 Committers: {fpj, kishoreg, leoneu, mmorel, robbins}@apache.org Presented by Leo Neumeyer (@leoneu) 1

2 About Me Born in Buenos Aires, Argentina, studied EE. School/Work in Canada (Signal Processing, Speech Coding). SRI Int'l (Menlo Park) Speech Lab, DARPA benchmarks, lab founded speech recognition spin-off Nuance Comm Inc. Mindstech: Startup to teach spoken English in Asia using web audio/video (before 2-way media was widely available). Yahoo! Labs: Search advertising (optimization, auctions). Quantbench: mission is to create a marketplace for data scientists, data providers, and investment funds. 2

3 S4 Project History Started as a research project at Yahoo! Labs in August 2008 out of the need to personalize search ads in real-time. Open sourced in September Moved to Apache Incubator in October

4 Motivation Personalized Search Twitter Trends Online Parameter Optimization Predict Market Prices Automatic Trading Network Intrusion Detection given multiple event streams extract information using data driven models in real time with low latency at scale It's Fun! Spam Filtering Sensor Networks 4

5 S4 Architecture Node App Server App App PE Prototype App PE Instance App Stream App Unlimited number of nodes. Each node has one process. There is one server process per node. The server loads/unloads apps. Apps encapsulate units of work. They can consume and produce event streams. An app is a graph composed of PE prototypes and streams that produce, consume, and transmit msgs. PE instances are clones of the prototype. They are associated with a unique key and contain the state. S4 is a general-purpose, real-time, distributed, decentralized, robust, scalable, event driven, pluggable platform that allows programmers to easily implement applications for processing continuous unbounded streams of data. 5

6 Latency vs. Accuracy Zero Errors Real-Time Latency Unconstrained Constrained Why? Reproducible results Limited control over inbound data rate and computing complexity Use Debug Train Models Process unstructured data Tolerance to small errors Graceful recovery from inbound data streams 6

7 Design Actors programming model. Probabilistic thinking in both algorithms and systems. Run on commodity hardware. All in-memory, no disk bottlenecks. Pluggable (Protocols, applications, serialization, etc.) Object oriented design POJOs Static typing, no string literals, minimize type casting. Science friendly constant change, ease of use. 7

8 Programming Model Example: estimate clickthrough rate in a web application after applying a filter to remove bot traffic. 8

9 Coding an App 9

10 Research Areas: Systems Checkpointing strategies Replication strategies Dynamic load balancing Adaptive load management Query languages 10

11 Fault Tolerance Problem Approaches S4 High Availability State Loss (Crashes, system updates) Warm/hot failover Cold failover Lossy checkpointing Lossless checkpoint. Warm failover Standby nodes + Apache Zookeeper Lossy checkpointing Low Latency Decouple stream processing from checkpointing Asynchronous writes Uncoordinated checkpointing Approach: checkpoints are count or time based, pluggable backend to support any data store, lazy PE restore, tuning is application dependent. Research by M. Morel, F. Junqueira, Yahoo! Research Europe,

12 Resilience in a Distributed Word Count Task 12

13 Research Areas: Algorithms Self-adaptive models: adaptive language models using small amounts of data. Personalization: learn from user feedback (clicks, location, behavior) to deliver relevant information in RT. Trend detection: find personal Twitter trends relevant to you. Intrusion detection: summarize high level state of the network and detect unusual patterns. Sensor networks: large amounts of audio/video and other sources require processing, recognition, detection, and tracking. Detect events across sensors. 13

14 Personalized Search Ads Goal is to maximize: Revenue Click yield User experience By controlling: Ranking Pricing Filtering Placement S. Schroedl, A. Kesari, and L. Neumeyer, Personalized ad placement in web search, in ADKDD 10: Proceedings of the 4th Annual International Workshop on Data Mining and Audience Intelligence for Online Advertising,

15 Personalized Search Ads Model ad click intent using recent user activity. More likely to click show more North ads. Example 1 First query is digital slr camera Next query is canon slr More likely than average to click another ad Example 2 Repeated query without previous clicks Less likely to click another ad 15

16 Personalized Search Ads Modeling user session Typical features: Number of searches/clicks by user past 24 hrs User COPC: Ratio of observed clicks to predicted clicks Identical query searched before / clicked before Time (seconds) since last search/click Similarity measures: current vs. previous queries Modeling technique: stochastic gradient-descent boosted trees (GDBT) 16

17 Personalized Search Ads Target P[CLICK ad,query,user] Approximation P[CLICK ad,query]*ucp[user,session] Non-personalized long-term model computed using Hadoop User Click Propensity (UCP) for user session computed using S4 17

18 Personalized Search Ads Results: We can reduce the average number of ads (ad footprint) by 7% without decreasing click yield and revenue. - OR - For a given ad footprint we can increase click yield by ~2%. 18

19 Thank you! Join the Apache S4 project: 19