Big Data on Google Cloud

Big Data on Google Cloud Using Cloud Dataflow, BigQuery, and friends to process data the Cloud way William Vambenepe, Lead Product Manager for Big Data, Google Cloud Platform @vambenepe / vbp@google.com

Agenda 1 Big Data at Google 2 Managing data through its lifecycle 3 Google Cloud Dataflow 4 Friends of Cloud Dataflow: BigQuery, Pub/Sub, Hadoop/Spark... 5 Optimizing your time 6 References and follow-up

Big Data at Google

Building on Google s infrastructure 1.5 million devices activated every day (over a billion devices) 6 billion hours watched every month (10h uploaded every minute) 20 billion pages crawled every day

Hardware and data center innovation

Software innovation BigQuery MapReduce Cloud Dataflow Flume Dremel GFS Big Table MillWheel Spanner Pregel Colossus 2002 2004 2006 2008 2010 2012 2013

Managing data through its lifecycle

Data lifecycle Re a an l tim al e & a ytics ler ts Google App Engine Cloud Pub/Sub Cloud Logs Google Analytics Premium Google Cloud Storage Stream BigQuery Storage BigQuery Analytics (SQL) (tables) Batch Cloud Dataflow Cloud Storage (files) Cloud Dataflow

Data usage organization maturity lifecycle Prescriptive + Predictive Exploratory Descriptive Predictive + Exploratory Descriptive Exploratory + Descriptive Descriptive

Supporting organizations with operational ease of use no administration most powerful tools in the easiest way constant experimentation with low risks & cost easy collaboration across teams and organizations low costs without requiring usage commitments best performance & virtually unlimited scale always on

Google Cloud Dataflow

What is Cloud Dataflow? Cloud Dataflow is a collection of SDKs for building parallelized data processing pipelines Cloud Dataflow is a managed service for executing parallelized data processing pipelines Download from GitHub: Use on Google Cloud: https://github.com/googlecloudplatform/dataflowjavasdk https://cloud.google.com/dataflow/

Cloud Dataflow SDK - Logical Model Pipeline{ Who => Inputs What => Transforms Where => Windows When => Watermarks + Triggers To => Outputs } Unified programming model for both batch & stream processing.

Cloud Dataflow Pipeline A Direct Acyclic Graph of data processing transformations Can be submitted to the Dataflow Service for optimization and execution or executed on an alternate runner e.g. Spark May include multiple inputs and multiple outputs May encompass many logical MapReduce operations PCollections flow through the pipeline

Life of a Dataflow Pipeline Managed Service Job Manager Progress & Logs ph tion a Gr iza tim p o Deploy & Schedule User Code & SDK Work Manager Monitoring UI Google Cloud Platform

Worker Lifecycle Management throughout batch execution Deploy Schedule & Monitor Tear Down

Worker Optimization 100 mins. vs. 65 mins.

Continuous worker scaling for long-lived streaming pipelines 800 RPS 1,200 RPS 5,000 RPS 50 RPS time

Portability: Cloud Dataflow Runners Run the same code in multiple modes using different runners Direct Runner For local, in-memory execution. Great for developing and unit tests Cloud Dataflow Service Runner Runs on the fully-manage Dataflow Service Your code runs distributed across GCE instances Community sourced Spark runner @ github.com/cloudera/spark-dataflow Flink runner coming soon from dataartisans The most productive and portable Data pipeline SDK.

Friends of Cloud Dataflow: BigQuery, Pub/Sub, Hadoop/Spark...

Cloud Pub/Sub Many-to-many asynchronous messaging Fast and reliable

BigQuery Ingest data via streaming (100K rows/second/table) or file loader Process interactive SQL queries on TB or PB of data Zero administration; just upload data and send queries Pay for storage and query separately, based on actual usage Non-technical analysts can drive queries on massive datasets using BI tools (e.g. Tableau) Highly Available: Data replication in multiple geographies. Secure and easy collaboration: access to data is controlled using customer-owned ACLs

Hadoop and Spark Master Node BigQuery Connector Connectors GCS Connector Name Node (optional) Local SSD PD SSD Work Nodes Work Nodes Work Nodes HDFS (optional) HDFS (optional) bdutil orchestration PD standard

Optimizing your time

Optimizing Your Time

References and follow-up

Getting Started Cloud Dataflow Service: https://cloud.google.com/dataflow Questions: https://stackoverflow.com/questions/tagged/google-cloud-dataflow SDK: https://github.com/googlecloudplatform/dataflowjavasdk BigQuery https://cloud.google.com/bigquery/ Cloud Pub/Sub https://cloud.google.com/pubsub/ Hadoop and Spark https://cloud.google.com/hadoop/ Contact me Twitter: @vambenepe email: vbp@google.com

Thank You! cloud.google.com