Databricks A Primer
Who is Databricks? Databricks was founded by the team behind Apache Spark, the most active open source project in the big data ecosystem today. Our mission at Databricks is to dramatically simplify big data processing and free users to focus on turning their data into value. We do this through our product, Databricks, that is powered by Spark. For more information on Spark, download the Spark Primer. Data Databricks Value We ve had great success using Apache Spark on Databricks to compute the billions of data points behind our predictive models guiding consumers to the right health insurance plan. The simplicity and interactivity of Databricks makes it easy for developers and data scientists new to Spark to get up to speed very quickly, and not have to worry about the minutiae of managing clusters. Ani Vemprala, CTO & Co-founder, Picwell 2
What is Databricks? Databricks is a hosted end-to-end data platform powered by Spark. It enables organizations to seamlessly transition from data ingest through exploration and production. There are four foundational components that comprise Databricks: Managed Spark Clusters Exploration and Visualization Production Pipelines Third-Party Apps The Foundational Components of Databricks 3
Managed Spark Clusters Fully managed Spark clusters in the cloud that helps enterprises focus on their data and not operations. Easily Provision Clusters: Launch, dynamically scale up or down, and terminate clusters with just a few clicks. We automate management so you can focus on your data. Harness the Power of Spark: Configured and tuned by the people who built it. Import Data Seamlessly: Import data from S3, your local machine, or a wide variety of data sources, including HDFS, RDBMS, Cassandra, and MongoDB. Exploration and Visualization An interactive workspace for exploration and visualization so users can learn, work, and collaborate in a single, easy to use environment. Explore: Use interactive notebooks to write Spark commands in R, Python, Scala, or SQL and reuse your favorite Python, Java, or Scala libraries. Collaborate: Work on the same notebook in real time or send it around for offline collaboration. Visualize: Leverage a wide assortment of point-and-click visualizations. Or use powerful scriptable options like matplotlib, ggplot, and D3. Publish: Build rich dashboards that present key findings to share with your colleagues and customers. 4
Production Pipelines A production pipeline scheduler that helps users get from prototype to production without re-engineering. Schedule Production Workflows: Schedule any existing notebook or locally developed Spark code to run periodically using existing or newly-provisioned clusters. Implement Complete Pipelines: Build production pipelines that span data import and ETL, complex conditional processing, and data export. Monitor Progress and Results: Set up custom alerts for job completion and failure, and easily view historical and in-progress results. Third-Party Apps A platform for powering Spark-based applications that helps users leverage a growing ecosystem of applications, and re-use their favorite tools. 5
What are some of the technical and operational bottlenecks faced by data scientists, data engineers and analysts with their data pipeline? Over last few years, Spark has made great strides in helping enterprises overcome some of their big data processing challenges, however many enterprises are still struggling to extract value from their data pipelines. Capturing value from big data requires capabilities beyond data processing; enterprises are finding out that there are many challenges in their journey to operationalize their data pipeline: 1. Infrastructure issues requiring data teams to pre-provision, setup and manage on-premise clusters that are both costly and time consuming. 2. Once the infrastructure challenges have been addressed, data scientists and engineers still have to contend with siloed workspaces where working with data, code, and visualization requires switching between different software, and sharing work amongst peers means manually copying data. 3. Sharing of insights to non-engineering stakeholders and the hand-off to the production team. 6
Problem: the journey is complex and costly. Get a cluster up and running Import and explore data Build a Production Pipeline Expensive to build and hard to manage Disparate and difficult tools Months of re-engineering to deploy Your Data Pipeline: the journey is complex and costly In all this, enterprises are required to cobble various components together, making it not just highly inefficient, but also difficult to track data lineage and usage patterns over the various components within the stack. With this current model, enterprises are not able to implement complete pipelines - this severely inhibits innovation and value creation. Why Databricks? Given the challenges faced by data professionals and enterprises in managing their data pipeline, we saw the need for a single platform that can enable customers to easily deploy Spark as-a-service while providing a rich set of tools out-of-the-box. Key attributes: Managed Spark Clusters in the Cloud Notebook Environment Production Pipeline Scheduler 3rd Party Applications 7
Our key differentiators are: Unified Platform With Databricks, enterprises are able to go from data ingest through exploration and production on a single data platform. This significantly minimizes the integration pains they currently face when cobbling together multiple tools and systems, and helps streamline entire pipeline deployments. With a unified platform, data professionals are able to reuse their code base by utilizing the same notebooks for exploration and production, resulting in tremendous time savings. Zero Management Databricks provides powerful cluster management capabilities which allow users to create new clusters in seconds, dynamically scale them up and down, and share them across users. This obviates the need to set up and maintain the clusters. As such organizations do not need to have dedicated DevOps teams - their data teams can now create self-service Spark clusters and import their data seamlessly. This allows them to focus on their core mission understanding and gaining insights from their data, not in managing day-to-day operations. Real-Time Databricks provides real-time capabilities in several dimensions. 1. The notebook feature allows users to perform interactive queries and visualize results in real-time. This can dramatically increase their productivity when performing explorations and gain additional insights. 2. The interactive workspace feature enables real-time collaboration amongst multiple users. Team members can seamlessly share code, plots, and results, leveraging each other s work far more effectively. Open Platform Databricks is a platform for powering Sparkbased applications and comes with a thirdparty API in addition to JDBC connectivity, so users can plug in their favorite BI tools directly to their Databricks clusters, as each cluster comes with a JDBC server. This enables users to reuse their favorite tools, leverage our growing application ecosystem and to maximize their investments and knowledge base, leading to improved time to value and productivity. 3. The streaming feature provides low-latency and fault-tolerant processing of continuous data streams. This enables organizations to rapidly take action in response to live data in real-time. 8
How are enterprises typically using Databricks? Enterprises deploy Databricks to achieve a wide variety of objectives, including: Prepare Data Import data using APIs or connectors Clean mal-formed data Aggregate data to create a data warehouse Databricks is powered by Spark, giving it the ability to ingest data from a diverse set of sources and perform simple yet scalable transformations of data. The real-time interactive querying environment and data visualization capability of Databricks makes this typically slow process much faster. Build Data Products Rapid prototyping Implement advanced analytics algorithms Create and monitor robust production pipelines Databricks allows teams of developers and data scientists to efficiently experiment with new product ideas through the interactive workspace. Advanced analytics libraries such as MLlib and GraphX also provide an easy way for teams to deploy sophisticated algorithms in Spark. Once a prototype has been built, one can seamlessly deploy it in production at scale using the Jobs feature. Perform Analytics Explore large data sets in real-time Find hidden patterns with advanced analytics algorithms Publish customized dashboards With Databricks, developers and data scientists can work in SQL, Python, Scala, Java, and R with a wide range of advanced analytics algorithms at their disposal. Teams can be instantly productive with real-time analysis of largescale datasets on topics ranging from user behavior to customer funnel. Databricks can easily publish these results and complex visualizations as part of notebooks, integration with third party BI tools, or customized dashboards for consumption with a few clicks. 9
How will Databricks benefit data professionals and enterprises? Databricks helps data professionals and enterprises to focus on finding answers from their data, building data products, and ultimately capture the value promised by big data. Evaluate Databricks with a trial account now. The platform delivers the following key benefits to data professionals and enterprises: Higher productivity Maintenance-free infrastructure Real-time processing Easy to use tools Faster deployment of data pipelines Zero management Spark clusters Instant transition from prototype to production Data democratization within enterprises One shared repository Seamless collaboration Easy to build sophisticated dashboards and notebooks databricks.com/registration The fact that explorations by our data science team now take less than an hour, rather than days, has fundamentally changed how we ask questions and visualize changes to the index. Darian Shirazi, CEO, Radius Intelligence 10