No-SQL Databases for High Volume Data

Target Conference 2014 No-SQL Databases for High Volume Data Edward Wijnen 3 November 2014

The New Connected World Needs a Revolutionary New DBMS Today The Internet of Things 1990 s Mobile 1970 s Mainfram e Cloud ClientServer SemiConnected 2014 DataStax Confidential. Do not distribute without consent. Isolated Social Radically Connected

Businesses Must Close the Gap and Fast Your Business Your Customers

If You Started With This Connected Employees Connected Partners Connected Products Connected Customers Connected Devices

You Would End With This Connected Employees Connected Partners Connected Products Distributed Transactional Database Connected Customers Connected Devices

Apache Cassandra Apache Cassandra is a massively scalable, open source, NoSQL, distributed database built for modern, mission-critical online applications Written in Java and is a hybrid of Amazon Dynamo and Google BigTable Masterless with no single point of failure Distributed and data centre aware 100% uptime Predictable scaling Dynamo Cassandra BigTable BigTable: Dynamo: http://research.google.com/archive/bigtable-osdi06.pdf http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

Apache Cassandra You are already using it!

Distributed Transactional Database Advantages The Hague

Distributed Transactional Database Advantages The Hague London

Delivers 150+ Billion Content Recommendations Per Month Serves content for largest media brands in the world: Reuters, Wall St Journal, USA Today Needed a massively scalable data store High velocity of data with 58,000 links to content per second Always-on data architecture Use Case: Recommendations / Personalization

Distributed Transactional Database Advantages The Hague London

Distributed Transactional Database Advantages The Hague London Groningen

Netflix Delights Customers with Personal Recommendations World s leading streaming media provider with digital revenue $1.5BN+ Tailors content delivery based on viewing preference data captured in Cassandra Increased market cap by 600% since 2012 Introduction of Profiles drove throughput to over 10M transactions per second Replaced Oracle in six data centers, worldwide, 100% in the cloud Use Case: Personalization

Cassandra Always On, No Matter What The Hague London Groningen l Transactional Backbone l Industry Leading Performance l Predictable Scalability l Operational Simplicity l Business Flexibility

Cassandra Operational Simplicity

CAP Theorem Oracle vs Cassandra

Cassandra Tunable Consistency Consistency Level (CL) Client specifies per read or write Handles multi-data center operations 5 μs ack 12 μs ack Write CL=QUORUM Node 1 1st copy Parallel Write Node 5 ALL = All replicas ack QUORUM = > 51% of replicas ack LOCAL_QUORUM = > 51% in local DC ack ONE = Only one replica acks Plus more. (see docs) Node 2 2nd copy 12 μs ack Node 4 Node 3 3rd copy 500 μs ack Blog: Eventual Consistency!= Hopeful Consistency http://planetcassandra.org/blog/post/a-netflix-experiment-eventual-consistency-hopeful-consistencyby-christos-kalantzis/

Common Use Cases Product Catalogs and Playlists Fraud detection Internet of things/ Sensor data Recommendation/ Personalization Messaging

Product Catalogs and Playlists A product catalog is an organized collection of products or services. Playlists refer to userdefined queues of songs, movies, games and lessons. Examples: Shopping carts, gift registries, media playlists. Challenges Rigidity of relational databases Increase in volume and diversity of data Application must have zero downtime Predictable scalability is hard Desire to operate in the cloud Why DataStax? Real-time database infrastructure Rich analytics for flexible access to information Fast search and indexing of data Add new features while the application is online Multiple data centers to ensure applications and data have 100% uptime Customers

Recommendation Engine Recommendation and Personalization Engines understand each person s unique habits and preferences and bring to light products and items that a user may be unaware of and not looking for. Examples: News sites, shopping carts. Challenges Large volumes of user data makes accuracy challenging Merging real-time and historical information Cross-product information Response times need to be fast Predictable scalability is hard Why DataStax? Rich query language and enterprise search to store, search and analyze user activity data Integrations with data lakes allow for the merging of real time and historical data Multi-data center replication ensures applications and data suffer no downtime Linear scalability is predictable Customers

Fraud Detection Fraud detection solutions identify out-of-the-ordinary patterns to prevent malicious attacks on digital and physical assets from unauthorized applications and individuals. Examples: credit card monitoring, application infiltration Challenges Increasing volume of fraudulent attacks across all industries Technology sophistication Limited historical and trend information Information is stored across multiple channels The customer can be the first to spot the fraud Why DataStax? Easy management of high-data volumes Real-time monitoring across channels, sites and data centers Integrations with data lakes allow for the merging of real time and historical data Ease of use in managing and monitoring data Multi-data center replication ensures applications and data suffer no downtime Customers

Internet of Things/ Sensor Data IOT refers to the revolution of a growing number of internet-connected devices that can network and communicate with each other. Challenges Vast and diverse amounts of unstructured data from internet enabled devices Volume of sensors is increasing exponentially Fast-changing technology Support multiple channels with varying data types Predictable scalability is hard Why DataStax? Easy management of high-data volumes Rich query language and enterprise search to store, search and analyze data Dynamic database schema Linear scalability offers predictability Customers

Messaging Messaging facilitates communication, interaction and collaboration between diverse user-groups and applications via social networks, cloud services and more. Examples: SMS, email and instant messaging. Challenges Managing large data volumes at a reasonable cost Real-time updates and information, getting detailed alerts and notifications Predictable scalability is hard Information is stored across multiple platforms and systems Agility Why DataStax? Easy management of high-data volumes Real-time monitoring across channels, sites and data centers Multi-data center replication ensures applications and data suffer no downtime Ease of use in managing and monitoring data Dynamic database schema Customers

The Weather Channel on Learning to use Cassandra If you had a look in the past, you may have found Cassandra had a high learning curve and a fair amount of complexity. CQL3, the native drivers, and virtual nodes have changed the game entirely, making Cassandra a much more accessible and friendly platform. While I have years of experience using Cassandra, my team was mostly new to it; CQL made their transition essentially painless. But where Cassandra really shines is in speed and operational simplicity, and I would say those two points were critical. Robbie Strickland Software Dev Manager

API s and Drivers Application drivers and connectors for all popular developer languages exist for Cassandra and DataStax Enterprise. CQL (Cassandra Query Language) is the primary API Drivers/connectors include: Java C++ Python Ruby PHP

DataStax Enabling The Future Connected Employees Connected Partners Connected Products Distributed Transactional Database Connected Customers Connected Devices

Thank you! edward.wijnen@datastax.com @edwardwijnen