BASHO DATA PLATFORM SIMPLIFIES BIG DATA, IOT, AND HYBRID CLOUD APPS

Transcription

1 WHITEPAPER BASHO DATA PLATFORM BASHO DATA PLATFORM SIMPLIFIES BIG DATA, IOT, AND HYBRID CLOUD APPS INTRODUCTION Big Data applications and the Internet of Things (IoT) are changing and often improving our lives. These applications strive to be simple to use, however the technology stack required to make them work can be very complex. Enterprise applications require data to be highly available and massively scalable, but they also need to be easy to manage. This whitepaper shows how Basho Data Platform addresses challenges in Big Data, IoT, and hybrid cloud applications. We first start with an example of how a company can use integrated services to meet their business requirements. We then outline how Basho Data Platform can enhance your enterprise application by integrating NoSQL with caching, real-time analytics, and search. Finally, we illustrate flexible deployment options. WHY USE BASHO DATA PLATFORM? Basho Data Platform provides a new way to build, deploy, and manage your Enterprise Applications. It integrates Riak KV software with Apache SparkTM, Redis, and Apache SolrTM, and controls the replication and synchronization of data between these components, simplifying management of your applications. BASHO DATA PLATFORM BENEFITS Reduce complexity with integrated NoSQL databases, caching, in-memory analytics, and search components Enhance high availability and fault tolerance across components Integrate real-time analytics with Apache Spark and Riak KV Increase application performance with integrated Redis caching and Riak KV Optimize search with Apache Solr and Riak KV integration 1

2 EXAMPLE Picture an advertising company that runs an ad-exchange. In this model, they provide ad-serving, a marketplace of advertisements, and billing & reporting capabilities. SERVING ADS Let s begin with the core of the business, serving advertisements. In a world where impressions drive revenue, latency matters. Many have solved this by tuning their database read and write parameters. However, at massive scale, even a highly efficient database will incur too much latency. Many companies address high latency by placing a caching mechanism in front of their data persistence solution. Caching allows control of the latency profile of an application, but requires custom code to enable replication from database into the cache. Basho Data Platform solves the problem of replication by intelligently pairing Redis with Riak KV. Basho Data Platform provides both speed and high availability. Auto-sharding and cluster management capabilities ensure that the environment is stable and easy to manage, turning Redis into an enterprisegrade solution. Redis then handles the ad serving while Riak KV provides the distributed, scalable, and available data store for ad persistence. Since latency matters, the data location is also very important the closer the data is to the end user, the faster it will be served. Basho Data Platform s multi-cluster replication ensures ads are near the presentation endpoint, which significantly reduces latency. SEARCH FOR ADS The advertising marketplace must allow customers to search for either their own advertisements or for ads to place into rotation onto their websites. With Basho Data Platform, Redis can be used as a caching solution for type-ahead prediction (auto-complete), and Solr can be added to search for characteristics that have been tagged to each advertisement. This implementation differs greatly from what s described in the Serving Ads section above. In this case, the customer would use multi-cluster replication to serve advertisements from cluster A while providing search capabilities from cluster B, something no other solution offers. BILL FOR ADS Placing ads and finding ads is the core of an advertising exchange, but the business wouldn t survive for long without the ability to bill and generate revenue. The advertising exchange tracks advertisement impressions in a very simple fashion a date-time of impression per ad. The data must be correlated and analyzed over time intervals determined by the business (minute, hour, day, week, etc.). The process of correlating this data, performing the analysis for time ranges, and writing the data back into Riak is provided by a periodic running Spark job. The Spark Add-On handles both reading data from Riak KV and writing the result set back to Riak KV for persistence and consumption by the billing application. 2

3 WHAT IS BASHO DATA PLATFORM? Basho Data Platform provides a comprehensive set of data services that take the complexity out of manually deploying and managing separate clusters and instances of Riak KV with Spark, Redis, and Solr. These data services are integrated as a set of Core Services, Storage Instances, and Service Instances, which jointly form Basho Data Platform. This is illustrated in the origami graphic. BASHO DATA PLATFORM CORE SERVICES Big Data applications require a set of core services to keep them running smoothly. Manually deploying and managing separate clusters and instances of NoSQL databases, caching, and in-memory analytics is difficult and complex. The Basho Data Platform Core Services provide a distributed, scalable, fault-tolerant framework and resource manager for integrating databases and other key components of Big Data applications. These services impact data accuracy, high availability, scalability, and operational simplicity. Basho Data Platform Core Services deploy, manage, and synchronize data in and between Storage Instances (Riak KV, Riak S2) and Service Instances (Apache Spark, Redis, Apache Solr). DATA REPLICATION & SYNCHRONIZATION In addition to replicating and synchronizing data within and across Riak clusters, Redis and Spark Clusters are now also highly available. For Redis queries, when the data is not found in the Redis cluster it is read from Riak KV and synchronized across the client application query and Redis. Spark data is persisted in Riak KV so Spark now executes queries against imported data from Riak KV and existing Spark RDDs. CLUSTER MANAGEMENT & MONITORING Automated cluster management downloads, builds, and deploys clusters of Riak KV, Riak S2, Apache Spark, and Redis. Monitoring will auto-detect incidents with and across clusters and auto-restart clusters. It also auto-scales clusters as data grows. For Spark, cluster management plus the Riak KV Ensemble for leader election eliminates the need for Zookeeper. INTERNAL DATA STORE Built-in distributed data store for speed, fault tolerance, and ease of operations. It is used to persist configurations as well as static and dynamic data (port number, IP address) for sessions running across the Basho Data Platform. MESSAGE ROUTING A high throughput distributed message system for speed, scalability and availability. The data platform enhanced message system will persist and route messages across platform clusters. LOGGING AND ANALYTICS Event logs provide valuable information to assist with enhanced tuning of clusters and to analyze dataflow across the cluster. 3

4 BASHO DATA PLATFORM STORAGE INSTANCES Big Data applications need multiple data models to support different use cases in the same enterprise and often in the same application. Integrating these into applications requires additional development and operational skills that make it more complex. RIAK STORES AND MANAGES DATA EFFICIENTLY AND EFFECTIVELY Basho Data Platform simplifies this by supporting Storage Instances that include today s most flexible NoSQL database, Riak KV, and large object storage software, Riak S2, that are architected for high availability and horizontal scale. Making it easy to deploy and manage these Storage Instances with Service Instances (Spark, Redis, and Solr), Basho Data Platform also replicates and synchronizes data between them. DATACENTER #1 MULTI-CLUSTER REPLICATION DATACENTER #2 DATACENTER #3 RIAK KV DISTRIBUTED NOSQL DATABASE A key/value data store that is highly available, scalable, and easy to operate. Automatic data distribution across the cluster ensures fast performance and fault tolerance. Multi-cluster replication delivers low-latency global performance and robust business continuity. RIAK S2 OBJECT STORAGE SOFTWARE Simple, available, distributed large object storage for public, private, or hybrid clouds. Cost effective compared to traditional storage at petabyte scale. Plus it s compatible with Amazon S3 and OpenStack Swift for easy integration into existing production workloads. BASHO DATA PLATFORM SERVICE INSTANCES Big Data applications never stand alone. They are highly distributed and comprised of multiple components that include NoSQL databases, caching, and in-memory analytics, as well as separate configuration and resource management. Just keeping it all running and available takes a considerable commitment of effort and resources. Basho Data Platform takes the difficulty out of doing this by integrating Riak KV with these Add-On Service Instances: APACHE SPARK Integrated real-time analytics REDIS Faster application performance with integrated Redis caching APACHE SOLR Optimized search with Apache Solr 4

5 APACHE SPARK ADD-ON FOR BASHO DATA PLATFORM Integration of Riak KV with Apache Spark provides realtime analytics using the Spark connector. Built-in cluster management eliminates the use of Zookeeper. The rapid growth of unstructured data has changed the way that modern Big Data applications are designed and deployed. These unstructured data sets must be processed fast, in realtime, to reveal patterns, trends, and associations. The Spark connector for Basho Data Platform connects directly to Riak KV instances and moves required data to the Spark cluster. back in Riak KV. The ability to persist these results to Riak KV retains flexibility for future data processing. As part of Basho Data Platform, Spark Cluster deployments with Riak KV are as simple as specifying where code should be deployed. Both static information (configuration) and dynamic information (port numbers, etc.) are managed at installation time for newly deployed instances and existing Spark clusters. This makes it easy to manage Spark clusters without the use of Zookeeper. When data is required for analysis in Spark, that data is read from Riak KV, processed in Spark, and the results can be stored WRITE IT LIKE RIAK, ANALYZE IT LIKE SPARK CLUSTER MANAGEMENT Eliminate Zookeeper Built-in leader election makes it easy to manage Spark clusters at scale. FAST DATA MOVER Add Spark to your Riak data Intelligently load data from Riak KV into Spark clusters to minimize network traffic and processing overhead. RIAK WRITE-BACK Make persistence simple Store intermediate and final results in Riak KV for further processing by Spark or other components of your Big Data application. PERFORMANCE AT SCALE Process Big Data fast Architected for high performance, real-time analysis, and persistence of your Big Data. AUTOMATED DEPLOYMENT Run Spark easily Quickly deploy and configure Spark clusters with Riak KV. Auto-start failed Spark instances to reduce manual operations. APPLICATION SIMPLICITY Don t DIY Systematically integrate and update analytics, caching, and search technologies to simplify the design and operations of your Big Data application. 5

6 REDIS ADD-ON FOR BASHO DATA PLATFORM Redis caching with Riak KV improves application performance by reducing latency. Built-in cluster management, high availability, automatic data sharding, and the ability to replicate and sync data between Riak KV and Redis makes Redis enterprise grade. The combined power of Redis caching and Riak KV reduces latency to improve application performance. Basho Data Platform adds high availability and fault tolerance to Redis, and extends the operational simplicity of Riak KV to Redis for instance management and auto-sharding. Redis doesn t have the data in cache, it is accessed from Riak KV. Data is also automatically synchronized between Redis and Riak KV, increasing availability by allowing read-failures in Redis to be resolved by Riak KV and written back to the Redis cache. As part of Basho Data Platform, Redis deployment with Riak KV is as simple as specifying where the code should be deployed. Both static (configuration) and dynamic information (port numbers, etc.) are managed at the time of installation for both newly deployed instances and existing Redis installations. Since any Redis client can query the cache, no changes are required for existing Redis clients to access data in Riak KV. If WRITE IT LIKE RIAK, CACHE IT LIKE REDIS HIGH AVAILABILITY Ensure Uptime Integration with Riak KV makes the high-performance caching capabilities of Redis also highly available. FAST CACHE Optimize for milliseconds The speed of Redis is combined with the power of Riak KV to ensure low latency at scale. AUTOMATIC DATA SYNCHRONIZATION Get your data when and where you need it Data is automatically synchronized between Redis and Riak KV, and Basho Data Proxy resolves cache misses without requiring custom code to populate the cache. AUTOMATIC SHARDING Eliminate painful manual sharding Easily shard data automatically between multiple cache servers to reduce the time and errors of implementing manual sharding. AUTOMATED DEPLOYMENT Save time Easily deploy and configure Redis instances with Riak KV. Automatically restart failed Redis instances or disable on failure to reduce manual processes. APPLICATION SIMPLICITY Improve Efficiency Systematically integrate and update caching, analytics, and search technologies to simplify your Big Data application. 6

7 APACHE SOLR ADD-ON FOR BASHO DATA PLATFORM The inclusion of integrated search means it s easy to query Riak KV data sets using Apache Solr. As data changes, search indexes are automatically synchronized. Get the full-text search power of Solr with the availability and scalability of Riak KV. Storing unstructured data in Riak KV is only one component of a Big Data application. It is also necessary to retrieve that data for application consumption. The Solr Add-On brings together the strengths of Riak KV s scalable, distributed database with the powerful full-text search functionality of Apache Solr. This allows for transparent indexing and querying of Riak KV data values. In addition, there is direct support for Solr client query APIs, which enables integration with existing software solutions (either homegrown or commercial). With the Solr Add-On, Riak KV is responsible for the data and Solr is responsible for the indexes. Riak KV monitors for changes to data and propagates those changes to indexes managed by Solr. This data synchronization is critical to ensuring that full-text search results are up to date as data changes. WRITE IT LIKE RIAK, QUERY IT LIKE SOLR DISTRIBUTED FULL-TEXT SEARCH Connect to one, talk to all Standard full-text Solr queries are automatically expanded into distributed search queries to provide a complete result set across instances. AD-HOC QUERY SUPPORT Ask complex questions of your data Broad support for a wide range of Solr query parameters: exact match, range queries, and/or/not, sorting, pagination, scoring, ranking, etc. INDEX SYNCHRONIZATION Automate index updates Automatically synchronize data between Riak KV and Solr. Intelligent monitoring picks up changes to data and propagates those changes to Solr indexes. SOLR API SUPPORT Integrate with existing software Query data in Riak KV using existing Solr software, adding a powerful data source to Big Data applications. AUTO-RESTART Reduce or eliminate slow manual restarts Monitor the Solr OS processes and automatically start or restart processes when failures are detected. APPLICATION SIMPLICITY Make the complex simple Systematically integrate and update search, caching, and analytics technologies to simplify the design and operations of your Big Data application. 7

8 CONFIGURATION FLEXIBILITY INSTALLATION CHOICES Basho Data Platform provides customers with the flexibility to install one, some, or all available components of Basho Data Platform. Often, customers choose to install Spark and Redis with Riak KV for a fully managed implementation that ensures high availability and scalability for all of the data components in the solution (i.e. Riak KV, Spark, and Redis). Again, customers can choose to install any combination of Riak KV, Spark, Redis, and/or Solr that best fits their needs. Customers can add the Basho Data Platform to their existing installation of Spark and/or Redis. However, we recommend a fully managed configuration as shown below. Basho Data Platform Fully Managed Configuration 8

9 FULLY MANAGED SPARK ADD-ON Basho Data Platform includes a Spark Connector to implement real-time analytics seamlessly. This creates a 1:1 mapping between Riak KV and Spark data and optionally allows for query results to be persisted back into Riak KV. This Spark Connector provides both power and flexibility. It does this by providing high availability for Spark using Riak KV, rather than Zookeeper, for leader election. Also, for greater flexibility, the Basho Spark Connector does not require you to run Spark on the same node as the source database. You can run Spark anywhere you want, plus have either Riak KV or Solr (or both) query results. FULLY MANAGED REDIS ADD-ON Basho Redis Proxy supports multiple caching scenarios, including read-through cache. The diagram below shows a client application attempting to read a value from cache. The proxy service first tries to retrieve the value from Redis using a readthrough cache. If that value isn t found in Redis, then the value is read from Riak KV. This method of caching is called readthrough cache. 9

10 CONCLUSION Architecting Big Data applications that rely on multiple data stores requires a clear vision about the specific components and integration points in the data flow pipeline. Modern solutions can include hybrid cloud, IoT streams, and many other components or flavors of Big Data. Rather than assuming that all data components easily fit together, effective use of well-designed integration services are key to successful implementation. Basho Data Platform is designed to deliver maximum data availability, to scale linearly using commodity hardware, and to provide operational simplicity at production scale. COMPLEX DATA PROJECTS SIMPLIFIED CHALLENGES: Complex data models Complex interactions Complex fault tolerance Complex query patterns BASHO DATA PLATFORM: Supports multiple database models Integrates NoSQL with real-time analytics & caching Ensures high availability and fault tolerance Provides rich query capabilities YOU GAIN: Faster time to market More uptime Faster application performance Integrated real-time analytics Basho specializes in solving distributed systems challenges, and integrated approaches such as Basho Data Platform help ensure that applications are highly available, massively scalable, and easy to deploy at production scale. Mac Devine, VP CTO IBM Cloud Services at IBM ABOUT BASHO TECHNOLOGIES Basho is a distributed systems company dedicated to developing disruptive technology that simplify enterprises most critical data management challenges. Basho has attracted one of the most talented groups of engineers and technical experts ever assembled devoted exclusively to solving some of the most complex issues presented by scaling distributed systems. Basho s distributed database, Riak KV, the industry leading distributed NoSQL database, and Basho s cloud storage software, Riak S2, are used by fast growing Web businesses and by one third of the Fortune 50 to power their critical Web, mobile and social applications. The Basho Data Platform helps enterprises reduce the complexity of supporting Big Data applications by integrating Riak KV and Riak S2 with Apache Spark, Redis, and Apache Solr. Basho is the organizer of RICON a distributed systems conference. Riak is the registered trademark of Basho Technologies, inc. BASHO TECHNOLOGIES, INC NE 8TH STREET SEATTLE, WA // 10