Real-World Analytics with Solr Cloud and Spark

Transcription

1 Real-World Analytics with Solr Cloud and Spark Solving Analytic Problems for Billions of Records Within Seconds Johannes Weigend Apache Big Data North America 2016 May 2016 Vancouver, May 2016 Johannes Weigend QAware GmbH

2 Any Question? Ask or Twitter with the Hashtag #cloudnativenerd

3 The Problem We Want to Solve Interactive applications with runtimes lower than a second! Processing of billions of records ( rows / records) Continuously import data (near realtime) Applications on top of the Reactive Manifesto

4

5 Apache Big Data North America Vancouver Johannes Weigend QAware GmbH

6 Horizontal Scalability can be difficult! Horizontal Scalability of functions Trivial Loadbalancing of (stateless) services (makro- / microservices) More users! more machines Not trivial More machines! faster response times Horizontal Scalability of data Trivial Linear distribution of data on multiple machines More machines! more data Not trivial Constant response times with growing datasets

7 Hadoop Gives Answers for Horizontal Scalability of Data and Functions

8

9 The Processing of Distributed Data can be Quite Slow! Reduce foreach() -> Minutes / Hours Map Map Map Filter Filter Filter Read Read Read Data Flow HDFS / NFS / NoSQL 9

10 With Former Indexing and Searching, Less Data has to be Read and Filtered. Reduce Map Map Map foreach() -> Seconds/Minutes Filter Filter Filter Data Flow Search Search Search Search / NoSQL 10

11 Frontend Spark Reduce Map Map Map Business Layer Cluster Processing Search Search Search Distributed Data

12 DEMO

13 Reduce Map Map Map Spark Filter Filter Filter Search Search Search Data Flow Search / NoSQL 1. Solr Cloud for Analytics

14 Cloud Document based NoSQL database with outstanding search capabilities A document is a collection of fields (string, number, date, ) Single und multiple fields (fields can be arrays) Nested documents Static und dynamic scheme Powerful query language (Lucene) Horizontal scalable with Solr Cloud Distributed data in separate shards Resilience by the combination of zookeeper and replication Powerful aggregations (aka facets) Stable > V 6.0

15 The Architecture of Solr Cloud Leader Zookeeper Zookeeper Zookeeper Zookeeper Cluster Solr Server Solr Server Solr Server Solr Cloud Collection Shard1 Shard2 Shard3 Shard4 Shard5 Shard6 Shard7 Shard8 Shard9 Shards Replica4 Replica8 Replica9 Replica7 Replica2 Replica3 Replica1 Replica5 Replica6 Replicas Scale Out

16 Solr Stores Everything in a Single Table (BigTable). Searching is Extremely Fast and Powerful. * Customer Order Name Address 1 * Amount Product SolrDocument SolrDocument SolrDocument SolrDocument Type ID Name Address Amount Product C2O Customer 1 K 1 A [3,5] Customer 2 K 2 A [4] Order Z 1 P 1 [1] Order Z 2 P 2 [2]... (*) With 100 million documents per shard, runtimes of queries and aggregations are normally less then 100ms

17 A Solr Cloud can be Started in Seconds. Create a scheme by reusing an existing set of solr config files There are examples in the installation directory $SOLR_HOME/server/solr/configsets which can be copied and modified cp $SOLR_HOME/server/solr/configsets/basic_configs \ $SOLR_HOME/server/solr/configsets/bigdata2016 Start solr When the wizzard asks for a collection name use bigdata2016 (see above) $SOLR_HOME/bin/solr start e cloud Make a first test curl localhost:8983/solr/jax2016/query?q=*:*

18 With the Solr Cloud Collection API, Shards can be Created, Changed or Deleted. Create a collection <<SOLR URL>>/solr/admin/collections?action=CREATE& name=<<name of collection>>& numshards=16& replicationfactor=2& maxshardspernode=8& collection.configname= <<name of uploaded zookeeper configuration>> Delete a collection <<SOLR URL>>/solr/admin/collections?action=DELETE& name=<<name of collection>>

19 Zookeeper has to be Started First and the Solr Configuration must be Uploaded to Use a Solr Cloud. 1. Start zookeeper on 2n+1 nodes (odd number) $ZOO_HOME/bin/zkServer.sh start 2. Upload the solr configuration into zookeeper $SOLR_HOME/server/scripts/cloud-scripts$./zkcli.sh -cmd upconfig -zkhost :2181, :2181, confname ekgdata -solrhome /opt/solr/server/solr -confdir /opt/solr/ server/solr/configsets/ekgdata_configs/conf 3. Start solr on n-nodes connected to the zookeeper cluster $SOLR_HOME/bin/solr start c -z :2181, :2181, Create a collection with a number of shards and replicas

20 Example: Solr Cloud for Analytics of Insurance Data IBM Watson insurance sample data containing the following fields Education Gender Income...

21 DEMO

22 Executing Facet Queries

23 Term Facets Group and Count a Single Field. Apache Big Data North America Vancouver Johannes Weigend QAware GmbH 23

24 Function Facets Aggregate Fields. 24

25 Pivot Facets Compose Facets to Hierarchies. 25

26 Solr 6 Supports SQL Solr 6 supports distributed SQL The JDBC Driver is part of the solrj client library A collection is currently mapped as single table. Collection -> Table SolrDocument -> Row Field -> Column The Solr 6.0 JDBC Driver is very limited, but more functionality is expected in upcoming versions No database metadata, no prepared statements, no mapping to tables per type field

27 Resilience The number of replicas per shard is configurable (replication factor) This number corresponds with the number of nodes which can silently fail Zookeeper is the single source of failure, but can also be failsafe by running multiple instances Solr knows all zookeeper instances and can silently switch over to the next available leader if last connected zookeeper crashes

28 You Got Everything for Analytics Applications! Or Not? Client side processing of Solr documents does not scale No possibility to run parallel business logic inside Solr with a strong separation of concerns between Solr and your code The Solr index is not a general purpose store for huge data Images Videos Binaries / large text documents No Interface to machine learning or typical statistics libraries (R)...

29 Reduce Map Map Map Spark Filter Filter Filter Search Search Search Data flow Search / NoSQL Distributed In-Memory Computing mit Apache Spark

30 READ THIS: Distributed computing (100x faster than Hadoop (M/R) Distributed Map/Reduce on distributed data can be done in-memory Written in Scala (JVM) Java/Scala/Python APIs Processes data from distributed and non-distributed sources Textfiles (accessible from all nodes) Hadoop File System (HDFS) Databases (JDBC) Solr per Lucidworks API...

31 Driver Node Cluster JVM Driver Application JVM Master Host Master / Yarn / Mesos creates Spark Context Resilient Distributed Dataset RDD uses MasterURL Task Slave Slave JVM Worker JVM Worker start start JVM Executor JVM Partition Task(s) Executor Partition Task(s) Application JVM JVM Slave Worker start Executor Partition Task(s)

32 A Very First Spark Application

33 Spark Pattern 1: Distributed Task with Params

34 Spark Pattern 2: Distributed Read from External Sources

35 Spark Pattern 3: Caching and Further Processing with RDDs

36 DEMO

37 Reduce Map Map Map Spark Filter Filter Filter Search Search Search Datenfluss Search / NoSQL Putting All Together Solr & Spark in Action

38 How to Implement readfromshard()? There are several possibilities: SolrJ: SolrCloudStream /export Handler can stream mass data (with limitations) Supports only JSON (No binary or xml) Or: SolrJ cursor marks Or: Build your own custom export handler

39 LucidWorks has released a Spark/Solr Integration Library.

40 1 Lucidworks Solr-Spark Adapter V

41 Logfile Analytics with Solr and Spark Histogram of exceptions from hosts A,B,C during time interval D Step 1: Search with Solr Solr Query q=*exception AND (server: A OR server:b OR server:c) AND timestamp between [ , ] Step 2: Create a map key = << exception name >>, value = count

42

43 + DEMO

44 Specifications Intel NUC6i5SYK CPU 6th generation Intel Core i5-6260u processor with Intel Iris graphics (1.9 GHz up to 2.8 GHz Turbo, Dual Core, 4 MB Cache, 15W TDP) RAM 32 GB Dual-channel DDR4 SODIMMs 1.2V, 2133 MHz DISK 256 GB Samsung M.2 internal SSD Total 8 Cores, 16 HT Units, 128 GB RAM, 1 TB Disk! This case is as powerful like four notebooks Apache Big Data North America Vancouver Johannes Weigend QAware GmbH

45 Technical Cluster Architecture Zeppelin Zookeeper #1 Spark Spark Master JVM #4 Slave JVM #4 Master JVM Slave JVM Executor JVM #4 Solr Cloud 4 1 Executor JVM #1 Solr Cloud s13 s14 s15 s16 s1 s2 s3 s4 Ubuntu Linux Ubuntu Linux hdfs Spark Zookeeper #3 Zeppelin Spark Zookeeper #2 Master JVM #3 Slave JVM #3 Master JVM #2 Slave JVM #2 Executor JVM #3 Solr Cloud 3 2 Executor JVM #2 Solr Cloud s9 s10 s11 s12 s5 s6 s7 s8 Ubuntu Linux Ubuntu Linux

46 You Can Build a Solr/Spark Cloud on Odroid 70$ Computers ODROID XU4: 8 Cores, 2GB RAM, 64 GB emmc Disk ~1/10 CPU performance in comparison to Intel NUC 6 / Core i5 Apache Big Data North America Vancouver Johannes Weigend QAware GmbH

47 40 Cores 10 GB RAM 320 GB emmc Disk SPARK Worker SOLR 5.3 SPARK Worker SOLR 5.3 Odroid XU4 2 GB RAM 64 GB emmc Disk Ubuntu Linux 70$ SPARK Master SPARK Worker SOLR 5.3 ZOOKEEPER SPARK Worker SOLR 5.3 SPARK Worker SOLR

48

49 Summary Solr Cloud / Spark are a powerful combination for interactive analytics and data intense applications Writing distributed software stays hard. Only distribute if you have to. 100% Open Source A simple integration of Solr and Spark is easy. For high performance applications things could be challenging. If professional product support is needed, customers can switch to Lucidworks Fusion to get a pre integrated and supported Solr/Spark platform

50 slideshare.net/qaware blog.qaware.de

51 51