Leveraging the Power of SOLR with SPARK. Johannes Weigend QAware GmbH Germany pache Big Data Europe September 2015

Transcription

1 Leveraging the Power of SOLR with SPARK Johannes Weigend QAware GmbH Germany pache Big Data Europe September 2015

2 Welcome Johannes Weigend - CTO QAware GmbH - Software architect / developer - 25 years of experience - Custom enterprise solutions (Java, JS, ) - Lecturer for UI development at the University of Applied Science in Rosenheim - Focus on performance and scalability - SOLR user since

3 Advantages of Search Based M/R Brute Force Map/Reduce Input Filter Map Reduce Search based Map/Reduce Input Search Map Reduce 3

4 Agenda SOLR cloud Demo SPARK cluster Demo Importing data into SOLR with SPARK Demo Analysis with SOLR and SPARK Demo 4

5 Horizontally scalable, distributed NoSQL (Index) Database Document oriented A document is a collection of fields (string, number, date, ) Simple and multiple fields (similar to arrays) Schema and schema less Powerful query language (Lucene) Distributed data in shards Replication Powerful full text search capabilities Aggregation functions (aka facets) Stable > V 5.3 5

6 AIR Aftersales Information Research ZEBRA Part explosion for complex products EKG Software Electro Cardiogram QAsearch Enterprise search across all repositories including history 6

7 7

8 8

9 9

10 Apache SOLR for BigData Analysis? Distributed Processing? Text Search Engine? Aggregations? Slice and Dice? 10

11 Demo: SOLR Cloud Installing and configuring SOLR Cloud Searching, sorting and filtering Facets Terms (count by term) Ranges (count in range) Functions (avg, sum, ) Sub-Facets (pivot) 11

12 Counting as Term Facet 12

13 Statistics as Function Facet 13

14 Pivots as Sub Facets 14

15 careerbuilder.com 15

16 Banana 16

17 17

18 What s Missing? Client-side processing of SOLR results does not scale No built-in M/R support Where to store really big data? Images Videos Binaries / large text documents No interfaces to R / ML 18

19 Distributed job execution engine Map/Reduce framework Scala based (runs on JVM) Java/Scala/Python APIs Processes data from various data sources Textfiles (accessible from all nodes) Hadoop File System (HDFS) Databases (JDBC) SOLR! Must Read: 19

20 Combining Spark with SOLR Use Cases Distributed ETL Importing data into SOLR- Cloud Our Usecase: importing N logfiles into SOLR Distributed processing data analysis Statistics on binary data Map/Reduce 20

21 Four Ways to Import Data into SOLR 1. Using built-in functions post script Dataimport handler, Admin-UI 2. Writing custom parallel code using the SOLRJ API 3. Using and customizing Apache Nutch (Hadoop!) 4. Using and customizing Apache Spark 21

22 Demo: Import Logfiles with Spark Writing a Spark job which imports a bunch of logfiles in one directory Using Lucidwork s Solr-Spark library 22

23 23

24 Demo: Distributed Analysis with Spark Write a Spark Job which calculates the Duration of Business Actions Use Spark to access SOLR per SQL / JDBC 24

25 SolrRDD - The Spark Abstraction to process SOLR Results 25

26 SPARK Supports Parallel SQL 26

27 Dataframe API 27

28 40 Cores 10 GB RAM 320 GB emmc Disk SPARK Master SPARK Worker SOLR 5.3 SHARD #0 ZOOKEEPER NFS SPARK Worker SOLR 5.3 SHARD #1 Odroid XU4 2 GB RAM 64 GB emmc Disk Ubuntu Linux 70$ SPARK Worker SOLR 5.3 SHARD #2 SPARK Worker SOLR 5.3 SHARD #3 SPARK Worker SOLR 5.3 SHARD #4 28

29 Summary 29

30 Any Questions? 30