Search-based business intelligence and reverse data engineering with Apache Solr

Transcription

1 Search-based business intelligence and reverse data engineering with Apache Solr M a r i o - L e a n d e r R e i m e r C h i e f T e c h n o l o g i s t

2 This talk will o Give a brief overview of the AIR system s architecture o Show reverse data engineering using Solr and MIR o Talk about the fight for our right to Solr o Describe solutions for the problem of combinatorial explosion o Outline a flexible and lightweight ETL approach for Solr Mario-Leander Reimer 2

3 A <<System>> AIR Central I <<Subsystem>> Spring Framework I <<Subsystem>> JEE 5 WS Clients A <<Ext. System>> AIR Bus I <<Ext. System>> Backend Systems Vehicles Service Bulletins Independent Workshop A <<Client>> Browser Search and Display Retrofits JSF Web UI REST API Protocoll Watchlist File Storage AIR DB Document Storage Mechanic A <<Anwendungscluster>> A <<Client>> Masterdata Documents AIR Repository AIR Client I <<Subsystem>> I <<Subsystem>> Measures Solr Access Apache Solr.NET WPF Solr Extensions Defects Maintenance Launch Parts Flat Rates 20 Languages 800 GB Solr Index Service Technician A <<Ext. System>> 3rd Party Application AIR Fork DLL AIR Call DLL Query A <<Application Cluster>> AIR Loader I <<Subsystem>> Spring Framework Vehicles Service Bulletins Load Execute A <<System>> AIR Control I <<Subsystem>> Jenkins Maintenance Defects A <<Ext. System>> 3rd Party ios App Flat Rates Documents AIR ios Lib Repair Overview... Backend Databases and Systems Mario-Leander Reimer 3

4 A <<Anwendungscluster>> AIR Repository 20 Languages 800 GB Solr Index I <<Subsystem>> Apache Solr Slave Solr Extensions Replicate I <<Subsystem>> Apache Solr Master Solr Extensions 20 Languages 800 GB Solr Index A <<Application Cluster>> AIR Loader A <<System>> AIR Control I <<Subsystem>> Spring Framework Load I <<Subsystem>> Jenkins Vehicles Service Bulletins Execute Maintenance Defects Flat Rates Repair Overview Documents... Backend Databases and Systems Search A <<System>> MIR Mario-Leander Reimer 4

5 Let s go back to when it all began Source: 5

6 The project vision: find the right information in less than 3 clicks. The situation: o Users had to use up to 7 different applications for their daily work. o Systems were not really integrated nicely. o Finding the correct information was laborious and error prone. The idea: o Combine the data into a consistent information network. o Make the information network and its data searchable and navigable. o Replace existing application with one easy to use application. Mario-Leander Reimer 6

7 But how do we find the originating system for the desired data? System A System B System C System D Where to find the vehicle data? Vehicle data 60 potential systems and 5000 entities. Other data Mario-Leander Reimer 7

8 And how do we find the hidden relations between the systems and their data? System A System B System C System D How is the data linked to each other? Customer Vehicle potential relations. Documents Other data Mario-Leander Reimer 8

9 Meta Information Research (MIR) Source: 9

10 MIR is a simple and lightweight data reverse engineering and analysis tool based on Solr. o MIR manages meta information about the source systems (the data models and record descriptions) o MIR allows to navigate and search in the metadata, you can drill into the metadata using facets o MIR also manages the target data model and Solr schema description A <<System>> Meta Information Research 25MB MIR User Interface Metadata Index I <<Subsystem>> Apache Solr MIR Loader Read Backend Databases and Systems Sources (Java, XML) Magic Draw MIR Generators Mario-Leander Reimer 10

11 Facetted drill down Wildcard queries Search results Tree view of systems, tables and attributes Found potential synonyms for the chassis number 11

12 Solr entities for each release Solr schema attributes EAT YOUR OWN DOG FOOD. The AIR Solr schema definition is modelled and defined within MIR. 12

13 def sourcegenerator = MIR + Solr + Maven; Mario-Leander Reimer 13

14 But Solr is a full text search engine. You have to use an Oracle DB for your application data! NO! 14

15 Some of the AIR requirements were... o Focus is on search. Transactions are not required. o High demands on request volume and performance. o Free navigation on data model and content. o Support for full text search and facetted search. o Offline capabilities. o Scalability from low-end device to server to cloud. Mario-Leander Reimer 15

16 Apache Solr outperformed Oracle significantly in query time as well as index size. SELECT * FROM VEHICLE WHERE VIN='V%' INFO_TYPE:VEHICLE AND VIN:V* 383 ms 384 ms 383 ms 038 ms 000 ms 000 ms SELECT * FROM VEHICLE WHERE VIN='%X%' INFO_TYPE:VEHICLE AND VIN:*X* 389 ms 387 ms 386 ms 092 ms 000 ms 000 ms SELECT * FROM MEASURE WHERE TEXT='engine' INFO_TYPE:MEASURE AND TEXT:engine 859 ms 379 ms 383 ms 039 ms 000 ms 000 ms Test data set: records Disk space: 132 MB Solr vs. 385 MB Oracle Mario-Leander Reimer 16

17 Dirt Race Use Case: o Low-end devices o No Internet 28. Source: September

18 Running Solr and AIR-2-Go on Raspberry Pi Model B worked like a charm. Running Debian Linux + JDK8 Jetty Container with the Solr and AIR WARs deployed Reduced Solr data set with approx ~1.5 Mio documents Model B Hardware Specs: o ARMv6 CPU at 700Mhz o 512MB RAM o 32GB SD-Card Mario-Leander Reimer 18

19 YOU GOTTA FIGHT FOR YOUR RIGHT TO SOLR! 19

20 No silver bullet. A careful schema design is crucial for your Solr performance. Mario-Leander Reimer 20

21 Naive data denormalization can quickly lead to combinatorial explosion. m FRU Groups Flat Rate Units m Packages n n n Measures Repair Instructions Vehicles m m n n Types m n n Technical Documents Parts n Fault Indications Num Docs: Relationship Navigation Mario-Leander Reimer 21

22 Multi-valued fields can efficiently store 1..n relations but may result in false positives. { } "INFO_TYPE":"AWPOS_GROUP", "NUMMER" :[ " ", " " ] "BAUSTAND" :[" T23:00:00Z"," T23:00:00Z"] "E_SERIES" :[ "F10", "E30" ] Index 0 Index 1 q=info_type:awpos_group AND NUMMER: AND E_SERIES:F10 q=info_type:awpos_group AND NUMMER: AND E_SERIES:E30 In case this doesn t matter, perform a post filtering in your application. Note: latest Solr versions support nested child documents. Use instead. Mario-Leander Reimer 22

23 Technical documents and their validity were expressed in a binary representation. o Validity expressions may have up to 46 characteristics. o Validity expressions use 5 different boolean operators (AND, NOT, ) o Validity expessions can be nested and complex. o Some characteristics are dynamic and not even known at index time. Solution: transform the validity expressions into the equivalent JavaScript terms and evaluate these terms at query time using a custom function query filter. Mario-Leander Reimer 23

24 Binary validity expression example. Type( ) = Brand, Value( ) = BMW PKW Type( ) = E-Series, Value( ) = F10 Type( ) = Transmission, Value( ) = MECH 24

25 Transformation of binary validity terms into their JavaScript equivalent at index time. AND(Brand='BMW PKW', E-Series='F10' Transmission='MECH') ((BRAND=='BMW PKW')&&(E_SERIES=='F10')&&(TRANSMISSION=='MECH')) { "INFO_TYPE": "TECHNISCHES_DOKUMENT", "DOKUMENT_TITEL": "Getriebe aus- und einbauen", "DOKUMENT_ART": " reparaturanleitung", "VALIDITY": "((BRAND=='BMW PKW')&&((E_SERIES=='F10')&&(...))", BRAND": [ BMW PKW"],... } Mario-Leander Reimer 25

26 Base64 decode The JavaScript validity term is evaluated at query time using a custom function query. &fq=info_type:technisches_dokument &fq=dokument_art:reparaturanleitung &fq={!frange l=1 u=1 incl=true incu=true cache=false cost=500} jsterm(validity,eyjnt1rpul9lukfgvfnut0zgqvjux01pve9sqvjcruluu 1ZFUkZBSFJFTiI6IkIiLCJFX01BU0NISU5FX0tSQUZUU1RPRkZBUlQiOm51bG wsilnjq0hfukhfsvrtrkfiulpfvucioiiwiiwiqu5uuklfqii6ikfxrcisikv kjbvvjfsuhfijoiwccifq==) { BRAND":"BMW PKW", "E_SERIES":"F10", "TRANSMISSION":"MECH" } Mario-Leander Reimer 26

27 How often do we load data? How do we ensure data consistency? 27

28 A traditional approach using a DWH and ETL: too inflexible, heavy weight and expensive. System A File DB ETL ETL jobs would usually be implemented with Informatica System B ETL ETL Data Warehouse ETL AIR Solr ETL System C ETL File DB Significant business logic required depending on the source database Mario-Leander Reimer 28

29 Start Build Run Flexible and lightweight ETL combined with Continuous Delivery and DevOps. H <<System>> AIR Loader Slave H <<System>> AIR Search Developer Solr Index Solr Index Build & Deploy I <<System>> Apache Solr A <<System>> AIR Loader Load Replicate I <<System>> Apache Solr I <<System>> Nexus Repository I <<System>> Apache Maven Extract I <<System>> Jenkins Slave Data Source A Operations I <<System>> Jenkins Master Data Source n Mario-Leander Reimer 29

30 30

31 Apache Solr has become a powerful tool for data analytics applications. Be creative. Our next big project using Apache Solr is already on its way. High performance application to predict and calculate the bill of materials for all required parts and orders. Apache Solr as a compressed, scalable and high performance time series database. FOSDEM 15 Florian Lautenschlager, QAware GmbH Leveraging the Power of SOLR and SPARK Apache: Big Data 2015 Johannes Weigend, QAware GmbH Mario-Leander Reimer 31

32 Business intelligence is about asking the right questions about your data. 32

33 And with Apache Solr you can search and find the answers you are looking for. 33

34 & Mario-Leander Reimer Chief Technologist, QAware GmbH