The Value of Advanced Data Integration in a Big Data Services Company. Presenter: Flavio Villanustre, VP Technology September 2014

Transcription

1 The Value of Advanced Data Integration in a Big Data Services Company Presenter: Flavio Villanustre, VP Technology September 2014

2 About LexisNexis We are among the largest providers of risk solutions in the market today LexisNexis Reed Elsevier is a world leading provider of information solutions. LexisNexis Risk Solutions has seen sustained revenue and profit growth LexisNexis Risk Solutions is a leading provider in the U.S. across Business Services, Insurance and Government segments. LexisNexis Risk Solutions Revenue: $M LexisNexis Risk Solutions Revenue by Segment Government 8% Business Services: 39% Insurance: 53% The Value of Advanced Data Integration in a Big Data Services Company 2

3 About LexisNexis We are among the largest providers of risk solutions in the market today LexisNexis LexisNexis Risk Solutions is a leading provider in the U.S. across Business Services, Insurance and Government segments. LexisNexis Risk Solutions Revenue by Segment Our customers include: 99 of the top 100 US banks 90% of the Fortune % of US P&C insurance carriers All 50 US states, 70% of local governments and 80% of US federal agencies 97 of Am Law 100 firms Government 8% Business Services: 39% Insurance: 53% The Value of Advanced Data Integration in a Big Data Services Company 3

4 About LexisNexis We have a unique set of capabilities: Big data, linking, analytics, and product development Big Data Technology Vast Data Resources Linking & Analytics Industry-Specific Expertise & Delivery = Customer-Focused Solutions Speed Capacity Cost savings Process Sources Coverage Advanced linking & analytics Accuracy & efficiency Protect private information Aligned with our customers industries Deep industry expertise Predict, manage and assess risk across many industries. The Value of Advanced Data Integration in a Big Data Services Company 4

5 Entity Resolution in practice Examples of some questions we answer for our customers: 1. Are you who you say you are? 2. Who else might you be, or claim to have been in the past? 3. What other people and/or assets are associated with you? 4. What kind of a risk do you represent in a given context? 1. in calculating your insurance premium, or 2. in processing your claim, or 3. in granting you access to credit, or in doing business with you as a vendor or customer 5. Can I quantify the risk that you represent in the form of a score? 6. Which of these millions of transactions should I look at in case there s something suspicious? 7. What small subset of these 1000s (or millions) of events have something in common - which will cause me to look more closely without wasting time on false positives? The Value of Advanced Data Integration in a Big Data Services Company 5

6 It all starts with data LexisNexis collects many different types of data, from sources such as: Bureaus, utility connections, and student lists. Directory assistance and cell phones. Motor vehicle, watercraft, and aircraft registrations. Property deeds, assessments, and foreclosures. Driver s licenses, professional licenses, and voter registrations. Corporation and UCC filings, and other business registrations. Bankruptcy filings, liens, judgments, and accident reports. Departments of Corrections, criminal courts, and watch lists. The Value of Advanced Data Integration in a Big Data Services Company 6

7 And significant volumes of it Access to over approximately 25 billion public record filings Break-down of record counts for the more popular data sets: Data Source # of records Data Source # of records Associates/Relatives 1.8 Billion People at Work 1.5 Billion Bankruptcy 23 Million Private Phones 172 Million Business BDID's 283 Million Professional Licenses 94 Million Business People Links 959 Million Property 2.5 Billion Canadian Phones 62 Million Sex Offenders 550,000 Consumer Header 10.8 Billion SSN's 7.2 Billion Criminal 216 Million Student Records 38 Million Date of Birth 5.2 Billion TIN 2.9 Million Death 98 Million Unique ADLs - active 257 Million Drivers Licenses 397 Million Utility 645 Million EDA Phones 124 Million Vehicle Titles 635 Million FEINs 10.4 Million Vehicle Registrations 2.5 Billion Historical Phones 800 Million White Pages 116 Million Hunting and Fishing Licenses 67 Million Wireless Phones 101 Million Liens and Judgments 244 Million Yellow Pages 14 Million The Value of Advanced Data Integration in a Big Data Services Company 7

8 How do we harness all this content? We build a Header File A credit header contains the following fields derived from financial transactions: Name Name variations Address Previous addresses Telephone number DOB Social Security number The LN Master Person Header contains the same fields, but derived from many more sources: Credit Header Property records Bankruptcy filings Vehicle registrations Voter registrations Driver s licenses Utility information Phone records The Value of Advanced Data Integration in a Big Data Services Company 8

9 We can therefore build unique identifiers such as LexisNexis LexID : Non-SSN, non-fein-dependent identifier that brings together information from disparate databases offering an up-to-date unique profile for an individual or a business. Securely and intelligently analyzes billions of partial and complete records. Filters and links that information based on relevance to connect seemingly unrelated data. Faster access and better results/match-rates due to pre-linking of data. Addresses Government and societal concerns regarding collection and usage of SSN. The Value of Advanced Data Integration in a Big Data Services Company 9

10 The data linking/integration flow The Value of Advanced Data Integration in a Big Data Services Company 10

11 The LexisNexis Open Source HPCC Systems platform WHT/ The Value of Advanced Data Integration in a Big Data Services Company 11

12 Detailed HPCC Systems Platform Architecture WHT/ The Value of Advanced Data Integration in a Big Data Services Company 12

13 Enterprise Control Language (ECL) Declarative programming language: Describe what needs to be done and not how to do it Powerful: High level data primitives as JOIN, TRANSFORM, PROJECT, SORT, DISTRIBUTE, MAP, etc. are available. Extensible: As new attributes are defined, they become primitives that other programmers can use Implicitly parallel: Parallelism is built into the underlying platform. The programmer needs not be concerned with it Maintainable: A high level programming language, without side effects and with efficient encapsulation, programs are more succinct, reliable and easier to troubleshoot Complete: ECL provides for a complete data programming paradigm Homogeneous: One language to express data algorithms across the entire HPCC platform, data integration, analytics and high speed delivery WHT/ The Value of Advanced Data Integration in a Big Data Services Company 13

14 Enterprise Control Language (ECL) The ECL optimizer generates an optimal execution plan Sub-graphs run activities in parallel, automatically persist the results and spill to disk as needed Lazy evaluation avoids re-computing results when data and code haven t changed Graphs are dynamic and display the number of records traversing the edges and data skew percentages WHT/ The Value of Advanced Data Integration in a Big Data Services Company 14

15 Scalable Automated Linking Technology (SALT) The acronym stands for Scalable Automated Linking Technology Declarative linking paradigm Compiles to ECL, which compiles to C++ Provides for automated data profiling, QA/QC, parsing, cleansing, normalization and standardization Sophisticated specificity based linking and clustering Links entities to attributes Data Sources and entities together Iterative process leveraging inference gained in prior steps Profiling Parsing Cleansing Normalization Standardization Data Preparation Processes (ETL) 42 Lines of SALT 3,980 Lines of ECL 482,410 Lines of C++ Matching Weights and Threshold Computation Blocking/ Searching Weight Assignment and Record Comparison Record Match Decision Linked Data File Additional Data Ingest Linking Iterations Record Linkage Processes The Value of Advanced Data Integration in a Big Data Services Company 15

16 Rules based vs. probabilistic record linkage Through statistical methodologies, SALT has the two significant advantages over rules: 1. Recall INPUT SALT Match, because the system has learnt that Villanustre is specific because the frequency of occurrence is small and there is only one present in Atlanta Flavio Villanustre, Atlanta Javio Villanustre, Atlanta Record 1 Record 2 Error RULES NO MATCH, because the rules determine that Flavio and Javio are not the same 2. Precision INPUT SALT NO Match, because the system has learnt that John Smith is not specific because the frequency of occurrence is large and there are many present in Atlanta John Smith, Atlanta John Smith, Atlanta Error RULES MATCH, because the rules determine that John Smith and the city for both the records match The Value of Advanced Data Integration in a Big Data Services Company 16

17 Entity linkage and associations SALT also links entities to form networks of relationships To clustering From disparate data To showing relationships The Value of Advanced Data Integration in a Big Data Services Company 17

18 LexID Linking A process that links together all records common to a single individual. Assigns a unique identifier (AKA LexID) to those records. Not rules based. Calculates a weight (the specificity) for each value of each field, relative to all of the other values for that field found within the data. Includes fuzzy matching and other capabilities to help find the best matches. Regional diversity or cultural variations can affect linking results. LexID John Doe Header data sources requires high standard of quality and reliability. The Value of Advanced Data Integration in a Big Data Services Company 18

19 Putting the big picture together Business Logic Step 1 Step 2 Request 1 or more Lex ID Pull Data PII [{ID, Score}, ] Linking Subsystem Source Data The Value of Advanced Data Integration in a Big Data Services Company 19

20 Why is Linking a Big Data problem? Candidate File 73 separate blocking criteria. 55 hours of joins across 400 machines. Every possible match is scored for specificity. 11 Billion match candidates identified. Match By Date of Birth and locale 10 Billion Records Match Candidates 16 Million Matches 400 Machines 2 hours Comparisons The Value of Advanced Data Integration in a Big Data Services Company 20

21 Linking is resilient to new data sources As records and data-sources are merged into the consumer database, the total number of identities continues to track with the population Millions 12,000 10,000 8,000 6,000 4,000 Total Records Total Identities Core Identities 2,000 0 Months During this 2-year period: Almost doubled the number of records The number of Lex IDs remained stable Core Lex ID count on track (3% net increase) The Value of Advanced Data Integration in a Big Data Services Company 21

22 All other data services naturally derive from the core linked data Source Data Matched Data Sets Scores & Attributes Primary attributes Tax Liens, Felonies, Bankruptcies Summarize a particular characteristic of a consumer Property Deeds Examples include: Professional Licenses Landline & Cell Phone Court Judgments Compiled records for more than 250M identities Number of addresses Tax assessed residence value Composite attributes Summarize aggregate consumer behavior (wealth, income, mobility) Voter Registration Created by combining primary attributes Bureau Header Data Scores Other (education, etc) Industry specific scores Custom scores The Value of Advanced Data Integration in a Big Data Services Company 22

23 Useful Links LexisNexis Open Source HPCC Systems Platform: Free Online Training: SALT: Machine Learning portal: The HPCC Systems blog: Community Forums: Our GitHub portal: JIRA: The Value of Advanced Data Integration in a Big Data Services Company 23

24 Questions? Thank you! The Value of Advanced Data Integration in a Big Data Services Company 24