Enhancing Search with Predictive Analytics

Transcription

1 Enhancing Search with Predictive Analytics Andrew Fast Chief Scientist Elder Research, Inc. Text Analytics World Boston 2013

2 The Elephant Test It is difficult to describe, but you know it when you see it. Lord Justice Stuart Smith, Cadogan Estates Limited v. Morris (1998) Likewise, most textual concepts cannot be easily defined with a single keyword query Quote from: h,p://

3 Combining Search and Predictive Models Search and Predictive modeling each provides a different trade-off between power and generality. Document Classification A predicive model can answer one query well, especially a complex query Power Generality Keyword Search Keyword queries can answer any query, but with limited depth for complex queries.

4 Our Approach A search ensemble ranking function that boosts keyword relevance based on a predictive model Keyword Relevance High Keyword Relevance, Low Model Ranking Low Keyword Relevance, Low Model Ranking High Keyword Relevance, High Model Ranking Low Keyword Relevance, High Model Ranking Model Ranking

5 The Problem The Goal: Explore NEW interesting ideas using OLD social entrepreneurship contest entries The Data: A collection of contest entries from 19 different contests sponsored by our client Contests cover a range of topics such as health, education, literacy, finance, technology, and geotourism. The Challenge: Emphasize high-quality entries in the results as entry quality varies widely

6 Combining Search and Predictive Models Keyword ranking does not help you find highquality entries but Model Ranking is not topic centric. Complimentary strengths Search for exploration and discovery Predictive Models for trends and correlations

7 THE MODEL

8 Target Variable Identify characteristics of past entries that are correlated with that proposal being Shortlisted by the Contest Judges Rankings: 1 Likely Finalist 2 Top Tier 3 Honorable Mention 4 Passed Screening 5 No Shortlisted Note: Not every contest used all 5 rankings

9 The Inputs Learn a logistic regression model to fit the feature weights Inputs: Taxonomy Textual Features Structured Data Budget Size Maturity Impact Auto- tagging taxonomy terms Length Lexical Diversity

10 The Taxonomy Joint work with Beth Maser and Richard Iams at PPC Non-traditional, general approach Broad, flexible taxonomy Focus on the range of interests of the organization

11 Using the Taxonomy Each contest emphasizes different branches of the taxonomy Taxonomy features need to be contest specific Step 1: Use the Wisdom of Crowds to find the center of each contest Step 2: Rate each entry based on the distance from the center

12 Evaluation: Area Under the ROC Evaluate the overall ranking provided by the model. Higher means more Shortlisted entries at the top of the list

13 Evaluation: Lift Evaluates the improvement using the model at a fixed amount of work How much more efficient are the judges using our model alone? Every contest showed positive lift. Maximum lift of 3.3 Average lift of 1.67

14 THE SEARCH APPROACH

15 Our Approach A new search ranking function that boosts keyword relevance for probable shortlisted entries Keyword Relevance High Keyword Relevance, Low Model Ranking Low Keyword Relevance, Low Model Ranking High Keyword Relevance, High Model Ranking Low Keyword Relevance, High Model Ranking Model Ranking

16 The Prototype Platform Custom Search Interface Search Index Model (PredicIve + Taxonomy) ERI Text Mining Data

17 Faceted Search with Solr Apache Solr is an open-source faceted search engine (

18 Blind Men and the Elephant Text mining can be viewed from many different perspectives No single view provides a complete solution Must consider the entire beast to get the best solution

19 Contact Information Andrew Fast, Ph.D. Chief Scientist (434)

20 Practical Text Mining Winner of the 2012 PROSE award for Computing and Information Science Written for a technical audience seeking more text experience Includes trial versions of major software tools

21 Andrew Fast" Chief Scientist, Elder Research, Inc. Dr. Andrew Fast leads research in Text Mining and Social Network Analysis at Elder Research, the nation s leading data mining consultancy. ERI was founded in 1995 and has offices in Charlottesville VA and Washington DC, ( ERI focuses on Federal, commercial, investment, and security applications of advanced analytics, including stock selection, image recognition, biometrics, process optimization, cross-selling, drug efficacy, credit scoring, risk management, and fraud detection. Dr. Fast graduated Magna Cum Laude from Bethel University and earned Master s and Ph.D. degrees in Computer Science from the University of Massachusetts Amherst. There, his research focused on causal data mining and mining complex relational data such as social networks. At ERI, Andrew leads the development of new tools and algorithms for data and text mining for applications of capabilities assessment, fraud detection, and national security. Dr. Fast has published on an array of applications including detecting securities fraud using the social network among brokers, and understanding the structure of criminal and violent groups. Other publications cover modeling peer-to-peer music file sharing networks, understanding how collective classification works, and predicting playoff success of NFL head coaches (work featured on ESPN.com). With John Elder and other co-authors, Andrew has written a book on Practical Text Mining, that was awarded the prose Award for Computing and Information Science in