RANDOM PROJECTIONS FOR SEARCH AND MACHINE LEARNING

Transcription

1 = + RANDOM PROJECTIONS FOR SEARCH AND MACHINE LEARNING Stefan Savev Berlin Buzzwords June 2015

2 KEYWORD-BASED SEARCH Document Data 300 unique words per document words in vocabulary Data sparsity: 99.9% Words are strong query features STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

3 IMAGE SEARCH BY EXAMPLE Image Data 800 pixels 200 black pixels Data sparsity 80% Pixels are weak query features STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

4 TWO KINDS OF SEARCH Data Sparse Dense Query Size Short Long Use Cases Keyword Search Image Search, Semantic Search, Recommendations SEARCH METHOD INVERTED INDEX (LUCENE) RANDOM PROJECTIONS STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

5 OUTLINE 1. High dimensional data (images, text, clicks) 2. Random Projections: Why? What? How? 3. Image Search Benchmark STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

6 HIGH-DIMENSIONAL DATA Images STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

7 HIGH-DIMENSIONAL DATA Images Pixels STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

8 HIGH-DIMENSIONAL DATA Images Pixels Dimensions STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

9 MATRIX REPRESENTATION OF DATASET STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

10 MATRIX REPRESENTATION OF DATASET STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

11 HIGH-DIMENSIONAL DATA Text A beer can do everyone a favor. a abstract be bee beer can do everyone Dimensions STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

12 HIGH-DIMENSIONAL DATA Clicks STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

13 DENSE VS SPARSE MATRIX IMAGES 1000 PIXELS TEXT/CLICKS WORDS img_1 img_3 doc_1 doc_3 img_n doc_n CANNOT SEARCH WITH LUCENE CAN SEARCH WITH LUCENE STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

14 DENSE VS SPARSE MATRIX IMAGES 1000 PIXELS TEXT/CLICKS WORDS TEXT/CLICKS 500 DIMENSIONS img_1 img_3 doc_1 doc_3 img_n doc_n CANNOT SEARCH WITH LUCENE CAN SEARCH WITH LUCENE WITH LUCENE STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

15 DENSE VS SPARSE MATRIX IMAGES 1000 PIXELS TEXT/CLICKS WORDS TEXT/CLICKS 500 DIMENSIONS img_1 img_3 doc_1 doc_3 img_n doc_n CANNOT SEARCH WITH LUCENE CAN SEARCH WITH LUCENE WITH LUCENE WITH RANDOM PROJECTIONS WITH RANDOM PROJECTIONS WITH RANDOM PROJECTIONS STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

16 DIMENSIONALITY REDUCTION beer wine beer + wine doc_1 doc_3 SVD doc_n dimensions 500 dimensions STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

17 DIMENSIONALITY REDUCTION = PATTERN DISCOVERY = + STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

18 APPLICATIONS Random Forest Search (Nearest Neighbor) Random Projections SVM SVD Boosting STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

19 RANDOM PROJECTIONS IN INDUSTRY Spotify for music recommendations Etsy for user/product recommendations Style in the Long Tail: Discovering Unique Interests with Latent Variable Models in Large Scale Social E-commerce, Diane Hu, Rob Hall, Josh Attenberg Referred to as Locality Sensitive Hashing STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

20 OUTLINE 1. High dimensional data (images, text, clicks) 2. Random Projections: Why? What? How? 3. Image Search Benchmark STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

21 HOW DOES IT WORK? = + STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

22 ILLUSTRATION ON ARTIFICIAL DATASET id_1 id_3 id_n STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

23 ILLUSTRATION ON ARTIFICIAL DATASET Nearest neighbor problem: Find the closest point STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

24 ILLUSTRATION ON ARTIFICIAL DATASET Approximation: Neighbors are in a circle with small radius STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

25 ILLUSTRATION ON ARTIFICIAL DATASET Core idea: random grids Dynamic grids (variably sized) Random = cheap STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

26 PARTITION BY RANDOM SPLITS Hyperplane splits the dataset STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

27 PROJECTION = ONE DIMENSIONAL VIEW OF DATASET A Random Direction (1 Dimension) STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

28 PLATO S ALEGORY OF THE CAVE A Random Direction (1 Dimension) STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

29 PARTITIONING BY PROJECTING ON RANDOM LINES STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

30 PARTITIONING BY PROJECTING ON RANDOM LINES STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

31 ALGORITHM VISUALIZATION STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

32 ALGORITHM VISUALIZATION STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

33 ALGORITHM VISUALIZATION STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

34 ALGORITHM VISUALIZATION STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

35 ALGORITHM VISUALIZATION When two points are close, they are likely to end up in the same partition, even under random partitioning STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

36 ALGORITHM VISUALIZATION When two points are close, they are LIKELY to end up in the same partition, even under random partitioning STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

37 MULTIPLE TREES Random tree partitioning is noisy. Build more trees. STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

38 ADDING MORE TREES Tree 1 Trees 1 and 2 Trees 1, 2 and 3 STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

39 ADDING MORE TREES Trees 1-50 Trees STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

40 CONSTRAINING THE NEAREST NEIGHBOR REGION THRESHOLD = In how many trees a NN candidate appears Threshold = 1 Threshold = 60 Threshold = 20 STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

41 WE VE GOT HIM STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

42 OUTLINE 1. High dimensional data (images, text, clicks) 2. Random Projections: Why? What? How? 3. Image Search Benchmark STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

43 IMAGE SEARCH FOR PREDICTION STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

44 IMAGE SEARCH FOR PREDICTION Results Image Label STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

45 STAGE 1: INDEXING examples with labels 784 pixels per image STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

46 STAGE 1: INDEXING examples with labels 784 pixels per image STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

47 STAGE 1: INDEXING examples with labels 784 pixels per image index = Index(...) for image in training_examples: labels[image.id] = image.label index.add_item(image.id, image.vector) index.build(...) STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

48 STAGE 2: SEARCH TO PREDICT THE LABEL Results Image Label index.load('.../file.index')... image_vec = read_image(...) nn = 100 #number of nearest neighbors results = index.get_nns_by_vector(image_vec,nn) top_result = results[0] predicted_label = labels[top_result.id] test examples STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

49 TOOLS stefansavev.com/randomtrees STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

50 BENCHMARK IMAGE SEARCH DIMENSIONS # TREES # NN CANDIDATES ACCURACY ON TEST SEARCH TIME [ms/query] BOTTLENECK Reference: STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

51 BENCHMARK IMAGE SEARCH DIMENSIONS # TREES # NN CANDIDATES ACCURACY ON TEST , after SVD SEARCH TIME [ms/query] Reference: STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

52 BENCHMARK IMAGE SEARCH DIMENSIONS # TREES # NN CANDIDATES ACCURACY ON TEST , after SVD SEARCH TIME [ms/query] BOTTLENECK Reference: STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

53 BENCHMARK IMAGE SEARCH stefansavev.com/ randomtrees DIMENSIONS # TREES # NN CANDIDATES ACCURACY ON TEST , after SVD 100, after SVD (mean) SEARCH TIME [ms/query] Reference: STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

54 SAME FOUNDATION FOR SEARCH AND MACHINE LEARNING + Data point id STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

55 SAME FOUNDATION FOR SEARCH AND MACHINE LEARNING Histogram + label frequency STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

56 CONCLUSION = + Search in Dense Data CHEAP method to explore data; Makes algorithms FASTER (e.g. SVD, SVM) STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

57 THANK YOU! Stefan Savev Blog: stefansavev.com STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

58 EXTRA SLIDES STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

59 REFERENCES STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

60 ADVANTAGES/DISADVANTAGES DIMENSIONALITY REDUCTION Advantages Semantic Similarity Concentrate the signal Disadvantages No exact matches possible Adds noise STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

61 RANDOM PROJECTIONS AS HASH FUNCTIONS * = = -1 STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

62 EXAMPLE-BASED IMAGE SEARCH MNIST image dataset of digits Images are digits from 0 to images for training images for test 28 x 28 pixels with values from 0 to (=28*28) features Dataset is already preprocessed Goal is to predict the label of an image STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

63 SOME IMPLEMENTATION TRICKS sparse random projections combine not all dimension but a small number of them apply dimensionality reduction first pick better random projections the projected histogram is wide project on multiple lines simultaneously Hadamard matrix trick reuse random projections via recombination cut out a ball from the densest region PCA may help use the neighbors of neighbors of a point as additional similarity candidates advanced search inside the trees with backtracking (KD-tree style) generate multiple versions of query/training data (i.e. for image data translate or rotate the image) [Link to Blog] STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

64 PROJECTION Dot product (cosine, similarity) View of the dataset Overlap with Pattern Geometry of Search Foundation for Search and Machine Learning STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

65 WHY RANDOM? Cheap (replace optimization with randomization); Simply build more trees If we build the perfect tree, it s just one tree. We need more and DIVERSE trees In high dimensions all algorithms will have problems, so do the cheapest With a few points/dimensions random is not the best choice STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

66 DIMENSIONALITY REDUCTION doc_1 doc_3 doc_n beer wine SVD beer + wine MIXING THE COLUMNS OF THE ORIGINAL MATRIX TO MAXIMIZE SIGNAL TO NOISE WORDS 500 DIMENSIONS STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

67 PROJECTION = OVERLAP WITH A PATTERN OVERLAP, =SUM = 1, more red -1, more green STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

68 PROJECTION = OVERLAP WITH A PATTERN OVERLAP, =SUM STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

69 PROJECTION = OVERLAP WITH A PATTERN OVERLAP (, ) = SUM STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

70 PROJECTION = OVERLAP WITH A PATTERN OVERLAP (, ) = SUM = 1, if more red pixels than white -1, if more white pixels than red STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

71 PROJECTION = OVERLAP WITH A PATTERN OVERLAP (, ) = SUM = 1, if more red pixels than white -1, if more white pixels than red STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

72 DATA POINT SVD FEATURE RANDOM FEATURE STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

73 PLATO S ALEGORY OF THE CAVE He then explains how the philosopher is like a prisoner who is freed from the cave and comes to understand that the shadows on the wall do not make up reality at all, as he can perceive the true form of reality rather than the mere shadows seen by the prisoners Source: STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE

74 SECRET SAUCE How to make it faster? How to make it more accurate? When does it work? Is there something better? STEFAN SAVEV ~ RANDOM PROJECTIONS FOR BIG DATA ~ BERLIN BUZZWORDS ~ JUNE