Reti di Calcolatori! Web Search

Transcription

1 Reti di Calcolatori! Web Search

2 Search use (iprospect Survey, 4/04,

3 Without search engines the web wouldn t scale 1. No incentive in creating content unless it can be easily found other finding methods haven t kept pace (taxonomies, bookmarks, etc) 2. The web is both a technology artifact and a social environment The Web has become the new normal in the American way of life; those who don t go online constitute an ever-shrinking minority. [Pew Foundation report, January 2005] 3. Search engines make aggregation of interest possible: Create incentives for very specialized niche players Economical specialized stores, providers, etc Social narrow interests, specialized communities, etc 4. The acceptance of search interaction makes unlimited selection stores possible: Amazon, Netflix, etc 5. Search turned out to be the best mechanism for advertising on the web, a $15+ B industry. Growing very fast but entire US advertising industry $250B huge room to grow Sponsored search marketing is about $10B

4 The coarse-level dynamics Advertisement Editorial Feeds Crawls Content creators Content aggregators Subscription Transaction Content consumers

5 Brief (non-technical) history Early keyword-based engines Altavista, Excite, Infoseek, Inktomi, ca Paid placement ranking: Goto.com (morphed into Overture.com Yahoo!) Your search ranking depended on how much you paid Auction for keywords: casino was expensive!

6 Brief (non-technical) history 1998+: Link-based ranking pioneered by Google Blew away all early engines save Inktomi Great user experience in search of a business model Meanwhile Goto/Overture s annual revenues were nearing $1 billion Result: Google added paid-placement ads to the side, independent of search results Yahoo follows suit, acquiring Overture (for paid placement) and Inktomi (for search)

7 Ads vs. search results Google has maintained that ads (based on vendors bidding for keywords) do not affect vendors rankings in search results Sponsored Links CG Appliance Express Discount Appliances (650) Same Day Certified Installation San Francis co-oakland-san Jose, CA Miele Vacuum Cleaners Miele Vacuums - Complete Sele ction Free Shipping! Miele Vacuum Cleaners Miele -Free Air shipping! All models. Helpful advice. -vacuum.com Search = miele Web Results 1-10 of about 7,310,000 for miele. (0.12 seconds) Miele, Inc -- Anything else is a compromise At the heart of your home, Appliances by Miele.... USA. to miele.com. Residential Appliances. Vacuum Cleaners. Dishwashers. Cooking Appliances. Steam Oven. Coffee System... www. miele.com/ - 20k - Cached - Similar pages Miele Welcome to Miele, the home of the very best appliances and kitchens in the world. www. miele.co.uk/ - 3k - Cached - Similar pages Miele - Deutscher Hersteller von Einbaugeräten, Hausgeräten... - [ Translate this page ] Das Portal zum Thema Essen & Geniessen online unter -tisch.d e. Miele weltweit...ein Leben lang.... Wählen Sie die Miele Vertretung Ihres Landes. www. miele.de/ - 10k - Cached - Similar pages Herzlich willkommen bei Miele Österreich - [ Translate this page ] Herzlich willkommen bei Miele Österreich Wenn Sie nicht automatisch weitergeleitet werden, klicken Sie bitte hier! HAUSHALTSGERÄTE... www. miele.at/ - 3k - Cached - Similar pages

8 Ads vs. search results Other vendors (Yahoo, MSN) have made similar statements from time to time Any of them can change anytime We will focus primarily on search results independent of paid placement ads Although the latter is a fascinating technical subject in itself

9 Web search basics Sponsored Links User CG Appliance Express Discount Appliances (650) Same Day Certified Installation San Francisco -Oakland-San Jose, CA Miele Vacuum Cleaners Miele Vacuums - Complete Selection Free Shipping! Miele Vacuum Cleaners Miele -Free Air shipping! All models. Helpful advice. est -vacuum.com Web Results 1-10 of about 7,310,000 for miele. (0.12 seconds) Web spider Miele, Inc -- Anything else is a compromise At the heart of your home, Appliances by Miele.... USA. to miele.com. Residential Appliances. Vacuum C leaners. Dishwashers. Cooking Appliances. Steam Oven. Coffee System... www. miele.com/ - 20k - Cached - Similar pages Miele Welcome to Miele, the home of the very best appliances and kitchens in the world. www. miele.co.uk/ - 3k - Cached - Similar pages Miele - Deutscher Hersteller von Einbaugeräten, Hausgeräten... - [ Translate this page ] Das Portal zum Thema Essen & Geniessen online unter -tisch.de. Miele weltweit...ein Leben lang.... Wählen Sie die Miele Vertretung Ihres Landes. www. miele.de/ - 10k - Cached - Similar pages Herzlich willkommen bei Miele Österreich - [ Translate this page ] Herzlich willkommen bei Miele Österreich Wenn Sie nicht automatisch weitergeleitet werden, klicken Sie bitte hier! HAUSHALTSGERÄTE... www. miele.at/ - 3k - Cached - Similar pages Search Indexer The Web Indexes Ad indexes

10 User Needs Need [Brod02, RL04] Informational want to learn about something (~40% / 65%) Low hemoglobin Navigational want to go to that page (~25% / 15%) United Airlines Transactional want to do something (web-mediated) (~35% / 20%) Access a service Downloads Shop Gray areas Find a good hub Exploratory search see what s there Seattle weather Mars surface images Canon S410 Car rental Brasil

11 Web search users Make ill defined queries Short AV 2001: 2.54 terms avg, 80% < 3 words) AV 1998: 2.35 terms avg, 88% < 3 words [Silv98] Imprecise terms Sub-optimal syntax (most queries without operator) Low effort Wide variance in Needs Expectations Knowledge Bandwidth Specific behavior 85% look over one result screen only (mostly above the fold) 78% of queries are not modified (one query/session) Follow links the scent of information...

12 Query Distribution Power law: few popular broad queries, many rare specific queries

13 How far do people look for results? (Source: iprospect.com WhitePaper_2006_SearchEngineUserBehavior.pdf)

14 The Web corpus The Web No design/co-ordination Distributed content creation, linking, democratization of publishing Content includes truth, lies, obsolete information, contradictions Unstructured (text, html, ), semistructured (XML, annotated photos), structured (Databases) Scale much larger than previous text corpora but corporate records are catching up. Growth slowed down from initial volume doubling every few months but still expanding Content can be dynamically generated

15 The Web: Dynamic content A page without a static html version E.g., current status of flight AA129 Current availability of rooms at a hotel Usually, assembled at the time of a request from a browser Typically, URL has a? character in it AA129 Application server Browser Back-end databases

16 Dynamic content Most dynamic content is ignored by web spiders Many reasons including malicious spider traps Some dynamic content (news stories from subscriptions) are sometimes delivered as static content Application-specific spidering Spiders commonly view web pages just as Lynx (a text browser) would Note: even static pages are typically assembled on the fly (e.g., headers are common)

17 The web: size What is being measured? Number of hosts Number of (static) html pages Volume of data Number of hosts netcraft survey Monthly report on how many web hosts & servers are out there Number of pages numerous estimates (will discuss later)

18 The Web as a Directed Graph Page A Anchor hyperlink Page B Assumption 1: A hyperlink between pages denotes author perceived relevance (quality signal) Assumption 2: The anchor of the hyperlink describes the target page (textual context)

19 Anchor Text WWW Worm - McBryan [Mcbr94] For ibm how to distinguish between: IBM s home page (mostly graphical) IBM s copyright page (high term freq. for ibm ) Rival s spam page (arbitrarily high term freq.) ibm ibm.com IBM home page A million pieces of anchor text with ibm send a strong signal

20 Indexing anchor text When indexing a document D, include anchor text from links pointing to D. Armonk, NY-based computer giant IBM announced today Joe s computer hardware links Compaq HP IBM Big Blue today announced record profits for the quarter

21 Indexing anchor text Can sometimes have unexpected side effects - e.g., evil empire. Can index anchor text with less weight.

22 Anchor Text Other applications Weighting/filtering links in the graph HITS [Chak98], Hilltop [Bhar01] Generating page descriptions from anchor text [Amit98, Amit00]

23 Citation Analysis Citation frequency Co-citation coupling frequency Cocitations with a given author measures impact Cocitation analysis [Mcca90] Convert frequencies to correlation coefficients, do multivariate analysis/clustering, validate conclusions E.g., cocitation in the Geography and GIS web shows communities [Lars96 ] Bibliographic coupling frequency Articles that co-cite the same articles are related Citation indexing Who is a given author cited by? (Garfield [Garf72]) E.g., Science Citation Index ( ) CiteSeer ( ) [Lawr99a] Pagerank preview: Pinsker and Narin 60s

24 Query-independent ordering First generation: using link counts as simple measures of popularity. Two basic suggestions: Undirected popularity: Each page gets a score = the number of in-links plus the number of out-links (3+2=5). Directed popularity: Score of a page = number of its in-links (3).

25 Query processing First retrieve all pages meeting the text query (say venture capital). Order these by their link popularity (either variant on the previous page).

26 Spamming simple popularity Exercise: How do you spam each of the following heuristics so your page gets a high score? Each page gets a score = the number of inlinks plus the number of out-links. Score of a page = number of its in-links.

27 Pagerank scoring Imagine a browser doing a random walk on web pages: Start at a random page 1/3 1/3 1/3 At each step, go out of the current page along one of the links on that page, equiprobably In the steady state each page has a longterm visit rate - use this as the page s score.

28 Not quite enough The web is full of dead-ends. Random walk can get stuck in dead-ends. Makes no sense to talk about long-term visit rates.??

29 Teleporting At a dead end, jump to a random web page. At any non-dead end, with probability 10%, jump to a random web page. With remaining probability (90%), go out on a random link. 10% - a parameter.

30 Result of teleporting Now cannot get stuck locally. There is a long-term rate at which any page is visited (not obvious, will show this). How do we compute this visit rate?

31 Markov chains A Markov chain consists of n states, plus an n n transition probability matrix P. At each step, we are in exactly one of the states. For 1 i,j n, the matrix entry P ij tells us the probability of j being the next state, given we are currently in state i. P ii >0 is OK. i P ij j

32 Markov chains Clearly, for all i, n j= 1 P ij = 1. Markov chains are abstractions of random walks. Exercise: represent the teleporting random walk from 3 slides ago as a Markov chain, for this case:

33 Ergodic Markov chains A Markov chain is ergodic if you have a path from any state to any other you can be in any state at every time step, with non-zero probability. Not ergodic (even/ odd).

34 Ergodic Markov chains For any ergodic Markov chain, there is a unique long-term visit rate for each state. Steady-state distribution. Over a long time-period, we visit each state in proportion to this rate. It doesn t matter where we start.

35 Probability vectors A probability (row) vector x = (x 1, x n ) tells us where the walk is at any point. E.g., ( ) means we re in state i. 1 i n More generally, the vector x = (x 1, x n ) means the walk is in state i with probability x i. n i= 1 x i = 1.

36 Change in probability vector If the probability vector is x = (x 1, x n ) at this step, what is it at the next step? Recall that row i of the transition prob. Matrix P tells us where we go next from state i. So from x, our next state is distributed as xp.

37 Steady state example The steady state looks like a vector of probabilities a = (a 1, a n ): a i is the probability that we are in state i. 1/4 3/ /4 3/4 For this example, a 1 =1/4 and a 2 =3/4.

38 How do we compute this vector? Let a = (a 1, a n ) denote the row vector of steady-state probabilities. If we our current position is described by a, then the next step is distributed as ap. But a is the steady state, so a=ap. Solving this matrix equation gives us a. So a is the (left) eigenvector for P. (Corresponds to the principal eigenvector of P with the largest eigenvalue.) Transition probability matrices always have largest eigenvalue 1.

39 One way of computing a Recall, regardless of where we start, we eventually reach the steady state a. Start with any distribution (say x=(10 0)). After one step, we re at xp; after two steps at xp 2, then xp 3 and so on. Eventually means for large k, xp k = a. Algorithm: multiply x by increasing powers of P until the product looks stable.

40 Pagerank summary Preprocessing: Given graph of links, build matrix P. From it compute a. The entry a i is a number between 0 and 1: the pagerank of page i. Query processing: Retrieve pages meeting query. Rank them by their pagerank. Order is query-independent.

41 Resources IIR Chapter 19