The concept of web search engines and metacrawlers within the Internet environment

Transcription

1 The concept of web search engines and metacrawlers within the Internet environment Andrej Danko Nowadays, the Internet is rife with data requiring every user to use specialized tools and software engines to search and find appropriate information relevant to a set of keywords. This task is of a complicated nature since, on one hand, it involves many complex processes to accomplish a simple request (from the point of a user), and on the other hand, it requires enormous computational capacity (processor ti me and disk storage) to perform these processes. Since this challenge can be considered as a new one and scientific (emerged in mid 90 s of the last century), there are many open issues to be solved or optimized to bring better results and usability [VOJ03]. The aim of this article is to briefly introduce search engines, describe their fundamental components, and discuss some of the open points and possible optimizations. Search engine (SE) is a powerful tool for obtaining links to relevant web sites (or, more generally, documents) based on a group of keywords entered by a user that are somehow related to the results (documents). We can identify following types of search engines based on the method of document (web site) indexing they use: 1. Crawler-based search engines (automatic robots search the web and create index out of the searched pages) 2. Human-Powered Directories (manually added links and site description to the index by the web owner and/or directory operators) 3. Hybrid search engines (these are a combination of 1 and 2 and provide results based on the automatically created index and also by filing in the directory) In addition to these mentioned types, first of all, there are so-called metacrawlers, whose principle is based on using more than one crawler-based SE (thus using multiple indexes). Secondly, new search engines are emanating in the Internet, whose primary information target is not web-based (e.g. peer-to-peer networks), but the contents of the underlying database is not hidden or invisible to metasearch engines [HAM03]. The most important and of the greatest interest are, however, the crawler-based SE, because they can produce the largest base of data against which queries can be executed. The subsequent paragraphs contain a brief description how the index is produced, how the pages are analyzed and how they are ranked in the means of importance (i.e. which is more relevant to a user search than another). Information retrieval in the Internet environment is very different than that operating on well controlled document collections, since documents contained on web are very different in every aspect: language of any kind (human, programmatic), vocabulary, format, etc. Other aspects connected to this issue are relevancy, no-control over document publishing, copyright policies, and other. Crawler-based search engine This type of engine, as mentioned above, is based on automated reading of web pages and processing of results in the means of creating the index. There are three major elements within this type of engine: 1. Spider (crawler, robot) - visits a web page, reads it and downloads it into a storage system, and then follows links to other pages within the site. 2. Index (catalog) - the results of a spider are stored in the index of the search engine, where new pages are added but also existing ones updated. Andrej Danko

2 3. Search engine this is the interface between user and the index that searches the index for a set of particular keywords and returns results. It is primary a set of algorithms whose main purpose is to identify the best pool of documents to be presented as search results [ROS02]. Depending on different implementation approaches mostly in (1) and (3), the search engines return different results. Many SE have a more complicated architecture, as for example Google ( see Figure 1, for details, read more in [BRI98]). In addition to these basic elements, SE can contain other subsystems, like history caching, lexicons, etc., that can provide for users additional functionality and thus bring more usability. Figure 1. High Level Google Architecture [BRI98] Searching pages and index population Each web page crawled by a robot is ranked by a ranking algorithm, which means that each page gets points for relevance according to criteria programmed in the algorithm. Since these algorithms are different in each search engine (and are usually not public, or are patented), the search engines produce for the same keywords different results. There are several initiatives going on today to discover better algorithms for page indexing and search engines in general [KOV03]. Since web pages are processed by engines and cached centrally in a database, any search performed by users is done over data that are not actual. This is an important point, since a page cached within a SE might not be available when users perform a search and get results. This issue is connected with cash update and web document updates in general. Proper propagation and re-caching of already cached pages is crucial and plays a vital role in a SE success. Architecture of the searching subsystem is usually distributed, since crawlers must search millions of pages. In addition to this, there is a number of problems the algorithm has to cope with, as are dead links, incorrect syntax, DNS issues, and other. The following criteria are the most important when evaluating a web page and are commonly used by many SE: 1. Page Criteria 1.1. Location and frequency of the keywords (hit lists). 2. Non-page Criteria 2.1. Click count of the found pages Popularity of the web page Thematic relevancy of other web pages related to the searched page. Page criteria are based on the content of the page, i.e. keywords, title page, URL, etc. During the indexing (crawling) process, engine analyzes content of a page (in the means of words the page contains) and the frequency in which the words are appearing in the web page (most of the HTML tags and other system formatting words are ignored). Depending on the algorithm, pages with the search terms appearing in the HTML title tag are often assumed to be more relevant than others to the topic. Title tag, even in frameset documents, is very important, and one of the most relevant. Other tags, besides the content of the web page itself that provides some summary of the web page are Meta tags. However, the importance of the Meta tags is not as high as it used to be, Andrej Danko

3 since the most used engines do not support them any more. Nevertheless, there are some which are still used by many search engines (see Table 1.) Meta tag Usage & description Remark Robots Specifying that a page should be NOT indexed or page linked from the search page should NOT be followed up. (no)index sets whether the page should be indexed; (no)follow sets whether the links on the current page should be followed up Description Contains an abstract of the Length between characters Keywords Table 1 Meta tags webpage Additional keywords describing the web page is indexed at most If used by engine, 25 keywords or less are optimum to state (separated by a comma), other are ignored Other tags that have a special meaning when web pages are analyzed, are the ALT tag and the A (anchor) tag (i.e. citation). The content of the A tag text of link (that is displayed as a hyperlink on web pages) is associated with the page that the link is on. Some engines (like Google) associate it also with the page the link points to. The basic assumption made by citation or link analysis is that a link is often created because of a subjective connection between the original document and the cited, or linked to document [ GLO03]. The other factor of words on page is how often they appear in relation to other words in a web page. Those with a higher frequency are often deemed more relevant than other web pages. In addition to this, formatting is sometimes also taken into consideration. Words in bold letters are most of the times more important than regular words. Some search engines might ignore those words that are contained in a page to often, i.e. the page wants to spam the search engine. There are several of ways the engines determine this, usually, a threshold and some proximity rules are defined. Some words are ignored totally (called stop list), because they are to o commonly used to be included (e.g. the ) in the index. Today s trend is not to search and parse only basic HTML documents, but also documents in other format that contain textual data, like word documents, spreadsheeds, presentations, and PDF files. The content of these documents is often very relevant and usually enhances a SE s usability. In addition to searching for documents, SEs are offering, based on the citation (link) paradigm, the possibility to search for pictures. Non-page criteria evaluate a page within the context the page is located (i.e. links to the page, content of pages linked to the page). Click count, one of the frequently used, updates the page relevancy depending on the number of recorded clicks that go from search result to the found page. This means, when a link to a page is provided in the result set, it is not directly pointing to the original document, but, instead, to a system within SE that counts the clicks. Afterwards, when this information is recorded, the user is redirected to the Figure 2 - Page ranking by Google appropriate document (web page). Andrej Danko

4 Page popularity is another widely used technique to measure relevancy. Most of the non-page criteria are based on the link analysis. The number of links pointing to the page is taken into account, i.e. calculating how many backward links are pointing to web page/site. In addition to this, some engines (like Google) take in account the ranking (points) of those pages (parent) when calculating the ranking of the actual page. The basic idea behind this is that pages, which are very relevant (e.g. cnn.com), and have a high page rank, point to pages that are also relevant. On the other hand, if a page is not relevant, there is a high probability, that a page like cnn.com will not have a reference to this page. More details are available in [BRI98]. Search engine The search engine is the third and a very important part of SE that directly interact with a user. Its goal is to find instantly documents matching a number of keywords. This is a demanding task since the engine must run over millions of records and return results in real time. Usually, the user can enter more words that are joined together with a set of Boolean operators (AND, OR, NOT, NEAR) that specify how the engine should perform the search. For example, for the NEAR operator, the index database must contain word proximity in order to be able to perform such a query. The provided results are sorted by document relevancy, which determines the quality of a search engine. The more relevant documents are returned (from the subjective point of a user), the better the search engine works. The problem of search keywords lays their unstructured format (not counting the Boolean operators concatenating single terms) and lack of semantic context. When searching e.g. for Washington, SE don t know whether the user means ~ the first president or ~ city or ~ state. A future challenge could be to add such a semantic context and thus bring results more to the point. Search engines applications and their future There are many specific areas where search engines can specialize and bring new information sources for users. At first, there are some attempts to bring personalization to search itself by extending the page ranking algorithm with the usage of personal information about a user and thus customize search by specific user s needs, interests, geographical location, or customs. Personalization could dramatically higher the relevance of search results. Next, search engines are used as data sources for other systems operating with sets of keywords. One of the first initiatives was to provide additional informational content for realtime broadcasting, where the broadcasting (TV, Radio, News on internet) provided keywords that entered the search engine and returned some relevant documents about the information being broadcasted at that moment [BRI03]. Last but not least, search engines start to specialize on specific information retrieval in such areas as product price comparison or recruitment data. A user enters a desired product and the engine finds stores with the best price. However, use of those engines in such conditions will require additional work and perhaps a different approach since the information is becoming more structured. As a final point, economical and commercial aspects of search engines are becoming more pertinent and some types of search engines are already charging their customers for being listed. However, the more relevant searches get (personalization), the more probable it is that the service will not be free of charge even for the regular users. For a fee, they will get value added services, which, of course, require much more effort to create and maintain. Andrej Danko

5 Abstrakt Prostredie Internetu obsahuje vzhľadom na svoju povahu obrovský objem heterogénnych údajov (v zmysle svojej formy, obsahu, štruktúry a väzieb). Na to, aby bolo možné využiť danú dátovú základňu na získavanie relevantných informácií, sú potrebné nástroje, ktoré dokážu efektívnym spôsobom prehľadávať a indexovať webový priestor a poskytovať používateľom rýchly a jednoduchý prístup k vytvoreným databázam s obsahom Internetu. Cieľom tohto článku je oboznámiť sa s niektorými prístupmi v oblasti vyhľadávania na Internete, a to konkrétne s nástrojmi, architektúrami softvérových systémov a inými súčasťami technického i iného charakteru. Literatúra [BRI03] Brin, Sergey, et al. Query-Free News Search. In Proceedings of The Twelfth International World Wide Web Conference May 2003, Budapest, Hungary. Budapest: ACM Press, [BRI98] Brin, Sergey Page, Lawrence. The Anatomy of a Large-Scale Hypertextual Web Search Engine [online]. c1998, [cit ]. Dostupné z: < pub/papers/google.pdf> [GLO03] Glover, Eric J., et al. Using Web Structure for Classifying and Describing Web Pages. In Proceedings of the eleventh international World Wide Web conference May 7-11 May 2002, Honolulu, Hawaii, USA. Honolulu: ACM Press, [HAM03] Hamilton, Nigel. The Mechanics of a Deep Net Metasearch Engine. In Proceedings of The Twelfth International World Wide Web Conference May 2003, Budapest, Hungary. Budapest: ACM Press, [KOV03] Kováč, Martin. Microsoft najal popredných matematikov proti Google [online] , posledná revízia [cit ]. Dostupné z: < H/Spravodajstvo/Ar.asp?ARI=106292&CAI=2107>. [ROS2002] Rosenfeld, Louis Morville, Peter. Information Architecture for the World Wide Web. 2. vyd. Sebastopol, CA, USA: O Reilly & Associates, Inc., ISBN Kapitola 8, Search Systems, s [VOJ03] Vojtek, Peter. Vyhľadávanie na internete: problém aj pre vedcov [online] , posledná revízia [cit ]. Dostupné z: < h/spravodajstvo/ar.asp?ari=105992&cai=2107> Autor Ing. Andrej Danko Katedra aplikovanej informatiky FHI, EU Bratislava andrej@danko.sk Andrej Danko