The Rise of Recommendation-Type Search Engines Osamu OSHIMA

Transcription

1 No. 127 April 1, 2008 The Rise of Recommendation-Type Search Engines Osamu OSHIMA

2 NRI Papers No. 127 April 1, 2008 The Rise of Recommendation-Type Search Engines Osamu OSHIMA I Expanding Search Market II Changes in Search Technology Environment III Intelligent Search Services IV The ASSOCIÉ Association Search Engine V Prospects of Recommendation Engines According to the Ministry of Economy, Trade and Industry, the size of the Internet search market in Japan is expected to grow from billion in 2006 to 549 billion in The search engine market will continue to lead the growth of the information technology (IT) service market in the future. This growth means that search engines constitute the basis of the Internet and that they are coming to play roles that are more significant. Previously, search engines have generally operated by displaying results that match an entered keyword. However, these search results do not always provide the exact information the user really wants. Future search engines are, therefore, required to provide the ability of accurately deducing the user s intention and guiding the user to the exact information desired, which can be described as intellectual navigation. Search engines with a recommendation function have begun to appear. These search engines offer a higher level of ability by tracking the user s behaviors in the past and referring to feedback provided by the user to better understand his or her preferences. Nomura Research Institute (NRI) has developed an association search engine that automatically proposes a relationship between one topic and another. This search engine enables the user to discover and learn new knowledge by predicting the user s intentions and presenting relationships of which the user may not be aware. Recommendation search engines are also expected to evolve from the proposal of fragmentary information such as keywords and commodities toward the ability to feed knowledge to the user. While search technologies have actually been developing in this way, the establishment of the appropriate legal environment including an amendment of the Copyright Act is of urgent necessity in Japan to accommodate such new technologies. Copyright 2008 by Nomura Research Institute, Ltd. 1

3 I Expanding Search Market How many times a day do you input a keyword into a search engine? According to the US research company comscore 1, every month about 750 million users around the world make 61 billion searches. Whatever the actual figures may be these days, whenever we want to find out about something, it is second nature for us to input a keyword into a search engine such as Google or Yahoo!, regardless of whether we are at work or at home. Our attachment to such searches probably stems from the explosive growth of the amount of information available on the Web with individuals actively publishing information through means such as blogs. Given the vast amount of information available on the Internet, it seems as though it would no longer be possible to seek out the information we need without the assistance of a search engine. The way in which search engines have become so important to us is hinted at by recent TV commercials. While it was common practice up until several years ago to present the URL (universal resource locator) of a company or product brand, from around the year 2006 we have been seeing more and more commercials presenting a keyword that consumers could input into a search engine. This trend stems from the fact that, during a short TV commercial, it is much easier to emphasize an easily remembered keyword to consumers than it is to have them remember a long alphanumeric URL beginning with This method can guide a greater number of consumers to a company s website. At this point, it is worth mentioning that search engines that are widely popular are based on one of two main earnings models. One search engine business model is advertising. In general, the majority of search engines that are offered free of charge to consumers rely on pay for performance (P4P) for their main revenue. With this model, advertisements for products and services that are in some way related to the input keywords are displayed simultaneously with the search results. Another search engine business model relies on charging companies a fee to access the search engine. In addition to electronic commerce (EC) sites such as for online mail orders, this type is also used within companies. The internal use of the search engine is called enterprise searches, which offer an excellent means of improving the efficiency of business operations by targeting an assortment of information stored within the company. According to a survey report entitled Current Status and Prospects of the New IT Market, published in June 2007 by the Ministry of Economy, Trade and Industry, the size of this search-related market in Japan is expected to grow from billion in 2006 to 549 billion in 2011, achieving an annual growth rate of 30 percent. As such, the importance of search engines to the Internet will continue to increase in the future. They are also projected to propel growth in the information technology (IT) service market (Figure 1). Figure 1. Trends in New IT Service Market ( Billion) Blogs SNS Q&A, user-written and edited encyclopedia Product/service comparison Photo sharing Video and music sharing Map sharing Online gaming Web-mail applications Searches (Year) Note: IT = information technology, SNS = social networking service Source: The Ministry of Economy, Trade and Industry, Current Status and Prospects of the New IT Market, June 2007 (Original source: the Ministry of Economy, Trade and Industry, Survey and Research on Actual Status of the New IT Market in 2006) Copyright 2007 by Nomura Research Institute, Ltd. 2

4 II Changes in Search Technology Environment 1 Changes in the Internet Environment At the same time as we become more dependent on search engines, we can see signs of changes occurring in the environment in which search engines are used. Recently, the so-called Web 2.0 has been talked about a great deal, and the following four major changes have actually been taking place on the Internet. The first change relates to the explosive growth in the amount of information available. As mentioned in Chapter I, individuals are now easily able to publish their own information by using media such as blogs and social networking services (SNS). Furthermore, the communities formed by these sites have been accelerating such trends towards the publication of information by individuals. Unfortunately, this proliferation of information makes it more difficult for a person searching for specific information to isolate it from the vast amount available on the Web. The second change concerns the fact that the history of an individual s behavior and preferences on the Internet are collected. In our daily activities such as communication and shopping, opportunities have been increasing where individuals act in the world of the Internet. As a result, information that we can call an individual identity is now being accrued on the Internet. This allows a user to be offered a higher level of service that is tailored to the preferences of the individual by using such information, rather than providing the same service to all users. Similar moves have also occurred in search services. Efforts are being made to optimize search results by providing those that meet the preferences of a user based on the user s attribute information and search history. The third change involves the emergence of a new service that links the functions of different websites. While the link to an outside site by a hyperlink (the function that causes the display to change from the currently displayed page to another specified page) can be thought of as a kind of link, such links can only enable static site reference. Rather than such static site reference, what has become noteworthy recently is the appearance of services offering so-called dynamic links that can call a website s dynamic functions from an outside system by providing an application programming interface (API) to website services. Many search engines are already using APIs to offer their search functions. As a result, the entire Web can be accessed as if it were a database. The fourth change is the use of collective knowledge. This collective knowledge refers to meaningful information that consolidates the large amount of data that users generate. Wikipedia is often quoted as being representative of an online encyclopedia to which a user can freely contribute knowledge. By consolidating fragmentary knowledge that individuals have, an encyclopedia that presents a huge volume of information as a whole is generated. We can regard Wikipedia as being a model that explicitly consolidates the wisdom of a group of people because users enter their knowledge directly in Wikipedia. On the other hand, search engines that cover all resources on the Web are expected to tacitly consolidate collective knowledge. That is, through the consolidation of information accumulated on the Web, expectations are given to search engines to extract by means of statistical techniques tendencies that are hidden in the vast amount of information and are invisible and knowledge consisting of other than specific statements. 2 Roles Expected of Search Engines in the Future Such changes on the Internet also promote a new evolution of search engines. To date, search engines have generally operated by seeking and displaying typically text documents that contain the input search keywords. Unfortunately, however, such simple keyword retrieval often fails to provide the user with the information sought after. Furthermore, if the user inputs other than the optimum keywords, too many results may be returned, making it even more difficult to isolate the required information. In the worst case, the search may fail to find any information. To improve this situation, there is a need for a more advanced search technology. We can summarize the future roles and expectations of search engines as follows: Users want to be able to isolate the exact information they need from the vast amount of information available. Users want to be able to retrieve information that semantically matches their intentions, rather than that simply matches the entered keywords. Users want to be able to perform analyses targeting the Web. In view of these expectations, it is clear that future search engines will be required to offer more than simple keyword-based retrieval. We can say that such search engines will have to be able to accurately deduce information that corresponds to the user s intentions, that is, they should have intellectual navigational skills (Figure 2). III Intelligent Search Services In fact, search engines capable of navigation that are more intelligent have already begun to appear. Particularly Copyright 2008 by Nomura Research Institute, Ltd. 3

5 Figure 2. Evolutionary Trends of Web Search Technology Underlying changes in the Web 2.0 era Changes in expectations of search technology Trends of technological evolution Explosive increase in amount of information Accumulation of user preferences and histories on the Internet Linked websites and the Web serving as a database Utilization of group wisdom I want to be able to find the exact information I need among the mountain of information that is out there. I want to be able to find information that contains the meaning I particularly want. Analyses targeting the Web. From Keyword search to Intellectual navigation (intelligent search) noticeable are those that offer a recommendation function that deduces what the user is looking for and shows candidate information, in addition to the existing function of matching the conditions. A good example of such a search engine is that offered by the major e-commerce company, Amazon.com. By analyzing what customers have purchased in the past, it proposes products that correspond to the customer s preferences. In addition, it proposes a recommended product to a person who has bought a certain product based on the data on the products purchased by this person in the past. With this mechanism, Amazon.com has succeeded in prompting users to purchase more products. Other examples of recommendations include Google Suggest, which automatically looks ahead and displays candidate keywords as the user inputs a search keyword, and the related search offered by both Google and Yahoo! that, based on the keyword input by the user, displays other candidate keywords that are often requested for narrowing down the conditions. Search objects are not limited to text information. The Pandora Internet Radio site, which is called the music discovery engine, automatically chooses and plays similar music when the user inputs the title of a favorite piece of music or the name of an artist. If feedback is provided about which tunes a user likes or dislikes, the system learns the user s preferences, selects and plays music based on such data. In this way, recommendation searches are also beginning to spread among media other than simple text. IV The ASSOCIÉ Association Search Engine 1 A Search Engine that Proposes a Related Topic As mentioned above, some recommendation-type search engines have already begun to appear in the general consumer market. As yet, however, a general-purpose recommendation engine that can be used in the fields where specialized knowledge is required and that can propose accurate information based on expertise, such as a sommelier that draws on knowledge of wine, has yet to appear. To enable new searches in these fields, Nomura Research Institute (NRI) has developed an association search engine called ASSOCIÉ that is capable of this kind of search. ASSOCIÉ is a recommendation-type search engine that is notable for its ability to discover other topics that are related to an input topic and propose them to the user. For instance, if the user inputs obesity as a keyword, ASSOCIÉ will return related topics such as metabolic syndrome, fat and diabetes. As such, related words that the user may not have otherwise been aware of can be displayed. In addition, ASSOCIÉ has a feature of automatically generating an association dictionary based on documents in a specialized field in which relationships between keywords are defined. Specifically, it has a mechanism whereby it notes the relationship between words, and automatically extracts expertise from the document. It then refers to this association dictionary to automatically propose keywords associated with the search keyword. 2 Value of Association Searches This system, whereby the computer can propose topics related to a keyword input by the user, can be applied in two important ways. (1) Predicting the user s intentions This feature helps the user quickly arrive at the information that is being sought. For example, when the input keywords fail to produce the desired document, the user can select more suitable search keywords and/or keywords that can be added to the conditions to narrow down the target from among candidate keywords suggested by the computer. This Copyright 2007 by Nomura Research Institute, Ltd. 4

6 feature is effective as a means of improving the user s efficiency in gathering information. (2) Discovery of unanticipated relationships and acquisition of knowledge A user can be steered toward acquiring new knowledge by being presented with relationships of which he or she was previously unaware. Take, for example, the above case where the user inputs obesity. The system presents metabolic syndrome as a related topic, and then, for which the system presents the related keyword adiponectin (that is, protein in the blood and whose secretion is affected by the amount of visceral fat). Upon being presented with this keyword and if interested in knowing more about adiponectin, the user can refer to documents that contain this keyword (Figure 3). Based on the results of this search, relationships evolve towards new related topics. Thus, the process builds on the ideas of the user and leads to the acquisition of new knowledge. This kind of outcome is valuable not only in that it produces results that are close in meaning to the search word, but in that it also proposes related topics that are supported by expertise. In addition to search services targeting consumers, this technology can also be used for search engines within companies, where it is expected to enhance business efficiency. (3) Utilization of an association search engine One possible application of an association search engine is for patent searches. A company has a need to be able to access a patent database to check whether another company has attempted to register a patent that infringes on a patent that has already been registered by the company. Conversely, prior to embarking on a technological development program or registering a patent, the company wants to determine whether another company has already applied for a similar patent. However, the task of identifying similar patents from among the huge number registered has always been very time consuming and has required expert knowledge in the subject field. Simple keyword-based retrieval would present insurmountable problems. Because different people express a single idea or phenomenon in different ways, and because one given method can be applied to different fields, there is always the possibility that a required document cannot be retrieved by a keyword search alone. In applying the association search engine to patent searches, an association dictionary is first created with the official patent gazette being used as the data source. This makes it possible to tie together those keywords appearing simultaneously in the same document and having semantic relationships, depending on their closeness. As a result, besides the documents that contain the entered keyword, it becomes possible to retrieve documents that contain many keywords having close semantic relationships with the entered keyword. This means that it becomes possible to retrieve patents in the same field as well as those in other fields but having some of the same technological elements. Although we have considered patents here, the application range is not limited to patent retrieval. The association search engine can similarly be used for retrieving documents from the huge library of documents that a company might have accumulated. In particular, we can say that such recommendation functions will come to play a much greater role in those fields where more advanced expertise is needed. V Prospects of Recommendation Engines As described so far, in addition to search engines that provide search results by means of simple keyword matching, those called recommendation engines that offer navigation to make the user aware of unanticipated yet semantically related results started to emerge. Much is expected of recommendation technology, even though it is currently only being applied to recommending Figure 3. Example of Using Association Search Recently, I ve been worried about my weight. Obesity Association Yeah, we often hear about that now. Metabolic syndrome Fat Association What on earth is adiponectin? Diabetes Adiponectin Visceral fat Arteriosclerosis Documents Copyright 2008 by Nomura Research Institute, Ltd. 5

7 fragmentary information such as keywords and products. In the future, however, it is expected to be applied as a tool to support the user s work, in particular the user s intellectual endeavors. In the future, recommendation engines specializing in specific domains of expertise and offering recommendations that are more advanced will appear in a variety of fields. As represented by recommendation engines, search engines will further evolve in the direction of extracting semantically related information from the resources available on the Web and/or within an organization, thus giving the user access to new knowledge. However, the development of more advanced functions for web search engines not only presents the issue of technological development, but also raises copyright and other legal issues, making the establishment of legal infrastructure essential. As the law currently stands, some experts still point out that merely extracting data to create a search index from publications appearing on the Internet violates reproduction rights under the Copyright Act. In this respect, recently, there have been moves to amend the Copyright Act to ease restrictions on the reproduction of publications for the purpose of creating an index for search engines. In light of the current situation, these moves are considered most appropriate. From the perspective of fostering new search technology developed in Japan as well, the establishment of a legal environment is a pressing need. To nurture the search engine market where remarkable growth is expected to continue in the future, discussions that fully consider the reality of the Internet society are sincerely hoped for. Bibliography Billion Searches Conducted Worldwide in August ( Osamu OSHIMA is Technical Engineer at NRI s Technology Research Department. His specialties include the development of natural language processing and application systems. Copyright 2007 by Nomura Research Institute, Ltd. 6

8 As a leading think tank and system integrator in Japan, Nomura Research Institute is opening new perspectives for the social paradigm by creating intellectual property for the benefit of all industries. NRI s services cover both public and private sectors around the world through knowledge creation and integration in the three creative spheres: Research and Consulting, Knowledge Solutions and Systems Solutions. The world economy is facing thorough structural changes led by the dramatic growth of IT industries and the rapid expansion of worldwide Internet usage the challenges of which require new concepts and improvement of current systems. NRI devotes all its efforts to equipping its clients with business strategies for success by providing the best in knowledge resources and solutions. NRI Papers present selected works of NRI s 3,000 professionals through its worldwide research network. The mission of NRI Papers is to contribute new ideas and insights into business management and future policy planning, which are indispensable for overcoming obstacles to the structural changes in our society. All copyrights to NRI Papers are reserved by NRI. No part of this publication may be reproduced in any form without the prior written consent of NRI. Inquiries to: Corporate Communications Department Nomura Research Institute, Ltd. nri-papers@nri.co.jp FAX: