The Internet s immense growth has led to dramatic

Transcription

1 Editor: Siobhán Clarke Spotlight Shopbots: A Syntactic Present, A Semantic Future Maria Fasli University of Essex Despite high expectations, shopbots have yet to significantly facilitate a richer, more satisfying online shopping experience for users. By taking advantage of Semantic Web and Web services technologies, however, researchers can overcome current technological limitations and finally realize the shopbot s significant potential. The Internet s immense growth has led to dramatic changes in business-to-consumer commerce (B2C). E-commerce sites abound, offering users a wide range of services and products. People can now shop from their homes for everything from flowers to cars; the traditional shopping trip has been transformed into a virtual one. Although users have product and services information at their fingertips, they increasingly require assistance, guidance, and support to navigate the Internet s vast information space and make informed purchasing decisions. Shopbots are specialized agents that help users sift through this information by finding product specifications and reviews; comparing products, vendors, and services according to user-defined criteria; finding the best-value products or services; monitoring online shops for product availability or special offers and discounts and sending user alerts; recommending services and products; and identifying new products of potential interest. Once a shopbot receives a product request from a user, it queries vendors and providers. It then retrieves and compares or differentiates and evaluates, according to Yun Wan and colleagues 1 relevant information such as price, delivery options, and warranty. By performing such comparisons for the users, shopbots ease the purchasing process and enhance users shopping experience, while also saving them time and money. Despite the initial hype, 2 however, shopbots have yet to fulfill their immense potential. This is primarily due to technical limitations: shopbots don t yet understand the information that they retrieve and (moreover) can t yet dynamically discover and query vendors. Here, I discuss current shopbot limitations and how using the Semantic Web and Web services can solve these problems. The Online Shopping Experience At the onset of e-commerce, users were wary of online shopping and reluctant to provide personal and financial details to online vendors for fear of credit-card fraud and loss of privacy. In recent years, vendors have better addressed security and privacy issues, and users have become more familiar with the technology. As a result, online shopping sales have increased massively. Existing Approaches Users typically look for products online by visiting familiar sites or by identifying potential vendors using standard search engines and keyword retrieval. Users then visit each site and search for the product, its price, and other related information. This basic approach has several shortcomings. 3 First, hundreds of vendors might sell the same or similar products. Unless users are aware of partic- IEEE INTERNET COMPUTING /06/$ IEEE Published by the IEEE Computer Society NOVEMBER DECEMBER

2 Spotlight ular vendors, they must decide which and how many sites to visit to acquire the necessary information. Although visiting multiple vendors requires considerable time, not visiting enough can produce suboptimal decisions. Second, identifying vendors through search engines might simply return the largest vendors, not necessarily those with the best prices. Third, if users need several items that aren t typically sold together, the search time increases for each new product category. Finally, each time users visit new vendors, they must acquaint themselves with new interfaces. This not only further increases search time but also unavoidably hinders impulse shopping. To help customers stay informed, vendors sometimes let users sign up for product alerts, which might inform customers about price changes, for example, or about the availability of a forthcoming or out-of-stock product. Although these alert services aren t personalized, users must often complete long surveys that sometimes request personal data. Furthermore, submitting an address for notifications in itself reveals an aspect of the user s identity and thus weakens his or her privacy. Shopbot Benefits Delegating product research, comparison, and evaluation to shopbots can clearly offer users significant benefits. First, shopbots can save users time by searching numerous vendors quickly. By querying multiple vendors including those that users are unfamiliar with shopbots are more likely to get the best deals for their users. Shopbots can also uncover special deals that users would be otherwise unaware of. The second potential benefit is in psychological burden shifting. 4 Users often feel uncertain about buying products but, by using shopbots, users can shift some of the psychological costs of decision making onto the agent. If a decision turns out to be less than ideal, the user can blame the shopbot and thus minimize his or her own psychological risk. (Indeed, because they re using shopbots, users might remain unaware that a recommendation is suboptimal, further minimizing the psychological burden.) Finally, shopbots can create marketplaces that are more efficient. Shopbots let users easily compare vendor prices and services and thereby increase marketplace competition. Users also gain access to smaller vendors offering competitive prices and thus reduce larger vendor monopolies. Shopbot Technology When shopbots appeared in the mid 90s, technology analysts predicted they d have a huge impact on vendors and on business conduct in the years to come. 2 However, shopbots have yet to meet this potential. Characteristic Shopbots The first shopbot for price comparisons was Andersen Consulting s BargainFinder, 5 which compared prices on music CDs from online stores. However, because BargainFinder simply evaluated vendors based on their prices ignoring all other site features many music retailers started blocking access to the agent. Eventually, BargainFinder stopped operating. PersonaLogic 6 helped users make decisions on products and services based on their needs and preferences. Given a user profile, PersonaLogic could identify products with features the user valued. However, vendors had to provide an interface that explicitly disclosed product features in a way that matched the agent s user profiles. In 1998, AOL acquired PersonaLogic and withdrew it from the market. ShopBot 7 submitted queries to e-commerce sites and parsed the results to extract product information. The agent exploited regularities in e- commerce sites, using an automatic process for building wrappers to parse semistructured HTML documents. It then extracted product features from the documents, such as product description and price. Shopbot, which was later renamed Jango, avoided the vendor-blocking problem that BargainFinder faced by sending product information requests from the user s browser, rather than the agent s server. In 1997, Excite acquired Jango and incorporated it into its search technology. Excite has since evolved into a vendor-biased comparison site. A more recent shopbot example is IntelliShopper, 3 which observes users actions and unobtrusively attempts to learn their preferences and help them with their shopping. IntelliShopper also monitors various vendors for products that might match user needs and preferences. The agent also preserves user privacy: it conceals users IP addresses by passing requests through one or more anonymizer servers. Moreover, it uses shopping personas to hide all user information from the IntelliShopper server. The system stores logic about the different target vendors in modules. These modules specify how the agent can query vendors, 70 NOVEMBER DECEMBER IEEE INTERNET COMPUTING

3 Shopbots interpret the results, and extract the information users require from the returned HTML pages. Technical Limitations Shopbots operate like meta search engines, using a form of screen-scraping 8 to retrieve information from different vendors. The agents typically let users sort product and vendor information along some desired dimension, such as price. Essentially, shopbots interact with vendors through HTML pages that were originally designed and generated for humans to read and understand. Shopbots submit queries to vendors and attempt to process the resulting HTML pages by parsing them and searching for the target item s name. They then search the nearest set of characters with a currency sign (presumably, the item s price). To retrieve such information, shopbots rely on the following regularities across e-commerce Web sites: 7 Navigation sites make it easy for users to navigate and find what they need. Uniformity Web pages of a particular site look similar. Regular vertical separation sites use spaces to separate products, and each new product description starts on a new line. In fact, the vendor s database has the information that shopbots need in a machine-processable and well-structured format. However, as Figure 1 shows, the shopbot can t interact with the database: it has access only to the Web pages that are generated from the database. Consequently, developers must implement clever heuristics that can extract the original, well-structured data from the HTML pages implicit information. Such heuristics are inevitably ad hoc, difficult, and time-consuming to develop. They re also error prone and can retrieve only limited information. Inevitably, developers must update the shopbot s heuristics whenever the layout of the vendor s site changes. This makes shopbot development cumbersome, and the resulting systems inflexible and vendor-specific. Shopbots therefore can t discover and query new vendors at runtime: the shopbot s developers must first identify vendors, then tailor methods for retrieving information from their Web pages. Also, because current techniques rely on syntax, shopbots can retrieve only limited information from vendors, ignoring harder to retrieve attributes, such as warranty and shipping options. Users Task Shopbot Wrapper1 Wrapper2 Wrapper3 HTML pages HTML pages HTML pages Figure 1. Shopbots at work. Shopbots use vendor-specific wrapper programs to wrap up and submit a user s request for product information to multiple vendors through HTML pages. The wrappers parse the returned HTML pages and extract the required product information. This is a severe limitation. Although price is often the most important factor in purchasing decisions, other attributes also factor in, including delivery and payment options and the vendor s reputation. Such value-added services can be the deciding factor when several vendors offer a product at the same price. Also, vendors naturally object to a comparison of price alone because it fails to reflect the full range of vendor services and value. Consequently, many vendors block shopbot price requests. Shopbots are thereby restricted in their ability to compare multiple vendors and help users find the best deal around. Finally, syntax-dependent shopbots can inadvertently retrieve inaccurate pricing information, creating discrepancies between the prices the shopbots report and those listed at the vendor s site. The reason is that shopbots don t understand the retrieved Web pages content. Consequently, they can t distinguish between such things as vendors that include shipping charges and taxes in their prices and those that don t. These technological limitations have led to the demise of vendorindependent shopbots. They ve also contributed to the emergence of comparison sites (see the Comparison Shopping Sites sidebar). Shopbots and the Semantic Web Despite high expectations of shopbots impact on retail markets, their true potential hasn t been realized. This is mainly because of inherent limitations in the shopbots information extraction technique that is, screen scrapping. Web services offer a highly promising alternative to this approach. Database Vendor 1 Database Vendor 2 Database Vendor 3 IEEE INTERNET COMPUTING NOVEMBER DECEMBER

4 Spotlight Web Services Overview A Web service is a collection of functions that a developer packages as a single entity and publishes on a network for use by other programs. 9 Developers can describe Web services programmatic interfaces using the XML-based Web Services Description Language (WSDL; wsdl). Service descriptions include the input and output parameters and their data types, the format for message exchange, and the service s operations, network address (the Uniform Resource Identifier, or URI), and protocol bindings. To facilitate a service s discovery by requesters, developers can describe and register it in directories. Universal Description, Discovery, and Integration (UDDI; describes a mechanism for registering and locating services. Once a requester knows the service s URI, it can use SOAP ( to invoke the service with its parameters. Web services power lies in their ability to interact seamlessly and transparently to achieve combined functionality and produce the required result. 9 Vendor Gateways Because Web services can retrieve information directly from the vendor in machine-processable form, shopbots can use them as gateways to vendor sites. For example, Amazon ( com) offers Web services that let programs perform operations such as retrieving product information and adding items to a shopping cart. Agents can thus employ Web services to acquire, aggregate, and compare information on products, services, and vendors. This greatly enhances shopbots potential for comparisons based not only on price, but on various other attributes as well. Shopbots can potentially access numerous vendors, provided that they use programmatic interfaces to facilitate information retrieval. However, this process could be hindered if agents don t understand references to product or service attributes. Unless an agent understands that warranty and guarantee mean the same thing to different vendors, for example, it won t be able to correctly retrieve information and present and compare it in a uniform or meaningful way. To facilitate this common understanding among agents, underlying domain ontologies are required. Such ontologies can be used by developers to describe Web services, and shopbots to understand and process the retrieved information and present it to users. Therefore, to take advantage of Web services, developers must endow shopbots with the ability to use and reason about ontologies. Limitations of Existing Standards In an increasingly interconnected world, hundreds of providers might offer a particular service or product. Consequently, shopbots need some means of locating providers. One obvious solution here is to use a UDDI registry to discover providers and their services. This would give shopbots a service description and location, and instructions for invoking the service. However, current XML-based standards for describing Web services such as SOAP and WSDL are designed to provide descriptions of transport formats and mechanisms, address binding, and the service s interface, but not descriptions of a service s capabilities. Similarly, UDDI fails to represent service capabilities and thus can t facilitate automatic Web services location based on capability specifications. 10,11 Thus, while shopbots could search a UDDI registry for US vendors, for example, they couldn t search for vendors that ship outside the US. Searching for services with such specific capabilities is extremely difficult. UDDI registries also lack an explicit semantics: although two XML service descriptions might look exactly the same, they might mean different things in different contexts. Enriching Service Descriptions For shopbots to achieve their full potential, the matching between a service request and potential providers must take into account semantics and capability specifications and must be done in context. Hence, we need both richer Web services descriptions and a way to process those descriptions. The DARPA Agent Markup Language (DAML; 10 and its most recent incarnation, the Ontology Web Language for Services (OWL-S; provide a more suitable framework for describing services capabilities than SOAP or WSDL. In OWL-S, developers describe a Web service using three interrelated sub-ontologies: ServiceProfile describes a service s capability specification, which is essential in determining whether or not it meets the requester s needs. ServiceModel describes how a requester can use 72 NOVEMBER DECEMBER IEEE INTERNET COMPUTING

5 Shopbots Ontologies Web services Vendors Users Task Request Publish Register Shopbot Middle agent Service invocation (SOAP) Figure 2. How shopbots could use the Semantic Web and Web services. Upon receiving a user s product query, the shopbot contacts one or more middle agents and requests information on matching vendors that have registered their Web services. Shopbots, middle agents, and developers all consult ontologies to facilitate common understanding and information exchange. the service, how to ask for it, and what happens when it s executed. ServiceGrounding specifies the communication protocol, message formats, and other servicespecific details, such as the port numbers used to contact the service. Selecting providers that use these enhanced descriptions would require a registry that lets agents search it using semantics that UDDI doesn t currently offer. 11 Alternatively, middle or broker agents could act as ontology-based search engines and take into account semantics. 12 In this case, specialized middle agents that store, maintain, and provide connection information would match shopbot service requests with one or more providers. Vendors would publish and register their Web services with these middle agents, and shopbots would query the middle agents when receiving user requests for products. Realizing the Vision Figure 2 shows how shopbots could use the Semantic Web and Web services: Vendors create Web services that allow access to their sites, and they publish and register these services with middle agents. Users delegate the task of finding and comparing products to shopbots. Shopbots contact middle agents who identify and return a set of providers offering the service along with information on how to contact and interact with the service. Given a user s product query, the shopbot invokes the appropriate service and retrieves the required information. In this setting, shopbots could dynamically retrieve information from various vendors without having to depend on static (and sometimes unreliable) links. They could also cache Web services locations for common requests, thereby improving efficiency. The end result would be easily digestible information that users could consider to make their final purchasing decisions. If shopbots are to retrieve semantic information on vendors products and services, researchers must address several outstanding challenges, including the development of efficient algorithms for matching requests with Web services; efficient ontology services, including ontology consolidation services; and reasoning mechanisms that let shopbots perform true comparison shopping on the basis of user preferences, the retrieved information, and the underlying ontologies. Although standardizing ontologies to describe cer- IEEE INTERNET COMPUTING NOVEMBER DECEMBER

6 Spotlight Comparison Shopping Sites C omparison shopping sites don t typically use agent technology. Usually, they either rely on vendors to provide required information on catalogues and products or operate as meta search engines and search vendor sites. Sites like MySimon ( and DealTime (www. dealtime.com), for example, collate product catalogues of vendors who pay for site listings. Meta search engine sites, such as Kayak ( and Side- Step ( can have formal or informal vendor relationships. Vendors can simply let such sites access and retrieve information from their Web sites. Or, they might form explicit partnerships and pay a commission either for each hit resulting from the listing or for sales resulting from clickthrough purchases.vendors can also place advertisements on comparison shopping sites. Froogle ( Google s product search engine, indexes products from multiple e-commerce sites in two ways: sellers can submit product information directly; or Froogle uses Google Web searches to identify Web pages that offer the target products and services. Clearly, comparison shopping sites might not offer impartial advice or the best deals. In most cases, comparison shopping sites don t retrieve and evaluate attributes such as delivery options, warranty, or other value-added services; users have to access the online stores directly to retrieve and compare such information. tain retail sectors would be ideal, ontology standardization is a general problem. Consequently, shopbots and middle agents must be able to reason with and compare multiple ontologies. So far, work on shopbots that can use Web services to retrieve vendor information has been rather limited. One recent approach, the Intelligent Product Information Search (PIS) system, offers meta search and comparison for products. 13 IPIS assumes that shopping malls offer Web services to describe their available products. Upon receiving a user request, IPIS configures appropriate vendorspecific queries using an ontology and submits them to the vendors Web services. The IPIS system follows some of the principles described here, but has two important limitations. First, it can retrieve information only from a predefined set of vendors whose programmatic interfaces (that is, Web services) are known in advance. There are no directory services or middle agents to let shopbots discover and query vendors at runtime. If none of the predefined vendors provides the requested product, IPIS cannot search for alternative vendors. Second, while IPIS uses an ontology, it is a centralized ontology containing the predefined vendors vocabularies. To enable dynamic product information retrieval from multiple vendors and present it consistently to users, agents must have access to and reason with multiple distributed ontologies. Despite its limitations, IPIS is a step in the right direction and illustrates that the use of the Semantic Web and Web Services can greatly enhance shopbot functionality. Toward a Semantic Future Transitioning shopbots to the Semantic Web and Web services would alleviate existing problems with shopbots and comparison shopping sites in several ways. Expanding the Marketplace Through the use of Web services, Shopbots could retrieve various product attributes well beyond price. Expanding the comparison attributes could in turn motivate more vendors to provide programmatic interfaces to their sites, thus increasing their accessibility to shopbots and their visibility to customers. Although this inevitably increases competition, it also improves overall market efficiency. Smaller vendors could also increase their visibility and benefit from the large traffic volumes offered through shopbots. 14 Larger vendors with a bigger, more loyal customer base might have less incentive to join or allow shopbot access. Still, some will choose to do so to further increase their customer base. In any case, shopbots and comparison sites would be liberated from dependencies on specific vendors and would thus be better able to offer impartial advice. Protecting Users Shopbots can facilitate Web anonymity by processing product information requests without revealing users identities. This would make it difficult for vendors to run price discrimination schemes. Such schemes depend on the vendors being able to identify users and build profiles which they can subsequently use to measure a product's desirability to specific users. Shopbots can also use reputation systems to filter out unreliable vendors. Such systems collate information on vendors by aggregating feedback on past transactions from users (or agents), and then use various means to measure vendor trustworthiness. 15 Shopbots might be able to steer users 74 NOVEMBER DECEMBER IEEE INTERNET COMPUTING

7 Shopbots away from dubious vendors and help tackle market fraud. Finally, users could indirectly interact with shopbots through personal agents. For example, users could delegate product-finding tasks to their personal agents, who could in turn contact shopbots and request information. Personal agents could then filter the information returned by shopbots even further by considering additional user preferences or other restrictions. Assisting Business-to-Business Transactions In addition to benefiting B2C users, shopbots can assist organizations in business-to-business (B2B) transactions. A shopbot can perform searches and comparisons for suppliers and components on behalf of businesses and organizations, such as those in a supply chain. Shopbots can locate the cheapest suppliers or those that satisfy certain conditions, such as delivery deadlines. B2B users would receive the same benefits from the Web services infrastructure, including reduced costs and improved market efficiency. My colleagues and I are currently working on two shopbot-related problems. We are applying graph network analysis to the problem of automatic Web-service discovery and composition. We are also exploring how we can use the OWL-S service profile description to enhance semantic matchmaking. In future work, we aim to investigate the problem of reasoning with multiple ontologies and ontology consolidation services. In the years to come, using the Semantic Web and Web services will lead to the development of increasingly sophisticated shopbots. In addition, given that mobile devices such as smart phones and PDAs have become indispensable to many people s daily lives, the next step will be to develop shopbots that can operate on these Webenabled devices. Web for Bargains, CIO Magazine, 2000; archive/110100/et.html. 3. F. Menczer, W.N. Street, and A.E. Monge, Adaptive Assistants for Customized E-Shopping, IEEE Intelligent Systems, vol. 17, no. 6, 2002, pp V. Rajiv and P. Aggarwal, The Impact of Shopping Agents on Small Business E-Commerce Strategy, J. Small Business Strategy, vol. 13, no. 1, 2002, pp B.T. Krulwich, The BargainFinder Agent: Comparing Price Shopping on the Internet, Bots and other Internet Beasties, J. Williams, ed., Macmillan, 1996, pp A.G. Guttman, R.H. Moukas, and P. Maes, Agent-Mediated Integrative Negotiation for Retail Electronic Commerce, Proc. 1st Int l Workshop on Agent Mediated Electronic Trading (AMET 98), LNCS 1571, Springer, Berlin, 1999, pp R.B. Doorenbos, O. Etzioni, and D.S.Weld, A Scalable Comparison-Shopping Agent for the World-Wide Web, Proc. 1st Int l Conf. Autonomous Agents (Agents 97), ACM Press, 1997, pp M.P. Singh, The Service Web, IEEE Internet Computing, vol. 4, no. 4, 2000, pp M.P. Singh and M.N. Huhns, Service-Oriented Computing: Semantics, Processes, Agents, John Wiley and Sons, M. Paolucci and K. Sycara, Autonomous Semantic Web Services, IEEE Internet Computing, vol. 7, no. 5, 2003, pp K. Verma et al., METEOR-S WSDI: A Scalable P2P Infrastructure of Registries for Semantic Publication and Discovery of Web Services, Information Technology and Management, vol. 6, no. 1, 2005, pp H.C. Wong and K. Sycara, A Taxonomy of Middle-Agents for the Internet, Proc. 4th Int l Conf. MultiAgent Systems (ICMAS-00), IEEE CS Press, 2000, pp W. Kim et al., Development of a Meta Product Search Engine with Web Services, Proc. 2nd Asia Information Retrieval Symp. (AIRS 2005), LNCS 3689, Springer-Verlag, 2005, pp G. Iyer and A. Pazgal, Internet Shopping Agents: Virtual Co-Location and Competition, Marketing Science, vol. 22, no. 1, 2003, pp P. Resnick et al., Reputation Systems, Comm. ACM, vol. 43, no. 12, 2000, pp Acknowledgments The discussion on the problems associated with current shopbots is derived from my book, Agent Technology for E- Commerce (forthcoming from Wiley & Sons, 2007). Reproduced with permission. References 1. Y. Wan, S. Menon, and A. Ramaprasad, A Classification of Product Comparison Agents, Proc. 5th Int l Conf. Electronic Commerce (ICEC 03), ACM Press, 2003, pp J. Edwards, Is That Your Best Offer? Shopbots Search the Maria Fasli is a lecturer in the Department of Computer Science at the University of Essex. Her research interests are in agents and multiagent systems particularly formal theories for reasoning agents, group formation, and social order as well as in applying agent technology in e-commerce. Fasli has a PhD in computer science from Essex University. She is the author of Agent Technology for E-Commerce (forthcoming from Wiley and Sons, 2007) and was awarded a Higher Education Academy National Teaching Fellowship in 2005 for her innovative approaches to learning and teaching. Contact her at mfasli@essex.ac.uk. IEEE INTERNET COMPUTING NOVEMBER DECEMBER