WISE-Integrator: An Automatic Integrator of Web Search Interfaces for E-Commerce

Transcription

1 WSE-ntegrator: An Automatc ntegrator of Web Search nterfaces for E-Commerce Ha He, Wey Meng Dept. of Computer Scence SUNY at Bnghamton Bnghamton, NY Clement Yu Dept. of Computer Scence Unv. of llnos at Chcago Chcago, Zonghuan Wu Center for Adv. Compu. Studes Unv. of ousana at afayette afayette, A zwu@cacs.lousana.edu Abstract More and more databases are becomng Web accessble through form-based search nterfaces, and many of these sources are E-commerce stes. Provdng a unfed access to multple E- commerce search engnes sellng smlar products s of great mportance n allowng users to search and compare products from multple stes wth ease. One key task for provdng such a capablty s to ntegrate the Web nterfaces of these E- commerce search engnes so that user queres can be submtted aganst the ntegrated nterface. Currently, ntegratng such search nterfaces s carred out ether manually or sem-automatcally, whch s neffcent and dffcult to mantan. n ths paper, we present WSE-ntegrator - a tool that performs automatc ntegraton of Web nterfaces of Search Engnes. WSE-ntegrator employs sophstcated technques to dentfy matchng attrbutes from dfferent search nterfaces for ntegraton. t also resolves doman dfferences of matchng attrbutes. Our expermental results based on 20 and 50 nterfaces n two dfferent domans ndcate that WSE- ntegrator can acheve hgh attrbute matchng accuracy and can produce hgh-qualty ntegrated search nterfaces wthout human nteractons. 1. ntroducton More and more databases are becomng Web accessble through form-based search nterfaces. Among these web Permsson to copy wthout fee all or part of ths materal s granted provded that the copes are not made or dstrbuted for drect commercal advantage, the VDB copyrght notce and the ttle of the publcaton and ts date appear, and notce s gven that copyng s by permsson of the Very arge Data Base Endowment. To copy otherwse, or to republsh, requres a fee and/or specal permsson from the Endowment Proceedngs of the 29 th VDB Conference, Berln, Germany, 2003 sources, E-commerce search engnes (ESEs) account for a large proporton. t s of great mportance to provde a unfed access to multple ESEs sellng smlar products because ths would allow users to search and compare products from multple stes wth ease. n ths paper, we call a system that supports unfed access to multple ESEs as an E-commerce metasearch engne (EMSE for short). Currently, there are a number of EMSEs on the nternet, such as and However, ther technques are not publcly avalable. To the best of our knowledge, most exstng EMSEs are bult manually or sem-automatcally. Furthermore, as ESEs operate autonomously, changes/upgrades to them may affect the operaton of the EMSE. As a result, mantanng the operaton of an EMSE s a costly long-term effort. Our E-Metabase proect ams to automate the process of buldng large-scale EMSEs so as to sgnfcantly reduce the cost of buldng and mantanng EMSEs. Ths proect conssts of a number of components. Frst, a specal crawler s used to crawl the Web and dentfy ESEs from the fetched Web pages. Second, the found ESEs are clustered nto dfferent groups such that ESEs n the same group sell the same type of products (.e., n the same doman). Thrd, the nterfaces of the ESEs n the same group are ntegrated nto a unfed nterface that becomes the nterface of the EMSE for ths group. Fourth, a global query submtted to the EMSE s mapped to queres for the underlyng ESEs. Ffth, a component that s responsble for connectng to each ESE s bult so that a query can be passed to and results can be returned back from each ESE. Sxth, nformaton of every product returned by each ESE needs to be correctly extracted from the returned result pages by an nformaton extracton program. Fnally, the extracted results from dfferent ESEs need to be fltered accordng to the global query and then combned nto a sngle lst for presentaton to the user based on some desred features, say prce. WSE- ntegrator s desgned to automate the nterface ntegraton step. n ths paper, we present our technques used to buld WSE-ntegrator. WSE-ntegrator s appled to each group of ESEs to produce an ntegrated

2 nterface for ths group of ESEs. Wthout loss of generalty, n ths paper, we assume that all ESEs under consderaton are n the same product doman. (Our technques for clusterng ESEs can be found n [PMH03]) Ths paper has the followng contrbutons. Frst, we provde a comprehensve soluton to the nterface ntegraton problem. nterface ntegraton ncludes schema ntegraton, attrbute value mergng, format ntegraton and layout generaton of global attrbutes n the global (ntegrated) nterface. n contrast, related exstng works deal wth only schema ntegraton (see Secton 2). Second, we propose an automated soluton for nterface ntegraton usng only general (.e., doman-ndependent) knowledge. Most exstng works employ manual or semautomatc technques. One of the key ssues n nterface ntegraton s to dentfy matchng attrbutes from dfferent nterfaces and we propose a clusterng and weght-based two-step method to tackle ths problem. Furthermore, ths method also solves a rarely addressed ssue,.e., fndng approprate names for attrbutes n the global nterface automatcally. Our expermental results based on 20 and 50 nterfaces n two dfferent domans ndcate that WSE-ntegrator can acheve hgh attrbute matchng accuracy and can produce hgh-qualty ntegrated search nterfaces wthout human nteractons. The rest of ths paper s organzed as follows. Secton 2 brefly revews prevous research works related to our work. Secton 3 dscusses nterface representaton used by WSE-ntegrator. n Secton 4, we present our method for matchng attrbutes. n Secton 5, we dscuss mergng attrbute domans. n Secton 6, we dscuss global nterface constructon. Expermental results are reported n Secton 7. Secton 8 concludes the paper. 2. Related work A thorough survey of approaches for automatc schema matchng can be found n [RB01]. [GKD97] predefnes the mappng rules for each attrbute and assembles these rules nto a knowledge base for nterpretaton when a query s handled. [RO96] uses a world vew to represent all sources but t does not dscuss how to construct the world vew automatcally. [DEW96] predefnes each doman descrpton that ncludes nformaton about product attrbutes, and then uses some heurstcs and mappng functons for the felds of each search nterface but t does not provde much detal about user nterface. [BBB01, BCV01] use Descrpton ogcs, Common Thesaurus and clusterng technques for semantc schema ntegraton. WordNet [WDNT] s also used to dentfy semantc relatonshps between schema terms. Ths s a sem-automatc approach as the ntegraton process stll nvolves human nteracton. Furthermore, the approach used for matchng attrbutes s manly based on name affnty and structure affnty, and only a few metadata (such as key, foregn key) of schemas are used. [C00] uses neural network technques and focuses on utlzng both schema level and data contents level metadata to automatcally dentfy matchng attrbutes. Our approach has adopted some deas from [C00] but there are sgnfcant dfferences (see Secton 4.3 for more comparson wth [C00]). [MBR01] nvestgates algorthms for generc schema matchng. t combnes a number of past technques, such as lngustc-based matchng and some metadata of schemas. t proposes structure-matchng algorthms for herarchy schemas (tree structures) n whch a structural smlarty s computed between each par of schema elements. However, how a global schema s obtaned s not dscussed. [DDH01] uses and extends machne-learnng technques to semautomatcally fnd mappngs between source schemas and the medated schema. Ths approach needs human users to manually construct the semantc mappngs between a small set of data sources (tranng) and the medated schema. [HC03] uses a statstcal approach for schema ntegraton of query nterfaces of the deep web. t argues that as the Web sources prolferate the aggregate schema vocabulary of sources n the same doman tends to stablze at a relatvely small sze, and that underlyng these sources, there exsts a unfed hdden schema model. Then t uses statstcal probablty and goes through three steps (hypothess modelng, generaton and selecton) to obtan the hdden schema model. t uses only attrbute names for statstcs, and t does not apply other schema nformaton such as doman type, default value and attrbute values whch, we fnd, based on our experments, to be very effectve n nterface ntegraton. t s not clear how semantc relatonshps between names (such as synonymy and hypernymy) are obtaned n ths work. n addton, t dscusses only schema ntegraton, but not attrbute mergng and global nterface generaton. [MGR02] uses the dea of P packet floodng to flood the smlarty of elements. t converts each schema nto a drected labeled graph. On the bass of the graph model, a part of the smlarty of two elements propagates to ther respectve neghbors. The smlarty floodng algorthm termnates after a fx pont s reached and some flters are used to get a subset of the result mappng. No lngustc name matchng s done beyond utlzng a smple strng matcher to compare common prefxes and suffxes of lterals. Ths approach s not sutable for search nterfaces because name matchng plays an mportant role n the ntegraton of search nterfaces. [DR02] dscusses combnng dfferent matchng algorthms n a flexble way and supports dfferent ways to combne match results. n [DR02], schemas are represented as rooted drected acyclc graphs. t mantans a matcher lbrary for smple matchers such as approxmate strng matcher, synonym matcher, data type matcher and hybrd matchers (e.g. name matcher and structural matchers). t uses data type but not other schema and doman nformaton to help fnd matches. The man dfference between our work and exstng works s that we am to perform comprehensve nterface

3 ntegraton automatcally whle others perform only schema ntegraton, employng mostly manual or semautomatc technques. There are bascally no publshed work, to the best of our knowledge, on automatc attrbute value mergng, format ntegraton and layout generaton. Furthermore, compared wth other approaches for Web sources ntegraton, we utlze a rcher set of schema and doman nformaton to fnd matchng attrbutes and we utlze the nformaton dfferently (see Secton 4). 3. nterface representaton A search nterface for E-commerce s usually presented through an HTM form n a Web page [HT4]. t may contan elements such as text box, rado button, check box and selecton lst, and each element usually has a label (descrptve text) assocated wth t. Users fll out the form and then submt the flled form as a query through the browser to the remote server. The server then returns to users the results that satsfy the query condtons. n general, much useful nformaton s embedded n the HTM source fle of each local nterface and such nformaton needs to be extracted for nterface ntegraton. n ths paper, we do not dscuss how to extract the needed nformaton (some related work can be found n [RGM01, DEW96], and our work on ths wll be reported n another paper). nstead, we focus on what nformaton should be used to represent each search nterface for the purpose of nterface ntegraton. Each local ESE nterface can be conceptually vewed as a partal export relatonal schema of the underlyng product database. n our nterface representaton, each label s consdered as the name of an attrbute of the underlyng products. Each attrbute has one or more assocated elements. Each element has a format whch s the nput format of the element. There are generally four types of formats: text box, rado button, check box and selecton lst. Each element also has a doman that defnes the set of values that can be used to nstantate the element when formng a query. Text box allows users to nput whatever value they want and thus the correspondng doman can be consdered to be nfnte. A selecton lst provdes a fnte number of pre-determned values for users to select whle a check box and rado button have one assocated value. These three formats thus have a fnte doman. Often multple check boxes or multple rado buttons are used together to accomplsh the same functon as a selecton lst. n addton, each element or a group of elements may have ts or ther default value, whch s used to help formng a query when a user does not make a dfferent selecton. For each attrbute, there s a type for ts values. Sx value types are consdered and they are date, tme, currency, number, char and d. The d type ndcates that the attrbute s used for dentfcaton purpose (e.g., product number, order number). The type nformaton can be obtaned through analyzng attrbute name (contanng date, tme, prce etc.) and the pattern or format of attrbute values (that are vewable on the nterface). For example, $300 for currency and 3:00PM for tme. When the value type s dffcult to determne, a default value type,.e., char, s used. Whenever possble, the scale/unt of the attrbute values s also extracted. For example, all values wth US$ are consdered to have the same unt but US$ and CAN$ have dfferent unts even though they are both of currency value type. Fnally, each attrbute has ts layout poston n the nterface. The poston value s determned by the layout order of attrbutes n an nterface. More mportant attrbutes are usually arranged ahead of less mportant ones. n addton to the label of an attrbute, each element of the attrbute may have ts own label. For example, n Fgure 1, attrbute publcaton year has two text box elements wth ther own labels after and before, respectvely. Such label helps defne the semantc meanng of the element. Fgure 1: Examples of element relatonshp type When an attrbute has multple elements, these elements are related n some way. We dentfy the followng four relatonshp types among related elements based on our observatons. Range type: t refers to the stuaton where two or more elements are used to specfy the range semantcs for an attrbute. For example, n Fgure 1, the prce range has two related elements ndcatng the mnmum and the maxmum values allowed. Part type: t refers to the part-of relatonshp. For example, n Fgure 1, author has two elements frst name and last name and each of them s part of author. Range type s a specal case of part type. Group type: Multple check boxes/rado buttons are sometmes used together to form a sngle semantc concept (attrbute). n ths case, the labels assocated wth the check boxes/rado buttons are values of the attrbute. n Fgure 1, attrbute Platform has a group of check boxes. Constrant type: An element can be used as a constrant for another element. For example, for a text nput box, a check box may be used to specfy whether or not the nput s case senstve. n ths case, the check box s meanngless wthout beng related to the text nput box.

4 To summarze, n our approach, each attrbute A s represented as A = (N, P, DT, DF, VT, SU, ES, R), where N s the name (label) of A, P s the layout poston of A, DT s the doman type of A, DF s the default value of A (possbly null and there s at most one default value for each group of check boxes and rado buttons),vt s the value type of A, SU s the scale/unt of A, ES s the set of elements assocated wth A and R s the relatonshp type between the elements n ES. For example, for attrbute Prce Range n Fgure 1, ts ES contans two text box elements labeled between US$ and and US$, and ther relatonshp type s range type. f ES contans only one element, then R s null. Each element E n ES s tself represented as a quadruplet E(, F, V, DV), where s ts label (possbly empty), F s the format, V s the set of values (for fnte doman type of elements only), and DV s the default value of the element (possbly null ). 4. Matchng attrbutes n ths secton, we present our method for matchng attrbutes from multple local nterfaces. 4.1 Semantc relatonshps Semantc relatonshps between concepts or obects are very mportant n the database schema ntegraton and Web source ntegraton. n our approach, we dentfy the followng three semantc relatonshps between terms (attrbute names or element s values): Synonymy, Hypernymy/Hyponymy and Meronymy [M95, WDNT, BCV01, BBB01]. Gven a term, we use WordNet [M95, WDNT] to get ts synonyms, hypernyms and meronyms, f applcable. Synonymy. Term T 1 s a synonym of term T 2, denoted by S(T 1,T 2 ), f T 1 s n the synonym-set of T 2. Hypernymy/Hyponymy. Term T 1 s a hypernymy of term T 2, denoted by H(T 1,T 2 ), f T 1 s more generc than T 2. For example, H(tree, maple) and H(format, hardcover). Meronymy. Term T 1 s a meronym of term T 2, denoted by M(T 1,T 2 ), f T 1 s a part of T 2. For example, M(frst name, name) and M(last name, name). However, hypernymy and meronymy terms that can be found from WordNet are very lmted. n WSE- ntegrator, we also dentfy hypernymy and meronymy relatonshps of two terms usng the nformaton n the nterface representatons. For example, suppose we have two nterfaces, one has a hardcover attrbute and the other has a format attrbute that contans a value hardcover. From ths, we can dentfy the hypernymy between the two attrbutes: H(format, hardcover). For meronymy, we use the part relatonshp of elements. For example, f a search nterface contans an author attrbute that has two parts: frst name and last name, we can say M(frst name, author) and M(last name, author). Other nterfaces that contan frst name or last name wthout author can use the relatonshp to match. 4.2 Normalzaton Before ntegraton, attrbute names and element values are normalzed as follows to reduce msmatches. Convert each name or value strng to lower case equvalents. Remove all content n parentheses, ncludng parentheses. Replace all characters that are not alphanumerc wth a space character. Tokenze each strng usng space, replace abbrevaton and acronym (f any) [MBR01] and use WordNet to get the base form of each token. Remove stop words when a name or a value conssts of multple words. 4.3 Mergng attrbutes Mergng attrbutes has two tasks: one s to fnd the matchng attrbutes from search nterfaces to be ntegrated, and the other s to determne what global attrbute name should be used for each group of matchng attrbutes. To the best of our knowledge, no n-depth dscusson of the second task has been reported n the lterature. The SEMNT approach n [C00] utlzes and extends the metadata characterstcs n [NE89] to determne matchng attrbutes. SEMNT ntroduces three levels of metadata that can be used: attrbute names (the dctonary level), feld specfcaton (the schema level, e.g., data type and prmary key) and attrbute values and patterns (data content level). SEMNT ust focuses on usng the metadata at the schema level and data content level to determne attrbute correspondences. t descrbes 20 characterstcs at the two levels, such as data length, data type, nullable, prmary key, default scale, mnmum, maxmum, average and so on. We adopt the basc dea of the SEMNT approach for the attrbute-matchng task n the sense we also use metadata characterstcs n multple levels. Our approach dffers from the SEMNT approach n four aspects. Frst, the set of characterstcs used s dfferent. For example, prmary key nformaton and maxmum value are readly avalable n a database context but they are not avalable for nterface ntegraton. On the other hand, nformaton such as element format apples to only nterface ntegraton. Second, we utlze all three levels of metadata nstead of ust two. Thrd, we classfy matches based on dfferent metadata nto postve matches and predctve matches (see below). Fourth, SEMNT uses neural network technques but we don t. Furthermore, the SEMNT approach does not address the second task of mergng attrbutes. As mentoned above, n our approach, we use the three levels of metadata to determne matchng attrbutes. At the dctonary level, we explore sx possble matches on attrbute names: exact match, approxmate strng match [WM92], vector space smlarty match [FB92] (see secton 4.3.2), synonymy match, hypernymy match and meronymy match. At the schema level, scale, value type,

5 doman type, default value and Boolean property are used. At the data content level, we focus on comparng values n the elements. n our approach, we classfy the dfferent matches nto two types: postve matches and predctve matches. postve matches nclude exact name match, semantc (synonymy, hypernymy and meronymy) matches and value-based match. For value-based match, we employ exact match, approxmate strng match, synonymy match and hypernymy match to compare values. When enough values from the two attrbutes are matched (a threshold s used), value-based match s recognzed as succeeded. When one of the postve matches occurs durng our ntegraton process, the correspondng attrbutes are recognzed as matched. Predctve matches consst of approxmate name match, vector space smlarty match of names, and matches based on scale, doman type, value type, default value, Boolean property and value pattern. Predctve matches must be suffcently strong (based on a weght threshold) for two attrbutes to be recognzed as matched. Our approach for accomplshng the two tasks of mergng attrbutes s descrbed n the next two subsectons Clusterng (postve match) Ths s to group attrbutes nto clusters based on the postve matches between attrbutes. Ths step consders all nterfaces. There are three steps for the clusterng: Group attrbutes nto clusters based on the exact match of attrbute names n all nterfaces of the same doman. Thus, after ths step, all attrbutes n the same cluster have the same attrbute name. For each dstnct attrbute name, the number of nterfaces havng the attrbute s counted. Values of all attrbutes n each cluster, f any, are unoned. Merge the clusters produced n the frst step based on the matchng of values n each cluster and the semantc (synonymy, hypernymy and meronymy) matches of attrbute names. New clusters are generated n ths step. Determne the representatve attrbute name of each cluster produced n the second step. Ths attrbute name s a canddate to be the global attrbute name to whch other attrbutes n the cluster are mapped. To determne the representatve attrbute name of each cluster, generally we employ the maorty rule. n other words, the attrbute name that appears n most nterfaces n a cluster would be chosen as the representatve attrbute name of the cluster. However, we also consder the semantc relatonshps among attrbute names n the cluster. For example, f a cluster contans four dfferent attrbute names: format, bndng type hardcover and paperback, we do not choose hardcover or paperback as the representatve name of the cluster even f they appear n more nterfaces. The reason s that hardcover or paperback s a knd of format or bndng type. Therefore, durng the clusterng we buld hypernymy herarchy trees for attrbute names n the cluster. We then choose the representatve attrbute name among the roots usng the maorty rule. For the prevous example, we ust need to compare the number of occurrences between format and bndng type. n our approach, the clusterng step does prelmnary attrbute matchng and representatve attrbute dentfcaton. The step ust collects the knowledge about what attrbutes should be matched based on the postve nformaton. No ntermedate ntegrated nterface s yet constructed after ths step. There are several reasons to perform clusterng. Frst, count the number of nterfaces an attrbute name appears n; ths nformaton s mportant for determnng the global attrbute names. Second, determne the representatve attrbute name of one cluster n advance. Thrd, make sure that attrbutes that should be matched (based on postve matches) are matched together. Ths can smplfy the comparsons n the weghtbased match step and avod msmatches. Our experments ndcate that ths two-step approach s effectve Weght-based match (predctve match) Ths step s to utlze the knowledge obtaned n the frst step and the predctve matches to construct the ntegrated nterface and fnalze the global attrbute names. ntally, there s no ntermedate ntegrated nterface. n ths case, gven a local nterface, our approach takes an attrbute n t and looks up the representatve attrbute name of the cluster n whch the attrbute appears. Then the representatve attrbute name s added to the ntermedate nterface (t s empty ntally) as the global attrbute name. These two operatons are repeated untl all attrbutes n the local nterface are handled. The global attrbute names may be adusted later. Once the frst ntermedate nterface s generated, weght-based match begns to work. Frst, we present the defnton of weght-based match as follows: Defnton: Gven an ntegrated ntermedate nterface ={A 1, A 2, A 3,, A n } and a local nterface ={ A 1, A 2, A 3,, A m }, where A s an attrbute of, A s an attrbute of, the mapped attrbute n for an attrbute A s the one wth the hghest weght: W( A, A ) > W( A, A ) > w, = 1,..., n, k= 1,..., n, k, where W( A, k A ) s the weght of attrbutes A and A, w s the weght threshold. n our approach, the weght-based match computes the matchng weght between two attrbutes and then predcts whether the two attrbutes are matched based on the weght. The weght between attrbutes A and A can be computed based on the followng metrcs (predctve matches):

6 1. Approxmate strng match Compare the two attrbute names to fnd out f the edt-dstance between the two strngs s wthn the allowed threshold. We use an approxmate strngmatch algorthm [WM92] to fnd the match. f the edt-dstance s wthn the allowed threshold T, assgn a postve weght Wam; otherwse Wam s Vector space smlarty The vector space smlarty s the smlarty between two text strngs based on the Vector Space Model [FB92]. The approach s also used n [Coh98]. We tokenze each strng and get the term frequency of each term n each strng. The weght of ths metrc s the Cosne smlarty of two strngs. Wvss ( v, w ) = = 1 m v = 1 m ( v ) 2 m ( w where m s the number of unque terms n the two strngs, w s the term frequency of the th term n attrbute strng w and v s the term frequency of the th term n attrbute strng v. 3. Compatble doman We consder four doman types: fnte, nfnte, hybrd and range. The doman type of an attrbute s derved from ts assocated element(s). f the element(s) of the attrbute has the range semantcs, the doman type of the attrbute s range. Hybrd s the combnaton of fnte and nfnte. f an attrbute doman s hybrd, users can ether select from a lst of pre-compled values or fll n a new value. n our approach, the hybrd type s only lmted to the ntermedate nterface and the global nterface. Hybrd s compatble wth fnte and nfnte; the same types are compatble. f two attrbutes have compatble doman types, assgn a weght Wcd; otherwse Wcd s 0. n addton, we observed that range type s used much less often than fnte and nfnte types. Thus, f two attrbutes have range doman type, we double Wcd. 4. Value type match As mentoned n Secton 3, we consder sx value types: date, tme, currency, number, char and d. f two attrbutes have the same value type, assgn a weght Wvtm; otherwse Wvtm s 0 5. Scale/unt match Consder two attrbutes that have the same value type. f they also have the same scale or unt, assgn a weght Wcs; otherwse (.e., f they have dfferent value types or dfferent scales/unts), Wcs s 0. For example, f two attrbutes are both of currency type and ther values are n US$, then Wcs s assgned to the overall match of the two attrbutes. 6. Default value n a search nterface, some elements may have ther default values. n some cases, an element may have w = 1 ) 2 no assocated label, but t has a default value whch s mportant for the element to fnd a matchng attrbute. n addton, f an attrbute s n a cluster, then ts default value s consdered as one of the default values of the cluster. So when we check default values of two attrbutes we check default values of the two attrbutes themselves as well as ther clusters. f two attrbutes have the same default value, assgn a weght Wdv; otherwse Wdv s 0; 7. Boolean property f an attrbute has ust a sngle check box, ths check box s usually used to mark a yes-or-no selecton. Such an attrbute s consdered to have a Boolean property. f both attrbutes have the Boolean property, assgn a weght Wbp; otherwse Wbp s Value pattern We apply value pattern only to the numerc attrbutes. We compute the average of all numerc values n each attrbute. f the two averages are close, assgn a weght Wvp; otherwse Wvp s 0. The weght between attrbutes A and A s the sum of the above eght metrc weghts (the values of these weghts are determned expermentally, see Secton 7.2.2): W( A, A ) = Wam + Wvss + Wcd + Wvtm + Wcs + Wdv +Wbp+ Wvp Gven the ntermedate nterface and an attrbute A n a local nterface, the approach frst looks up the attrbute thesaurus to see f the attrbute s already mapped to a global attrbute n the ntermedate nterface. The attrbute thesaurus s establshed ncrementally durng the weght-based matchng process. f t has been mapped, the attrbute A would drectly be mapped to the global attrbute name. f t has not, the representatve attrbute name would be found usng the name of attrbute A. Then recheck the attrbute thesaurus usng the representatve attrbute name to see f the representatve attrbute name s mapped to a global attrbute. f the mappng s found, A s mapped to the global attrbute; otherwse compute the weghts between A and all attrbutes n the ntermedate nterface. After these weghts are computed, the attrbute wth the hghest weght s selected. f ths weght s greater than the threshold w, the selected attrbute s consdered as the matchng attrbute of the attrbute A ; otherwse, we assume that no matchng attrbute s found. n the former case, we have to determne the global attrbute name between the two attrbutes. n our approach, the attrbute name that appears n more local nterfaces would be selected. f applcable, the correspondng entry n the attrbute mappng table (whch keeps mappngs between

7 each global attrbute and ts correspondng local attrbutes) s changed and so s the thesaurus. n the latter case, the attrbute A s added as a new attrbute to the ntermedate nterface and a new entry for the attrbute s added n the attrbute mappng table Mantenance of ntegrated nterface After a global nterface s generated, t s lkely that new local nterfaces need to be added to or some exstng local nterfaces need to be removed from the global nterface from tme to tme. Ths requres mantanng the global nterface. For addng new local nterfaces, the frst step of clusterng needs to be performed on the new local nterfaces, followed by the second step of clusterng to cluster the output of the frst step to the exstng clusters. The representatve attrbute name may need to be updated based on the current and prevous statstcal and semantc knowledge. Then, the weght-based match s performed. For removng some local nterfaces from the global nterface, we remove the attrbute names and ther correspondng values from the clusters and the related mappng nformaton from the attrbute mappng table. n both cases, the count ndcatng the number of nterfaces contanng an attrbute needs to be updated accordngly, and f applcable, the global attrbute name of the global nterface may also need to be changed. 5. Mergng attrbute domans When a local attrbute s mapped to a global attrbute n the ntermedate nterface, we must determne the global attrbute doman after the mappng. Ths ncludes the followng two aspects: 1) The global doman type. As mentoned prevously, four doman types are supported n our approach and they are fnte, nfnte, hybrd and range. A compatble doman type between the two attrbutes should be used. 2) The attrbute values. Need to merge the values that represent the same concept and provde a set of values for the global attrbute. To deal wth these two ssues, dfferences between the two domans should be dentfed and resolved, ncludng format dfference, semantc conflct, scale dfference, range dfference and constrant dfference. Here we need to take a closer look at the range dfference. n Fgure 3, we can see that there are varous range formats. Two aspects need to be consdered n resolvng range conflct, one s about range modfers such as from, to, less than, under and so on, and the other s about range wdth. Fgure 3 shows that dfferent range domans may have dfferent range modfers and dfferent range wdth. The resoluton of range conflcts s to generate a global range doman that s compatble wth the range domans of the matchng attrbutes (see detals n secton 5.3). 5.1 Determne global doman type For a gven local attrbute A and a matchng global attrbute A, we use the followng rules to determne the new doman type for A : 1) fnte + fnte fnte 2) nfnte + nfnte nfnte 3) range + any type range 4) (fnte + nfnte) or (hybrd + fnte) or (hybrd + nfnte) hybrd The frst rule can be explaned as follows: f the local attrbute A s fnte and the global attrbute fnte, then the new global doman type of Other rules can be explaned smlarly. 5.2 Mergng alphabetc domans A s also A s fnte. f a local attrbute and ts matchng global attrbute are fnte or hybrd and have alphabetc values, we should consder how to merge ther values and form a new value set for the global attrbute. n WSE-ntegrator, ths s carred out n two phases. The frst phase s n the clusterng step dscussed n Secton n ths phase, attrbutes that have some values n common are grouped nto the same cluster. Furthermore, due to the matchng technques employed (exact match, approxmate strng match, synonymy match and hypernymy match), semantc relatonshps between values are dentfed. n the second phase, we use the knowledge of the relatonshps between values to merge values and generate a global value set. Phase 2 conssts of the followng steps. Frst, we cluster all values nto categores based on approxmate strng match, vector space smlarty match, synonymy match and hypernymy match. Thus, all values that are smlar, synonymy or hypernymy are clustered nto the same category. Next, we solve the followng two problems: (1) Whch value should be chosen as the global value f multple smlar and synonym values are n the same category? (2) How to provde values to users f the values n the same category have hypernymy relatonshps? For the frst problem, we can keep a counter for each value and use the maorty rule to choose the most popular value. As to the second problem, we need to make a tradeoff between choosng generc concepts and choosng specfc concepts as the choce would have dfferent effects on query cost and nterface frendlness. The cost of evaluatng a global query ncludes the cost of nvokng local ESEs to submt sub-queres, the cost of processng sub-queres at local ESEs, result transmsson cost and post-processng cost (e.g., result extracton and mergng). f we choose only generc concepts as global values and do not use specfc concepts, a query aganst the global nterface may need to be mapped to multple values (correspondng to specfc concepts) n some local

8 nterfaces, leadng to multple nvocatons to the local search engnes. On the other hand, f we keep only specfc concepts and gnore generc concepts, users who want to query generc concepts (.e., have broader coverage) may have to submt multple queres usng specfc concepts, resultng n less user-frendly nterface. Our approach s to provde a concept herarchy of values to users, ncludng generc and specfc concepts. Ths remedes the problems of the prevous two optons and gves the users more flexblty to form ther queres. Value clusterng may produce multple categores and a value herarchy s created for each category. Each herarchy s lmted to at most three levels to make t easer to use. After these two phases, the mappngs between global values and local values are establshed. Subects Network Databases Programmng - anguages Subect TCP/P Wrelessnetwork Oracle Sybase Sql server C C++ Java Pascal Network --TCP/P --Wreless network Databases --Oracle --Sybase --Sql server Programmng- anguages --C --C++ --Java --Pascal Fgure 2: Example of mergng doman values Example 1: Consder two Web bookstore nterfaces, one has an attrbute subects wth values Network, Databases, Programmng anguages and so on and the other has a correspondng attrbute subect wth values TCP/P, Wreless network, Oracle, Sybase, Sql server, C, C++, Java, Pascal and so on. After clusterng the values, some semantc herarches between the values from the two nterfaces can be dentfed. There are three possble ways to generate the global doman values for subect. One s to use only generc concept values,.e., values from the frst nterface, namely Network, Databases, Programmng anguages etc. n ths case, suppose a user wants to fnd nformaton about Oracle. Snce Oracle s not avalable, the user has to select Databases on the global nterface and submt the query. Ths global query would have to be mapped to three sub-queres for the second nterface, namely Oracle, Sybase, and Sql server. Obvously, searchng based on Sybase and Sql server wastes the resources at the second ste and returns more useless results to the user. The second opton s to use only the values wth more specfc concepts,.e., the values from the second nterface. n ths case, a user who wants to fnd nformaton about database (not of any specfc type) needs to submt three queres respectvely usng Oracle, Sybase and Sql server. Ths s nconvenent to the user. Our approach wll organze related values nto a herarchy (see the box on the rght sde n Fgure 2). n ths case, f the user selects Databases, then the metasearch engne wll generate three sub-queres for the second ste on behalf of the user. On the other hand, f any of the three sub-concepts of Databases s selected, only that concept wll be used for the second ste but Databases wll be used for the frst ste. Ths soluton remedes the problems of the frst two solutons. We should pont out that not every category of values can form a herarchy. n that case, we ust provde a lst of values. 5.3 Mergng numerc domans To merge numerc domans, we need to perform the followng tasks: 1) Resolve scale dfference. We assume the dentfcaton of the scale/unt of a numerc attrbute has already been done by the nterface extractor. n our approach, we buld a scale relatonshp dctonary n advance for some popular scales. The system can look up the dctonary to fnd out how to map one scale to another scale. The numerc values n those attrbutes are transformed to the same global scale durng value mergng. 2) Understand the semantc dfferences nvolved. 3) Generate a global doman wth query cost taken nto consderaton. We dentfy two types of numerc domans: range numerc doman and non-range numerc doman. Nonrange numerc doman attrbutes may come from the numerc attrbutes that are ether fnte or nfnte. f the domans of the matchng local attrbutes are non-range numerc, we ust unon all values of these attrbutes for the global attrbute. For the rest of ths subsecton, we focus on range numerc doman. For the range numerc doman, three types of formats can be dentfed as shown n Fgure 3. 1) One selecton lst. The range type conssts of only one selecton lst, for example, the frst four selecton lsts n Fgure 3. 2) One selecton lst and one text box. The range doman s lke the publcaton date n Fgure 3, whch has two elements, one s a selecton lst for range modfer and the other s a text box for numerc value. 3) Two textboxes or two selecton lsts. The type conssts of two elements and each of them may be a textbox or a selecton lst. The examples are prce range, publcaton year n Fgure 3. From Fgure 3, we can see that numerc values are mostly combned wth other semantc words. To help the system understand such formats, we need to let the system know the meanng of the range modfers such as less

9 than, from, to and over. For ths purpose, we buld a semantc dctonary that keeps all possble range modfers for numerc domans. n addton to these range modfers, we also save the meanng of other terms related to numerc values. For example, n Fgure 3, we can see that baby and teen are n reader age. We have to specfy the real meanngs of these words to help the system know what they are. We can say that baby represents under 3 years, teen s years and adult s over 18 years. Then, we desgn a specal extractor that can extract the range modfers and the numerc values, and use the semantc meanngs n that dctonary to buld a semantc range table that can be understood by the system. The semantc range table keeps multple ranges correspondng to the orgnal ranges n the element(s). Ths table can be used n query mappng and submsson. ths end, we keep a lst for the matchng numerc attrbutes. Every tme when a numerc attrbute s mapped or added to the ntermedate nterface, the numerc values that are prevously extracted from the numerc attrbute are added to the lst (scale conflcts are resolved before ths step). When all attrbutes are matched, the lst s sorted n ascendng order of the values. The ranges are generated usng every two consecutve numerc values n the lst. For the mnmum and the maxmum values, under and over range modfers are used, respectvely. Range modfers Meanng ess than < Over > Under < Greater than > From* >= To* <= Between* >= And* <= After > Before < All All range Any All range o H nternal value 0 5 lessthan lessthan lessthan lessthan lessthan lessthan50 0 allrange Table 1: Range modfers dctonary * modfers to be used n pars Table 2: A range element semantc table Fgure 3: Examples of dfferent range formats Suppose we handle the element that has less than range modfers n Fgure 3. From ths element, we can obtan numerc values: 5, 10, 15, 20, 25 and 50 by extracton. We can also get the semantc words: all prce ranges and less than. Wth the nformaton, we can buld a range semantc table as shown n Table 2. The nternal values are the values n the HTM text that correspond to the values of the element. So far we have solved the frst two problems of mergng range numerc domans. The last thng we need to do s to generate a global range format that s compatble wth the local doman formats of the matchng attrbutes. And the global range format should consder query effcency as much as possble. ntutvely, a larger range condton n the global nterface would lead to more nvocatons to some local stes, causng more local server processng effort, more data transmsson and more postprocessng effort. Therefore, we am to reduce the range wdth of each range condton n the global nterface. To ocal range Under $10 From $10 to $20 From $20 to $30 From $30 to $40 From $40 to $50 Over $50 ocal range ess than $5 ess than $10 ess than $15 ess than $20 ess than $25 ess than $50 Global range Under $5 From $5 to $10 From $10 to $15 From $15 to $20 Form $20 to $25 From $25 to $30 From $30 to $40 From $40 to $50 Over $50 Fgure 4: Example of a global range doman Example 2: Suppose n Fgure 3 two attrbutes wth from and less than range modfers are matched, then the lst of values kept for the two matchng attrbutes s: 5, 10, 15, 20, 25, 30, 40 and 50. The global range format s shown n Fgure 4. From Fgure 4, we see that one global range condton s translated to only one approprate local range condton. For example, from $10 to $15 n the global range format s respectvely mapped to from $10 to $20 and less than $15 n the local range formats. Thus multple query nvocatons to local nterfaces are avoded and other costs ncludng post-processng tme are also reduced.

10 6. Generatng global nterface WSE-ntegrator uses the results of both the attrbute matchng and the attrbute doman mergng to generate the global nterface and show the nterface n HTM format. t also has to decde whch attrbute should appear n the global nterface and the layout of all the attrbutes. 6.1 Attrbute poston Each attrbute has ts layout poston n a gven local nterface. These layout postons reflect the mportance of the attrbutes as perceved by local nterface desgners and ther users, and they may nfluence users behavors of selectng attrbutes to use. To be user-frendly, we aggregate local mportance of each attrbute and arrange more mportant attrbutes ahead of less mportant ones. n WSE-ntegrator, the global layout poston of a global attrbute s computed as follows. P ( A ) = m = 1 P ( A where P (A) denotes the poston value of the -th global attrbute A, m s the number of local nterfaces to be ntegrated, P (A ) s the layout poston of the local attrbute n the -th local nterface that s mapped to A ; P(A ) s assgned the total number of global attrbutes when no matchng local attrbute exsts n the -th local nterface. All global attrbutes are lad out n ncreasng order of ther poston values. Clearly, usng ths method, attrbutes that appear n hgh postons (the frst poston s the hghest) n many local nterfaces are lkely to appear n hgh postons n the global nterface. 6.2 Attrbute selecton When a large number of local nterfaces are ntegrated, the global nterface may have too many attrbutes to be user-frendly. Whle some key attrbutes about the underlyng products appear n most or all local nterfaces, some less mportant attrbutes appear n only a small number of local nterfaces. One way to remedy ths problem s to trm some less mportant attrbutes from the global nterface. We use the global poston of each global attrbute to trm off less mportant attrbutes,.e., those that have large global poston values. A user-adustable threshold can be used to control ths. 7. mplementaton and expermental results 7.1 mplementaton WSE-ntegrator s developed usng JDK1.4 and s operatonal. WordNet1.6 s embedded nto the system through APs based on C. The GU of the system s shown n Fgure 5. The system can read the nterface descrpton of each Web ste and then dsplay the nterface descrpton vsually n a tree structure. From the ) tree vew, users can see all nformaton on each search nterface. The global nterface and the attrbute matchng nformaton are shown after the ntegraton s fnshed. Through the GU, users can remove or add any nterface at any tme on the fly. And the new global nterface s generated wthout startng from scratch. n addton, a user can choose any parameter value to trm attrbutes from the global nterface. To see a demo of WSE-ntegrator, go to the Web ste: Fgure 5: WSE-ntegrator nterface 7.2 Experments Evaluaton crtera Three qualtatve crtera for measurng the qualty of a global conceptual schema n the context of database schema ntegraton are proposed n [BN86] and they are Completeness and Correctness, Mnmalty and Understandablty. We rephrase these crtera and propose the followng prncples to gude the evaluaton of search nterface ntegraton. Correctness. Attrbutes that should be matched are correctly matched; attrbute domans for the matchng attrbutes are correctly merged and constructed. Completeness. f a result can be retreved drectly through a local nterface, then the result can also be retreved through the global nterface. Effcency. Global nterface constructon should consder query cost. Whle query cost s usually consdered at the query evaluaton tme, a bad global nterface may cause a hgh query cost despte of good query evaluaton algorthms. For example, supportng only very wde range condtons n the global nterface may cause too many local queres to be submtted to a local engne and too useless results to be transmtted to the metasearch engne. Frendlness. A global nterface should be smple and easy to understand and use by users. As an example, t s better to provde users a lst of values for an attrbute

11 Doman The number of Total Case 1 Case 2 Case 3 ama(%) amc(%) nterfaces Attrbutes 10 (1 st round) (2 nd round) Book 30 (3 rd round) (4 th round) (5 th round) Electroncs 10 (1 st round) (2 nd round) Average when these values are avalable for the attrbute rather than let users fll out wthout any knowledge. As another example, frequently used attrbutes should be arranged ahead of less frequently used ones. Effcency and frendlness of the global schema are taken nto consderaton by WSE-ntegrator (see Sectons 5 and 6). n the next subsecton, we report our expermental results for completeness and correctness for matchng attrbutes Expermental results To perform the experments, we collected the search nterfaces of 50 book Web stores and 20 electroncs Web stes, and then constructed the nterface representaton for each search nterface by hand (Tools for automatc constructon of ESE nterface representaton s under development and wll be reported n another paper). Correctness requres that attrbutes that should be matched across all search nterfaces be matched and that attrbutes that should not be matched not be matched. t also requres that the attrbutes n the global nterface be semantcally unque. To help measure the correctness of attrbute matchng, the global attrbute name and semantcs are used as a reference to measure how well local attrbutes are matched to the global attrbute. f there exst multple global attrbutes that are semantcally the same n the global nterface, the global attrbute wth more local attrbutes matched s consdered as the only real global attrbute whle others should be matched to t. There exst three cases for attrbute matchng: 1) Attrbutes are correctly matched to a unque global attrbute. 2) Attrbutes are ncorrectly matched to a global attrbute. 3) Attrbutes are correctly matched to a global attrbute, but they should belong to another matched group that has more matchng attrbutes. Our evaluaton metrc s called Attrbute Matchng Accuracy (ama), whch defnes what percentage of all attrbutes s correctly matched. ama n m = 1 = n a = 1 Table 3: Attrbute matchng correctness and completeness where n s the number of all local nterfaces used for ntegraton, m s the number of correctly matched attrbutes n the -th nterface (case 1), a s the number of all attrbutes n the -th nterface. Completeness requres that all contents and capabltes of each local nterface be preserved n the global nterface. As we mentoned above, three cases exst for attrbute matchng. Among these three cases, case 2 would reduce the completeness because some attrbutes are msmatched to a global attrbute; usng such global attrbutes may lead to ncorrect results from some local search engnes. For case 3, although the unqueness requrement s not satsfed, usng these global attrbutes can stll retreve results from the matchng local nterfaces. Therefore, case 3 matches do not affect completeness. We defne the Attrbute Matchng Completeness (amc) measure as follows: amc n ( a ) r = 1 = n a where r s the number of msmatched attrbutes n the -th nterface (case 2). We performed 5 rounds of experments on book nterfaces. n the frst round, 10 nterfaces were randomly selected and a global nterface was generated for them. n each subsequent round, 10 addtonal nterfaces were randomly selected and added to prevously selected nterfaces. Then a global nterface was generated from all selected nterfaces from scratch. Then we manually checked how well the attrbutes are matched. We also performed 2 rounds of experments usng nterfaces of electroncs stes. The expermental results are shown n Table 3. We can see that, on the average, the overall correctness and completeness of our approach for the two domans are 95.25% and 97.91%, respectvely. n addton, the results are remarkable stable (wth all correctness and completeness values wthn a narrow range) despte the dfferences n the number of nterfaces used and the product types. n all experments, the weghts for the seven metrcs n secton (the other metrc, smlarty match, has no fxed weght) are: Wam=0.5, Wcs=0.2, Wcd=0.1, Wvtm=0.4, Wdv=0.6, Wbp=0.1 and Wvp=0 (value pattern match s not used n our experments) and the weght = 1

12 threshold w s These values are obtaned from the experments usng the book nterfaces and they are appled to the electroncs nterfaces wthout changes. As the nterfaces for books are very dfferent from those for electroncs, the expermental results ndcate that the above parameter/threshold values are robust. 8. Conclusons n ths paper, we provded a comprehensve soluton to the problem of automatcally ntegratng the nterfaces of E-commerce search engnes. The problem s sgnfcantly dfferent from schema ntegraton for tradtonal database applcatons. Here we need to deal wth not only schema ntegraton, but also attrbute value ntegraton, format ntegraton and layout ntegraton. n ths paper, we descrbed our technques used to buld WSE-ntegrator. Wth approprate nterface representaton of local nterfaces, WSE-ntegrator automatcally ntegrates them nto a global nterface usng only doman (applcaton) ndependent knowledge. Our two-step approach based on postve matches and predctve matches for mergng attrbutes was shown to be very effectve by our experments. We beleve that the proposed approach can also be appled to other domans of E-commerce or ones beyond E-commerce such as dgtal lbrary and some professonal databases on the nternet. Whle good results were obtaned usng our method, there s room for mprovement. One possblty s to use the Open Drectory Herarchy to fnd more hypernymy relatonshps. One possble way to reduce the case 3 problem n Secton s to allow an attrbute n a local nterface to match more than one attrbute n the ntermedate nterface n Secton We wll nvestgate these possbltes n the near future. Acknowledgements Ths work s supported n part by the followng grants from Natonal Scence Foundaton: S , S , EA and ARO We thank Mr. eond Botsov for provdng us hs agrep algorthm mplementaton (tman.narod.ru/englsh/aboutnotser.html). References [BN86] C. Batn, M. enzern, S. Navathe. A Comparatve Analyss of Methodologes for Database Schema ntegraton. ACM Computng Surveys, 18(4): , December [BBB01]. Benett, D. Beneventano, S. Bergamasch, F. Guerra and M. Vncn. S-Desgner: An ntegraton Framework for E-Commerce. 17 th JCA-01, Seattle. [BCV01] S. Bergamasch, S. Castano, M. Vncn, D. Beneventano. Semantc ntegraton of Heterogeneous nformaton Sources. Journal of Data and Knowledge Engneerng, 36(3): , [Coh98] W. Cohen. ntegraton of Heterogeneous Databases Wthout Common Domans Usng Queres Based on Textual Smlarty. ACM SGMOD Conference, Seattle, WA, [DR02] H. Do, E. Rahm. COMA- A System for Flexble Combnaton of Schema Matchng Approaches. The 28 th VDB conference, Hong Kong, [DDH01] A. Doan, P. Domngos, A. Halevy. Reconclng Schemas of Dsparate Data Sources: A Machne-learnng Approach. ACM SGMOD Conference, May [DEW96] R. B.Doorenbos, O. Etzon, and D. S.Weld. A Scalable Comparson-Shoppng Agent for the World Wde Web. Techncal Report UW-CSE , Unversty of Washngton, [FB92] W. Frakes and R. Baeza-Yates. nformaton Retreval: Data Structures & Algorthms. Prentce Hall, Englewood Clffs, N.J [GKD97] M. Genesereth, A. Keller, O. Duschka. nfomaster: An nformaton ntegraton System. ACM SGMOD Conference, May 1997 [HC03] B. He, K. Chang. Statstcal Schema ntegraton Across the Deep Web. ACM SGMOD Conference, [HT4] HTM4: [NE89] J. arson, S. Navathe, R. Elmasr. A Theory of Attrbute Equvalence n Databases wth Applcaton to Schema ntegraton. EEE Transactons on Software Engneerng, Vol.15, No.4, Aprl [RO96] A. evy, A. Raaraman, J. J.Ordlle. Queryng Heterogeneous nformaton Sources Usng Source Descrpton. The 22 nd VDB Conference, nda, 1996 [C00] W., and C. Clfton. SEMNT: A Tool for dentfyng Attrbute Correspondences n Heterogeneous Databases Usng Neural Networks. Data & Knowledge Engneerng, 33: 49-84, [MBR01] J. Madhavan, P. Bernsten, E. Rahm. Generc Schema Matchng wth Cupd. VDB Conference, [MGR02] S. Melnk, H. Garca-Molna, and E. Rahm. Smlarty Floodng: A Versatle Graph Matchng Algorthm and ts Applcaton to Schema Matchng. EEE Conference on Data Engneerng, San Jose, [M95] A. Mller. WordNet: A excal Database for Englsh. Communcatons of the ACM, 38(11): 39-41,1995. [PMH03] Q. Peng, W. Meng, H. He, and C. Yu. Clusterng of E-Commerce Search Engnes. Submtted for publcaton, [RGM01] S. Raghavan, H. Garca-Molna. Crawlng the Hdden Web. The 27 th VDB Conference, [RB01] E. Rahm, P. Bernsten. A Survey of Approaches to Automatc Schema Matchng. VDB Journal, 10: , [WDNT] WordNet: [WM92] S. Wu and U. Manber. Fast Text Searchng Allowng Errors. Communcatons of the ACM, 35(10):83-91, October 1992.