Characterizing the Influence of Domain Expertise on Web Search Behavior

Transcription

1 Characterizing the Influence of Domain Expertise on Web Search Behavior Ryen W. White Microsoft Research One Microsoft Way Remon, WA Susan T. Dumais Microsoft Research One Microsoft Way Remon, WA Jaime Teevan Microsoft Research One Microsoft Way Remon, WA ABSTRACT Domain experts search ifferently than people with little or no omain knowlege. Previous research suggests that omain experts employ ifferent search strategies an are more successful in fining what they are looking for than non-experts. In this paper we present a large-scale, longituinal, log-base analysis of the effect of omain expertise on web search behavior in four ifferent omains (meicine, finance, law, an computer science). We characterize the nature of the queries, search sessions, web sites visite, an search success for users ientifie as experts an non-experts within these omains. Large-scale analysis of real-worl interactions allows us to unerstan how expertise relates to vocabulary, resource use, an search task uner more realistic search conitions than has been possible in previous small-scale stuies. Builing upon our analysis we evelop a moel to preict expertise base on search behavior, an escribe how knowlege about omain expertise can be use to present better results an query suggestions to users an to help non-experts gain expertise. Categories an Subject Descriptors H.3.3 [Information Storage an Retrieval]: Information Search an Retrieval query formulation, search process General Terms Experimentation, Human Factors Keywors Domain expertise, Web search 1. INTRODUCTION Web searchers iffer from each other in many ways that can greatly influence their ability to carry out successful searches. One way in which they can iffer is in their knowlege of a subject or topic area. Domain expertise is not the same as search expertise [22], as it concerns knowlege of the subject or topic of the information nee, rather than knowlege of the search process. Domain expertise has been stuie extensively in the information science community (see [23] for a review). Stuies of omain expertise have highlighte several ifferences between experts an non-experts, incluing: site selection an sequencing [4], task completion time [3], vocabulary an search expression [2], the number an length of queries, an search effectiveness [24]. These stuies involve small numbers of subjects with carefully controlle tasks, making it ifficult to generalize their finings. Permission to make igital or har copies of all or part of this work for personal or classroom use is grante without fee provie that copies are not mae or istribute for profit or commercial avantage an that copies bear this notice an the full citation on the first page. To copy otherwise, to republish, to post on servers or to reistribute to lists, requires prior specific permission an/or a fee. WSDM 09, February 9-12, 2009, Barcelona, Spain Copyright 2009 ACM $5.00. In this paper we buil on this previous research via a large-scale log-base stuy of web search behavior. We contrast the search strategies of omain experts with those of non-experts through analysis of naturalistic interaction log ata over a three-month perio of time. This large-scale analysis allows us to ientify greater iversity in vocabulary, web site visits, an user tasks than is possible with smaller-scale laboratory stuies. In aition we evelop methos for ientifying omain experts using online search interaction patterns rather than offline tests of expertise. We focus on four omains meicine, finance, law, an computer science with complex subject matter an a large potential benefit to non-experts in ientifying effective search strategies. In aition to highlighting ifferences in the search behavior of experts an non-experts, we escribe the possible benefits of being able to ientify omain experts an leverage their querying strategies an source selection abilities. Search tools currently provie the same experience to users regarless of expertise. A cariologist searching for the latest research stuies on heart isease gets the same search results for the query heart isease as a newly-iagnose patient with little backgroun in the area. We believe that by unerstaning how people s omain expertise affects their search behavior, we can better support interactions at the appropriate level, an help non-experts gain expertise. The remainer of this paper is structure as follows. Section 2 presents relate work on omain expertise. Section 3 escribes the search logs, an Section 4 the approach use to ientify experts within them. In Section 5 we iscuss ifferences an commonalities in the interaction behavior of omain experts an omain non-experts. Section 6 presents a classifier that can ientify users, actions, an sessions as expert or non-expert base on observable behavior, an iscusses how such a classifier can be use to improve the Web search experience for people of varying omain expertise. We conclue in Section RELATED WORK Research on omain expertise has examine ifferences between experts an non-experts in three main classes of search behavior: query attributes (choice of search terms, query length an syntax), search strategies an tactics (resource selection, sequence of actions, mix of querying an browsing), an search outcomes (accuracy, time). Many of these stuies were conucte in the context of library systems rather than the general web search, an involve small numbers of participants in laboratory settings. Allen [2] examine the relationship between topic knowlege (in the area of Voyager explorations of Neptune) an search behavior in an early online library catalog. He foun the searchers with high omain knowlege ha greater familiarity with the vocabulary for the topic an foun more items. Hsieh-Yee [13] foun that library science stuents use more of their own search

2 terms on a library science topic, an use more terms suggeste by external sources (thesaurus an synonym list) for a topic in which they ha little omain knowlege. They also spent more time preparing queries an examining results when they ha little omain knowlege. More recently, Hembrooke et al. [11] investigate the effects of omain knowlege on users search term selection an reformulation strategies for web searches. They foun that self-rate omain experts issue longer an more complex queries than novices. Further, experts use elaborations as a reformulation strategy more often compare with the simple stemming an backtracking moifications use by novices. Wilemuth [23] examine the search behavior of meical stuents in microbiology. In this experiment, stuents were observe at three points of time (at the beginning of the course, at the en of the course, an six months after the course), uner the assumption that omain expertise changes uring a semester. Some search strategies, most notably the graual narrowing of the results through iterative query moification, were the same throughout the observation perio. Other strategies varie over time as iniviuals gaine omains knowlege. Novices were less efficient in selecting concepts to inclue in search an less accurate in their tactics for moifying searches. Vakkari et al. [19] also examine stuents at multiple points in time, as they were eveloping their thesis proposal. One important change in behavior was the use of a more varie an more specific vocabulary as stuents learne more about their research topic. An important methoological issue with stuies that examine changes in expertise of an iniviual over time is that iniviuals also acquire many other kins of knowlege uring that time (e.g., in Wilemuth s stuy stuents also acquire search knowlege of the atabase of microbiology facts that was use in the stuy), so it is ifficult to isolate omain expertise from search expertise. Bhavnani [3,4] conucte several stuies examining the search strategies of omain experts an novices. In these stuies, five healthcare experts an five shopping experts performe web search tasks in their omain of expertise as well as in the other omain. Important ifferences were ientifie in site selection an knowlege of goal sequencing. Domain experts knew about key resources for their omain an often went irectly to these sites rather than starting with general web search engines. In aition, omain experts ha a general strategy for performing tasks e.g., in the shopping tasks experts visite sites with etaile prouct reviews, comparative shopping sites, an iscount sites. Novices, in contrast, went to a general search engine an examine a few items in the results list, often terminating their session before ientifying goo sites or thorough information. Using these insights they evelope a search system, Strategy Hubs, to provie support for novices to ientify comprehensive information from high-quality sites in the meical omain [5]. Hölscher an Strube [12] examine both search an omain expertise. Search expertise was etermine by interview an a pre-test, an omain experts were unergrauates in economics (the topic of the search task). They foun that both variables affecte search behavior. Iniviuals with both omain an search expertise accesse web sites irectly, but others always use search engines. Search novices were more likely to reformulate their queries, especially when they were also omain novices. Search experts were more likely to use richer query syntax an shorter queries. Kelly an Cool [14] investigate the relationship between topic familiarity an information search behavior (in the form of ocument reaing time an efficacy). They showe that as searcher s topic familiarity increases, search efficacy increases an reaing time ecreases. They claim that it may be possible to infer topic familiarity from search behavior. Marchionini et al. [15] compare the performance of search specialists, omain specialists, an novices (stuents) using a fulltext hypertext system. Both types of experts were more successful than novices in completing search tasks. Zhang et al. [24] examine relationships between engineering omain knowlege, search behavior, an search success. Expertise was assesse by having unergrauate an grauate stuents rate their familiarity with two hunre terms from the Heat an Thermoynamics category in the Engineering Information Thesaurus. Experts foun slightly more relevant ocuments. Experts issue more queries per task an longer queries, an their vocabulary overlappe somewhat more with thesaurus entries, although these ifferences were not reliable statistically. In a recent stuy, Duggan an Payne [9] looke at web search performance for iniviuals with varying expertise in the omains for music an football. Domain knowlege was assesse using answers to 30 simple fact-base questions. Search performance was assesse for these same questions when a web search engine was available to searchers. For the music omain, there was little effect of omain expertise, perhaps because of a relatively narrow range of expertise scores. For the football omain, several interesting associations with omain knowlege were observe. Expertise was positively correlate with search accuracy (even for questions that they i not know the answer to), an negatively correlate with time spent on web pages an mean query length. Experts coul process pages relate to their omain more quickly, which is to be expecte. However, that experts issue shorter queries is inconsistent with previous work an not well explaine by the authors. Query length may vary epening on the nature of the task (chosen by participants vs. assigne by experimenters) or on the content source (web vs. omain-specific resources). Freun an Toms [10] reporte some interesting ifferences between software engineering consultants performing work-task scenarios an general web search behavior. They foun that the software engineers issue longer queries on average (4.4 wors), an use technical terminology an acronyms in 66% of their searches. Although this work oes not explicitly compare omain experts an novices, it oes suggest that experts performing realistic work-relate tasks exhibit ifferent search behavior than is reporte in general web log analyses (e.g., [17]). As summarize above, previous research has shown ifferences in search queries, strategies, an search outcomes as a function of omain expertise. However, much of this research examine search behavior in controlle laboratory settings using small numbers of searchers, experimenter-specifie tasks, an require the explicit measurement of omain expertise. Our research generalizes this along several imensions, with the goal of eveloping methos that can be broaly applie to unerstaning an supporting omain experts in naturalistic task environments. We use a large-scale log analysis of web search behavior allowing us to ientify greater iversity in search vocabulary, interaction patterns, an tasks than have previously been reporte. We stuy experts in four ifferent omains (meicine, finance, law, an computer science), enabling us to ientify similarities an ifferences in interaction patterns across omains. An, we evelop a moel for preicting omain expertise base on these patterns (rather than requiring tests to assess omain knowlege), an escribe how these preictions can be use at web scale.

3 3. DATA SOURCES To perform this stuy, we examine the querying an browsing behavior of many searchers in the four omains of interest. We obtaine fully-anonymize logs of URLs visite by users who opte in to provie ata through a wiely istribute browser toolbar. The information containe in these log entries inclues a unique ientifier for the user, a timestamp for each page view, a unique browser winow ientifier (to resolve ambiguities in etermining which browser a page was viewe), an the URL of the web page visite. Intranet an secure (https) URL visits were exclue at the source. In orer to remove variability cause by geographic an linguistic variation in search behavior, we inclue only entries generate in the English speaking Unite States ISO locale. The results escribe in this paper are base on a sample of URL visits uring a three-month perio from May 2007 to July 2007 inclusive, representing more than 10 billion URL visits from more than 500 thousan unique users. From these logs we extracte aroun 900 million browser trails an aroun 90 million search sessions, as efine by [20]. Browser trails consist of a temporally-orere sequence of URLs comprising all pages viewe per web browser instance or browser tab. Search sessions are a subset of these browser trails, which begin with a query to a search engine such as Google, Yahoo!, or Live Search, an terminate with a perio of user inactivity of 30 or more minutes. This threshol has been use previously to emarcate search sessions in logs [8,20]. These sessions are use as the basis of the comparison between experts an non-experts. We compare the groups base on aspects of querying, navigation, overall search success, an changes in expertise, within topical areas. In the next section we escribe in more etail how experts an non-experts were ientifie. 4. IDENTIFYING DOMAIN EXPERTS We selecte four omains for our investigation: meicine, finance, law, an computer science. We selecte these omains because, in aition to being of general interest to the population at large, there are large professional groups in each area whose omain expert members commonly use the web as a source of information. The selection of computer science was avantageous since the authors are omain experts an coul manually verify the valiity of the queries issue an web sites visite. In our analysis, we first ientifie a set of people in the logs that appeare intereste in each of the four omains, regarless of their expertise. This ensure that all of the behavior stuie relate to people intereste in the omain an helpe control for topical ifferences. From this group of intereste people we then separate experts from non-experts base on whether they visite specialist web sites. This simple yet broaly-applicable metho for ientifying expertise allows us to exten on a large-scale the unerstaning of interaction patterns previous researchers have evelope in laboratory stuies to real-worl situations. We now escribe how we ientify omain experts in more etail. 4.1 Ientifying Users with Topical Interest To ientify users intereste in each omain we classifie pages in the browser trails into the topical hierarchy from a popular web irectory, the Open Directory Project (ODP). Given the large number of pages involve, the classification neee to be automatic. Our classifier assigne labels to pages base on the ODP in a similar way to Shen et al. [16], by starting with URLs that were in the ODP an backing-off to cover other URLs. Using this classifier we calculate the proportion of pages that each user visite that were relate to each omain via the classification. We use the following ODP categories for each omain: Meicine: Health/Meicine Finance: Business/Financial_Services Legal: Society/Law/Legal_Information Computer Science: Computers/Computer_Science Although the objective was to ientify people with some egree of interest in each of the topics, we also wante to remove outliers who viewe only a few pages. Thus, we selecte people who viewe 100 or more pages of any type over the three-month uration of the stuy, an whose page views containe 1% or more of omain-relate pages. Table 1 shows the number of selecte users, the total number of search sessions, an the number of search sessions in each omain from these users. Table 1. Number of users, sessions, an in-omain sessions Domain # users # sessions # in-omain sessions Meicine 45,214 1,918,722 94,036 Finance 194,409 6,489, ,471 Legal 25,141 1,010,868 36,418 CS 2, ,037 3,706 There are aroun 40 search sessions per user an aroun 5% of search sessions were on each of the four omains of interest. These selecte users an their in-omain an out-of-omain sessions extracte from the logs form the basis for the analysis in the remainer of this paper. 4.2 Separating Experts from Non-Experts Researchers have ientifie omain experts using user surveys or the completion of an acaemic course in the omain in question. These techniques can suffer from low participation rates an small sample sizes. To conuct a large-scale stuy of omain expertise it is necessary to ientify experts an non-experts base solely on observable behavior. For this reason, we ivie users base on whether they ha ever visite one or more specific web sites. These expert sites (also referre to as expert URL filters) were ientifie through iscussion with omain experts in each of the four subject areas. They were as follows: Meicine: Visits to the U.S. National Library of Meicine s webbase PubMe service. PubMe is use primarily by meical researchers an physicians, an provies access to citations an abstracts of biomeical research articles. A similar filter was use in earlier work to separate meical experts from non-experts [21]. Finance: Visits to online financial services Bloomberg, Hoovers, Egar Online, an the U.S. Securities an Exchange Commission (SEC) that are use by financial professionals such as investment bankers an accountants. These sites provie market reports, analysis, an financial regulatory information. Legal: Visits to major online legal research services Westlaw an LexisNexis that are use by lawyers an other members of the legal services profession. These services assist in fining, unerstaning, an applying the law an legal concepts. Computer Science: Visits to the Association for Computing Machinery s (ACM) Digital Library. The igital library contains the full-text repository of scientific papers that have been publishe by the ACM. These publications are typically rea by computer science researchers in acaemia an inustry, as well as stuents in computer science programs.

4 With the exception of PubMe an SEC, these sites require user subscriptions that can be prohibitively expensive for the general population. While some of the sites also contain valuable free content, it is likely that even those people who visit them for the free content have greater than average omain knowlege. Using a combination of URL filters an a minimum percentage of URL visits in the topic of interest mitigates the risk of associating an occasional visit to a efining URL as an inication of expertise. Table 2 presents the expert filters use an the number of experts an non-experts ientifie in each omain using this technique. Table 2. Expert URL filters an user group sizes Domain Expert URL filters Expert Non-expert Meicine ncbi.nlm.nih.gov/pubme 7,971 37,243 pubmecentral.nih.gov (17.6%) (82.4%) Finance Legal CS bloomberg.com egar-online.com hoovers.com sec.gov lexis.com westlaw.com acm.org/l portal.acm.org 8,850 (4.6%) 2,501 (9.9%) 949 (39.1%) 185,559 (95.4%) 22,640 (90.1%) 1,478 (60.9%) As we can see from the table, the use of expert filters resulte in a large number of experts in each of the four omains. These large samples allow us to examine a wie range of naturally-occurring search sessions within each of these omains. This is important if we want to generalize to the breath of search tasks an strategies that omain experts exhibit. There may be alternative methos for ientifying in-omain sessions an omain experts, an it woul be easy for us to apply them in a similar framework. In aition to iviing users into expert an non-expert groups we also classifie their search sessions as in-omain an out-ofomain. We performe this classification for each session base on whether it containe a page tagge with the omain ODP label liste in Section 4.1 by our classifier (e.g., search sessions containing at least one page classifie as Health/Meicine were regare as meical). This provie us with omain experts an non-experts, engage in in-omain an out-of-omain sessions. In the next section we characterize the search behavior an relate attributes of experts an non-experts in all four omains. 5. CHARACTERIZING EXPERTISE In this section we compare an contrast the search behavior of experts an non-experts both within their omains of expertise, an outsie of them. We analyze several characteristics of search behavior: querying (attributes of textual queries that are submitte to search engines), sessions (attributes of users interaction behavior uring search sessions), an source selection (attributes of the pages that users visit). We then look at how omain expertise affects search success an how this expertise evelops. Finally, since task ifferences coul account for some of the observe variation we compare behavior across common tasks. Our finings on search behavior are summarize in Table 4 an iscusse in greater etail below. In Table 4 we present the means (M) an the stanar eviations (SD) of the query an session attributes for experts an non-experts separately for sessions in their omain of expertise an those outsie the omain. Given the large sample sizes, most of the observe ifferences in the means between user groups were statistically significant with inepenent measures t-tests (at p < 0.01). We use Cohen s tests to etermine the effect size of each betweengroup comparison [6]. Table 4 also shows the obtaine -values. In general, ifferences between omain experts an non-experts are much larger insie the omain than outsie of it. This means that the ifferences observe for the in-omain sessions reflect omain expertise an not general iniviual ifferences or search skills. We now escribe the results in more etail. 5.1 Queries We begin with a iscussion of the ifferences in the queries issue by omain experts an non-experts. The queries submitte to search engines may provie clues about the expertise of users. For each of the four omains we examine features of the query length (in tokens an characters) an the query vocabulary. Base on previous laboratory stuies (e.g., [10,11]) we conjecture that those with expertise within a particular omain woul issue longer queries an use a more technical query vocabulary. To quantify query vocabulary we obtaine a omain-specific lexicon for each of the four omains. In Table 3 we efine the lexicons use for each omain an state the number of entries in each. Table 3. Domain lexicons an number of entries Domain Lexicon # entries Meicine MelinePlus meical encyclopeia (available from 3,535 Finance Financial ictionary (available from 2,476 Legal Legal ictionary (available from 2,463 CS 1998 ACM Computing Classification System (available from 1,361 For sessions in their omain of expertise, experts queries, as hypothesize, were longer an containe more of the vocabulary from the omain-specific lexicon. The vocabulary feature was consistently among the most important features, as measure by Cohen s. Experts generate queries containing wors from these lexicons fifty percent more often than non-experts (37% vs. 26% of queries). In aition to being able to generate more technically-sophisticate queries, experts also generate longer queries in terms of tokens an characters. It may be that because omain experts are more familiar with the omain vocabulary, it was easier for them to self-generate content to inclue in the query. The magnitue of the ifferences was fairly consistent across omains, except for the legal omain where the ifferences were smaller. It is possible that legal terminology is use comfortably by both legal experts an non-experts. For sessions outsie their omain of expertise, the ifferences between experts an non-experts were much smaller. The Cohen s values were generally less than.10. Thus, the ifferences observe for the in-omain sessions reflect omain expertise an not general iniviual ifferences or search skills. 5.2 Search Sessions We conjecture that experts an non-experts woul exhibit ifferences in their search behavior. To compare groups we examine the following features of in-omain search sessions: 1. Session length (pages): Number of pages visite in session, incluing search engine home pages an result pages. 2. Session length (queries): The number of web search engine queries issue in session. 3. Session length (secons): The total time spent in session.

5 Table 4. Features of web search interaction for experts an non-experts in each of the four omains (larger means are bole) Session Domain Features In omain Out of omain Meicine User group Expert Non-expert Expert Non-expert Number of sessions 26,000 68, ,571 1,495,114 Number of queries 362, ,882 1,577,898 6,463,764 Query Length Tokens Characters % queries w/ tech. term. Exact Substring Session Length Pages (inc. result pages) Queries Secons Branches Unique omains Average page isplay time (secs.) Ratio of querying to browsing Finance In omain Out of omain User group Expert Non-expert Expert Non-expert Number of sessions 23, , ,207 5,663,988 Number of queries 147,018 1,029,101 2,355,477 22,397,513 Query Length Tokens Characters % queries w/ tech. term. Exact Substring Session Length Pages (inc. result pages) Queries Secons Branches Unique omains Average page isplay time (secs.) Ratio of querying to browsing Legal In omain Out of omain User group Expert Non-expert Expert Non-expert Number of sessions 6,346 30, , ,731 Number of queries 75, , ,330 3,581,834 Query Length Tokens Characters % queries w/ tech. term. Exact Substring Session Length Pages (inc. result pages) Queries Secons Branches Unique omains Average page isplay time (secs.) Ratio of querying to browsing CS In omain Out of omain User group Expert Non-expert Expert Non-expert Number of sessions 1,609 2,097 28,210 81,121 Number of queries 26,768 23, , ,198 Query Length Tokens Characters % queries w/ tech. term. Exact Substring Session Length Pages (inc. result pages) Queries Secons Branches Unique omains Average page isplay time (secs.) Ratio of querying to browsing

6 4. Branchiness: The number of re-visits to previous pages in the session that were then followe by a forwar motion to a previously unvisite page in the session. 5. Number of unique (non-search engine) omains: The number of unique non-search engine omains in a session gives a sense of the breath of coverage. 6. Average page isplay time (secons): The average length of time for which a web page is viewe uring a session. 7. Ratio of querying to browsing: The proportion of the session that is evote to querying versus browsing pages retrieve by the search engine or linke to from search results. A high number (much greater than one) means that the session was query-intensive. In contrast, a low number (much less than one) means that the session was browse-intensive. These features offer insight into how the ifferent user groups interacte with web search engines an the web pages they visite uring their search sessions. In Table 4 we show the results of the session analysis for each of the four omains. For in-omain sessions, the search behavior of experts in all four omains iffere from non-experts on most measures. An, for sessions outsie the omain of expertise, the ifferences between experts an non-experts are smaller. The - values reporte in Table 4 for the out-of-omain comparisons suggest that the magnitue of the treatment effects is small. The most noticeable ifferences were in the length of the sessions, the number of unique omains visite, an the branchiness of the session. The sessions conucte by omain experts were generally longer than non-expert sessions. Domain experts consistently visite more pages in a session, an in three of the four omains they spent more time an issue more queries. This coul inicate a greater investment in the topics by experts than non-experts. The information being sought may be more important to the experts, making them more likely to spen time an effort on the task. An interesting aspect of the experts sessions is that they also appear to be more iverse than non-expert sessions, with experts exhibiting more branchiness an visiting more unique omains in all cases (as also in [4]). It may be that experts have evelope strategies to explore the space more broaly than non-experts. The smallest ifferences were consistently in page isplay time, an in two cases (meicine an computer science) omain experts were faster than non-experts. This suggests that omain experts are more aept at reaing omain-relevant pages, as others have sometimes observe [9,14]. 5.3 Source Selection Given the variation in session behavior, we investigate the nature of the web sites visite by experts an non-experts within each omain. To o this formally we analyze the nature of visite page URLs an collecte human jugments of the expertise level of popular pages visite in each of the omains. This section escribes the results of our analysis URL-base We first examine the features of the URLs of web pages visite by experts an non-experts in each omain. The objective was to etermine whether there were any noticeable ifferences in the type of pages the user groups visite. Table 5 shows the top-ten most popular top-level web omains visite by users, an the istribution of omain name extensions (e.g.,.com,.gov,.eu). Table 5. Most popular omains an istribution of URLs over top-level omains. Top-level omains with ifferences < 2% are groupe into the other category Domain Expert Non-expert Meicine nih.gov meicinenet.com mayoclinic.com mescape.com emeicine.com healthline.com rxlist.com nejm.org cc.gov americanheart.org meicinenet.com mayoclinic.com implantinfo.com about.com locateaoc.com emeicinehealth.com rugs.com plasticsurgery.org justbreastimplants.com webm.com Extension % of pages Extension % of pages com 46% com 61% org 26% org 23% gov 14% gov 6% eu 8% eu 5% other 6% other 5% Finance Legal CS citibank.com americanexpress.com ml.com gs.com citigroup.com jpmorgan.com ms.com wachovia.com visa.com nb.com capitalone.com citibank.com americanexpress.com sovereignbank.com iscovercar.com nationwie.com visa.com scotiabank.com bankofamerica.com wachovia.com Extension % of pages Extension % of pages com 89% com 87% other 11% other 13% finlaw.com uspto.gov hhs.gov lawguru.com lexisone.com laborlawtalk.com eeoc.gov alllaw.com expertlaw.com ilrg.com finlaw.com uspto.gov freeavice.com freepatentsonline.com lawguru.com nolo.com ivorcenet.com workerscompensation.com alllaw.com lectlaw.com Extension % of pages Extension % of pages com 48% com 62% org 6% org 8% gov 37% gov 23% other 9% other 7% acm.org ieee.org nist.gov sigmo.org columbia.eu cornell.eu cmu.eu msn.com computer.org coeplex.com microsoft.com ownloa.com msn.com coeproject.com nist.gov sun.com coeplex.com ell.com w3schools.com aobe.com Extension % of pages Extension % of pages com 57% com 71% org 22% org 9% eu 11% eu 8% other 10% other 12% The ifferences between experts an non-experts in URL selection appears to vary by omain. Legal experts are more likely to visit government pages than non-experts, which may reflect the irect use of laws an statues by legal experts. Meical experts visit more government an eucational sites, reflecting a preponerance of public ata available at those locations. Computer scientists visit a relatively large proportion of organizational an eucational sites, reflecting visits to conference

7 web sites (that typically have.org omain name extensions), major U.S. computing societies (ACM an IEEE), an the large U.S. computer science acaemic community. In all of the preceing cases, experts visite fewer commercial sites. They appeare to focus on technical etail, while nonexperts focuse on more consumer-oriente or avisory aspects. In contrast, financial experts visit approximately the same proportion of commercial web sites as non-experts. This may reflect the commercial nature of finance. In aition to looking at the omains visite by each of the user groups, we also investigate the omains unique to each group. The finings showe clear ifferences in the types of web sites that are unique to experts an non-experts within each omain. From the finings, we hypothesize that meical experts visite websites containing information on specific conitions relevant to their specialty (e.g., acc.org, a resource offering benefits an services for cariologists). In contrast, it seems that meical nonexperts visite sites relate to conitions or meical proceures that were relevant to them (e.g., obesity, breast augmentation). Finance experts visite sites on funs, investments, an securities, whereas non-experts visite creit unions an creit avisory sites. Legal experts appeare concerne with regulations an legal preceents, while non-experts were intereste in particular legal scenarios such as speeing tickets an rente accommoation. Computer science experts visite sites that were specific to a programming language (e.g., lyx, smalltalk). In contrast, computer science non-experts appeare intereste in the customization of their operating system, their esktop environment, or protecting their personal computer from viruses. From this analysis it appears that the web sites visite by omain experts were more technical in nature. However, analyzing URLs oes not allow us to formally compare the content of the web sites that omain experts an non-experts visite when searching within their omain of interest. For this reason, in aition to stuying web sites URLs, we also evaluate the content of popular pages visite by users in within each of the four omains Content The goal of analyzing page content was to see if human jugments of the technical epth of pages were associate with our automatic methos of ientifying experts. To o this we ientifie the top 150 most popular URLs within each omain visite by experts an non-experts, for a total of 600 pages. Two of the authors of this paper inepenently juge the technical etail of the pages, rating each page as either expert or not. Although these authors were omain experts in computer science, they juge all omains in the interest of jugment consistency. The juge URLs containe a 20% overlap in pages to etermine inter-rater reliability. There was substantial agreement between raters (Cohen s Kappa =0.72), with the agreement ranging some accoring to omain from moerate (Finance, =0.45) to almost perfect (Computer Science, =0.89). Given that the authors are omain experts in the area of computer science, it is not surprising that this is the area of greatest overlap. Table 6 shows the number of popular pages from each omain that were visite by either experts or non-experts, broken own by whether the pages were rate as expert or not. Pages visite by users in both groups are represente in both the expert an nonexpert columns. Pages for which the raters isagree or for which no jugment coul be obtaine (e.g., the page i not loa) are exclue from analysis. Table 6. Number (an percentage) of pages rate as being a resource for experts or non-experts, broken own by group Domain Expert visitor Rate expert Rate non-expert Non-expert visitor Rate expert Rate non-expert Cohen s Kappa Meicine 100 (68%) 46 (32%) 66 (46%) 79 (54%) 0.75 Finance 30 (22%) 108 (78%) 10 (8%) 121 (92%) 0.45 Legal 79 (56%) 63 (44%) 65 (47%) 72 (53%) 0.64 CS 107 (89%) 13 (11%) 18 (21%) 66 (79%) 0.89 In all cases, omain experts visite more web sites rate expert than omain non-experts i. Overall, 58% of the common pages visite by experts were juge expert, an 42% were juge nonexpert. In contrast, only 32% of the pages commonly visite by non-experts were classifie as expert, while 68% were juge non-expert. The tren is most pronounce for meicine an computer science. For those omains the inter-rater reliability is also highest (as measure by Cohen s Kappa). The tren may be less pronounce for the finance an legal omains because experts visit many sites also visite by non-experts, an the explicit jugments in those omains are of lower quality since the human juges were not omain experts. 5.4 Search Success In the previous sections we observe several important ifferences in the queries experts an non-experts use, the search session behavior, an the resource selection. In this section we investigate how successful users from both groups appeare to be when searching in omain an out of omain. Since our stuy was log-base we coul not control user task or confirm whether searchers ha been successful in their search session. Instea we ha to approximate success heuristically. Search result click-through ata has been use previously to evelop moels of user preferences [1]. We use our logs an looke at the final action in a session. If the final event in a search session was a URL click we score the session as a success, an if the final action was a query we score the session as a failure. In Table 7 we present the proportion of search sessions that were eeme successful with this metric. Table 7. Percentage of successful sessions, by expertise. The Pearson s R with omain expertise is reporte In omain Out of omain Pearson s Nonexperexpert coefficient (R) Non- correlation Expert Expert Domain Meicine 84.9% 76.8% 75.4% 78.6% 0.55 Finance 84.6% 81.4% 80.1% 82.1% 0.38 Legal 82.3% 79.9% 79.3% 81.1% 0.49 CS 83.6% 69.1% 72.8% 71.0% 0.66 Average 83.9% 76.9% 76.8% 78.2% 0.52 Although this was only an approximation for search success, an may result in an overestimation of search success in absolute terms, any overestimation shoul affect all groups equally. Finings showe that experts were more successful than nonexperts when searching within their omain of expertise. When searching in out-of-omain sessions, experts an non-experts ha comparable levels of search success, with non-experts being somewhat more successful on average. It is interesting that for each omain, experts performe better in omain than out of omain, while non-experts actually performe worse in omain

8 than out of omain. Non-experts exhibite clear interest in the omain, with at least 1% of all of the pages they visite falling within the omain, but still appeare to have trouble working in that omain. It may be that they lacke the technical expertise necessary to succee in their searches. To further probe the relationship between search success an omain expertise we examine the correlation between the two variables for in-omain sessions, as suggeste by [23]. For each user, we calculate a search success an omain expertise score. Search success was efine as the proportion of successful sessions, an omain expertise was represente as the proportion of query terms with omain-specific vocabulary (since this was the most important variable for istinguishing experts an nonexperts). For each omain we compute Pearson s correlation coefficient (R) between the two measures. The values for R are reporte in the last column of Table 7. They suggest that there is a goo correlation between level of expertise an egree of search success. That is, the more expert a user is the more likely they are to be successful when searching in their omain of interest. 5.5 Development of Expertise The evelopment of people s omain expertise over time has been observe in prior longituinal stuies [19,23]. Our logs affore us the opportunity to track an iniviual user over the three-month uration of our investigation. While users may have omain knowlege extening outsie the winow, we wishe to etermine whether there was any evience of users omain expertise eveloping over time. The presence of such evience woul suggest that we coul remotely estimate aspects of user learning which may be useful in offering tailore search support. To investigate omain expertise evelopment, we examine the proportion of queries containing omain-specific vocabulary (the strongest preictor of search expertise that emerge from our earlier analysis) at a series of time points across the three months. To begin, we ivie the three months into 13 one-week perios. To give us sufficient ata we restricte our analysis to users for whom we ha five or more weeks of ata in the uration of the stuy. We then compute the proportion of queries with inomain sessions that containe omain-specific vocabulary in each week. To gain a sense for whether query vocabulary was expaning we compute the Pearson s R across all available ata points for each user. This offere a sense for whether their query vocabulary was increasing, ecreasing, or remaining constant across the five or more weeks we stuie. In Table 8 we present the proportion of users whose query vocabulary usage trene ownwars, those where it remaine the same (i.e., R lies between 0.1 an 0.1), an those where it increase. Table 8. Percentage of users with increase ( ), ecrease ( ), or no change ( ) in query vocabulary Experts Non-expert Domain Meicine 9.8% 74.9% 15.3% 8.3% 43.1% 48.6% Finance 10.1% 75.8% 14.3% 9.9% 52.0% 38.1% Legal 13.2% 73.2% 13.6% 15.2% 54.1% 30.7% CS 9.4% 72.1% 18.5% 11.1% 51.7% 37.2% The results show that experts use of omain-specific vocabulary changes only slightly over the uration of the stuy. However, many non-expert users exhibit an increase in their usage of omain-specific vocabulary. This provies evience that suggests that non-expert omain expertise may be eveloping over time. However, such evelopment in users omain expertise also emonstrate the ifficulty in monitoring omain non-experts over a perio of time; omain expertise is ynamic. 5.6 Common Tasks A possible confoun in the above analysis is that the observe ifferences may be a function of task ifferences rather than expertise ifferences. Our observations woul be the same if experts ha inherently ifferent tasks than non-experts, an exhibite the same behavior as non-experts for the same tasks. To aress this concern we evelope two methos to ientify comparable tasks: (i) we ientifie search sessions that began with the same query, an (ii) we ientifie sessions that ene with the same URL. The number of sessions with the same initial query or same last URL varie epening on the omain, from for computer science to aroun ten thousan for finance. For these linke sessions we extracte the same set of features of the queries an sessions as were shown in Table 4. Our analysis of these sessions showe that for matche queries an sessions, the between-group ifferences note earlier in this paper hel true. It seems that task ifferences o not significantly impact user interaction patterns. However, more work is necessary to stuy the effects of user intent on search behavior. For example, in Teevan et al. [18] we show that some queries have more variation in user intent (efine by clicke results) than others. 6. SEARCHING BETTER VIA EXPERTISE We have seen in our stuy that omain experts employ ifferent search strategies an are more successful than non-experts in four ifferent omains. Given these ifferences, we believe we can help people search by consiering their omain expertise. One way our finings coul be use to improve the search experience woul be tailor the results shown or search ais such as query suggestions to match the expertise of the searcher. Alternatively, the search strategies employe by omain experts coul be use to support non-experts in learning more about omain resources an vocabulary. Regarless of the specific strategies employe, any search system that takes avantage of omain expertise nees to be able to ientify whether a user is an expert or a non-expert, an then moify the experience accoringly. For this reason, in this section we explore how well omain expertise can be automatically preicte. We focus on techniques that work using only observable search behavior an history as input, since these can be eploye wiely. We then iscuss how we can use preiction. 6.1 Preicting Domain Expertise To preict omain expertise we evelope a classifier base on the features of users interaction behavior liste in Table 4 within each of the omains. We employe a maximum-margin average perceptron [7] as the classifier, since it was appropriate for our binary classification task an has previously shown excellent empirical performance in many omains from natural language to vision. A separate version of the average perceptron was traine for each of the four omains. We focuse on three preiction challenges: (i) whether an in-omain session was conucte by a omain expert, (ii) whether we coul ientify omain experts uring the course of a session by preicting after successive actions (queries or page visits), an (iii) whether a user was a omain expert given multiple in-omain sessions.

9 6.1.1 Post-Session Expertise Preiction To train the classifier to preict if an observe in-omain session was conucte by a omain expert or not, we treate in-omain search sessions performe by omain experts as positive examples an search sessions performe by non-experts as negative examples. All of the query an session attributes shown in Table 4 were use as features to train the classifier. We performe a five-fol cross-valiation experiment across ten runs, an generate precision-recall curves that summarize the performance of the classifier traine on each omain (Figure 1). Precision Recall Computer Science Meicine Legal Finance Figure 1. Preiction accuracy for ifferent omains. The precision-recall curves show the number of search sessions accurately classifie as expert or non-expert at ifferent recall levels. The curves illustrate that it is possible to accurately preict whether an observe session was performe by a omain expert or a omain non-expert, especially at low recall levels. Recall from Table 2 that anywhere from 4.6% to 39.1% of all people involve in omain-specific sessions were experts. The classification accuracy is highest for computer science, an this most likely reflects the fact that that was also the omain with the highest percentage of ientifie experts Within-Session Expertise Preiction Post-session preiction has limite utility since the session must be complete before a preiction can be mae. An attractive alternative is to preict expertise on-the-fly uring the course of a session. While a web browser or client-sie application may inee know all of the interactions, search engines an other web applications typically o not. We traine the classifier using all of the features in Table 4. In aition, we also explore using only those features relate to the query, erive at each query iteration, an using only those features of pages visite (e.g., page isplay time, number of unique omains), erive at each web page view. To examine within-session preictions, we selecte the 2181 CS sessions (59% of the total in-omain sessions) that containe at least five queries an at least five non-result page visits. These sessions enable us to stuy accuracy for a sizeable number of actions over the same set of sessions. We use five-fol cross valiation over ten experimental runs to train our classifier an evaluate accuracy, computing preiction accuracy after each action. The average accuracy across the ten runs is reporte in Table 9, for all features an for query an web page features separately. Preiction accuracy after observing a full session with each feature set is also inclue for reference. Table 9. Mean preiction accuracy for all / queries / pages. Significant p-values from t-tests comparing accuracy with baseline (.566) are shown with * = p <.05 an ** = p <.01 Action type Action number Full session All.616 *.625 *.639 **.651 **.660 **.718 ** Queries.616 *.635 **.651 **.668 **.683 **.710 ** Pages *.608 *.617 *.634 **.661 ** The finings show that we can generally preict CS omain expertise within a session after only a few user actions, compare with a maximal margin baseline that always preicts non-expert (accuracy =.566). Our finings show that preictions base on queries yiele a higher accuracy than page-base preictions or preictions from all features given few observations. As observe earlier, queries are a goo source of omain expertise evience. Across full sessions, the use of all features slightly outperforms query features. These finings hol for the other omains stuie. Querying activity is reaily available to search engines an coul be classifie immeiately to rapily tailor the search experience User Expertise Preiction We also explore how well we coul preict whether a user is a omain expert given interaction history across multiple sessions. To o this we selecte users from each omain with at least five sessions in the interaction logs an monitore the improvement obtaine by using aitional sessions. For each omain we incremente the number of sessions use to compute the features an recore the accuracy obtaine at each iteration, up to at most five sessions. Sessions were use in chronological orer to mimic how they woul arrive in an operational setting. In aition to the single-session features, we also use several inter-session features incluing the time between in-omain sessions an average number of observe in-omain sessions per ay. Figure 2 shows the learning curves for each of the four omains generate from five-fol cross valiation experiments across ten runs. The curves plot accuracy (i.e., the proportion of times the classifier accurately etermines whether a user is an expert or a non expert) against the number of search sessions presente, average across all runs. The finings show that preiction accuracy for all classifiers improves with aitional search sessions, but the marginal improvement ecreases. Mean accuracy Computer Science Meicine Legal Finance Number of sessions per user Figure 2. Accuracy given more search sessions per user (±SE).

10 Clearly having a lot of information about a user s web search activity is valuable for preicting omain expertise. A web browser, Internet service provier, or large omain-specific web site may be able to collect session-base information as escribe above an use it to tailor the experience. However, such rich behavior information is often not available to popular search tools like online commercial search engines. To unerstan how performance of the classifier woul be affecte if interactions were limite to only to those visible from a search engine, we traine a classifier using only queries an result clicks. The finings show a ip in accuracy of 5-10% across all omains, but the general trens ientifie in this section remain. 6.2 Improving the Search Experience Knowlege of an iniviual s expertise level can be use to support the search process in many ways. For example, a search engine (or client-sie application) coul bias its results towars the web sites that people with similar expertise prefer, an provie query suggestions or query re-writing that use level-appropriate terminology. A challenge with this approach is that it reinforces behavior rather than encouraging people to learn over time. That is, non-experts are encourage to search more like non-experts, rather than to gain expertise. This approach stans in contrast to the evelopment of expertise over time observe in Section 5.5. For this reason, it may be worthwhile for search tools to consier how they can better help omain non-experts become omain experts over the course of time. Bhavnani et al. s work on Strategy Hubs [5] is an example of a system that supports an eucates non-experts by proviing critical search proceures an associate high-quality links. We believe that proviing such support coul be extene more broaly. One way a search engine coul o this woul be to provie non-expert efinitions for relate expert terms when a person searches. The results for a search for cancer, for example, may inclue a efinition of malignancy, which woul, in turn, help the non-expert better unerstan the technical vocabulary an the complete information space. Non-experts coul also be taught to ientify reliable, expert sites as they gain the necessary knowlege to unerstan them better, or to examine the broaer range of information that experts o. By helping non-experts move from tutorial information to more etaile information, search tools can help them evelop omain expertise through their search experience. 7. CONCLUSIONS AND FUTURE WORK In this paper we have escribe a large-scale, log-base stuy of the web search behavior of omain experts an non-experts. Our finings emonstrate that, within their omain of expertise, experts search ifferently than non-experts in terms of the sites they visit, the query vocabulary they use, their patterns of search behavior, an their search success. These ifferences were obtaine in naturalistic web search sessions in four omains, thus extening previous lab stuies in terms of breath an scale. We have also evelope moels to preict omain expertise using characteristics of search interactions. By focusing on attributes that are reaily available from web search behavior, we can apply our moels in real-time as part of a web search engine. The ientification of omain experts can allow us to provie expert query suggestions an site recommenations to non-expert users, an personalize results or suggestions base on expertise. Future work will involve eveloping such applications, testing them with human subjects, an eploying them at web scale. 8. REFERENCES [1] Agichtein, E., Brill, E., Dumais, S. & Ragno, R. (2006). Learning user interaction moels for preicting web search result preferences. Proc. SIGIR, [2] Allen, B.L. (1991). Topic knowlege an online catalog search formulation. Lib. Quart., 61(2), [3] Bhavnani, S.K. (2001). Important cognitive components of omain-specific search knowlege. Proc. TREC, [4] Bhavnani, S.K. (2002). Domain-specific search strategies for the effective retrieval of healthcare an shopping information. Proc. SIGCHI, [5] Bhavnani, S.K. et al. (2005). Strategy Hubs: Domain portals to help fin comprehensive information. JASIST, 57(1), [6] Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2n e.). Lawrence Earlbaum. [7] Collins, M. (2002). Discriminative training methos for hien markov moels: Theory an experiments with perceptron algorithms. Proc. EMNLP, 1-8. [8] Downey, D., Dumais, S. & Horvitz, E. (2007). Moels of searching an browsing: Languages, stuies an application. Proc. IJCAI, [9] Duggan, G.B. & Payne, S.J. (2008). Knowlege in the hea an on the web: Using topic expertise to ai search. Proc. SIGCHI, [10] Freun, L. & Toms, E.G. (2006). Enterprise search behaviour of software engineers. Proc. SIGIR, [11] Hembrooke, H.A., Gay, G.K. & Granka, L.A. (2005). The effects of expertise an feeback on search term selection an subsequent learning. JASIST, 56(8), [12] Hölscher, C. & Strube, G. (2000). Web search behavior of internet experts an newbies. Comp. Net., 33(1), [13] Hsieh-Yee, I. (1993). Effects of search experience an subject knowlege on the search tactics of novice an experience searchers. JASIST, 44(3), [14] Kelly, D. & Cool, C. (2002). The effects of topic familiarity on information search behavior. Proc. JCDL, [15] Marchionini, G., Lin, X. & Dwiggins, S. (1990). Effects of search an subject expertise on information seeking in a hypertext environment. Proc. ASIS, [16] Shen, X., Dumais, S. & Horvitz, E. (2005). Analysis of topic ynamics in web search. Proc. WWW, [17] Spink, A. & Jensen, B. (2004). Web search: Public searching of the web. Kluwer Acaemic Publishers. [18] Teevan, J., Dumais, S.T. & Liebling, D.J. (2008). To personalize or not to personalize: Moeling queries with variation in user intent. Proc. SIGIR, [19] Vakkari, P., Pennanen, M. & Serola, S. (2003). Changes in search terms an tactics while writing a research proposal: A longituinal case stuy. IP&M, 39(3), [20] White, R.W. & Drucker, S.M. (2007). Investigating behavioral variability in web search. Proc. WWW, [21] White, R.W., Dumais, S. & Teevan, J. (2008). How meical expertise influences web search interaction. Proc. SIGIR [22] White, R.W. & Morris, D. (2007). Investigating the querying an browsing behavior of avance search engine users. Proc. SIGIR, [23] Wilemuth, B.M. (2004). The effects of omain knowlege on search tactic formulation. JASIST, 55(3), [24] Zhang, X., Anghelescu, H.G.B. &Yuan, X. (2005). Domain knowlege, search behavior, an search effectiveness of engineering an science stuents. Inf. Res., 10(2), 217.