Determining the User Intent of Chinese-English Mixed Language Queries Based On Search Logs

Transcription

1 Determining the User Intent of Chinese-Engish Mixed Language Queries Based On Search Logs Hengyi Fu, Forida State University, City University of New York Abstract With the increasing number of mutiingua web pages on the Internet, mutiingua information retrieva has become an important research topic. Whie queries are the key eement of information retrieva process, mixed-anguage queries have not yet been adequatey studied. This study is to determine the user intents of Chinese-Engish mixed-anguage queries submitted to a Chinese search engine, and compares the user intents identified by query content to those identified using additiona user behavior data (e.g. cicked resuts, subsequent queries). The preiminary findings present the distributions of user intents by anayzing query ony and additiona user behavior data, suggesting a specific searching behavior of Chinese-Engish mixed-anguage queries users. The findings of this study coud provide usefu insights in understanding the searching behavior of Chinese-Engish mixed-anguage queries users, and enabe web search engines to provide users with more reevant resuts and more precisey targeted sponsored inks. Keywords: Information retrieva; mutiingua search query; search behavior; user intent; search og anaysis Citation: Fu, H., Wu, S. (2015). Determining the User Intent of Chinese-Engish Mixed Language Queries Based On Search Logs. In iconference 2015 Proceedings. Copyright: Copyright is hed by the authors. Contact: hf13c@my.fsu.edu, wu1@gmai.com 1 Introduction Athough Engish has remained the predominant content anguage used in websites, the amount of content in other anguages aso increases rapidy. In cutures where peope use both Chinese and Engish, using mixed-anguage in spoken anguage is very common. In genera, a mixed-anguage consists of terms or sentences mosty in the primary anguage with certain words in a secondary anguage. For exampe, Hong Kong peope typicay use Cantonese with Engish words. When such peope are doing web searching, they often empoy the Chinese-Engish mixed-anguage queries to approximate their information need rather than using Chinese-ony queries. A mixed-anguage query is a search query that is mixed with two or more anguages. For exampe, the query 3G 手机 is a Chinese-Engish mixed-anguage query that a user created to ook for information about the 3G mobie phones. Researches about Chinese search engines indicated that the number of Chinese-Engish mixed-anguage queries has increased in recent years (Chau, Fang, and Yang, 2007). However, ony a sma number of studies have examined the Chinese-Engish mixedanguage queries in depth (Chau, Lu, Fang, & Yang, 2009). In particuar, the Why question (why user performs such search), which is more essentia to understand user s information need (Rose & Levinson, 2004), has not been addressed. Identifying user intent of mixed-anguage queries can provide usefu insights in understanding the searching behavior of Chinese-Engish mixed-anguage queries users, and can be utiized in improving advertisement targeting and page ranking as we as the presentation of the search resuts. Most studies regarding user intent of web searching ony anayzed queries. However, queries themseves are not aways enough to determine user intent because they are sparse representations of user intents (Sivestri, 2010). Without additiona evidence from other sources, such as cicks from search resuts, think-aoud protocos, and documents viewed, it can be difficut to determine what the users were intending to do in a certain context. User intent of mixed-anguage query is supposed to be more difficut to infer just by anayzing queries, due to the difficuty of expressing non-specificabiity need using mixed anguages (Bekin, 1980). When user empoyed a non-native anguage in the search, the query submitted is more ikey to be deviated from the rea intent because the user does not know how to use anguage correcty to form a query. In a previous study, we examined the topic distribution of Chinese-Engish mixed-anguage queries and deveoped a typoogy of Engish terms used in those mixed anguage queries (Fu & Wu, 2014). In this foow-up study, we empoyed user behavior data such as cicked resuts, query modification patterns, and subsequent queries to identify user intent, and compared the resut to that using query content ony.

2 2 Reated Work Severa studies have attempted to cassify user queries in terms of users informationa actions with different methods, incuding queries (Broder, 2002; Jansen, Booth, & Spink, 2008, Rose & Levinson, 2004), cicked URL and subsequent queries (Rose & Levinson, 2004), ive ticker data (Lewandowski, 2006), and user survey (Broder, 2002; Lewandowski, Drechser, & Mach, 2012). A these studies foowed Broder s taxonomy of query intent: navigationa, informationa, and transactiona. The ratios of user intents varied across different studies and can be found in Tabe 1. Most studies used Engish queries except Lewandowski s (2006, 2012).Rose and Levinson (2004) reported that user intents coud be inferred with amost no information about the user s behavior. Lewandowski et a. (2012) concuded that the anaysis of cick-through data is adequate to identify navigationa queries but not for other query types. Study Data Number Method N I T Broder (2002) 1 AtaVista 3,190 User survey 39% 24.5% 36% Broder (2002) AtaVista 1,000 Query content 48% 20% 30% Rose & AtaVista 1,500 Query content, 61 63% 11 15% 21 27% Levinson (3*500) cicked URL and (2004) 2 subsequent queries Lewandowski Three German 1,500 45% 40% 15% Live ticker data (2006) search engines Jansen et a. Dogpie.com 1,523, % 80.6% 9.2% Query content (2008) 3 Lewandowski T-Onine % 38.6% 26.8% User survey et a. (2012) Tabe 1. Comparison of the resuts from query intent studies using Broder s (2002) taxonomy 3 Research Questions This poster addresses the foowing questions: RQ1: What are the user intents of web searching using the Chinese-Engish mixed anguage queries? RQ2: Is there any difference in user intents when anayzing subsequent user behavior data beyond the Chinese-Engish mixed-anguage queries? 4 Data Coection and Research Methods This study uses queries 3 submitted to the Sogou search engine ( which was estabished in 2004 and now is one of the most popuar search engines in China. The query-og data was coected in June Each record consists of six fieds: time of cick, user ID, query content, ranking of the cicked URL, ordering of user cick, and the cicked URL. The query-og data contains 86,538,613 non-empty queries. Query sessions are provided according to cookie information. There are 26,255,952 sessions. One of the researchers deveoped a C++ program to seect and pre-process a the Chinese- Engish queries that contain both ASCII and doube-byte characters aong with foowing user behavior data (cicked resuts, query modification patterns, subsequent queries) within a session. The fina dataset has 346,989 Chinese-Engish mixed-anguage queries, accounting for 7.98% of a queries. A random sampe of 384 Chinese-Engish mixed-anguage queries was drawn from this dataset to conduct content anaysis. The sampe size was determined using a technique introduced by Powe and Connaway (2004).Tabe 2 is an exampe of query with its foowing user behavior data. User behavior Initia query Resut cick Resut cick Query modify Detais Maxwe Render 教材 (i.e., Maxwe Render teaching materias) Maxwe Render v2.6 安装方法 (i.e., Maxwe Render v2.6 instaation) 1 The ratio of informationa and transactiona queries was estimated, based on the ower bounds derived from the user survey. 2 Resuts are not exact because they aggregate data from their three studies. 3 The dataset used in this study has been pubished and can be found at 2

3 Resut cick Tabe 2. User behavior foowing the query Maxwe Render teaching materias This study adopted Rose and Levinson s (2004) approach to manuay anayze user intent based on queries and subsequent user behavior. Two researchers independenty coded the same set of queries into three different categories of user intents based on query content and user behavior data. When empoying user behavior data, if the user modifies the initia query or cicks on more than one resut within a session, the ast modified query or the ast cicked resut in this session is used for intent anaysis, because if the initia query has been modified or more than one cicks existed, it woud impy that the initia query or earier cicked resut(s) have not satisfied the user need competey. Figure 1 and tabe 3 show the percentages of cicked resut numbers for a giving query and query modification patterns in the sampe. The researchers achieved an intercoder reiabiity of (Cohen s Kappa) using query content and with user behavior data. 40% 35% 30% 25% 20% 15% 10% 5% 0% or more Figure 1. Distribution of the number of cicked resuts for a given query Query modification pattern Percentage Adding terms 45.8% Deeting terms 11.5% Changing the entire query 37.0% Others 5.7% Tabe 3. Distribution of query modification patterns within a session 5 Preiminary Findings and Discussion Based on the anaysis of query content, of 384 queries in the sampe, 52.3% is informationa, 39.3% transactiona, and 8.3% navigationa. The anaysis of user behavior data found that 44.3% is informationa, 43.2% transactiona, and 12.5% navigationa. Figure 1 compares our user intent cassification with resuts reported by Broder. Whie the percentage of informationa queries reported in this study is simiar, much fewer navigationa queries are identified. To be considered navigationa, a query shoud have a singe authoritative web site that the user aready has in mind (Broder, 2002). The ower percentage of navigationa query may suggest users are ess certain of if the web pages exist or not, or the effectiveness of mixed-anguage queries compared to monoingua queries. 3

4 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 36.0% 30.0% 39.0% 48.0% 24.5% 20.0% Broder (user servey) Broder (query anaysis) 39.3% 43.2% 52.3% 44.3% 8.3% 12.5% Current Study (query anaysis) Current Study (user behavior data) Figure 2. Comparison of Broder s taxonomy to our user intent cassification Transactiona Informationa Navigationa Rose and Levinson (2004) argued user intents can be detected by using query content ony. However, examining additiona user behavior data in this study did resut in a substantia change in user intents. 26.5% (102 out of 384) queries changed to a different category when anayzing user behavior data. In most cases (86 out of 102), information queries changed to transactiona queries, and vice versa. Tabe 4 shows the distributions of user intent associated with different methods. Chi-square test aso indicates these two methods have statistica significances (X 2 (2, N = 768) = 6.500, p <.05) in cassifying user intent. This finding suggests the user intent of mixed-anguage querying is more difficut to infer. One possibe reason is that using non-native anguage makes a query more ambiguous. Since the Engish term in a query may not fuy represent what is meant by the user, additiona information, such as user behavior, is necessary for determining user intent. Metho d User Intent Tota Navigationa Informationa Transactiona Query Content Count User Behavior Data Expected Count Count Expected Count Tabe 4. Method * User Intent Crosstabuation 6 Concusion and Future Research This study anayzed the user intents of Chinese-Engish mixed-anguage queries submitted to a Chinese web search engine, and compared the user intents identified by query ony with those identified by additiona user behavior data. The distributions of user intents drawn from queries and using additiona user behavior data in this study differ from those of monoingua queries studies, suggesting a specific searching behavior of Chinese-Engish mixed-anguage queries users. Future research incudes anayzing how and why user intent changes during searching, investigating the context in which users desire to use Chinese-Engish mixed-anguage queries, and in which situations these queries woud be of beneficia. 4

5 7 References Bekin, N. J. (1980). Anomaous states of knowedge as a basis for information retrieva. The Canadian Journa of Information Science, 5, Broder, A. (2002). A taxonomy of web search. SIGIR Forum, 36(2), Chau, M., Fang, X., & Yang, C. C. (2007). Web searching in Chinese: A study of a search engine in Hong Kong. Journa of the American Society for Information Science and Technoogy, 58(7), Chau, M., Lu, Y., Fang, X., & Yang, C. C. (2009). Characteristics of character usage in Chinese web searching. Information Processing and Management, 45(1), Fu, H., & Wu, S. (2014). Anayzing Chinese-Engish mixed anguage queries in a Web search engine. Proceedings of the 77th Annua Meeting of the Association for Information Science and Technoogy. Jansen, B. J., Booth, D. L., & Spink, A. (2008). Determining the informationa, navigationa, and transactiona intent of web queries. Information Processing and Management, 44(3), Lewandowski, D. (2006). Query types and search topics of German web search engine users. Information Services and Use, 26(4), Lewandowski, D., Drechser, J., & Mach, S. (2012). Deriving query intents from web search engine queries. Journa of the American Society for Information Science and Technoogy, 63(9), Powe, R. R., & Connaway, L. S. (2004). Basic research methods for ibrarians (4th ed.). Westport, CT: Libraries Unimited. Rose, D., & Levinson, D. (2004). Understanding user goas in web search. In S. Fedman, M. Uretsky, M. Najork, & C. Wis (Eds.), Proceedings of the 13th Internationa Conference on Word Wide Web (WWW 2004) (pp ). New York: ACM Press. Sivestri, F. (2010). Mining query ogs: Turning search usage data into knowedge. Foundations and Trends in Information Retrieva, 4(1 2), Tabe of Figures Figure 1. Distribution of the number of cicked resuts for a given query... 3 Figure 2. Comparison of Broder s taxonomy to our user intent cassification Tabe of Tabes Tabe 1. Comparison of the resuts from query intent studies using Broder s (2002) taxonomy... 2 Tabe 2. User behavior foowing the query Maxwe Render teaching materias... 3 Tabe 3. Distribution of query modification patterns within a session... 3 Tabe 4. Method * User Intent Crosstabuation