Review Graph based Online Store Review Spammer Detection

Transcription

1 Review Gaph based Online Stoe Review Spamme Detection Guan Wang, Sihong Xie, Bing Liu, Philip S. Yu Univesity of Illinois at Chicago Chicago, USA Abstact Online eviews povide valuable infomation about poducts and sevices to consumes. Howeve, spammes ae joining the community tying to mislead eades by witing fake eviews. Pevious attempts fo spamme detection used eviewes behavios, text similaity, linguistics featues and ating pattens. Those studies ae able to identify cetain types of spammes, e.g., those who post many simila eviews about one taget entity. Howeve, in eality, thee ae othe kinds of spammes who can manipulate thei behavios to act just like genuine eviewes, and thus cannot be detected by the available techniques. In this pape, we popose a novel concept of a heteogeneous eview gaph to captue the elationships among eviewes, eviews and stoes that the eviewes have eviewed. We exploe how inteactions between nodes in this gaph can eveal the cause of spam and popose an iteative model to identify suspicious eviewes. This is the fist time such inticate elationships have been identified fo eview spam detection. We also develop an effective computation method to quantify the tustiness of eviewes, the honesty of eviews, and the eliability of stoes. Diffeent fom existing appoaches, we don t use eview text infomation. Ou model is thus complementay to existing appoaches and able to find moe difficult and subtle spamming activities, which ae ageed upon by human judges afte they evaluate ou esults. 1 Keywods- Spamme detection, eview gaph I. INTRODUCTION Online stoe eviews ae an impotant esouce to help people make wise choices fo thei puchases. Due to this eason, the eview system has become a taget of spammes who ae usually hied o enticed by companies to wite fake eviews to pomote thei poducts and sevices, and/o to distact customes fom thei competitos. Diven by pofits, thee ae moe and moe spam eviews in majo eview websites, such as PiceGabbe.com, Shopzilla.com, o Reselleatings.com. Spammes ae stating to coupt the online eview system and confuse the consumes. A. Challenges Automatically detecting spammes is an ugent yet unde exploed task. Unlike othe kinds of spam, e.g., Web spam o spam, eview spam is much hade to detect. The main eason is that spammes can easily disguise themselves. It is thus had fo a human use to ecognize them, while fo Web and spam, one can tell spam without much difficulty. 1 This wok is suppoted in pat by NSF though gants IIS , OISE , DBI , and IIS The task of detecting fake eviews and eviewes was fist poposed by Nitin and Liu in [3], which they call opinion spam detection. They built a classifie using cetain types of (nea) duplicates eviews as positive taining data and the est as the negative taining data. Thei appoach used featues about eviews, eviewes, and poducts. In [4], they also poposed anothe algoithm based on mining unexpected ules. They found that the top unexpected ules epesent abnomal eviews and eviewes. Howeve, thei method does not identify tue spammes. Using behavios of eviewes fo spamme detection was also studied in [8]. Howeve, this wok essentially also elies on duplications, i.e., multiple eviews fom the same eviewe tageting the same item o item goup. Thus, it is only suited fo a special kind of spamming. In [7, 9], eview contents wee also used fo spam detection, which we don t use. Although we also examine eviewes behavios, they ae only a small pat of ou solution. The novel eview gaph concept intoduced in this pape captues inte-elationships of eviewes, eviews, and stoes, which existing woks have not used. In this wok, we ae inteested in stoe eviews. Even if we can boow some ideas fom pevious studies, thei clues ae not sufficient to tell stoe eview spammes. Fo example, although it looks suspicious fo a peson to post multiple eviews to the same poduct, it may be nomal fo a peson to post moe than one eview to the same stoe due to multiple puchasing expeiences. Futhemoe, as one peson has the same witing style on eview witing, it may be nomal to have simila eviews fom one eviewe fo multiple stoes because unlike diffeent poducts, diffeent stoes basically povide the same types of sevices. Moeove, many odinay uses only wite eviews spoadically. It is easonable fo them to wite multiple eviews in a shot time peiod about diffeent shopping expeiences. Theefoe, eviewe behavios poposed in the existing appoaches fo poduct eviews ae insufficient fo catching spammes of stoe eviews. Thus thee is a need to look fo a moe sophisticated and complementay famewok. Howeve, the following challenges ae majo obstacles towads such a famewok. 1) Thee is no gound tuth whethe a eview is faked o not. By eading the eview text alone, we usually do not have enough clues to tell spam fom non-spam. 2) Spammes behavios may be had to captue. Fo example, in ode to successfully mislead customes, spammes can make thei witing styles and eview habits look vey simila to those of genuine eviewes 3) Spammes can also wite good and honest eviews,

2 because they could be eal customes of some online stoes themselves sometimes. Futhemoe, a cuent genuine eviewe could be a spamme befoe, and we do not know when a eviewe would wite a spam eview. These obstacles ae the coe challenges that we think make simple behavioal heuistics insufficient. To detect sophisticated spammes, we need to conside moe clues. B. Contibution Ou fist contibution is to popose a heteogeneous gaph model with thee types of nodes to captue spamming clues. We believe that clues fo telling if a eviewe is innocent include the eviewe s eviews, all the stoes (s)he commented on, and eviews fom othe eviewes who have shopping expeiences in the same stoes. Theefoe, we popose a novel heteogeneous gaph, efeed to as the eview gaph, to captue elations among eviewes, eviews, and stoes. These ae thee kinds of nodes in the eview gaph. A eviewe node has a link to a eview if (s)he wote it. A eview node has an edge to a stoe node if it is about that stoe. A stoe is connected to a eviewe via this eviewe s eview on the stoe. Each node is also attached with a set of featues. Fo example, a stoe node has featues about its aveage ating, its numbe of eviews, etc. Figue 1 illustates such a eview gaph. Fig. I Review Gaph Ou second contibution is the intoduction of thee fundamental concepts, i.e. the tustiness of eviewes, the honesty of eviews, and the eliability of stoes, and the identification of thei inteelationships: a eviewe is moe tustwothy if (s)he has witten moe honesty eviews, a stoe is moe eliable if it has moe positive eviews fom tustwothy eviewes, and a eview is moe honest if it is suppoted by many othe honest eviews. Futhemoe, if the honesty of a eview goes down, it affects the eviewe s tustiness, which has an impact on the stoe (s)he eviews. And depending on how this eviewe s opinion vaies with othe eviewes opinions about the same stoe, othe eviewes tustiness may change. These intetwined elations ae evealed fom the eview gaph. Ou thid contibution is the development of an iteative method to compute the thee concepts based on the gaph model. Stating fom the common senses that ae uniquely elated to the stoe eview system and its spamming scenaios, we deive how one concept impacts anothe. To the best of ou knowledge, we ae the fist to use this node einfocement method to analyze a eviewe s tustiness, a stoe s eliability, and a eview s honesty. Although the geneal idea of einfocement based on gaph links has been applied in many diffeent scenaios including authoity discovey [5] and tuth identification fom multiple conflicting souces [10], thei poblem settings and detailed techniques ae vey diffeent fom eview spam detection; theefoe they ae not applicable to ou wok. II. SPAM DETECTION MODEL A. Intuitive Assumptions and Obsevations We stat by exploing some possible easons fo spamming. Gounded on common sense, we fist make the following assumptions: Spammes ae usually fo pofits, so they have connections to stoes that would benefit spammes to pomote thei pominence o defame othe stoes. Spammes ae usually hied by low quality stoes. Such stoes have a stonge motivation to hie spammes to wite dishonest eviews. Stoes with good eputations and stable custome taffic may not hie spammes at all; since they lose much moe if they ae caught doing so. Even if good stoes eally entice spammes to say good things about them, it may not be vey hamful. Theefoe, we assume that less eliable stoes ae moe likely to be involved in eview spamming. Hamful spam eviews always deviate fom the tuth. Theefoe, they can be eithe positive eviews about lousy stoes, o negative eviews about good stoes. Not all eviews deviating fom mainsteam ae spam. People may feel diffeently o have diffeent expeiences about the same sevice. Based on these assumptions, we have the following obsevations: 1) We can judge the honesty of a eview given the eliability of the stoe it was posted to, plus the ageement (to be defined late) of the eview with its suounding eviews about the same stoe. 2) If we have the honesty scoes of all the eviews of a eviewe, we can infe his/he tustiness, because one is cetainly moe tustwothy if one wote moe eviews with high honesty scoes. 3) Now we go back to see how to depict a stoe s eliability. Again fom common sense again, a stoe is moe eliable if it is eviewed by moe tustwothy eviewes with positive eviews and less eliable if it is eviewed by moe tustwothy eviewes with negative eviews. Figue 2 shows the above influences among a stoe s eliability, a eview s honesty, and a eviewe s tustiness. They ae intetwined and affect each othe. These influences have some esemblance to the well-known authoity and hub elation [5], but the elations in ou case ae diffeent. Fist, a eviewe cannot gain moe tust by witing moe

3 eviews, no can we use the mean of eview honesty to captue a eviewe s tustiness. Second, eviews have impact on each othe. If a eview deviates fom most of the othes, it could be a clue of spamming. Fig. 2. Influences among diffeent types of nodes B. Basic Definitions Fom the above obsevations, we define vaiables that quantify the qualities of eviewes, eviews, and stoes. Definition 2.1 (Tustiness of eviewes): The tustiness of a eviewe (denoted by T()) is a scoe of how much we can tust. Fo ease of undestanding and computation, we limit the ange of T() to ( 1, 1). Definition 2.2 (Honesty of eviews): The honesty of a eview e (denoted by H(v)) is a scoe epesenting how honest the eview is, H(v) is within the ange ( 1, 1). Definition 2.3 (Reliability of stoes): The eliability of a stoe s (denoted by R(s)) is a scoe epesenting the quality of stoe s. R(s) is also within the ange ( 1, 1). Fo ease of pesentation, we give the notations of node featues that ae elated to the computation of the above definitions in Table I. Notation Definition a eviewe v a eview s a stoe i eviewe s i th eview s eviewe s eview on stoe s v eview v s autho id n the numbe of s eviews H honesty summation of all s eviews v eview v s ating v stoe id of v s t v eview v s posting time Us the set of eviews on stoe s TABLE I FEATURES ASSOCIATED WITH NODES AND THEIR NOTATIONS C. Reviewe Tustiness To model the tustiness, let s see how we can tell if a eviewe is tustwothy o not, based on which we can devise seveal computational ideas. 1) A eviewe s tustiness doesn t depend on the numbe of his/he eviews, but on the summation of thei honesty scoes. 2) A eviewe s tustiness scoe should be positive o negative if he/she wites moe eviews with positive o negative honesty scoes. 3) Fo a given eviewe, his/he tustiness does not gow/dop linealy with the numbe of high/low honesty eviews that (s)he wote. It gows/dops faste when the numbe of such eviews is smalle, and slows down when the numbe is lage. We define a eviewe s tustiness to be dependent on the summation of the honesty scoes of all s eviews, n = i H i = H(α 1 ) (1) whee n is the numbe of eviews fom. The poposed tustiness scoe of should be a function T satisfying the above intuitions, which ae fomally expessed with the following elations. T ( i) < T( j), if H i < H j (2) T ( ) < 0 if H < 0, T( ) > 0if H > 0 (3) dt( ) = T( )( K T( )) (4) dh Relation (2) epesents the fist idea: One eviewe is moe tustwothy than anothe if one has a lage honesty scoe. Relation (3) depicts the second idea: We don t tust a eviewe whose eviews tend to be dishonest, and we tust a eviewe whose eviews tend to be honest. Relation (4) epesents the gowing (o dopping) speed of tustiness is the poduct of cuent tustiness level and the oom of impovement. Suppose K is the uppe bound of the tustiness scoe, K T() is how much moe tustiness one can gain by witing moe honest eviews. It becomes smalle as T() gows. Solving these elations gives us the geneal fom of tustiness function K T( ) = (5) KH e Notice this geneal fom is in the ange (0, K). As we mentioned befoe, the uppe bound of the tustiness scoe is +1, and its ange should be ( 1, 1) to have moe pactical meanings. We then e-scale the tustiness scoe T fo a eviewe as 2 T ( ) 1 (6) = H e The geneal shape of the tustiness function is the wellknown logistic cuve and is shown in Figue 3. Fig. 3. Relation between tustiness and honesty In eality, eviews fom the same eviewe can have diffeent degees of honesty, due to many easons, e.g.,

4 subjective bias. A genuine eviewe p may post an uneasonable eview i with H(p(i)) < 0, while a spamme q may have a honest eview j with H(q(j)) > 0. Howeve, it is still possible to distinguish malicious eviewes fom benign ones, since the tustiness scoe is contolled by the summation of a eviewe s eview honesty, i.e., the geneal tend of a use s eviews. To calculate T(), we need the honesty values of s eviews, which we define next. D. Review Honesty How do we intepet a eview? When we ead a stoe eview, we usually have two things in mind. The fist one is the stoe we ae looking at. If the stoe is a good one such as Apple.com, we tend to tust the positive eviews. If we ae eading eviews about a stoe we have neve head of, o we knew it is a bad one, we tend to doubt the high ating eviews. The second facto is the suounding eviews, which ae othe eviews about the same stoe within a cetain time window t, e.g., 3 months befoe and afte the posting time of the taget eview. We ae likely to tust the mainsteam opinions held by most suounding eviews, athe than outlie opinions. Based on these obsevations, we model a eview s honesty with two factos. 1) The eliability of the stoe it eviews. 2) The ageement between this eview and othe eviews about the same stoe within a given time window. We will discuss eliability in the next subsection. We study ageement fist by intoducing the suounding set. Definition 2.4 (Suounding Set): The suounding set of eview v is the set of v s suounding eviews. S v ={i i = v, t i t v t} i, j S v agee with each othe when thei opinions about the same aspects of the stoe ae close. Howeve, opinion mining is too costly to let us assess opinion of a eviewe [2]. Fotunately, in addition to the eview text, eviews usually have ating infomation about a stoe. Even if two 5- sta atings may mean diffeent aspects of a stoe, such as custome sevice and delivey, they ae coelated with each othe. Theefoe, we make some assumptions hee. A eview s ating about a stoe eflects its opinion. Two eviews with simila ating scoes about the same stoe have simila opinions about the stoe. Fom the assumptions, i, j S agee with each othe if v i j < (7) whee is a given bound (we use 1 in a 5-sta ating system in this pape). Thus, we can patition the suounding set Sv = S S (8) S v v, a v, d S { i Ψ Ψ < } (9) v, a = i v δ S v, d Sv \ Sv, a = (10) We also take the eviewes tustiness scoes into consideation. One eview should be good even if it does not agee with any suounding eviews when it is witten by a tustwothy eviewe while the suounding eviews ae posted by untustwothy eviewes. Similaly, one eview may be bad even if it agees with the suounding eviews, since they may be all fom spammes. Theefoe, we define the ageement scoe of eview v within time window t as A( v, = T( κ i S i ) T( κ v a j S j ) (11), v, d Notice that tustiness scoe T could be eithe positive o negative. This equation means that if one eview agees with othe eviews by tustwothy eviewes, its ageement scoe inceases. On the othe hand, if it agees with untustwothy eviewes, its scoe deceases. This equation also pomotes a benign use s eview ageement scoe when it is suounded by spam eviews that it does not agee with, because if spammes tustiness scoes ae negative, a benign eview s ageement scoe gets pomoted by subtacting the negative numbes. A(v, t) can be positive o negative. Hee we nomalize it to ( 1, 1) to make late computation easie. 2 A n ( v, = 1 (12) A( v, e Review e s honesty H(v) is defined as follows: H( v) = R( Γ v ) A ( v, (13) n whee R( v ) is the eliability of stoe v, which we will define late. Hee we just want to note that, by definition, R( v ) can be positive (fo eliable stoes) o negative (fo poo quality stoes). We take its absolute value as an amplifie of A n (v, t). This is consistent with pevious discussions. If a stoe s R( v ) is lage, it is eithe quite good o quite bad. A eview on this stoe should have a high honesty scoe if it agees with many othe honest eviews. If R( v ) is small, the stoe s eliability is had to tell. And a eview s honesty gets diminished a little by that fact. E. Stoe Reliability To define eliability, we have simila intuitions to tustiness. A stoe is moe eliable if it has moe tustwothy eviewes saying good things about it, while it is moe uneliable if moe tustwothy eviewes complain about it. The inceasing/deceasing tend of eliability should also be a logistic cuve so as to be consistent with ou common sense. Theefoe, we define the eliability R of stoe s as 2 R ( s) = 1 (14) θ e whee θ = T( κ )( Ψ ) v Us, T ( κv ) > 0 v v μ (15) μ is the median value of the entie ating system, e.g., 3- sta in a 5-sta ating ange, and U s is all eviews of stoe s. Theefoe, a stoe s eliability depends on all tustable eviewes who post eviews on it, and thei atings. When consideing eliability, we only conside eviewes with positive tustiness scoe because thei atings eally eflect the stoe s quality. In contast, whateve a less tustwothy eviewe says about a stoe, it is less tustable. Fo example, we don t know the eal intention to ate a stoe as a good

5 one o bad one, if it comes fom a potential spamme. F. Iteative Computation Famewok Integating the pieces of infomation of the eview gaph togethe, we have an iteative computation famewok to compute eliability, tustiness, and honesty, by exploing the inte-dependencies among them. The algoithm is given in the figue below, which is self-explanatoy. The oveall time complexity of the algoithm is O(k(N + N v )), whee k is the numbe of iteations. k is small (usually 4 o 5) as the algoithm conveges quite fast in pactice. III. EVALUATION A. Data Set and Data Featues We use the stoe eview data fom a lagest host of stoe eviews, fo ou expeiments. The website povides a unique ul fo evey eviewe s pofile, containing meta-data such as eviewe id, all his/he eviews and atings with posting times about stoes, and links to those stoes. In each stoe page, thee is infomation about its aveage ating scoe, all eviewes with thei eviews. The data we cawled is a snapshot of all eviews fom the website on Oct. 6th, We clean the data by emoving uses and stoes with no eview. Afte that, we have eviewes who wote eviews on stoes. B. Spamme Detection Results Evaluation 1) Evaluation Citeia: We use IR-based evaluation stategy. Fist we let ou algoithm identify highly suspicious spamme candidates. Then we ecuit human judges to make the judgments on the candidates about whethe they seem to be eal spammes. Theefoe, we have pecision as ou pefomance measue. Simila evaluation appoaches have been used in pevious eview spam detection eseach [3, 8]. Theefoe this is a well-established way of pefomance evaluation. The details of ou evaluation stategy ae as follows. Human Judgments Consistency Citeion: Since thee is no spamme label in the eview data, human evaluation is necessay. A spamme detection algoithm is effective, if diffeent human evaluatos agee with each othe about thei judgments and concu with the system on the same set of esults. Theefoe, we use human judgment consistency as anothe evaluation citeion. Ou human evaluatos ae 3 compute science gaduate students who also have extensive online shopping expeiences. They wok independently on spamme identification. 2) Human Evaluation Pocess: Judging suspicious spammes is a complex task fo human and often involves intuition and seaching fo additional infomation, especially when we taget at moe subtle spamming activities. To decide if a candidate is a spamme equies human judges not only to ead his/he eviews and atings, but also to collect evidences fom elations with othe eviewes, stoes, and even the Intenet. To standadize the judgment pocess, ou human evaluatos agee upon thee conditions as ou evidence to claim that a eviewe is a potential spamme. A eviewe is suspicious if (s)he has a significant numbe of eviews giving opposite opinions to othes eviews about the same stoes. Fo example, if a eviewe gives high atings to all stoes he has eviewed, while othe eviewes ate these stoes low, the eviewe is poblematic. A eviewe is suspicious if (s)he has a significant numbe of eviews with opposite opinions about some stoes to the atings fom the Bette Business Bueaus (BBB) 2 Fo example, if a eviewe gives high atings to all the stoes (s)he eviewed, but BBB gives them Fs (the ating ange is fom A to F), the eviewe is clealy suspicious. A eviewe is suspicious if (s)he has a significant numbe of eviews saying opposite opinions about some stoes as compaed to evidences pesented by geneal web seach esults. Fo example, if a eviewe gives high atings to all stoes he/she eviewed, but Google seach esults about these stoes often contain infomation of them having fake eviews, this eviewe is poblematic. Note that each of the above steps involves human labo effot, intuition, and backgound knowledge, so the entie evaluation is vey had to be computeized. Fo example, fo the thid step, ou human judges actually need to go though seveal seach esult pages and undestand the content, in ode to make a judgment. Although evey single condition may not be convincing enough to pove spamming activities, all of them togethe can be a confident claim of spamming. We evaluate the top 100 suspicious eviewes identified by ou model. Note that no existing wok focused on such a lage scale and subtle individual cases. Ou human evaluatos gave thei independent judgments based only on infomation fom 2 Bette Business Bueau is a well-known copoation that endeavos to a fai and effective maketplace. It gathes epots on business eliability, alet the public to business o consume scams, and enfoce the mutual tustiness between consumes and companies

6 eselleatings, business honesty infomation fom BBB, and seach esults fom Google, and by eading eviews. 3) Pecision and Consistency: In ou evaluation, if moe than one evaluato egads a eviewe as a spamme, we label it as a suspicious spamme. Ou evaluatos identified 49 out of 100 suspicious candidates to be spammes. The pecision is 49%. Although ou pecision is not vey high, we ae dealing with much moe subtle and complex cases (not simple duplications), which existing studies could not handle. Besides, ou pecision is meaningful, since human evaluatos agee with each othe on thei judgments. Table II shows the ageement of human judges. Fo example, Evaluato 1 identified 49 suspicious eviewes, out of which 33 wee ecognized by Evaluato 2 and 37 wee caught by Evaluato 3. To study thei ageement, we used Fleiss kappa [1], which is an inte-evaluato ageement measue fo any numbe of evaluatos. The kappa among 3 evaluatos is 60.3%, which epesents almost substantial ageement [6]. It is also impotant to note that those eviewes who wee not identified as spammes ae not necessaily innocent. It is simply because ou judges did not found enough evidences to conclude they ae spammes. Evaluato 1 Evaluato 2 Evaluato 3 Evaluato Evaluato Evaluato 3 40 TABLE II: HUMAN EVALUATION RESULT 4) Compae with Baseline: Since ou wok is the fist to use a eview gaph and taget at subtle spamming activities, thee is no existing wok to compae. Given the diffeences between ou wok and pevious studies, we want to demonstate that suspicious eviewes found by ou method can hadly be identified by existing techniques. We choose to compae with the appoach in Lim et al. [8], because it is the state-of-at of behavio based spamme detection techniques. They use some types of duplicate eviews as a stong evidence of spamming. They fist look fo candidates who have multiple eviews about one taget (in ou case stoe), and then compute spamming scoes to captue spammes fom the candidates. In ou top 100 suspicious candidate list, only 3 candidates can be found based on the duplication citeion. Moeove, only 1 out of these 3 candidates is finally labeled as a spamme by ou human evaluatos. This esult means that ou wok detects diffeent types of spamming activities fom existing eseaches. They can seldom find the spamme types that we can find. By no means do we claim that the existing methods ae not useful. Instead, ou method aims to find those that cannot be found by pevious methods. 5) Suspicious Spamme Case Study: Hee we take a close look at eviewes anked highly suspicious. We pick two candidates: one often pomotes stoes while the othe demotes stoes. The fist case is the eviewe howcome 3, 3 whose eviews ae mostly positive. When we examined these eviews, we found many of them ae poblematic, such as those highly ated eviews fo UBid, OnRebate, ISquaed Inc, Batteies.com, and BigCazyStoe.com. Fo example, UBid is widely complained at diffeent eview websites like ConsumeAffais, epinions, CimesOfPesuasion, and ReselleRatings. OnRebate was sued fo failing to pay ebates to customes. ISquaed Inc was ated D by BBB. Batteies.com is geneally lowly ated at ReselleRatings. BigCazyStoe has vey few ecods about its quality on the Intenet. And this eviewe s eview is the only one about BigCazyStoe on ReselleRatings. All these evidences lead us to conclude that this eviewe is suspicious. The second case is shibbyjk 4. All eviews of this eviewe ae complaints. All stoes being complained ae high quality stoes accoding to BBB, e.g., 1SaleADay(A), StaMico(B+), 3B Tech(B), and Accstation(A ). Fom these unfai atings, one may ague that this eviewe may be still nomal but just picky. Howeve, afte eading all his/he negative eviews, we found that all complaints ae about tansaction poblems. It is fishy because the chance of all these quality companies having tansaction poblems and cedit cad fauds with this paticula custome is vey low. Besides, tansaction poblems ae a good excuse to make false complains, since the tuth is had to be veified. IV. CONCLUTION This pape poposed a novel eview gaph model and an iteative method utilizing influences among eviewes, eviews, and stoes to detect spammes. The method showed how the infomation in the eview gaph indicates the causes fo spamming and eveals impotant clues of diffeent types of spammes. Expeimental esults show that the poposed method can identify subtle spamming activities with good pecision and human evaluato ageement. REFERENCES [1] J.L. Fleiss and J. Cohen. The equivalence of weighted kappa and the intaclass coelation coefficient as measues of eliability. Educational and Psychological Measuement, [2] M. Hu and B. Liu. Mining and summaizing custome eviews. In KDD, [3] N. Jindal and B. Liu. Opinion spam and analysis. In WSDM 08, [4] N. Jindal, B. Liu, and E.P. Lim. Finding unusual eview pattens using unexpected ules. In CIKM, 2010 [5] J. Kleinbeg. Authoitative souces in a hypelinked envionment. In Jounal of the ACM (JACM), [6] J.R. Landis and G.G. Koch. The measuement of obseve ageement fo categoical data. Biometics, [7] F. Li, M. Huang, Y. Yang, X. Zhu. Leaning to identify eview Spam. In IJCAI, [8] E.P. Lim, V.A. Nguyen, N. Jindal, B. Liu, and H.W. Lauw. Detecting poduct eview spammes using ating behavios. In CIKM, [9] Ott, M., Y. Choi, C. Cadie, and J.T. Hancock. Finding deceptive opinion spam by any stetch of the imagination. In ACL, [10] X. Yin, J. Han, and P.S. Yu. Tuth discovey with multiple conflicting infomation povides on the web. In TKDE,