Reputation system design is the. Advanced Feedback Management for Internet Auction Reputation Systems. Trust & Reputation Management

Transcription

1 Advanced Feedback Management for Internet Auction Reputation Systems Electronic auction reputation systems have improved in recent years. However, most don t rely on user feedback but are still bound to old-fashioned comment counting while substantial information embedded in those comments is omitted. The authors system manages and learns from user feedback and considers auctions context, possible types of complaints, and the structure of connections between those complaints. They propose amplifying a reputation system algorithm to estimate the reported complaints harmfulness. Their results are based on a real-world dataset from a leading Eastern European online auction provider. Trust & Reputation Management Reputation system design is the most important thing in most social platforms. With Internet auction systems, especially where users invest great amounts of money it s crucial that those users get as much information as possible to improve the reliability of transactions. Reputation and trust management systems are based primarily on user feedback, but most platforms use old-fashioned comment counting and a [ 1, 0, 1] scale for comment valuation. However, such situations don t encourage sellers to improve their services. To address this problem, buyers should be able to explicitly express their detailed opinions about an auction and the seller. The Detailed Seller Report (DSR) ebay introduced in 2009 is still controversial (in part because the algorithm causes problems for low-volume sellers), but this approach moves one step closer toward extending the simple user-feedback interface. A well-designed feedback system and better-quality feedback can influence users trust toward the system itself. Here, we extend auction site feedback mechanisms by extracting information embedded in user comments, assigning this feedback value for use in reputation systems, and presenting the computed data to the user. To improve feedback, we suggest encouraging users to fill out a simple inquiry form at the end of an auction. This inquiry s subject should be adapted to take into account the item category, the feedback type, and the comments semantic meaning. Recent research shows that for positive comments, the added information is Tomasz Kaszuba Polish-Japanese Institute of Information Technology Albert Hupa University of Warsaw Adam Wierzbicki Polish-Japanese Institute of Information Technology SEPTEMBER/OCTOBER /10/$ IEEE Published by the IEEE Computer Society 31

2 Trust & Reputation Management negligible. 1 So, in our system, we consider only information from negative and neutral comments. Complaints Model Trust toward sellers is crucial for Internet auction platforms. Some trust enforcement mechanisms exist, such as DSR or escrow (an insurance system for auctions), but these don t build trust among a site s users. The auction platform provider should build trust in a bottom-up manner, starting with buyers, who express their feelings after the auction. The best possible way to improve trust is to ask buyers their opinions about sellers. Currently, ebay asks buyers the same four permanent questions after every auction, then computes the DSR value, which influences the sellers fees, promotions, the order of search, and so on. As our research shows, users can express much more subtle opinions about sellers than these questions might indicate. In our opinion, classification and understanding of users complaints can help build a better trust-management system. Additionally, existing reputation systems don t distinguish between different kinds of negative or neutral user feedback. They also use a very simple reputation algorithm. As long as we treat all negative feedback equally, we can t distinguish a purposeful from an accidental one. For example, a great difference exists between sending the wrong color or size of T-shirt to a user and not sending the T-shirt at all. To address these issues, we developed a model in which we can classify complaints against sellers and thus provide much more information about their inappropriate. The Dataset To create our classification model, we obtained a real-world dataset from Allegro ( pl), the leading Eastern European online auction provider. In this service, each auction has an explicit deadline, and all current bids are exposed to all participants. In most auctions, bidders can specify the maximum price they want to pay for an item, and Allegro s proxy bid system automatically raises the bid, using only the minimum amount necessary for the bidder to maintain the top position. Bidders can also increase their maximum price at any moment. When the auction terminates, the bidder with the highest bid wins. Allegro also has various multi-item ( Buy now! ) auctions in which sellers can sell more than one item (leading to more than one winner). In such auctions, every bid is a winning bid until no more items are available. For our model, we selected a subset of 9,500 sellers and buyers and their 285,000 auctions listed in 16,000 categories. We performed our research on a subset (15,000) of negative and neutral comments from 1.7 million total positive and nonpositive comments. The unequal amount of auction feedback versus the number of auctions is due to the multi-item type auctions. We mined the information from users comments using two independent classification approaches: top down and bottom up. These approaches helped us compare the outcomes that is, the different types of complaints on the basis of which we created a taxonomy by connecting those types according to different meanings. Classification Methods The top-down approach we used to develop our taxonomy involved creating a simple typology tree that employed a semiautomatic method using regular expressions. We designed and implemented the regex creator tool, which lets us create new regular expressions and assign the patterns to complaint types. In the bottomup approach, we used advanced data mining techniques to cluster the co-occurring words into groups. We applied the Newman-Girvan algorithm for community detection. 2,3 This approach is based on the measures of shortest path and betweenness centrality calculated for edges. Applying this algorithm generated sets of words that usually occurred together in our dataset. We treated these sets as meaningful types of complaints. Then, we examined the results from both methods and created the tree structures Figure 1 presents. More details about this work are available in our previous work. 4 Taxonomy of Complaints Within the taxonomy Figure 1 shows, we distinguish two kinds of losses due to fraud: time-related and money-related. We marked complaints related to lost time with striped lines, whereas those in light gray are related to monetary loss. In addition, we marked the four criteria that match those in ebay s DSR (the question about communication with the seller covers two of the categories in our taxonomy). We observed two general groups of complaints: seller--related and item-related. In the first group, we include the following seller : 32 IEEE INTERNET COMPUTING

3 Advanced Feedback Management Money related User related Seller related * Time related Main group DSR criteria Fraudulent Shill bidding Comm. misbehaving Overcharge shipping * No response * * sent late No payment info received not received No product to sell * * Poorly packaged Odd damaged Careless packaging different wrong not complete not as expected Fake or illegal Not sent or lost Low quality Not as described General Detailed Figure 1. Typology of complaints against sellers. We can discriminate between user- and item-related problems. Accusations against seller are mainly time-related, whereas item-related accusations are connected to a loss of money. Fraudulent includes shill bidding (that is, when a third party, often connected to the seller, raises the auction price intentionally) or shipping overcharges. We considered only explicitly formulated accusations, not those computed from historical auction data. No response complaints mean communicating with the seller after the auction was impossible. The seller didn t answer phone calls or respond to s. Odd involves the seller behaving in a completely unpredictable manner. For instance, communication with the seller was possible but handicapped, the seller sent the item with a delay, or he or she didn t define the payment method and shipping price. The second group of complaints is related strictly to the item: not sent or lost means the buyer never received the item, either because it was never sent, or because, as sellers sometimes argue, the courier or post office lost it. No product to sell occurs when the seller claims that the item was already sold to another buyer or that the item is no longer for sale. In this case, the buyer doesn t receive the item. Careless packaging indicates that the seller didn t take care in packaging the items and includes situations in which the received item arrived damaged. Verifying whether the seller sent a damaged item or the item was destroyed during shipment isn t possible. Wrong item means the seller sent the wrong item (or wrong color or type), or the received item wasn t complete. not as expected indicates that the item seems to be illegal goods (a fake, or pirated software) or just doesn t satisfy the buyer. Because the groups are disjoint, we can connect every comment to one or more of them. This provides more information about the seller s profile, which is crucial for sellers. The category tree covers only the meaningful accusations against the seller. We don t distinguish the most offensive comments (swearing and threats) unless we find at least one complaint from the list. Complaint Classification Using regular expressions prepared via our two classification methods, we partitioned all negative and neutral feedback from our dataset into the detailed types represented in the complaint taxonomy. Each complaint type has its own meaning and also a unique set of regular expression patterns. In our evaluation, we used the types classified into the general level only to obtain more legible results. We group SEPTEMBER/OCTOBER

4 Trust & Reputation Management Frequency of occurrence Negative Neutral Nonpositive 5 0 wrong not as expected Careless packaging Odd not sent or lost Fraudulent No product to sell No reponse Figure 2. Results for seller complaints. Buyers make neutral comments more often when there is a problem with the item or its state, whereas they make negative comments when the seller doesn t follow the rules (communication problems or lack of item). the patterns from the detailed level along the category tree. We tested all negative and neutral comments buyers made and assigned each to types in our taxonomy. We matched each comment against all patterns from our model. A comment could be assigned to more than one pattern from different types. In Figure 2, we present normalized results for all neutral or negative feedback separately. In addition, we present the percentage results jointly for all nonpositive feedback (negative or neutral comments). Our regular expression tool matched 68 percent of the negative comments and 54 percent of the neutral comments for sellers. The difference in classification quality between negative and neutral feedback occurs because neutral comments contain fewer complaints, which provide the most useful information. Unclassified comments contained mostly useless information (no specified reason or lots of spelling errors). We reduce the amount of such feedback by enabling users to choose one of our proposed complaint types from a list instead of editing comments themselves. However, users retain the ability to edit comments afterward to add more information if desired. Negative feedback. Figure 2 shows how frequently complaints occurred against sellers in our dataset. Most negatives are due to a lack of response from the seller or to buyers not receiving the item. This is predictable because users don t like to be uninformed, especially when they risk their money. A significant amount of negative feedback is due to problems with the item, such as the seller sending the wrong or a low-quality item. A few buyers made direct accusations of shill bidding or excess shipping costs. We also noticed some situations in which the seller refused to sell the item and informed the buyer about it. Neutral feedback. Buyers gave neutral feedback in most cases when the item didn t live up to their expectations or was different (for example, a different color or size) than described in the auction. Seller, such as a buyers problems understanding the seller or delays in sending the item, were also frequent reasons for neutral, rather than negative, feedback. Compared with negative feedback, we can observe a significant drop (almost 50 percent) in complaints related to not sending an item or ignoring the buyer. Complaints Grading To make our research more applicable to trust or reputation systems in general, we propose a simple method for rating the types of complaints according to their harmfulness. We based this method on the percentage points of negative and neutral comments in each complaint category. To verify our method and detect independent rules of grading, we conducted an opinion poll among real Internet auction users. We juxtaposed the results from both methods in Figure 3. All values for the computational method were generated from nonpositive feedback (negative or neutral). Less 34 IEEE INTERNET COMPUTING

5 Advanced Feedback Management Computational Opinion poll 6 Harmfulness No reponse not sent or lost No product to sell 1 Fraudulent Careless packaging wrong not as expected Odd Figure 3. Harmfulness grading. According to the opinion poll, problems with the item are the most harmful (that is, have the highest value). The computational method s results show that the sellers is also important harmful complaint types have lower values in the figure. Computational Method We define harmfulness as the balance between the frequency of occurrence of negative and neutral feedback (percentage points). We compute the harmfulness for every type in our complaint taxonomy. A type of complaint tends to be more harmful if we ve classified more negative than neutral feedback into that type. According to this computational method, the most harmful seller s are lack of response and not sending the item after receiving payment. Users are more forgiving with cases connected to the item s condition. Shipment delays and mistakes are more often graded as neutral than negative. Opinion Poll To verify our approach s correctness, we conducted an opinion poll among real Internet auction users from sites such as Allegro and ebay. We received 208 answers from people between the ages of 19 and 59. One hundred and forty-eight declared that they sold goods on auction systems. We asked respondents for their subjective opinions about each category s harmfulness. Results obtained from the opinion poll show that more frequent types of complaints aren t necessarily considered more harmful by respondents. Users seem to be more tolerant to a lack of response from the seller or to when sellers declare after the auction that they no longer have the item to sell. According to our respondents, the most harmful s regard the item s condition, as when sellers send a damaged, incomplete, or different item. Improving the Feedback System As mentioned previously, ebay introduced its DSR system to let buyers rate sellers on four different criteria: product description accuracy, communication between the seller and buyer, shipping time, and shipping charge. Buyers can leave one to five stars for each criteria. Although the system design is convenient for buyers, ebay doesn t instruct them on how to use it or reveal the reasons behind its criteria. Compared to our model, criteria used in DSR cover most user-related problems but don t take into consideration item-related complaints (only the question about the item description falls into this category). Moreover, DSR uses identical criteria for every type of category and value of an item, which can be a serious constraint in letting users express themselves. Our approach covers all complaints in greater detail with respect to the buyers harmfulness scale. We suggest adapting such questions to auction feedback systems, which should utilize userprovided information to create a seller profile, thereby influencing the reputation system. Complaint Co-Occurrence Our research shows that certain groups of complaints often occur together. Such connections can improve the description of the seller s profile when we can t presume it from a simple classification. To detect such clusters, we cre- SEPTEMBER/OCTOBER

6 Trust & Reputation Management (a) (b) _wrong _not_as_expected Careless_packaging Fraudulent_ No_product_to_sell Odd not_sent_or_ lost No_reponse _wrong _illegal_or_fake _not_sent_or_ lost _sent_late No_product_to_sell _damaged _not_sent_or_ lost Poorly_packaged _low_quality Communication_misbehaving Overcharge_shipping No_product_to_sell _not_as_described _not_complete _different Figure 4. Similarity tree of complaints. We applied a clustering algorithm to determine the co-occurrence of types of complaints for (a) general groups and (b) detailed groups from our complaint model. ated symmetric networks in which vertices represent types of complaints and edges stand for their co-occurrence. We used the frequency of co-occurences as weights for a Newman-Girvan algorithm. 2,3 We ran the algorithm twice to locate subgroups using the general and detailed groups from our complaint model (see Figure 1). General Groups We applied the Newman-Girvan clustering algorithm and generated a similarity tree for eight complaints. Figure 4a presents the results for the general groups. As we can see, two main groups occurred separately. One is related to an item s condition, which corresponds to the results from the opinion poll (users evaluated the three types in this group as more harmful than did our computational scale). These complaints frequently occur with accusations of shill bidding or shipping overcharges. The second group is connected with the seller s, such as no response, not sending or delay in sending an item, or no payment information. Detailed Groups We ran the algorithm again on the detailed groups (15 complaints) to detect more precise results. Figure 4b shows this similarity tree. We can spot two groups of co-occurring complaints. At the bottom, we can see a group connected strictly with item descriptions. The cluster includes situations such as sending an incomplete or different item, items with a poor description, or even the lack of the item itself. At the higher (less-connected) level, we see mis in communication or overcharge shipping, which can be characteristics of fraudulent sellers. The second strong cluster is related to late shipments and a lack of response from the seller, which is typical for sellers who aren t interested in communication with the buyer. We suggest using our solutions, including complaints grading and complaints cooccurrence, together to improve existing reputation systems. Our improvements are based on user-expressed feedback, so they re applicable to most auction platforms. Sites can adapt these improvements to their current reputation systems with little effort (they require recomputing the frequencies of occurrence and harmfulness). Internet auction platforms can also use our rating schemes to detect and fight against fraud and thereby gain more trust from users. Rating complaints can be a good alternative to the controversial DSR and can be easily implemented into auction platforms. Using types of complaints for the seller profile description can improve the service, and historical data about the seller s profile type can be available instead of or in conjunction with a list of comments. The seller s profile type can also affect search visibility or seller fee discounts. Auction services can use cooccurrence between the types of complaints as a building block for the adaptive questionnaire system. We suggest applying such a system to the feedback module in reputation management systems (instead of using an immutable list, as in ebay-like systems). Aggregated feedback or statistics can also be available to sellers in the dashboard or similar seller-support tool, so they can improve their services based on user comments. In the future, we plan to integrate our model with the ProtoTrust tool, 5 an interactive Web browser extension that helps users in the decision-making process using trust management 36 IEEE INTERNET COMPUTING

7 Related Work in Electronic Auction Fraud Detection Most recent work in electronic auction fraud detection has focused only on a seller s profile. 1,2 Researchers have devoted considerable work to inducing users to behave properly, 2,3 and to detecting fraudulent users. 4,5 Recently, Yatel Yang and his colleagues designed a system for detecting dishonest ratings. 6 Their system complements the one we describe in the main text because it can filter out unfair complaints. Some tools are dedicated to detecting fraudulent sellers 7 or entire cliques of fraudulent agents (such as NetProbe 8 ). Bezael Gavish and Christopher L. Tucci presented sellers swindling methods in Internet auctions, 9 whereas Dawn C. Gregg and Judy E. Scott have proposed a model of complaints against sellers, 1 although they haven t discussed a model against buyers. While their model is similar to ours, they use a manual process to classify feedback and don t employ bottom-up mining techniques to detect the complaint groups. They also haven t proposed a way to improve the grading of feedback types. Although we don t discuss it in this article, our recent work has created a model of complaints against buyers. 10 References 1. D.G. Gregg and J.E. Scott, A Typology of Complaints about ebay Sellers, Comm. ACM, vol. 51, no. 4, 2008, pp P. Resnick and R. Zeckhauser, Trust among Strangers in Internet Transactions: Empirical Analysis of ebay s Reputation System, The Economics of the Internet and E-Commerce, Advances in Applied Microeconomics series, M.R. Advanced Feedback Management Baye, ed., Elsevier Science, 2002, pp ; presnick/ papers/ebaynber/index.html. 3. C. Dellarocas, Immunizing Online Reputation Reporting Systems against Unfair Ratings and Discriminatory Behavior, Proc. 2nd ACM Conf. Electronic Commerce (EC 00), ACM Press, 2000, pp D.H. Chau and C. Faloutsos, Fraud Detection in Electronic Auction, European Web Mining Forum, Proc. European Conf. Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2005; seerx.ist.psu.edu/viewdoc/summary?doi= S. Rubin et al., An Auctioning Reputation System based on Anomaly Detection, Proc. 12th ACM Conf. Computer and Communications Security (CCS 05), ACM Press, 2005, pp Y. Yang et al., Defending Online Reputation Systems against Collaborative Unfair Raters through Signal Modeling and Trust, Proc ACM Symp. Applied Computing, ACM Press, 2009, pp T. Kaszuba et al., ProtoTrust: An Environment for Improved Trust Management in Internet Auctions, Local Proc. 13th East-European Conf. (ADBIS 09), JUMI Publishing House, 2009, pp S. Pandit et al., NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks, Proc. 16th Int l Conf. World Wide Web (WWW 07), ACM Press, 2007, pp B. Gavish and C.L. Tucci, Reducing Internet Auction Fraud, Comm. ACM, vol. 51, no. 5, 2008, pp T. Kaszuba, A. Hupa, and A. Wierzbicki, Comment Classification for Internet Auction Platforms, Local Proc. 13th East-European Conf. (ADBIS 09), JUMI Publishing House, 2009, pp techniques. ProtoTrust can compute more complex measures than simple reputation (risk, probability of fraud, and average selling price, for example) and takes into consideration an auction s context. By integrating with ProtoTrust, we hope to create a helpful, user-friendly tool that lets users detect unreliable contractors. Acknowledgments The Polish Ministry of Science and Higher Education funded the work reported in this article under research grant N N References 1. A. Wawer and R. Nielek, Remove Positive Comment Descriptions between Informativity and Sentiment, Proc. Associated Workshops and Doctoral Consortium of the 13th East European Conf. (ADBIS 09), LNCS 5968, Springer, M. Girvan and M.E.J. Newman, Community Structure in Social and Biological Networks, Proc. Nat l Academy of Sciences, vol. 99, no. 12, 2002, pp M.E.J. Newman and M. Girvan, Finding and Evaluating Community Structure in Networks, Physical Rev., vol. E 69, no , T. Kaszuba, A. Hupa, and A. Wierzbicki, Comment Classification for Internet Auction Platforms, Local Proc. 13th East-European Conf. (ADBIS 09), JUMI Publishing House, 2009, pp T. Kaszuba et al., ProtoTrust: An Environment for Improved Trust Management in Internet Auctions, Local Proc. 13th East-European Conf. (ADBIS 09), JUMI Publishing House, 2009, pp Tomasz Kaszuba is a fourth-year PhD student and assistant at the Polish-Japanese Institute of Information Technology (PJIIT). He has a master s degree in distributed computing from PJIIT. Contact him at kaszubat@pjwstk.edu.pl. Albert Hupa is a sociologist at the University of Warsaw. He has a PhD from the Institute of Applied Social Sciences at the University of Warsaw. Contact him at albert.hupa@gmail.com. Adam Wierzbicki is an assistant professor at the Polish-Japanese Institute of Information Technology. He has a PhD in Internet communications from Warsaw University of Technology. Contact him at adamw@pjwstk.edu.pl. SEPTEMBER/OCTOBER