Web Usage Mining. discovery of user access (usage) patterns from Web logs What s the big deal? Build a better site: Know your visitors better:

Transcription

1 Web Usage Mining

2 Web Usage Mining Web Usage Mining discovery of user access (usage) patterns from Web logs What s the big deal? Build a better site: For everybody system improvement (caching & web design) For individuals personalization For search engines SEO Know your visitors better: Customer behavior Be a better business

3 Web Usage Mining Applications Personalization Improve structure of a site s Web pages Aid in caching and prediction of future page references, Pre fetch of Proxy server ) Improve design of individual pages Improve effectiveness of e commerce (sales and advertising, Marketing decisions, Target Marketing )

4 Web Usage Mining: Data Source Typical data sources for web usage mining are: Web structure data (site map, links, etc.) Web content data User profile (may not be available) Web log Server access logs Server Referrer logs Agent logs Client side cookies User profiles Search engine logs Database logs

5 Transfer / Access Log The transfer/access log contains detailed information about each request that the server receives from user s web browsers. SERVER REQUEST REPLY CLIENT Time Date Hostname File Requested Amount of data transferred Status of the request

6 Agent Log The agent log lists the browsers (including version number and the platform) that people are using to connect to your server. SERVER REQUEST REPLY CLIENT Hostname Version Number Platform

7 Referrer Log The referrer log contains the URLs of pages on other sites that link to your pages. That is, if a user gets to one of the server s pages by clicking on a link from another site, that URL of that site will appear in this log. Page A B SERVER REQUEST REPLY CLIENT Page B URL REFERRER URL Note: Referrer logging is used to allow web servers to identify where people are visiting them from.

8 Error Log The error log keeps a record of errors and failed requests. A request may fail if the page contains links to a file that does not exist or if the user is not authorized to access a specific page or file. SERVER REQUEST REPLY CLIENT

9 Web Usage Mining Phases

10 Preprocessing: Challenges WHO are the users? IP vs. real people HOW LONG did the users stay? Measuring session time (L. Catledge and J. Pitkow. Characterizing browsing behaviors on the world wide web. Computer Networks and ISDN Systems, 27(6), 1995) (Berendt, B. Mobasher, M. Nakagawa, and M. Spiliopoulou. The impact of site structure and user environment on session reconstruction in web usage analysis. In Proceedings of the 4 th WebKDD 2002 Workshop, at the ACM SIGKDD Conference on Knowledge Discovery in Databases (KDD 2002), Edmonton, Alberta, Canada, July WHERE did the users go? Server side vs. Client side WHAT did the users view? Content processing Moe, Wendy W Buying, searching, or browsing: Differentiating between online shoppers using in store navigational click stream. J. Consumer Psych. 13(1, 2) For the best review on preprocessing methods, refer to: R. Cooley, B. Mobasher, J. Srivastava, Data preparation for mining world wide web browsing patterns, Knowledge and Information Systems 1 (1) (1999) 5 32

11 Data Preprocessing Steps To data mining algorithm

12 Data Preprocessing Steps Preprocessing includes four steps: Data Cleaning removes log entries that are not needed for the mining process User Identification associates page references with different users Session Identification groups user s page references into user sessions Path Completion fills in page references missing due to browser and proxy caching

13 Data Cleaning There are a variety of files accessed as a result of a request by a client to view a particular Web page. These include image, sound and video files, executable cgi files, coordinates of clickable regions in image map files and HTML files. Thus the server logs contain many entries that are redundant or irrelevant for the data mining tasks 1. User Request : Page1.html 2. Browser Request : Page1.html, a.gif, b.gif =>3 Entries for same user request in the Server Log redundancy Page1.html a.gif b.gif

14 Data Cleaning Hostnam e Date : Time Request SOLUTION: All the log entries with certain filename suffixes, such as gif, jpeg, GIF, JPEG, JPG, and map, are removed from the log.

15 Issues in User Session Identification A single IP address is used by many users different users Proxy server Web server Different IP addresses in a single session ISP server Single user Missing cache hits in the server logs Web server

16 User Identification Heuristics IP/Agent: Each different agent type for an IP address represents a different sessions Referring page: Uses site topology (web page linkage) If the referring page file for a request is not directly reachable by a hyperlink from any of the pages visited by the user, then is it a new user Combination with other information, such as machine name, temporal information,

17 IP/Agent Heuristic Two Users: - A-B-L-F-R-O-G-A-D - A-B-C-J

18 Example Referring page Heuristic Two Users: - A-B-L-F-R-O-G-A-D - A-B-C-J Three Users: -A-B-F-O-G-A-D -L-R -A-B-C-J

19 Session Identification Heuristics Timeout if the time between pages requests exceeds a certain limit, it is assumed that the user is starting a new session IP/Agent Each different agent type for an IP address represents a different sessions Referring page filed If the referring page file for a request is not part of an open session, it is assumed that the request is coming from a different session Same IP-Agent/different sessions (Closest): Assigns the request to the session that is closest to the referring page at the time of the request Same IP-Agent/different sessions (Recent) In the case where multiple sessions are same distance from a page request, assigns the request to the session with the most recent referrer access in terms of time

20 Session Identification Use timeout Three Users: -A-B-F-O-G-A-D -L-R -A-B-C-J Four Sessions: -A-B-F-O-G -A-D -L-R -A-B-C-J

21 Path Completion Refers to the problem of inferring missing user references due to caching Effective path completion requires extensive knowledge of the link structure within the site Referrer information in server logs can also be used in disambiguating the inferred paths Problem gets much more complicated in framebased sites

22 Path Completion Example Four Sessions: -A-B-F-O-G -A-D -L-R -A-B-C-J Four Sessions: -A-B-F-O-F-B-G -A-D -L-R -A-B-A-C-J

23 From Sessions to Knowledge What are the set of pages frequently accessed together by Web users? What page will be fetched next? What are paths frequently accessed by Web users? What is the page mostly used as entry point to the web? What is the average view length per page category? Is the user likely to buy or just navigating?

24 Web Mining System Architecture Data Cleaning Transaction Identification Data Integration Data Cleaning Pattern Discovery Pattern Analysis ===== Add Home Name Registration Data Clean log Document and Usage Transaction Data Integrated Data Database Query Language Formatted Data Path Analysis Association Rules Sequential Patterns Clusters & Classification Rules OLAP/ Visualization Tools Knowledge Query Mechanism Intelligent Agents Attributes

25 Usage Pattern Discovery Techniques statistics analysis path analysis association rules sequential patterns clustering and classification

26 Statistics Analysis A summary report of hits and bytes transferred A list of top requested URLs A list of top referrers A list of most common browsers used Hits per hour/day/week/month reports Hits per domain reports

27 Association Analysis Association analysis discovery correlation among references Examples 40% of clients who accessed /company/product1 also accessed /company/product2 30% of clients who accessed /company/special1 placed an online order in /company/product1

28 Classification and Clustering Classification and clustering similar to collaborative filtering approaches User user Item item develop a profile of items belonging to a particular group according to their common attributes Examples clients from state or government agencies who visit the site tend to be interested in the page /company/product1 50% of clients who placed an online order in /company/product2 were in the age group and lived on the West Coast

29 Sequential Patterns Sequential patterns find inter transaction patterns such that the presence of a set of items is followed by another item in the time stamp ordered transaction set. Examples 30% of clients who visited /company/products had done a search in Google, within the past week on keyword w 60% of clients who placed an online order in /company/product1 also placed an online order in /company/product4 within 15 days.

30 Path Analysis Examples 70% of clients who accessed /company/product2 did so by starting at /company and proceeding through /company/new, /company/products and /company/product1 80% of clients who accessed the site started from /company/products 65% of clients left the site after 4 or less page references

31 Data Structures Keep track of patterns identified during Web usage mining process Common techniques: Trie Suffix Tree Generalized Suffix Tree WAP Tree

32 Trie vs. Suffix Tree Trie: Rooted tree Edges labeled which character (page) from pattern Path from root to leaf represents pattern. Suffix Tree: Single child collapsed with parent. Edge contains labels of both prior edges.

33 Trie and Suffix Tree

34 Generalized Suffix Tree & WAP Tree Generalized Suffix Tree: Suffix tree for multiple sessions. Contains patterns from all sessions. Maintains count of frequency of occurrence of a pattern in the node. WAP Tree: Compressed version of generalized suffix tree

35 Types of Patterns Algorithms have been developed to discover different types of patterns. Properties: Ordered Characters (pages) must occur in the exact order in the original session. Duplicates Duplicate characters are allowed in the pattern. Consecutive All characters in pattern must occur consecutive in given session. Maximal Not subsequence of another pattern.

36 Pattern Types Association Rules None of the properties hold Episodes Only ordering holds Sequential Patterns Ordered and maximal Forward Sequences Ordered, consecutive, and maximal Maximal Frequent Sequences All properties hold

37 Episodes Partially ordered set of pages Serial episode totally ordered with time constraint Parallel episode partial ordered with time constraint General episode partial ordered with no time constraint

38 Build a Better Site: System Improvement Server side caching of web pages Y. H. Wu, A.L.P. Chen, Prediction of web page accesses by proxy server log, World Wide Web 5 (1) (2002) Preprocessing: No IP discussion, sessions split by time based heuristics Method: Sequential pattern mining Data: Usage Contribution: Use frequent sequence to predict candidate page, personalize based on user maturity

39 Build a Better Site: System Improvement Improvement of general web design Fang, X. and O. R. L. Sheng (2004). Link Selector: A web mining approach to hyperlink selection for web portals. ACM Transactions on Internet Technology 4, Preprocessing: No IP distinguished, sessions split by 25.5 minutes Method: Association mining Data: Usage & Structure Contribution: Combine structure info. and usage info. to optimize portal page design

40 Build a Better Site: Personalization Personalize the web site based on usage patterns A key research domain: recommender systems* Content clustering vs. users clustering vs. hybrid approach C. Shahabi and F. Banaei Kashani. Ecient and anonymous web usage mining for web personalization. INFORMS Journal on Computing, Special Issue on Data Mining, 2002 Method: Clustering of sessions Data: Client side usage data

41 Build a Better Site: SEO (Search Engine Optimization) Adding usage information into PageRank patterns Kalyan Beemanapalli, Ramya Rangarajan, Jaideep Srivastava, Usage Aware Average Clicks, In Proc. Of WebKDD 2006: KDD Workshop on Web Mining and Web Usage Analysis, in conjunction with the 12 th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2006), August Method: Association rule in spirit

42 Know your visitors better: : Customer behavior A favorite research stream by marketers and MIS researchers Statistical models are used most of the time Macro level behavior is often the focus Interesting questions related to firm performance and profitability

43 Know your visitors better: Customer behavior Johnson, E. J., Wendy Moe, Peter S. Fader, Steven Bellman, and Jerry Lohse. "On the Depth and Dynamics of Online Search Behavior," Management Science, Vol. 50, No. 3, March 2004, pp model an individual s tendency to search as a logarithmic process hierarchical Bayesian model with Depth of Search, dynamics of search and activity of search interested in the number of unique sites searched by each household within a given product category Preprocessing: Households identified by client side programs, session is month based Method: Statistical Modeling (log model) Data: Usage (search)

44 Know your visitors better: Customer behavior Moe, Wendy W Buying, searching, or browsing: Differentiating between online shoppers using in store navigational clickstream. J. Consumer Psych. 13(1, 2) WHY do the customers visit? Preprocessing: Content Processing Method: Clustering of sessions by visiting behavior parameters and content parameters Data: Usage & Content Conclusion:

45 Know your visitors better: Customer behavior Sismeiro, Catarina, Randolph E. Bucklin Modeling Purchase Behavior at an E Commerce Web Site: A Task Completing Approach. Journal of Marketing Research. 41 (3), How do the customers visit? Predicts online buying by linking the purchase decision to what visitors do and to what they are exposed while at the site. Preprocessing: Content Processing Method: Statistical Modeling Data: Usage & Content Conclusion:

46 Know your visitors better: Customer behavior Sismeiro, Catarina, Randolph E. Bucklin Modeling Purchase Behavior at an E Commerce Web Site: A Task Completing Approach. Journal of Marketing Research. 41 (3), browsing behavior (i.e., time and page views) repeat visitation to the site (return and total number of sessions) use of interactive decision aids Data input effort and information gathering and processing a series of page specific characteristics