Access Logs. Prof. Sheizaf Rafaeli Electronic commerce

Transcription

1 Access Logs Prof. Sheizaf Rafaeli Electronic commerce

2

3

4

5 Advantages of maintaining state Shopping cart applications Customizing and personalizing content Tracking navigation patterns Creating subscriber status Remembering pesky passwords Rewarding frequent or return visits Changing banners and bookmarks Games: remembering scores, high scores, skill levels Prof. Sheizaf Rafaeli E-Business 5

6 Maintaining state Stored in cookies Encoded in URL links Sent in hidden form variables Stored in variables in other (hidden) frames Stored on the web server (least desirable) Prof. Sheizaf Rafaeli E-Business 6

7 Cookies Magic cookies Persistent client state HTTP cookies A cookie is a small amount of information that a Web site sends to your browser. When your browser receives a cookie, it saves the cookie on your hard drive for future use When you re-visit a site, your browser checks for any pre-defined preferences (cookies) for that particular site. Prof. Sheizaf Rafaeli E-Business 7

8 Cookies Enable storing information on the client s browser for later retrieval Most powerful technique for maintaining state within a web site Prof. Sheizaf Rafaeli E-Business 8

9 Web sites use cookies in many different ways. Sites can accurately determine how many people actually visit the site. It turns out that because of proxy servers, caching, concentrators and so on, the only way for a site to accurately count visitors is to set a cookie with a unique ID for each visitor. Using cookies, sites can determine: How many visitors arrive How many are new vs. repeat visitors How often a visitor has visited The first time a visitor arrives, the site creates a new ID in the database and sends the ID as a cookie. The next time the user comes back, the site can increment a counter associated with that ID in the database. Sites can store user preferences (often referred to as customization). E-commerce sites can implement things like shopping carts and "quick checkout" options. It would be impossible to implement a convenient shopping mechanism without cookies or something like them. TRY THIS: Prof. Sheizaf Rafaeli E-Business 9

10 Prof. Sheizaf Rafaeli E-Business 10

16 Are YOU a voyeur? Visit (choose red) Was it interesting? Prof. Sheizaf Rafaeli E-Business 16

17 Problems, Constraints and Disadvantages (real) Cookie may not be persistent May be deleted by accident or on purpose may be disallowed or frozen Browser may impose limitations, distorting the information Unencrypted, may give away secrets Made to sound scary (see myths) Prof. Sheizaf Rafaeli E-Business 17

18 Where are cookies stored? By Netscape, as cookies.txt on Windows machines or as MagicCookies (on Macs) By Explorer in special directory named Windows/Cookies By other browsers - wherever they wish Prof. Sheizaf Rafaeli E-Business 18

19 Cookie Myths The biggest problem seems psychological Big brother violating privacy? Cookies seldom used for this purpose Cookies cannot be used to get data from your hard drive, your address or sensitive information about your person HOWEVER: look at delivering targeted REAL TIME marketing Prof. Sheizaf Rafaeli E-Business 19

20 WebBugs (doubleclick s secret) A hidden active link <img src= width=1height=1> </img> Prof. Sheizaf Rafaeli E-Business 20

21 The DoubleClick Controversy Profiling Prof. Sheizaf Rafaeli E-Business 21

22 DoubleClick Personal data sent to DoubleClick servers includes: My address My full name My mailing address (street, city, state, and Zip code) My phone number Transactional data sent to DoubleClick includes: Names of VHS movies I am interested in buying Details of a plane trip Search phrases used at search engines Health conditions See Richard Smith s Prof. Sheizaf Rafaeli E-Business 22

23 Cookie Myths (2) Early implementations of Java and JavaScript did allow awful things but for the most part these security leaks have been plugged. Software limits total size of cookie file: less than 1.2 MB no more than 80 KB per each web site each site can only access its own Prof. Sheizaf Rafaeli E-Business 23

24 Cookie Myths (3) A site can only access a cookie that has been set from its own domain, It cannot access any other cookies from your computer. Prof. Sheizaf Rafaeli E-Business 24

25 Morally Permissible Cookies Collection of Personal Information Customer preferences Online shopping Mr. Smith

26 Immoral Uses of Cookies Centralization of Personal Information In order to measure our browsing behavior, target marketers track us over the Internet by adding cookies to the advertisement banners on so many Web pages. Is such a use of cookies ethical? Does it fit within a reasonable expectation of privacy on the Web?

28 _hOEvuq1T.jpg

29 29

30 30

31 31

32 32

33 Online Filter Bubbles (Eli Pariser) /talks/eli_pariser_b eware_online_filter _bubbles.html 33

34

35

36

37 Why have an interest in hits, visitors, visits and sessions? Bragging rights Assessing advertising rates Pricing and billing Site design Feedback and navigational help for users Quality assurance Security concerns Copyright enforcement Marketing research Editorial and marketing quality improvements

38 Surveys and panels Media Metrix utilizes its patented, superior operatingsystem metering methodology to track Internet and Digital Media audience usage behavior in real-time - click-byclick, page-by-page, minute-by-minute. Media Metrix has a sample of more than 70,000 people under measurement worldwide, yielding monthly, weekly, and daily data collection and reporting. (

39

40 Information sources about the audience "clickstream": Speculate (data-less research) Surveys (ask 'em) and panels (watch em grow) Experiments (watch 'em in the lab) Counters (visible and hidden; internal and external) Cookies Javascript background processes Active-X background processes Java background processes CGI environment variables CGI events (info sent with POST commands) Eyeball-trackers and other physiological measures

41 Information sources about the audience "clickstream": ***Logs*** Non-server logs, such as history files created by the browsers, intermediate logs recorded by firewall and proxy devices, cookie logs on individual clients and intermediating networks. PC-Meters (measuring at the source, like TV-meters) (watch 'em at home) perhaps used in panels of participating households (likely to happen in major markets. unlikely to capture the essence of the web, as the sample is already biased).

42 Different types of LOGS: Error Security Referrer Browser (Agent) Access-Logs

43 What do log files count? Remember: sessions are "connectionless" or "stateless! The common access log file contains data entries for 1. Remote host name or IP number 2. User_logname - often not implemented and replaced by "-" 3. Authenticated_user - replaced by "-" if not an authenticated request 4. Date and time 5. Request from client (name of document, file or command) 6. HTTP status code returned to client (200 is success) 7. Number of bytes sent REFER and ERROR may have more

44 A snapshot from an access log: annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/giff2sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/giff1sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/watz2sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:54: ] "GET /vol1/issue4/mclaugh.html HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:55: ] "GET /vol1/issue4/watz1sm.gif HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /bluebar.jpg HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /address2.gif HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55:57 sample +0200] "GET of /names.gif a few HTTP/1.0" lines from an lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /Platsite.jpg HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /SCOUTSEL.GIF HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:56:15 access_log +0200] "GET / HTTP/1.0" of one 200 of 5469 the servers of port327.cjnetworks.com - - [04/Jan/1998:21:58: ] "GET / HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:21:58:49 the +0200] Journal "GET /vol1/issue3/crede.html of Computer HTTP/1.0" Mediated port327.cjnetworks.com - - [04/Jan/1998:21:59: ] "GET /journal.html HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:21:59:50 Communication +0200] "GET /vol1/issue3/vol1no3.html (JCMC): HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:22:00: ] "GET /vol3/issue2/ HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:22:00: ] "GET /vol3/issue2/head.jpg HTTP/1.0" cumin.mcc.ac.uk - - [04/Jan/1998:22:01: ] ( "GET /vol1/issue3/search.vol1no3.html and HTTP/1.0" shelley.gre.ac.uk - - [04/Jan/1998:22:01: ] "POST /cgi-jcmc/srch.vol1no3.cgi HTTP/1.0" cumin.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /vol1/ HTTP/1.0" saffron.lut.ac.uk - - [04/Jan/1998:22:01: ] "GET /icons/back.gif HTTP/1.0" sorrel.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /icons/blank.gif HTTP/1.0" sorrel.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /icons/folder.gif HTTP/1.0" saffron.lut.ac.uk - - [04/Jan/1998:22:01: ] "GET / HTTP/1.0"

45 A snapshot from an access log: annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/giff2sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/giff1sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/watz2sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:54: ] "GET /vol1/issue4/mclaugh.html HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:55: ] "GET /vol1/issue4/watz1sm.gif HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /bluebar.jpg HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /address2.gif HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /names.gif HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /Platsite.jpg HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /SCOUTSEL.GIF HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:56: ] "GET / HTTP/1.0" port327.cjnetworks.com - - [04/Jan/1998:21:58: ] "GET / HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:21:58: ] "GET /vol1/issue3/crede.html HTTP/1.0" port327.cjnetworks.com - - [04/Jan/1998:21:59: ] "GET /journal.html HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:21:59: ] "GET /vol1/issue3/vol1no3.html HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:22:00: ] "GET /vol3/issue2/ HTTP/1.0" ww-tp06.proxy.aol.com reading textual - - [04/Jan/1998:22:00:20 HTML files or requesting +0200] "GET graphic /vol3/issue2/head.jpg "gif" files. Other HTTP/1.0" "sessions" 200 are 7203 more cumin.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /vol1/issue3/search.vol1no3.html HTTP/1.0" shelley.gre.ac.uk - - [04/Jan/1998:22:01: ] "POST /cgi-jcmc/srch.vol1no3.cgi HTTP/1.0" cumin.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /vol1/ HTTP/1.0" saffron.lut.ac.uk - - [04/Jan/1998:22:01: ] "GET /icons/back.gif HTTP/1.0" sorrel.mcc.ac.uk an educational/academic - - [04/Jan/1998:22:01:53 institution +0200] while "GET /icons/blank.gif others are using HTTP/1.0" commercial and network sorrel.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /icons/folder.gif HTTP/1.0" saffron.lut.ac.uk - - [04/Jan/1998:22:01: ] "GET / HTTP/1.0" Seven simultaneous "hosts", "visiting" over a randomly chosen period of just under ten minutes in "an evening in the life of an online journal". Even in this small sample one can sense that some of the "sessions" are "directed", some "meandering". Some are passive, just interactive. One visitor generated a "POST" record that results in running a CGI program. Note how the visitors are from different countries (two in this case). Several visitors originates from accounts/connections. If we follow the log long enough, we can begin to discern a pattern of interests for individual visitors and in the aggregate for types of visitors.

46 A snapshot from an access log: annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/giff2sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/giff1sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:53: ] "GET /vol1/issue4/watz2sm.gif HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:54: ] "GET /vol1/issue4/mclaugh.html HTTP/1.0" annex bmi.net - - [04/Jan/1998:21:55: ] "GET /vol1/issue4/watz1sm.gif HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /bluebar.jpg HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /address2.gif HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /names.gif HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /Platsite.jpg HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:55: ] "GET /SCOUTSEL.GIF HTTP/1.0" lib21.lib.edinboro.edu - - [04/Jan/1998:21:56: ] "GET / HTTP/1.0" port327.cjnetworks.com - - [04/Jan/1998:21:58: ] "GET / HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:21:58: ] "GET /vol1/issue3/crede.html HTTP/1.0" port327.cjnetworks.com - - [04/Jan/1998:21:59: ] "GET /journal.html HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:21:59: ] "GET /vol1/issue3/vol1no3.html HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:22:00: ] "GET /vol3/issue2/ HTTP/1.0" ww-tp06.proxy.aol.com - - [04/Jan/1998:22:00: ] "GET /vol3/issue2/head.jpg HTTP/1.0" cumin.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /vol1/issue3/search.vol1no3.html HTTP/1.0" shelley.gre.ac.uk - - [04/Jan/1998:22:01: ] "POST /cgi-jcmc/srch.vol1no3.cgi HTTP/1.0" cumin.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /vol1/ HTTP/1.0" saffron.lut.ac.uk - - [04/Jan/1998:22:01: ] "GET /icons/back.gif HTTP/1.0" sorrel.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /icons/blank.gif HTTP/1.0" sorrel.mcc.ac.uk - - [04/Jan/1998:22:01: ] "GET /icons/folder.gif HTTP/1.0" saffron.lut.ac.uk - - [04/Jan/1998:22:01: ] "GET / HTTP/1.0"

47 Different measures Hit Page View Visit Session Status Code Date and Time Reach

48 And see:

49

50 To Measure is to Manage Commercial and semi-free analysis services Websidestory and Hitbox Another approach: Media metrix / PCMeter Nedstats NetRatings/Nielsen

51 It is important to distinguish Exposure between counting for: Click-through Interactivity Other outcome : learning, persuasion, employee monitoring, law enforcement, etc.

52 Online Marketing Metrics: Lexicon Metrics that focus on success of Web site in achieving audience or market share Impressions Click-through rate (CTR) View-through rate (VTR) Hits Page views Stickiness (duration) Unique visitors Loyalty Reach Recency

53 Online Marketing Metrics: Lexicon (cont d) Metrics that focus on conversion of visitor to customer Acquisition rate Conversion rate Browse-to-buy-ratio View-to-cart ratio Cart conversion rate Checkout conversion rate Abandonment rate Retention rate Attrition rate

54 Online Marketing Metrics: Lexicon (cont d) metrics Open rate Delivery rate Click-through rate ( ) Bounce-back rate Unsubscribe rate Conversion rate ( )

55 Problems and drawbacks with logs: Inappropriate, mistaken use Not aggregating, not separating wheat from chaff Deliberate "fudging" or worse Dynamic content Offline reading Proxies, robots, caches, forwarding, printed copies Multi casting Plug-ins and media objects No record of what happens offsite (inside java, etc.)

56 Ethical issues Should web-stats be made public? Would it be ethical to NOT make logs public? Should web stats be made public even if only in aggregate? Should decisions, allocation, pricing, strategy, etc. be based on log stats?

57

58

59

60

61 Getting Paranoid? Remember the paranoid s epitaph See CNET s paranoiac toolkit Use Window-washer for your cookie, cache, history and bookmarks.

62 Paranoid 2? Blackbook But remember that there is Prudence, and

63 Paranoid 3? LittleBrother, from

64 More on Little Brother

65 Moving into CRM CRM: Customer Relationship Management

66 What to look for in analysis tools Scalability: deal with 100 s of daily MB Manageability and Customizability Integration with rest of site Reactive and Proactive Analysis Departmental, compatmentalized and personalized views and cuts Data mining and drill down functionality

67 Further References See Junkbusters brief history of junk At: Please see access logs web page, on: NewMedia.com s How do you measure up, at: For some datamining and CRM pointers Yahoo s collection of Access log analysis tools Hypernews list: DevTools:

68 Further References Scientific American, April 1999 Watch the Watchers David Brin s The Transparent Society Addison Wesley, 1998.

69

70 Google Analytics

71 Analytics

72

73 Cookies and.

74

75

76

77 Expressions Instead of Impressions

78

79