Log Analyzer. August 1998 Edition. Copyright WebTrends Corporation

Transcription

1 Log Analyzer August 1998 Edition Copyright WebTrends Corporation

2

3 Disclaimer WebTrends Corporation makes no representations or warranties with respect to the contents or use of this manual, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, WebTrends Corporation reserves the right to revise this publication and to make changes to its content at any time, without obligation to notify any person or entity of such revisions or changes. Further, WebTrends Corporation makes no representations or warranties with respect to any WebTrends Corporation product, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, WebTrends Corporation reserves the right to make changes to any and all parts of WebTrends Corporation products at any time, without obligation to notify any person or entity of such changes. Trademarks Product Support WebTrends, WebTrends Log Analyzer, and FastTrends are trademarks of WebTrends Corporation. Other brands and their products are trademarks or registered trademarks of their respective holders. If you should have any questions during the installation or use of WebTrends Log Analyzer, please refer to the WebTrends Corporation Knowledge Base online at: If you are unable to resolve the problem, do not hesitate to contact our Product Support Department: Telephone: (503) , Fax: (503)

4 Software License Agreement GRANT WebTrends Corporation ("Developer") hereby grants you a non-exclusive license to use its accompanying software product ("Software") for use on a single computer. You may not: permit other individuals to use the Software, modify, translate, reverse engineer, decompile, disassemble (except to the extent applicable laws specifically prohibit such restriction), or create derivative works based on the Software; copy the Software (except for back-up purposes); rent, lease, transfer or otherwise transfer rights to the Software; or remove any proprietary notices or labels on the Software or documentation. SOFTWARE If you receive your first copy of the Software electronically, and a second copy on media, the second copy may be used for archival purposes only. This license does not grant you any right to any enhancement or update. TITLE Title, ownership rights, and intellectual property rights in and to the Software shall remain with the Developer or its suppliers. The Software is protected by the copyright laws of the United States and international copyright treaties. Title, ownership rights, and intellectual property rights in and to the content accessed through the Software is the property of the applicable content owner and may be protected by applicable copyright or other law. This License gives you no rights to such content. DISCLAIMER OF WARRANTIES THERE ARE NO WARRANTIES, CLAIMS OR REPRESENTATIONS MADE BY DEVELOPER, EITHER EXPRESS, IMPLIED, OR STATUTORY, WITH RESPECT TO THE SOFTWARE, INCLUDING WARRANTIES OF QUALITY, PERFORMANCE, NON INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE, NOR ARE THERE ANY WARRANTIES CREATED BY COURSE OF DEALING, COURSE OF PERFORMANCE, OR TRADE USAGE. DEVELOPER DOES NOT WARRANT THAT THE SOFTWARE WILL MEET YOUR NEEDS OR BE FREE FROM ERRORS, OR THAT THE OPERATIONS OF THE SOFTWARE WILL BE UNINTERRUPTED. THE FOREGOING EXCLUSIONS AND DISCLAIMERS ARE AN ESSENTIAL PART OF THIS AGREEMENT AND FORMED THE BASIS FOR DETERMINING THE PRICE CHARGED FOR THE PRODUCTS. SOME STATES DO NOT ALLOW EXCLUSION OF AN IMPLIED WARRANTY, SO THIS DISCLAIMER MAY NOT APPLY TO YOU. EXCLUSIVE REMEDY Developer's cumulative liability to you or any other party for any loss or damages resulting from any claims, demands, or actions arising out of or relating to this Agreement shall not exceed the license fee paid to Developer for use of the Software. In no event shall Developer be liable for any indirect, incidental, consequential, special, or exemplary damages or lost profits, even if Developer has been advised of the possibility of such damages. EXCLUSION OF CONSEQUENTIAL AND OTHER DAMAGES

5 DEVELOPER WILL NOT BE LIABLE TO USER OR ANY THIRD-PARTY CLAIMANT FOR ANY INDIRECT, SPECIAL, PUNITIVE, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, WHETHER BASED ON A CLAIM OR ACTION OF CONTRACT, WARRANTY, NEGLIGENCE, STRICT LIABILITY, OR OTHER TORT, BREACH OF ANY STATUTORY DUTY, INDEMNITY OR CONTRIBUTION, OR OTHERWISE, EVEN IF DEVELOPER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE EXCLUSION CONTAINED IN THIS PARAGRAPH SHALL APPLY REGARDLESS OF THE FAILURE OF THE EXCLUSIVE REMEDY PROVIDED IN THE FOREGOING PARAGRAPH. SOME STATES DO NOT ALLOW THE LIMITATION OR EXCLUSION OF LIABILITY FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THE LIMITATIONS OR EXCLUSIONS IN THIS AND THE FOREGOING PARAGRAPH MAY NOT APPLY TO YOU. TERMINATION This license will terminate automatically if you fail to comply with the limitations described above. On termination, you must destroy all copies of the Software. MISCELLANEOUS This Agreement represents the complete agreement concerning this license between the parties and supersedes all prior agreements and representations between them. It may be amended only in writing executed by both parties. If any provision of this Agreement is held to be unenforceable for any reason, such provision shall be reformed only to the extent necessary to make it enforceable. This Agreement shall be governed by and construed under Oregon law as such law applies to agreements between Oregon residents entered into and to be performed within Oregon, except as governed by Federal law. U.S. Government Restricted Rights. Use, duplication or disclosure by the Government is subject to restrictions set forth in subparagraphs (a) through (d) of the Commercial Computer-Restricted Rights clause at FAR when applicable, or in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS , and in similar clauses in the NASA FAR Supplement. Manufacturer is WebTrends Corporation, 621 SW Morrison, Suite 1300, Portland OR 97205, U.S.A.

6

7 Table of Contents Chapter 1. Before You Begin 13 WebTrends Features 13 Understanding WebTrends Log Analysis 14 WebTrends Log Analyzer Features 14 How does the WebTrends Log Analyzer Work? 15 Understanding Log Files 16 Terminology and Conventions 18 Getting Help 19 Giving Us Feedback 19 Chapter 2. Installing and Setting Up WebTrends 21 System Requirements 21 Installing WebTrends 21 Installing with Auto-run 22 Installing from the Run Dialog 22 Software License Agreement 22 Installing Extended Logging Plug-Ins 22 Installing TNG Integration Utility 23 Registering WebTrends 23 Uninstalling WebTrends 24 WebTrends Main Console 24 Chapter 3. Web Log Analysis Profiles and Reports 27 Scenario: Web Site Activity Reporting 27 Adding a Web Log Analysis Profile 28 Creating a Log Analysis Report 33

8 Formatting Log File Path Entries 36 Standard Log File URL Path Examples 36 Compressed Logs 36 Wildcards 37 Multiple Logs 37 File Name Macros 37 WebTrends Views 39 Creating a WebTrends View 39 Chapter 4. Filtering Log Analyses 45 Using Web Log Analysis Filters 46 Adding a Filter 46 Combining Filter Elements 47 Combining Include and Exclude Filters in a Profile 48 Filter Elements 48 Return Codes 48 User Address 48 Multi-Homed Domain 50 Directory 50 File Filter 52 Referrer 53 Advertising 54 Browser 55 Username 56 Day 56 Hour 57 Scenario: Filtering a Multi-Homed Log File 57 Verifying your Log File is Multi-Homed 57 Adding Profiles for Multi-Homed Log 58 Defining and Memorizing Report Settings 59 Chapter 5. Modifying Reports 61 Scenario: Modifying WebTrends Report Style 61

9 Adding or Editing Report Styles 62 Scenario: Modifying WebTrends Report Language 64 Adding or Editing Report Languages 65 Scenario: Modifying WebTrends Report Content 66 Chapter 6. Scheduling Reports 69 Scenario: Automating Web Log Analysis Reports for Speed 70 Adding a Log Analysis Profile for Scheduled Reports 70 Memorizing Settings for Scheduled Reports 72 Scheduling Web Log Analysis Reports 73 Scenario: Automating Web Log Analysis Reports Conserving System Resources 75 Adding a Log Analysis Profile for Scheduled Reports 75 Memorizing Settings for Scheduled Reports 76 Scheduling Web Log Analysis Reports 77 Changing a Scheduled Report 78 Deleting a Scheduled Report 78 Using the Schedule Log 79 Searching for a Text String 79 Changing Font 80 Changing Point Size 80 Printing the Schedule Log 80 Saving the Schedule Log 80 Clearing the Schedule Log 81 Running a Scheduled Report On Demand 81 Chapter 7. FastTrends and Real Time Reporting 83 Scenario: Creating Several Reports from One Log File 84 Adding a Profile for Multi-homed Domain 84 Scenario: Generating Reports in Real Time 84 Defining Real-Time Reporting Options 85 Defining a Real-Time Analysis Profile 86 Clearing Your FastTrends Cache 87

10 Chapter 8. Ad Views and Clicks 89 Scenario: Tracking Internet Advertising 89 Defining an Advertisement to Track 90 Adding a Log Analysis Profile for an Ad 93 Customizing and Memorizing Settings for Ad Tracking Reports 95 Chapter 9. Intranet Management 97 About Intranet Management 97 Scenario: Configuring WebTrends for your Intranet 98 Adding an Intranet Domain 98 Intranet Reporting 100 Chapter 10. Notes for Netscape Web Server Users 101 Samples of Netscape Web Server Log Files 101 Setting up WebTrends Netscape Plug-In 102 Extended Logging and Multi-Homing 104 Chapter 11. Notes for Microsoft IIS Users 105 Samples of Microsoft IIS Web Server Log Files 105 Microsoft IIS Plug-in for Extended Logging 107 WebTrends Plug-In Configuration Utility 107 Tips for Microsoft IIS Users 111 Multi-Homed Filtering 111 Home Page URL 112 Chapter 12. Remote Reporting 113 Scenario: Using Your Browser to Create a WebTrends Report 113 Creating a Report Remotely 114 View an Existing Report Remotely 115 Delete an Existing Report Remotely: 116 Remote Reporting Configuration Options 116

11 Running a New Report 117 Working With Profiles 117 Modifying Configuration Options 117 Chapter 13. Configuration Options 119 General Options 119 Locale 120 Selecting an HTML Browser 120 Running WebTrends as an NT Service 121 Proxy Server Settings 123 Options 124 Remote Reporting Options 125 Scenario: Making WebTrends Reporting Available to Remote Users 126 Defining Windows NT Accounts for WebTrends Remote Reporting Server Access _ 126 Setting up a Remote Reporting Server 127 SNMP Agent 128 Web Log Analyzer Options 129 File Types 129 Modifying Domain Name Groupings 131 General Web Log Analyzer Settings 132 Microsoft Word and Excel Report Options 134 Cookies 135 Log File Cache Options 137 Real-Time 138 Language 139 Chapter 14. Performance 141 Using Filters to Optimize Performance 141 Running Reports without DNS Lookup 141 Running Reports without DNS Lookup 142 Cache the Log File 142 Accessing the log file/site: file, ftp, http 142

12 Report Type 142 DNS Cache Size 143 Retrieving HTML Page Titles 143 Zip Archived Log Files 143 Appendix A. Glossary 145 Appendix B. Sample WebTrends Report 153 Web Log Analysis Report Sample 153 Appendix C. Reference 163 Report/View Ranges 163 Examples: Defining Log Files in a Log Analysis Profile 167 Working with Profiles 167 Local File 168 Remote (FTP) 168 Remote (HTTP) 168 Multiple (Rotated) 168 Product Support 170 Index 171

13 Chapter 1. Before You Begin In This Chapter: WebTrends Features Understanding WebTrends Log Analysis Terminology and Conventions Getting Help Giving Us Feedback The WebTrends User Guide provides the information you need to install, set up, and use WebTrends. In addition to general reference information and step by step instructions, we have included scenarios that describe how you can make the most of WebTrends. WebTrends Features WebTrends Log Analyzer is member of the WebTrends family of tools for the web professional. With WebTrends, you have the most powerful and comprehensive web site management tool on the market! WebTrends provides the following cartridges, each enabling you to manage and report on an aspect of your web site. Report Style, Language and Content Editors Designing your own WebTrends HTML reports is a breeze! Now you can not only choose which graphs and tables to include in your reports, but also the colors, fonts, layouts and text to use. Even adding your company logo to all of your reports is easy. Table of Contents Frame in HTML Reports Navigating WebTrends HTML reports is now simplified: each report automatically generates a hyper-linked Table of Contents so that you can jump directly to the section that interests you. 13

14 WebTrends Log Analyzer Remote Reporting/Configuration If you are running WebTrends Log Analyzer as a Remote Reporting Server, remote users now have all of the log analyzer configuration options available to them. Enhanced Company Database The company database included with your copy of WebTrends now contains more than 1 million domains and their corresponding addresses and regional information. This information is displayed in WebTrends reports. Understanding WebTrends Log Analysis Once you have set up your web server and connected to the Internet, anyone with a browser can access it to retrieve information and files. With WebTrends Log Analysis, you can measure that activity on your web site including who is visiting your site, which pages are the most popular, whether your Internet advertising is paying off, and what time of day you should perform server maintenance. WebTrends Log Analyzer Features Supports all major and most minor log file formats. Processes access log, referrer logs, and agent logs. Allows remote retrieval of log files so that WebTrends does not have to be installed on your Web server. Extensive filtering options make it possible to zero in on exactly the information that you need. Versatile report formats (HTML, Microsoft Word & Excel, ASCII & comma-delimited text). Advertisement tracking. Log file Viewer to view your log file data in color-coded, condensed or detailed format. 14

15 Chapter 1. Before You Begin How does the WebTrends Log Analyzer Work? Your Web server generates a file, the log file, that documents all activity on your site, including, but not limited to: The IP address or domain name of the visitor to your site. The date and time of their visit. What pages they viewed. Any errors that they encountered. Any files downloaded and the sizes. Any URL that referred to your site. The Web browser and platform (the operating system) that was used. You tell WebTrends where to find the log file, how you want the log file analyzed, and any filters to apply using a log profile. WebTrends analyzes the log file according to the Web log profile and stores the data to speed future analyses of the same log file. WebTrends applies the report settings to the analysis results and displays them in the Web log analysis report. The following graphic shows how WebTrends generates Web log analysis reports: 15

16 WebTrends Log Analyzer Understanding Log Files Web server log files contain all activity on the web site: who visited, when they visited, and what they did when they were there. They normally have a.log extension, and are located: On an FTP site: /logs/*.log Locally (Microsoft IIS): c:\winnt\system32\logfiles\in*.log Locally (Netscape): c:\netscape\servers\http-[server name]\logs\*.log If you retrieve your web server log file remotely from your ISP, you need to ask them for the following information to define your Web Log Analysis profile: What is the exact FTP or HTTP path to your log files? What is the naming convention for your log files? What are the user name and password you need to access them? What is the rotation of your log files (i.e., daily, weekly, monthly, etc.)? Samples: Common and Combined (Extended) Log File Format The NCSA recognizes two log file format standards: Common and Combined (Extended). Following is a sample line from a "common log file: - [16/Jan/1996:10:36: ] "GET /dept/arguides/dwgs/ jpg HTTP/1.0" The combined or extended log file format is similar to the common log file format with the addition of the referrer and user fields at the end of the line (inside quotes). The following is a sample line from a "combined (extended) log file: tarpon.gulf.net - [12/Jan/1996:20:37: ] "GET /buttons/next_p.gif HTTP/1.0" " "Mozilla/2.0b5 (WinNT; I)" 16

17 Chapter 1. Before You Begin Note: Both log file formats have the first eight fields in common; the Combined (Extended) log file format includes in addition Field 9, Referrer and Field 10, Browser/Platform : The following table defines each of the fields in the sample log file lines. Field # Field Name Description Field 1 User Address The numeric IP address or domain name of the user accessing the site. Field 2 Rfc931 This field is used to log the domain for multi-homed web servers. Field 3 User Authentication User name, if required. Field 4 Date/Time The date and time the user accessed the site. Field 5 GMT offset The number of hours from GMT (if this is it is logged in GMT time). Field 6 Action The particular operation (i.e., Get or Post ) of the hit (this must be in quotes). Field 7 Return Code The return code for the action tells whether it was successful and why. Field 8 Size The size of the file sent. Field 9 Referrer: The URL which linked the user to your site. Field 10 Browser/ Platform The Web browser and platform used by the visitor to your site. Note: If you are attempting to report on a Microsoft IIS log file, and if WebTrends returns an error stating that there is no data for the date range specified, you may need to change your date format (i.e., Microsoft IIS U.S., European, or Japanese Date Format). You can change the format in Log File Format field of the Add or Edit Log Profile dialog (Title - URL tab). 17

18 WebTrends Log Analyzer Terminology and Conventions The following table explains some common terms used throughout the program and documentation. If you see Click Dialog Doubleclick Field Select Tab Window It means Use the mouse to position the selector arrow on an a screen item and press the left mouse button once. A dialog is a program box that appears providing spaces (fields) for you to enter information or buttons for you to select. Use the mouse to position the selector arrow on an a screen item and press the left mouse button twice quickly. This is where you enter your information in a dialog. A field typically looks like a box. Use the mouse to position the selector arrow over a screen item so that it is highlighted, and click the left mouse button once. A tab is a part of the program that looks like a separation card in a Rolodex or recipe file. Programs are displayed through windows. A window may provide access to several dialogs or tabs. The following table describes the conventions used to present parts of the WebTrends program. This text bold Refers to A button or field name (New button, Description field). Italic Refers to a chapter or section in the manual (Chapter 12, Adding a Profile). Title Case courier 8 point in quotes Refers to the names of dialogs, windows, and fields (Add Web Log Profile dialog, Home Page URL) Refers to how text appears on the screen or in a file. Refers to a value or character that should be entered in a dialog (Enter 12 in the hours field). 18

19 Chapter 1. Before You Begin Refer to Appendix A. Glossary on page 145 for additional definitions of terms used in WebTrends and in the documentation, as well as for general Internet terminology. Getting Help WebTrends provides comprehensive online help to guide you through dialogs and procedures. Look for the Help button in WebTrends dialogs. If you re not sure about what to enter in a particular field click the Help button. At any time, you can access help by selecting the Help menu from the WebTrends main console. Giving Us Feedback We welcome your feedback. If you have questions that aren t answered through this manual, please tell us so by selecting the Give Us Feedback option from the Feedback menu main console. 19

20

21 Chapter 2. Installing and Setting Up WebTrends In This Chapter: System Requirements Installing WebTrends Uninstalling WebTrends WebTrends Main Console System Requirements You can install and run WebTrends on any computer running Microsoft Windows NT 4.x or Windows 95. Because WebTrends can retrieve your Web server log files and analyze web sites remotely, it is not necessary to install this program on the server itself. WebTrends works with the following minimum system requirements: 20 MB available disk space 16 MB memory for Windows 95 systems, 32 MB memory for Windows NT systems Microsoft Windows NT 4.x or Windows 95 Note: More space and memory is required to analyze large log files. Contact your Product Support representative for more information. Installing WebTrends WebTrends is installed using a straightforward Windows setup program. Choose the procedure that best suits your needs: 21

22 WebTrends Log Analyzer Installing with Auto-run If your system is set up to automatically run CD-ROMs when you place them in the CD-ROM drive, complete the following steps: 1. Insert the WebTrends CD-ROM into your CD-ROM drive. The setup program is launched. 2. Follow the on-screen instructions to install WebTrends. Installing from the Run Dialog 1. Insert the WebTrends CD-ROM into your CD-ROM drive, and choose Run from the File or Start Menu. 2. In the Run dialog box, type d:\setup where d: refers to the drive letter of your CD-ROM drive. The setup program is launched, and walks you through the installation of WebTrends. Software License Agreement Before the WebTrends program files are copied to your system, the WebTrends Software License Agreement is presented. If you agree to the stated terms, click Accept. Otherwise, click Cancel to exit the setup program without installing WebTrends. Note: The WebTrends Software License Agreement appears at the beginning of this guide. Installing Extended Logging Plug-Ins WebTrends is shipped with plug-ins for Microsoft IIS (3.0 and 4.0) and Netscape Web Server which allow you to log your web site visitors browser and platform (operating system) information (also known as extended logging ). If the WebTrends Setup program detects either Microsoft IIS or Netscape Web server software on the system it is being installed on, you have the option to install the plug-in to enable extended logging. For more information about the extended logging plug-ins including plug-in configuration options, see Microsoft IIS Plug-in for Extended Logging on page 107 or Setting up WebTrends Netscape Plug-In on page

23 Chapter 2. Installing and Setting up WebTrends Note: This option is available only if you are installing WebTrends on a system where Netscape Server or Microsoft IIS is installed. Note: You can install the plug-in later or on a different machine by running IIS Extended Log Setup or Netscape Extended Log Setup (extendlog.exe) from the WebTrends program group. Installing TNG Integration Utility The WebTrends TNG integration utility for Computer Associates TNG Unicenter and Computer Associates TNG Framework lets you manage WebTrends and monitor its status through the SNMP Agent. You can also monitor the status of WebTrends and perform certain actions to a given change of state. Unicenter and Framework are management consoles for IS infrastructure, which includes everything from networks, the computers attached to them, the software loaded onto those computers, as well as extraneous devices. To install the TNG integration utility complete the following steps: 1. From the Start menu, select Programs, WebTrends, and you'll see an entry called TNG Setup. 2. Click OK when the Welcome dialog is displayed to start the installation. 3. Select the repository where all WebTrends objects are stored for the TNG database. 4. A dialog is displayed telling you the importance of activating the SNMP trap server within Unicenter. Click OK. 5. Click OK to complete the installation. When you run Unicenter or Framework, WebTrends shows up as an icon. Note: You must have TNG Unicenter or TNG Framework set up and running before installing this plug-in. Registering WebTrends When you launch WebTrends for the first time, you can register your copy of the software online. 23

24 WebTrends Log Analyzer Note: If you choose not to register WebTrends at this time, you can access the registration form later by selecting Online Registration from the Help menu within the program. If you click Yes in the Online Registration dialog, a dialog appears allowing you to register your purchase. Uninstalling WebTrends WebTrends includes a standard Windows uninstall program. To remove the software from your system: 1. Select the Add/Remove Programs utility from your Control Panel. 2. In the Install/Uninstall tab, highlight WebTrends in the list of existing programs and click Add/Remove. 3. The Uninstall program walks you through the removal of the WebTrends program from your system. WebTrends Main Console All program functions can be accessed from the File and Configure menus. The main program activities are represented in the toolbar just below the menu. All of the analysis and reporting that you do with WebTrends is based on profiles. These are listed in the main area of the WebTrends main console. The profile tabs allow you to choose between the display of all profiles or just the profiles for a cartridge. Tip: By default, the profile lists are sorted alphabetically by the descriptions: to sort by either of the other list criteria (i.e., URL Path or Profile Type), click on the title button at the head of the column. Tip: Right-click anywhere in the profile list area to access a pop-up menu which includes a View > command. This submenu gives you the option to view your profile lists as large icons, small icons, list or detail. The default is Detail View. 24

25 Chapter 2. Installing and Setting up WebTrends Your first step when you initially install WebTrends is to add a profile (click New) telling WebTrends where your web site, Web server log file or proxy server log file is located. Then, when you are ready to analyze and report on that site or log file, you highlight the profile in the list, and click Report. Click here to launch the WebTrends scheduler. Click here to configure WebTrends for your environment. Click here to add a new link, web log or proxy analysis profile. Click here to edit the highlighted profile. Click here to remove the highlighted profile. Click here to generate a WebTrends report using the highlighted profile. Click here to access the WebTrends report style editor. List of the existing link, web log and/or proxy analysis profiles. Displays the status of the next scheduled report or a realtime analysis in progress. Displays statistics for the highlighted profile. Click and drag the lower right corner of the main console to enlarge it and display all of the button bar options. Balloon help appears as you drag your cursor over the button toolbar on the main console: 25

26

27 Chapter 3. Web Log Analysis Profiles and Reports In This Chapter: Scenario: Web Site Activity Reporting Formatting Log File Path Entries WebTrends Views Now you re ready to get down to business and see what kind of traffic your web site has been getting. All web site activity is recorded in a log file maintained by the Web server, so all you need to do is tell WebTrends where to find your Web server log file, what kind of information you want to retrieve from it, and WebTrends does the rest! Note: For a more general overview of how WebTrends Web log analysis works, see Understanding WebTrends Log Analysis on page 14. Scenario: Web Site Activity Reporting For this scenario, you are the new head of the marketing department of SuperSales, Inc. Your company has had a virtually hosted web site up for the last year which until now been an untapped resource. Your plan is to use WebTrends to show who is visiting the site, where they are coming from, and which pages are the most popular: in short, how to make the most of your internet real estate. 27

28 WebTrends Log Analyzer The procedures that follow walk you through these important steps: Adding a Web Log Analysis Profile tells WebTrends where to find your web site s log file and how to process and filter it. Creating a Log Analysis Report explains the WebTrends reporting options. Adding a Web Log Analysis Profile Your first step is to tell WebTrends where to find the log file that contains the activity information for your web site. 1. From the WebTrends main console, click New. The Add Log Profile Title, URL dialog appears. 2. In the Description field, enter a description to identify your Web server log file (for example, SuperSales, Inc. Web Site ). This description appears in the profile list in the WebTrends main console, and as the subtitle of reports and views that are based on this profile. 3. In the Log File URL Path field, select the ftp:// option to retrieve your log file via FTP since your site is hosted remotely. In the future you may select one of the following: File Retrieval Method Select the file://option from the drop-down list if you have a drive mapped to your web server. Then enter the path to your log file. Click the Browse button to search for the exact location and file name. 28

29 Chapter 3. Web Log Analysis Profiles and Reports Note: To add multiple log files (rotated log files), enter a pipe character ( ) between the log file path names. Or use the wildcard characters (* or?) to stand for several log files. FTP Retrieval Method Select the ftp:// option from the drop-down list if you must retrieve your log file using ftp (file transfer protocol). For example, if your web site is hosted through an ISP, you may have to retrieve your log file through ftp. When you specify an ftp URL, you must enter the full path from the root of the ftp server, not just from the default login directory. To add multiple log files (rotated log files), enter a pipe character ( ) between the log file path names. Or use the wildcard characters (* or?) to stand for several log files. To enter your user name and password for the FTP site click Details. Within this login dialog, you can click Browse to search for the each FTP location. HTTP Retrieval Method Select the http option to retrieve your web server log file over the Internet using HTTP. You must enter the username and password information only if you are accessing a secured web site. ODBC Retrieval Method If your web server log file data is stored in an ODBC data source, WebTrends can retrieve the information directly from this source. Click Browse to search for the exact location of the data source. Note: You can use wildcards, date macros, multiple logs, and compressed logs in your log file entry. See Examples: Log File Entries later in this chapter for details. 5. Click Details to access the FTP Server Login Info. dialog. 5. Since you are using FTP, you must supply a user name and password for the specified computer. Enter a username and password for the FTP server. Click Browse to specify the exact location and name of the log file, and click OK to continue. 29

30 WebTrends Log Analyzer 6. In the Log File Format field, specify the log file format if necessary. By default WebTrends automatically detects the type of log file that you are adding, but you can specify one here. 7. When you have completed defining the location of the log file, click Next to display the DNS Lookup dialog. Reverse DNS lookups translates the numeric IP address into a domain name. This can be done either by the web server when logging activity or by WebTrends when it analyzes the log files. DNS Lookups are performed more efficiently by the web server as the log is created, rather than by WebTrends. However, if you know your web server administrator or ISP disabled DNS Lookups for their servers, you can have WebTrends do them for you. Since you have determined that your Web server is not performing the DNS lookup, and you would like your reports to display domain names and corresponding geographic information, you will select Resolve Mode. In the future you can choose one of the following: Quick mode, using format from log Maintains the log format, domain name or numeric IP address, making this the fastest method for creating reports. If your web server doesn t perform DNS Lookups and you choose the Quick Mode in this dialog, WebTrends reports will display visitor s IP addresses rather than domain names. Resolve mode, lookup all numeric IPS If your web server does not perform DNS Lookups and you need geographic or other domain-related information, select this option. Note: Resolve mode may significantly slow reporting. Use it only if DNS lookups cannot be done on the server. Once a numeric address has been looked up, its text equivalent is stored in a permanent cache to expedite all subsequent reports. The default cache size is 7,000 entries; once this limit is reached, the oldest addresses are replaced with the new entries. You can change the cache size in the Web Log Analyzer General Options tab. See page 132 for details. 30

31 Chapter 3. Web Log Analysis Profiles and Reports Auto mode, if you re unsure If you don t know whether your log file contains IP addresses or domain names, you can let WebTrends decide. If WebTrends finds a domain name when it begins analyzing the log file, it uses the quick mode. If WebTrends finds an IP address when it begins analyzing the log file, it tries to translate, or resolve, all IP addresses. Note: For Tips on how to determine if your Web server is performing the DNS Lookup, see Creating a WebTrends View on page 39 later in this chapter. 8. Click Next > to proceed to the Add Log Profile, Home Page dialog. Although you are not required to complete the following two fields, we strongly recommend that you do. These fields define the location of your web server home page regardless of any sub-domains on which you are reporting. Note: If you want WebTrends reports to identify Web pages by their <title>, you need to specify the home page directory. See Retrieve HTML Page Title on page In the Home Page File Names field, specify the file names that the web server defaults to when a visitor requests a URL without a specific file name. Separate multiple file names with spaces. 10. In the Web Site URL field, specify the location of your web site. In the first field, select the method of accessing your web site and then enter the path to the web site. For our example, select ftp:// from the list, and click Details to enter your login information. Then click Browse to specify the exact location and file name of the HTML home page for your web site. Click OK. In the future you can choose from the following: file:/// If you have file system access to your web server log file (on your machine or through a mapped drive), this is the quickest retrieval method. Specify the drive and directory to your web page. For example, c:\inetpub\wwwroot\ 31

32 WebTrends Log Analyzer ftp:// If you access your web pages through FTP, select this method and enter the complete path to your web site. If you access your web pages through HTTP, select this method and enter the path to your web site. If the web site is secure, click Login to enter username and password information. Note: If you use each time a web page title is retrieved, it is counted as a new hit. Both ftp:// and file:/// avoid recording these false hits. Note: If WebTrends reports N/A hits to your home page, try specifying the IP address in this field. This field must contain the data as it is recorded in your log file. 11. Click Next > to display the Filters dialog. The filters dialog allows you to specify the log file data that you want to analyze. In this example, you will keep the default Include Everything filter so that all of your log file data will be included. Click Next > to proceed. For a complete explanation of WebTrends filters, (for example, reporting on one domain housed on a server with many others) see Chapter 4. Filtering Log Analyses on page In the Database and Real-Time dialog, you can set up analysis caching and real-time analysis and reporting options for your log analysis profile. See Chapter 7. FastTrends and Real Time Reporting on page 83 for details about FastTrends caching and real-time analysis and reporting options. 13. Click Finish. You are returned to the WebTrends main console, and your new log analysis profile appears in the profile list. Note: Statistics about the log file corresponding to the highlighted profile appear at the bottom of the main console. These include the log file size and when it was first and last updated. For remote FTP and HTTP log profiles, these fields will display N/A. 32

33 Chapter 3. Web Log Analysis Profiles and Reports Tip: WebTrends profiles are text-based to facilitate changing multiple profiles at one time with any text editor. The extension for web log analysis profile is.wlp. Profiles are located in the datfiles subdirectory of the wtm_log directory. Creating a Log Analysis Report Now that you have told WebTrends where to find your Web server log file and how to process the data, you can run a report. At the time that you create a WebTrends Web log analysis report, you define the following parameters: The time or date range of the report The report format (i.e., HTML, MS Word, etc.) The destination (file location or recipient) of the resulting report The report style and language to use The report content (which graphs and tables to include) All of these settings can be memorized so that you can apply them to other Web log analysis profiles. To create a Web log analysis report: 1. From the WebTrends main console, click once on your Web Log analysis profile to select it. 2. Click Report to display the Create Report window. 33

34 WebTrends Log Analyzer 3. To use previously memorized settings for reporting or editing, select the name of the memorized report from the Memorized Report Name dropdown list (and skip to step number 10). Otherwise, proceed to the next step. If you do not choose a Memorized Report Name, select the time or date range for your report from the Report Range drop-down list. See Report/View Ranges on page 163 for explanation of the predefined Report Ranges. Note: While WebTrends attempts to analyze and report on the specified Report Range within the log file processed, you need to ensure that the specified range corresponds to existing dates in the log file. For example, if the analyzed log file contains data for only the last week of activity, WebTrends is not able to provide meaningful monthly averages. 4. Select the Format tab. Select the format in which to produce the report. Note: HTML reports are the quickest to create. Use Word formats to created printed executive reports with page numbering, headers and footers. Note: You must have Microsoft Office installed to use the Word or Excel formats, rather than the stand alone versions. 5. Select the Save As/Mail To tab to identify the location and file name in which to save your report and select one of the following options: 34

35 Chapter 3. Web Log Analysis Profiles and Reports file:/// This option saves reports locally. ftp:// This option uploads files to another system. mailto This option automatically s the files to one or several recipients. Then, enter the URL path in which to save the report that you are creating. Note: WebTrends provides macros that you can use to define file or directory names that represents a date. For more information, see File Name Macros on page 37. To the completed report, select the mailto: option from the drop-down list and specify the addresses of the recipients as well as the report file name. Note: See Options on page 124 for more information about enabling and configuring the WebTrends feature. 6. Select the Style tab. In the Report Title field, enter the title that will appear at the top of your report above the log profile description. 7. In the Report Language field, select a language from the list. See Adding or Editing Report Languages on page 65 for more information about the Language Editor. 8. In the Report Style field, select from a list the default or your own predefined report styles. See Scenario: Modifying WebTrends Report Style on page 61 for more information about the Style Editor. 9. Select the Content tab to specify the report items to include, the number of elements for each table and graph, and the style of each graph. See Scenario: Modifying WebTrends Report Content on page 66 for more information about the Content Editor. 35

36 WebTrends Log Analyzer 10. Click Memorize if you would like to save your report settings in order to apply them to any log profile for future reports. Otherwise, proceed to the next step. 11. Click Start to accept the options that you have defined and generate your report. Note: While WebTrends is in the process of generating the report, you can click Stop Here to create the report with the data collected so far, or click Cancel to cancel the report generation. If your report settings are not memorized, however, you lose all the settings. 12. Once the report data is collected, WebTrends launches your browser and loads the generated HTML report. If you are generating a report in Microsoft Word or Excel, WebTrends automatically launches the appropriate program. See examples of Web Log Analysis Report on page 153. Formatting Log File Path Entries The examples in this section show how you might use wildcards or date macros to specify log file path entries, and how to specify compressed logs or zipped log files. Standard Log File URL Path Examples c:\winnt\system32\logfiles\ex log Specifies the file location of the log file. ftp.domain.com/logs/ex log Specifies the FTP log file path. Specifies HTTP log file path. Compressed Logs c:\domain.com/logs/ex zip ftp.isp.com/log/ex gz 36

37 Chapter 3. Web Log Analysis Profiles and Reports Wildcards c:\winnt\system32\logfiles\ex*.log c:\winnt\system32\logfiles\ex9802*.log c:\winnt\system32\logfiles\ex9802??.log Multiple Logs You can specify multiple log file names by separating the entries with the pipe( ) character. d:\logfiles\logfile.log logfile2.log Specifies "logfile.log" and "logfile2.log" in the "logfile" directory on drive d:. \server\*.log \server2\*.log Specifies all of the.log files in the "server" and "server2" directories on the current drive. c:\server\logfile.log d:\logfiles\*.log Specifies "logfile.log" in the "server" directory on drive c: and all of the.log files in the "logfiles" directory on drive d:. File Name Macros WebTrends provides built-in macros to assist in automating report process. You can also use these macros in the Save As paths of the Scheduler and the Create Report dialog. %mm% Returns the current two-digit month value. %dd% Returns the current two-digit day value. %yy% Returns the current two-digit year value. 37

38 WebTrends Log Analyzer %YYYY% Returns the current four-digit year. %MON% Returns the current three-letter month Note: This variable is now case-sensitive. For example, %Mon% will read Jan, and %mon% will read jan. %DATE-x% Subtracts x value from the current date (e.g., %DATE-1% returns one day before current date) Note: This variable will affect the month or year values. If, for example, you use %DATE-10% on the 10 th day of the month, you will be specifying the last day of the previous month. Example 1: To configure WebTrends to run a report every day which never overwrites the previous day s report, enter the following in the Create Report Window (Save As/Mail To tab) Save As field: C:\WEBTREND\DATFILES\REPORTS\%mm%%dd%%yy%.htm WebTrends creates a report file each day with file names like HTM Example 2: If your Web server maintains a separate log file for each day named by the date, and you would like to define a log profile which incorporates only the previous day s web site activity, enter a line like the following in the Log File URL Path: field in the Add Log Profile dialog: C:\WEBSRVR\LOGFILES\%DATE-1%%mm%%dd%%yy%.log Example 3: If your Web server maintains a separate log file for each day named by the date, and you would like to specify the previous three days worth of log files, enter a line like the following in the Log File URL Path: field in the Add Log Profile dialog: C:\WEBSRVR\LOGFILES\%DATE-1%%mm%%dd%%yy%.log %DATE-1%%mm%%dd%%yy%.log %DATE-1%%mm%%dd%%yy%.log Note the pipe characters between each file name, and that each %DATE-1% refers to the date immediately preceding it in the list: you do not have to enter DATE-2 and DATE-3, etc. 38

39 Chapter 3. Web Log Analysis Profiles and Reports WebTrends Views With WebTrends, you can peruse filtered data from your Web server log file in a color-coded, organized, and easily identifiable format referred to as a view. You can view a specific time or date range, order by different criteria such as country or domain type, and in a detailed or standard format. As with WebTrends reports, views are based on log analysis profiles, and once you have established the parameters for a view, you have the option to memorize the settings to use with any Web log analysis profile. Views Tip: WebTrends views can be especially helpful for gathering data about your log file and troubleshooting information. For example, if the Home field appears, you know that yours is a multi-homed log file. See Scenario: Filtering a Multi-Homed Log File on page 57 for an example. Or if you turn the WebTrends DNS lookup option off and the IP addresses are still being resolved (domain names are appearing in the Address field) then you know that your Web server is performing reverse DNS lookups. Creating a WebTrends View Complete the following steps to view your log file and memorize the view settings: 1. From the WebTrends main console, select your log analysis profile. 2. Choose View Log from the File menu. 39

40 WebTrends Log Analyzer 3. To use previously memorized settings for this view, select the name of the memorized view from the Memorized Views drop-down list. Otherwise, proceed to the next item. 4. In the View Type field, choose whether to generate a Detailed Log Report or a Standard Log Report (examples follow). The default is Standard Log Report. 5. In the Order View by field, choose whether to order the report by Time, Domain Type, Country, Protocol, Event, Date, Result or File. The default is to order by Time. Note: If you choose to order by any criteria other than Time, WebTrends creates a separate view for each item. For example, if you are sorting by Country and you have events from three countries, you will have three views. 6. In the View Range field, choose the time or date range that you would like to view within the selected log file. You can specify a date and time range, or you can choose between the predefined parameters. The default is All of Log. Click to view the list of predefined range options and then click on the range you want to select. View Range options are defined in Report/View Ranges section on page 163. Note: While WebTrends attempts to analyze and report on the specified Report Range within the log file processed, you need to ensure that the specified range corresponds to existing dates in the log file. For example, if the analyzed log file contains data for only the last week of activity, WebTrends is not able to provide meaningful monthly averages. 7. Click Memorize if you would like to save your view parameters in order to apply them to any log profile for future views. Otherwise, proceed to the next step. 8. Click Start to accept the options that you have defined and generate your log file view. Note: While WebTrends is collecting data for your log file view, you can click Stop Here to view the log data collected so far, or click Cancel to cancel the view. If your view settings are not memorized, however, you will lose all the settings. 40

41 Chapter 3. Web Log Analysis Profiles and Reports The following graphics show the same view in detailed format and then in standard format. Detailed view example: Standard view example: 41

42 WebTrends Log Analyzer For definitions of view items terminology, refer to the log file field definitions in Understanding Log Files on page 16 or Appendix A. Glossary on page Click the down and up arrows to scroll through additional pages of the view. When you are done, click Close to exit the view window and return to the WebTrends main console. 10. To create additional views, repeat steps 2-9. View Text Color Coding View text is color coded to help you identify the different items in a report. This Color Blue Black Green Red Indicates View title and file names Headings Data Warnings or errors Note: If you choose to order by any criteria other than by Time, WebTrends creates a separate view for each item. Click to display a list of views, and click on the item you want to view. 42

43 Chapter 3. Web Log Analysis Profiles and Reports Click here to view a list of available views. Within the WebTrends log file viewer, you can search for a specific text string, change the font and point size, save or print the log file view. The following sections details these activities. Searching for a Text String Complete the following steps to search the log file view for a text string: 1. Click in the Find Next text entry box to enable it and then type the text string you want to search for. 2. To begin searching, click Find Next. WebTrends finds the first instance of the text string and highlights it. 3. To search for the next instance of the text string, click Find Next again. When you have completely searched the view for the text string, a dialog box appears asking if you want to begin searching from the beginning of the view. 4. Choose Yes to continue or No to quit. 43