LogLogic Blue Coat ProxySG Log Configuration Guide

Transcription

1 LogLogic Blue Coat ProxySG Log Configuration Guide Document Release: September 2011 Part Number: LL ELS This manual supports LogLogic Blue Coat ProxySG Release 1.0 and later, and LogLogic Software Release 5.1 and later until replaced by a new edition.

2 2011 LogLogic, Inc. Proprietary Information Trademarks This document contains proprietary and confidential information of LogLogic, Inc. and its licensors. In accordance with the license, this document may not be copied, disclosed, modified, transmitted, or translated except as permitted in writing by LogLogic, Inc. LogLogic and the LogLogic logo are trademarks or registered trademarks of LogLogic, Inc. in the United States and/or foreign countries. All other company or product names are trademarks or registered trademarks of their respective owners. Notice The information contained in this document is subject to change at any time without notice. All warranties with respect to the software and accompanying documentation are set our exclusively in the Software License Agreement or in the Product Purchase Agreement that covers the documentation. LogLogic, Inc. 110 Rose Orchard Way, Suite 200 San Jose, CA Tel: Fax: U.S. Toll Free:

3 Contents Preface About This Guide Technical Support Documentation Support Conventions Chapter 1 Configuring Blue Coat ProxySG and the LogLogic Appliance Introduction to Blue Coat ProxySG Prerequisites Configuring Blue Coat ProxySG Enabling Access Logging Setting Up Access Log Uploads Enabling the LogLogic Appliance to Capture Log Data Configuring the LogLogic Appliance for File Collection Adding a Blue Coat ProxySG Device Importing the LogLogic SSL Certificate into the Blue Coat ProxySG Device Creating File Transfer Rules Verifying the Configuration Chapter 2 How LogLogic Supports Blue Coat ProxySG How LogLogic Captures Blue Coat ProxySG Log Data Supported Blue Coat ProxySG Log Data Formats LogLogic Real-Time Reports Chapter 3 Troubleshooting Verifying that Log Messages are Uploaded to the LogLogic Appliance Appendix A Blue Coat ProxySG Best Practices Blue Coat ProxySG Best Practices Optimized and Non-Optimized Report Fields Blue Coat Best Practices Reporting Package Appendix B Format Reference LogLogic Support for Blue Coat ProxySG Blue Coat ProxySG Log Configuration Guide 3

4 4 Blue Coat ProxySG Log Configuration Guide

5 Preface About This Guide The LogLogic Appliance-based solution lets you capture and manage log data from all types of log sources in your enterprise. The LogLogic support for Blue Coat ProxySG enables LogLogic Appliances to capture audit logs from machines running Blue Coat ProxySG. Once the logs are captured and parsed, you can generate reports and create alerts on Blue Coat ProxySG s operations. For more information on creating reports and alerts, see the LogLogic Users Guide and LogLogic Online Help. Technical Support LogLogic is committed to the success of our customers and to ensuring our products improve customers' ability to maintain secure, reliable networks. Although LogLogic products are easy to use and maintain, occasional assistance might be necessary. LogLogic provides timely and comprehensive customer support and technical assistance from highly knowledgeable, experienced engineers who can help you maximize the performance of your LogLogic Appliances. To reach LogLogic Customer Support: Telephone: Toll Free LOGS Local EMEA or APAC: + 44 (0) or +44 (0) support@loglogic.com You can also visit the LogLogic Support website at: When contacting Customer Support, be prepared to provide: Your name, address, phone number, and fax number Your company name and company address Your machine type and release version A description of the problem and the content of pertinent error messages (if any) Documentation Support Your feedback on LogLogic documentation is important to us. Send to DocComments@loglogic.com if you have questions or comments. Your comments will be reviewed and addressed by the LogLogic technical writing team. In your message, please indicate the software name and version you are using, as well as the title and document date of your documentation. Blue Coat ProxySG Log Configuration Guide 5

6 Conventions LogLogic documentation uses the following conventions to highlight code and command-line elements: A monospace font is used for programming elements (such as code fragments, objects, methods, parameters, and HTML tags) and system elements (such as filenames, directories, paths, and URLs). A monospace bold font is used to distinguish system prompts or screen output from user responses, as in this example: username: system home directory: home\app A monospace italic font is used for placeholders, which are general names that you replace with names specific to your site, as in this example: LogLogic_home_directory\upgrade\ Straight brackets signal options in command-line syntax. For example: ls [-AabCcdFfgiLlmnopqRrstux1] [-X attr] [path...] 6 Blue Coat ProxySG Log Configuration Guide

7 Chapter 1 Configuring Blue Coat ProxySG and the LogLogic Appliance This chapter describes the tasks you must perform to properly configure the LogLogic Appliance and Blue Coat ProxySG device to log web usage information such as HTTP traffic and Instant Messenger traffic. In addition, this chapter covers how to configure your systems to work with or without SSL authentication enabled. Introduction to Blue Coat ProxySG Prerequisites Configuring Blue Coat ProxySG Enabling the LogLogic Appliance to Capture Log Data Verifying the Configuration Introduction to Blue Coat ProxySG The Blue Coat ProxySG device is designed to integrate protection and control functions for Internet and intranet traffic without sacrificing performance and employee productivity. The device assists you in managing internet abuse by increasing security, limiting liability, and managing bandwidth usage. The LogLogic Appliance enables you to capture log data and report on critical points of your Blue Coat ProxySG internet access control solution including web cache usage. LogLogic provides an additional level of support by enabling you to generate reports and run searches on data to improve your ability to manage your Blue Coat ProxySG activity. Blue Coat ProxySG log data can be transferred to the LogLogic Appliance using one of the following methods: Continuous Uploads Data is uploaded directly to the Appliance using HTTP or HTTPS Periodic Uploads Data is uploaded to a Host Server and then transferred to the Appliance using FTP(S), SFTP, or SCP The configuration procedures for Blue Coat ProxySG and the LogLogic Appliance depend upon the method you select for your environment. For more information, see How LogLogic Captures Blue Coat ProxySG Log Data on page 27 and Defining the Upload Schedule on page 10. Blue Coat ProxySG Log Configuration Guide 7

8 Prerequisites Prior to configuring the Blue Coat ProxySG and LogLogic Appliance, ensure that you meet the following prerequisites: Blue Coat ProxySG installed Proper access permissions to make configuration changes LogLogic Appliance running Release 5.1 or later installed Administrative access on the LogLogic Appliance 3rd-party FTP, FTP(S), HTTP(S), CIFS, SCP, and/or SFTP server software installed for any platform that does not have these capabilities by default. For more information, see Configuring the LogLogic Appliance for File Collection on page 16. Note: If you are using a periodic upload schedule for Blue Coat ProxySG, you must configure a Host Server with the proper 3rd-party file transfer software. For more information, see Defining the Upload Schedule on page 10. Configuring Blue Coat ProxySG This section describes the configuration steps setting up general access logging on Blue Coat ProxySG. Enabling Access Logging Access logging enables you to monitor web traffic for your environment. The Blue Coat ProxySG device can be set up to generate real-time or schedule logs and reports. Once you set up the Blue Coat ProxySG device for access logging, you must enable it to send the logs to the LogLogic Appliance or a remote Host Server (e.g., SFTP Server). Note: The following steps walk you through the basic procedure for enabling access logging. For more information on setting up the Blue Coat ProxySG for access logging, see the Blue Coat ProxySG Product Documentation. 8 Blue Coat ProxySG Log Configuration Guide

9 To enable access logging: 1. In the Blue Coat Management Console navigation menu, select Access Logging > General. The Default Logging tab appears. This window might appear differently depending on the version of Blue Coat ProxySG you are running. Figure 1 Blue Coat ProxySG Access Logging > General > Default Logging Tab 2. Select the Enable Access Logging checkbox. 3. For each of the following protocols, click Edit, edit the entry as noted, then click OK. HTTP/HTTPS main FTP main SOCKS main TCP-Tunnel main ICP main Instant Messaging IM Windows Media streaming Real Media/QuickTime streaming Blue Coat ProxySG Log Configuration Guide 9

10 Setting Up Access Log Uploads After you enable access logging on Blue Coat ProxySG, you must define an upload schedule. The upload schedule dictates how and when logs are transferred to the LogLogic Appliance or a remote Host Server (e.g., SFTP Server). Defining the Upload Schedule The Blue Coat ProxySG device provides you the ability to upload messages continuously (near real-time) at schedule intervals (using a batch process). Messages sent continuously are available near real-time because of the processing time and network throughput between the Blue Coat ProxySG device and the LogLogic Appliance. If you intend to send messages continuously, then you must send the logs via HTTP or HTTPS using the HTTP Client. For more information, see Setting Up Logging for the HTTP Client on page 12. Messages sent periodically, such as hourly or once a day, are first batched, then saved to disk, and then uploaded as scheduled. If you intend to send messages periodically, then you can must send the logs via a 3rd-party FTP(S), SFTP, or SCP remote Host Server. Caution: If you choose to use periodic uploads, then you must ensure that the time between schedule uploads is greater than the amount of time it takes to upload the batch file containing multiple log entries. The batch file may become large over time, so uploading this file may not complete before the next scheduled upload. To define the upload schedule: 1. In the Blue Coat Management Console navigation menu, go to Access Logging > Logs > Upload Schedule tab. 10 Blue Coat ProxySG Log Configuration Guide

11 Figure 2 Blue Coat ProxySG Access Logging - Upload Schedule tab. 2. In the Log drop-down menu, select main. To enable continuous logging, go to Step 3. To enable periodic logging, go to Step Complete the following steps to modify the upload schedule settings for continuous logging: a. In the Upload type area, in the Upload access log section, select the continuously radio button. b. Set the timing for Wait between connection attempts and Time between keep-alive packets text fields. c. In the Rotate the log file section, set the scheduled time to Every 1 hour. LogLogic recommends that you specify the log rollover time to be greater than or equal to 1 hour so that the logs can be parsed and can be viewed in real-time reports. IMPORTANT! If you accumulated large legacy access log files, you must specify a large value in the Rotate the log file section to transfer files to your LogLogic Appliance. If the value selected is too small, then the access log feed is restarted before the transfer of the legacy data to the LogLogic Appliance is complete. d. Click Apply to save the changes. Blue Coat ProxySG Log Configuration Guide 11

12 2. Complete the following steps to modify the upload schedule settings for periodic logging: a. In the Upload type section, select the periodically radio button. Caution: If you select periodically, you do not see any logs from the Blue Coat ProxySG device on the Host Server (i.e., SFTP Server) until the time set in the Rotate the log file section. b. Set the timing for the Wait between connection attempts text field. c. In the Rotate the log file area, set the scheduled time to upload the log file. d. Click Apply to save the changes. The Blue Coat ProxySG device establishes a new connection when data is available. This connection is maintained for as long as the Blue Coat ProxySG device is sending data. The Blue Coat ProxySG access log data is parsed as soon as it is received by the LogLogic Appliance. Setting Up Logging for the HTTP Client Using HTTP Client for access log uploads enables the LogLogic Appliance to receive Blue Coat ProxySG messages near real-time (i.e., continuously). You can create an HTTP or an HTTPS upload client through the HTTP Client dialog. Note: If you are using a periodic upload schedule, then you do not need to configure the HTTP Client. To set up access logging for HTTP Client: 1. From the Blue Coat Management Console, click Access Logging > Logs. The Logs tab appears. 2. Click the Upload Client tab. 12 Blue Coat ProxySG Log Configuration Guide

13 Figure 3 Blue Coat ProxySG Access Logging - Upload Client Tab. 3. In the Log drop-down menu, select main. 4. In the Client Type drop-down menu, select HTTP Client. 5. Complete the following steps to configure the HTTP Client settings: a. In the Upload Client area, click Settings. The HTTP Client settings: Log main window appears. Blue Coat ProxySG Log Configuration Guide 13

14 Figure 4 Blue Coat ProxySG HTTP Client settings: Log main window b. In the HTTP Client settings: Log main window, set the following for HTTP server connection information: Settings for Select Primary HTTP Server from the drop-down menu Host IP address of the LogLogic Appliance. Note: If you select the Use secure connections (SSL) checkbox, the hostname must match the hostname in the certificate presented by the server. Port Use port number 4433 Path Directory path where the access log facility is uploaded to the server Username User name to log into the LogLogic Appliance Change Password Password to log into the LogLogic Appliance Filename Leave the default entry c. For SSL only, select the Use secure connections (SSL) checkbox. If you are not using SSL, proceed to Step 6. You must associate an SSL CA Certificate from the Host machine (i.e., LogLogic Appliance). For more information, see Importing the LogLogic SSL Certificate into the Blue Coat ProxySG Device on page 21. If you are not using SSL, then do not select this checkbox. 14 Blue Coat ProxySG Log Configuration Guide

15 6. Click OK. 7. In the Upload Client tab, click Apply. Note: If you run Test Upload, you might receive a failure message. However, this does not mean that you have configured the Blue Coat ProxySG device and LogLogic Appliance incorrectly. Verifying Access Log Formats After setting up the Blue Coat ProxySG device to upload messages to the LogLogic Appliance or a Host Server, verify the access log formats. To verify access log formats: 1. From the Blue Coat Management Console navigation menu, go to Access Logging > Formats. The Formats page appears. Figure 5 Blue Coat Access Logging - Formats Page 2. Verify that the Log Formats in the listing are correct and make changes as necessary. For more information on log formats, see the Blue Coat ProxySG product documentation. Blue Coat ProxySG Log Configuration Guide 15

16 Enabling the LogLogic Appliance to Capture Log Data The following sections describe how to enable the LogLogic Appliance to capture Blue Coat ProxySG log data. Configuring the LogLogic Appliance for File Collection The LogLogic Appliance captures Blue Coat ProxySG logs using file pull functionality via a file transfer rule. The deployment method you use to collect Blue Coat ProxySG file-based data depends on the upload schedule (i.e., periodic or continuous) you selected during the configuration procedure. For more information, see Setting Up Access Log Uploads on page 10. Blue Coat ProxySG File Collection Using a Periodic Upload Schedule If you defined a periodic upload schedule (see Defining the Upload Schedule on page 10), then you need to use the following deployment method for file collection: 1. Configure a remote Host Server with file transfer capability to capture log files from the Blue Coat ProxySG host machine. The following procedure explains, at a high-level, how to configure your environment to capture file-based log messages via SFTP. LogLogic recommends using SFTP for Windows-based systems, or SCP for Unix-based systems, to securely transfer files to the LogLogic Appliance from your log source. However, you can use any of the LogLogic-supported protocols in your environment (i.e., FTP(S), HTTP(S), SCP, etc.). Note: For more information on each supported protocol, including whether a Public Key Copy is needed and what search methods (i.e., CSV, Wildcard) are available, see the LogLogic Administration Guide. a. Make sure that a destination directory (i.e., log directory) exists and is accessible on the host machine where Blue Coat ProxySG is installed. The destination directory should contain the original log files that Blue Coat ProxySG generates. b. Transfer the Blue Coat ProxySG log files to a separate publishing directory on a remote Host Server. You can use a script or 3rd-party software that makes a copy of or moves the log files from the destination directory (i.e., log directory) to the publishing directory. In addition, if you are using a script, you can specify the schedule for when the script runs (e.g., hourly, daily, or weekly). Note: LogLogic recommends that you define a clean-up process to handle old log files that accumulate over time. 2. On the LogLogic Appliance, add Blue Coat ProxySG to the Appliance as a new device. For more information, see Adding a Blue Coat ProxySG Device on page Create a file transfer rule and specify SFTP as the Protocol. For more information, see Creating File Transfer Rules on page 24. IMPORTANT! SCP and SFTP have limitations in their ability to pull a large number of files (100 or more). LogLogic recommends that you compress the files into a single file (such as.tar or tar.gz) before the files are pulled by the LogLogic Appliance. 16 Blue Coat ProxySG Log Configuration Guide

17 Blue Coat ProxySG File Collection Using a Continuous Upload Schedule If you defined a continuous upload schedule (see Defining the Upload Schedule on page 10), then you need to use the following deployment method for file collection: 1. Properly configure the HTTP Client on Blue Coat ProxySG (see Setting Up Logging for the HTTP Client on page 12). 2. On the LogLogic Appliance, add Blue Coat ProxySG to the Appliance as a new device. For more information, see Adding a Blue Coat ProxySG Device on page Create a file transfer rule and specify HTTP or HTTPS as the Protocol. For more information, see Creating File Transfer Rules on page 24. Adding a Blue Coat ProxySG Device LogLogic captures Blue Coat ProxySG log files using the file pull method. You must add the server as a new device so LogLogic can properly handle the log file data to make it available through reports and searching. Once you have successfully added an Blue Coat ProxySG device, you must configure file transfer rules for file collection. For more information, see Configuring the LogLogic Appliance for File Collection on page 16. To add Blue Coat ProxySG as a new device: 1. Log in to the LogLogic Appliance. 2. From the navigation menu, click Management > Devices. The Devices tab appears. Blue Coat ProxySG Log Configuration Guide 17

18 Figure 6 Management > Devices > Devices Tab 3. On the Devices tab, click Add New. The Add Device tab appears. 18 Blue Coat ProxySG Log Configuration Guide

19 Figure 7 Add Device Tab 4. On the Add Device tab, complete the following information: Name Name for the Blue Coat ProxySG device. For example, BlueCoat_LX102, where LX102 is the LogLogic Appliance platform and last three digits of the IP address for the appliance. Description (optional) Description of the Blue Coat ProxySG device Device Type Select Blue Coat ProxySG from the drop-down menu Host IP IP address of the machine hosting the Blue Coat ProxySG log data (this can be a remote Host Server or a machine with Blue Coat ProxySG installed) Enable Data Collection Select the Yes radio button Refresh Device Name through DNS Lookups (optional) Select this checkbox to enable the Name field to be automatically updated. The name is obtained using a reverse DNS lookup on the configured refresh interval. The DNS name overrides any manual name you assign. Use User Authentication (optional) Select this checkbox to enable user name and password authentication for the Blue Coat ProxySG device. Use SSL (optional) If you are using an SSL Certificate to enable encryption of transferred files, then proceed with Step 5. If you are not using SSL Certificates, then go to Step 6. Blue Coat ProxySG Log Configuration Guide 19

20 Figure 8 Add Device tab populated with Example Content 1. (Optional) Complete this step only if you are using SSL to encrypt transferred files. You must enable SSL in the LogLogic Appliance, and then export the auto-generated SSL Certificate to the Blue Coat ProxySG device. a. Select the Use SSL checkbox. b. Click anywhere in the SSL Certificate text box. c. Press the Ctrl + A keys to select all text for the certificate, and then press the Ctrl + C keys to copy the text to the clipboard. 2. Verify that all fields are populated correctly, and then click Add to add the device. The Devices tab appears. 3. Verify that the device is in the list of devices. Click the device name to check the values you entered for the device. 4. If you are using SSL, then go to Importing the LogLogic SSL Certificate into the Blue Coat ProxySG Device on page 21 and complete the steps to import the LogLogic SSL Certificate into the Blue Coat ProxySG device. 20 Blue Coat ProxySG Log Configuration Guide

21 Importing the LogLogic SSL Certificate into the Blue Coat ProxySG Device This section only applies if you are using SSL certification to encrypt transferred files between the Blue Coat ProxySG device and the LogLogic Appliance. If you are using SSL, then you must import the SSL Certificate from the LogLogic Appliance into Blue Coat ProxySG. To import the LogLogic Appliance SSL Certificate into the Blue Coat ProxySG device: 1. Log in to the Blue Coat ProxySG web proxy appliance. a. Open a web browser using https and port For example, b. Enter the user name and password. The Blue Coat ProxySG web interface appears. Figure 9 Blue Coat ProxySG Web Interface 2. Click the Management Console link. The Blue Coat Management Console appears. Blue Coat ProxySG Log Configuration Guide 21

22 Figure 10 Blue Coat Management Console 3. From the Blue Coat Management Console, click SSL > CA Certificates. The CA Certificates tab appears. 22 Blue Coat ProxySG Log Configuration Guide

23 Figure 11 Blue Coat ProxySG SSL - CA Certificate Tab 4. In the CA Certificates tab click Import. The Import CA Certificate window appears. 5. Click in the CA Certificate text box, and then press and hold the Ctrl + V keys to paste in the LogLogic SSL Certificate. If you do not have the LogLogic SSL Certificate on your clipboard, return to the LogLogic Appliance and navigate to the Blue Coat ProxySG device details page. In the LogLogic interface, click Administration > Manage Devices, then click the Blue Coat ProxySG device name in the list. Use the Ctrl + A and Ctrl + C keys to select and copy all of the SSL Certificate text. Blue Coat ProxySG Log Configuration Guide 23

24 Figure 12 Example Import CA Certificate Entries. 6. Click OK to import the LogLogic SSL Certificate into the Blue Coat ProxySG device. Note: A dialog box may prompt you for the user name and password of the Blue Coat ProxySG device. 7. Verify that the LogLogic SSL Certificate appears in the list on the CA Certificates tab. 8. Click Apply to save changes. Creating File Transfer Rules After you have added your Blue Coat ProxySG device, you can create a File Transfer Rule for the access log files. File Transfer Rules enable the LogLogic Appliance to pull files from the Host Server publishing the Blue Coat ProxySG log files. LogLogic supports the following wildcards: * (asterisk),? (question mark), and [...] (open and close brackets) using directory queries. If you use wildcards, you must enable directory listing on your Host Server. Examples: file /foo/file, /bar/*.log /foo?/bar*/*.aud, /foo1/file1.tar.gz, /foo1/file2.z /foo[2-8]/bar*/net*.log LogLogic can pull and decompress archive files, extract individual files from the archive files, and then process the individual files. The following file types are supported:.tar.bz2,.tar.gz, tar.z,.tgz,.taz,.tar,.gz,.z,.z,.zip,.zip. For more information, see the LogLogic Administration Guide. 24 Blue Coat ProxySG Log Configuration Guide

25 To create a file transfer rule: 1. Log in to the LogLogic Appliance. 2. From the navigation menu, select Management > Devices. 3. Select the File Transfer Rule tab. 4. Add a rule for the Blue Coat ProxySG log files you want to capture by completing the following steps: a. From the Device Type drop-down menu, select the machine where Blue Coat ProxySG is installed. b. From the Device drop-down menu, select the appropriate Blue Coat ProxySG device. Note: If you have added only one Blue Coat ProxySG device, the device name is automatically added. c. Click Add New then enter the appropriate information for the following required fields: Rule Name Name of the transfer rule (e.g., Blue Coat ProxySG log files) Protocol Specify the appropriate protocol (e.g., HTTP(S) if you are using a continuous upload schedule from ProxySG, SFTP, FTP(S), SCP, etc. if you are using a periodic upload schedule) Note: LogLogic recommends using a secure file transfer protocol, such as SFTP for Windows-based devices or SCP for UNIX-based devices. If you are using SFTP or SCP, you must copy the Appliance s public key to the machine where the logs are located. For more information, see Configuring the LogLogic Appliance for File Collection on page 16 and the LogLogic Administration Guide. User ID Specify only if the protocol requires a User ID Password/Verify Password Specify only if required for the User ID Files Full path (after the IP address) to the Blue Coat ProxySG machine or the Host Server where the Blue Coat ProxySG log files are located. For example: /log/file_name.log To capture all logs in a specific directory specify the asterisk (*) wildcard. For example: /log/*.zip The server can be the host machine where the device is installed or a remote Host Server with file transfer functionality. For more information, see Configuring the LogLogic Appliance for File Collection on page 16. File Format Select W3C from the drop-down menu Collection Time Specify the time you want to retrieve the log file from the Host Server (i.e., a Host Server or the Blue Coat ProxySG machine) File Transfer History Click this button if you want to see the file transfer history. Use Advanced Duplication Detection Select the Yes radio button if you want the LogLogic Appliance to check for duplicate data while capturing the Blue Coat ProxySG logs. Enable Select the Yes radio button to enable the File Transfer Rule d. Click Add. Blue Coat ProxySG Log Configuration Guide 25

26 Verifying the Configuration The section describes how to verify that the configuration changes made to Blue Coat ProxySG and the LogLogic Appliance are applied correctly. To verify the configuration: 1. Log in to the LogLogic Appliance. 2. From the navigation menu, select Dashboards > Log Source Status. The Log Source Status tab appears. 3. Locate the IP address for each Blue Coat ProxySG device. If the device name (Blue Coat ProxySG) appears in the list of devices, then the configuration is correct. If the device does not appear in the Log Source Status tab, check the Blue Coat ProxySG logs for events that should have been sent. If events were detected and are still not appearing on the LogLogic Appliance, verify the Blue Coat ProxySG configuration and the LogLogic Appliance configuration. You can also verify that the LogLogic Appliance is properly capturing log data from Blue Coat ProxySG by trying to view the data in the reports. LogLogic recommends checking the reports to make sure that the data obtained is valid and matches expectations. For more information, see LogLogic Real-Time Reports on page 30. If the device name appears in the list of devices but event data for the device is not appearing within your reports, you need to verify that the LogLogic Appliance is actually capturing the logs properly. For more information, see Verifying that Log Messages are Uploaded to the LogLogic Appliance on page Blue Coat ProxySG Log Configuration Guide

27 Chapter 2 How LogLogic Supports Blue Coat ProxySG This chapter describes LogLogic s support for Blue Coat ProxySG. LogLogic enables you to track web usage information captured by the Blue Coat ProxySG device in real-time or on a scheduled basis. How LogLogic Captures Blue Coat ProxySG Log Data Supported Blue Coat ProxySG Log Data Formats LogLogic Real-Time Reports How LogLogic Captures Blue Coat ProxySG Log Data Access logs are made available in real-time (i.e., continuously) or at scheduled intervals (i.e., periodically) and contain log records that can be made specific to each protocol. For example, you can track the network traffic flowing through your environment and append an access log at the end of each transaction for a specified protocol such as HTTP or HTTPS. Access logs are directed to specified log facilities. Log facilities associate logs with a configured format and upload schedule. Log facilities can also be encrypted and digitally signed before they are uploaded. All log data stored in log facilities can be uploaded to a LogLogic Appliance for analysis and future archiving. Blue Coat ProxySG log data can be uploaded directly to the LogLogic Appliance using HTTP or HTTPS if you are uploading files continuously. Log data can also be uploaded to a remote Host Server and then transferred to the Appliance using FTP(S), SFTP, or SCP if you are uploading files periodically. For more information, see Defining the Upload Schedule on page 10. Figure 13 on page 28 provides a deployment example for capturing Blue Coat ProxySG access log messages continuously from the HTTP Client. Figure 14 on page 28 provides a deployment example for capturing Blue Coat ProxySG access log messages periodically from a remote Host Server. An SFTP server is used as a remote Host Server in the periodic upload example. For more information, see Configuring the LogLogic Appliance for File Collection on page 16. Blue Coat ProxySG Log Configuration Guide 27

28 Figure 13 Continuous File Upload Functionality (using HTTP(S)) and Connectivity for Blue Coat ProxySG and a LogLogic Appliance End User Intranet Internet Blue Coat ProxySG HTTP/S Upload (HTTP Client) Internet Website End User log data LogLogic Appliance Reports Database Alerts Figure 14 Periodic File Upload Functionality (using SFTP) and Connectivity for Blue Coat ProxySG and a LogLogic Appliance End User Intranet Internet Blue Coat ProxySG Internet Website End User Host Server LogLogic Appliance SFTP Upload log data Database Reports Alerts All Blue Coat ProxySG log data captured by the LogLogic Appliance is parsed and made available to the LogLogic Agile Report Engine and search. The Agile Report Engine provides report templates that can be run as-is or modified to create customized reports targeting specific information. For example, you can create customized reports that display the top cache server and status for a given time frame, the most requested file types for a given time frame, or the top URLs requested for a given time frame. For more information, see LogLogic Real-Time Reports on page 30. In addition, the data is made available so you can create alerts that notify you of issues on your Blue Coat ProxySG device. For more information on creating reports and alerts, see the LogLogic User Guide and LogLogic Online Help. Note: Blue Coat ProxySG records log data using GMT time while the LogLogic Appliance reports data using local time. In LogLogic releases prior to Hotfix 2, you must consider the time difference of when log data was captured by the Blue Coat ProxySG device and the current time of the LogLogic Appliance. If you have LogLogic Hotfix 2 or later, then you do not have to calculate the time difference. 28 Blue Coat ProxySG Log Configuration Guide

29 Supported Blue Coat ProxySG Log Data Formats LogLogic supports access logs in various formats such as ELFF, NCSA/Common, SmartReporter, SQUID, SurfControl, and Websense. Each log format applies to a specific type of log data. For example, ELFF (W3C Extended Log File Format) is a general format with specific implementations such as IM, Main, P2P, and streaming, NCSA/Common contains only basic HTTP access information, and SQUID contains cache statistics. Table 1 provides a list of the Blue Coat ProxySG fields that LogLogic parses. Table 1 Blue Coat ProxySG Fields Parsed by LogLogic bytes c-dns c-ip cached cs(content-type) cs(referer) cs(user-agent) cs(x-forwarded-for) cs-auth-group cs-auth-groups cs-method cs-types cs-uri cs-uri-query cs-uri-stem current-time date r-dns r-ip r-port rs(content-type) s-dns s-ip s-port sc-bytes sc-status time time-taken x-action x-connect-time x-hiercode x-play-time x-product x-protocol x-sc-contentlength x-smartfilter-categories x-smartfilter-result x-status x-timestamp x-transaction x-username x-virus-details x-virus-id Blue Coat ProxySG Log Configuration Guide 29

30 LogLogic Real-Time Reports LogLogic provides pre-configured Real-Time Reports for Blue Coat ProxySG log data. The following Real-Time Reports are available: All Unparsed Events Displays data for all events retrieved from the Blue Coat ProxySG log for a specified time interval Web Cache Activity - Displays locally-stored web cache information served during a specified time interval To access LMI 5 Real-Time Reports: 1. In the top navigation pane, click Reports. 2. Click Network Activity. The following Real-Time Reports are available: Web Cache Activity 3. Click Operational. The following Real-Time Reports are available: All Unparsed Events You can create custom reports from the existing Real-Time Report templates. For more information, see the LogLogic User Guide and LogLogic Online Help. 30 Blue Coat ProxySG Log Configuration Guide

31 Chapter 3 Troubleshooting This chapter contains troubleshooting information regarding the configuration and/or use of log collection for Blue Coat ProxySG. Verifying that Log Messages are Uploaded to the LogLogic Appliance Verifying that Log Messages are Uploaded to the LogLogic Appliance If you set messages to be uploaded continuously, then you can view the results of your tests immediately. If you set messages to be uploaded periodically, then the messages are sent at the schedule time. However, you can force uploads if necessary. It is possible to test that messages are being uploaded from the Blue Coat ProxySG device to the LogLogic Appliance in many ways. The following steps represent one example of how you can verify that you have set up your environment properly. Note: The Blue Coat ProxySG upload client uses GMT to when uploading data. In LogLogic versions prior to Hotfix 2, on the LogLogic Appliance, you must calculate the GMT time equivalent to your local time when searching and reporting on data. For example, if your Blue Coat ProxySG device is located in New York (EST), and you want to run a report on data that occurred from 13:00 to 15:00 (1pm to 3pm), then you must specify GMT time in your reports and searches as 17:00 to 19:00. If you have LogLogic Hotfix 2 or later, then you do not have to calculate the time difference. To verify that Log Messages are uploaded to the LogLogic Appliance: 1. On a Microsoft Windows machine, open an Instant Messenger window (e.g., Yahoo! Instant Messenger). 2. Send a message. 3. Log in to the LogLogic Appliance to view the test message sent by the Blue Coat ProxySG device. 4. From the navigation menu, select Summary Reports > Top Web Activity > Web Cache Activity. The Web Cache Activity tab appears. 5. In the Device Type drop-down menu, select Blue Coat ProxySG. 6. In the Source Device drop-down menu, select All Blue Coat ProxySG. 7. Set the Time Interval to capture the test message you sent. For example, set the specific time to be from 1 hour before you sent the message to the current time. 8. Click Run. The Web Cache Activity tab is populated with the appropriate data for the device. 9. View the test message and details to verify the message was sent properly. a. Locate the Source Device. The Source Device can be identified by the Source IP. Blue Coat ProxySG Log Configuration Guide 31

32 b. Drill down on a listed item to view the Blue Coat ProxySG message details. The Web Cache Activity tab is populated with the appropriate data. 10. Verify that the test message was sent. Note: If you do not see the test message in the Web Cache Activity tab, then verify that you completed the steps properly. 32 Blue Coat ProxySG Log Configuration Guide

33 Appendix A Blue Coat ProxySG Best Practices This appendix describes best practices recommended by LogLogic for use with Blue Coat ProxySG. It also provides a description of LogLogic s Blue Coat Best Practices Reporting Package. Blue Coat ProxySG Best Practices Optimized and Non-Optimized Report Fields Blue Coat Best Practices Reporting Package Blue Coat ProxySG Best Practices LogLogic recommends adhering to the following best practices while configuring Blue Coat ProxySG and the LogLogic Appliance: Continuous Log Collection vs. Periodic Log Collection It is possible to gather and transfer logs using continuous or periodic collection methods. If you intend to transfer over 100 messages per second, LogLogic recommends using the periodic method. For more information, see Defining the Upload Schedule on page 10. If you use the periodic collection method, make sure that each chunk of log data that is transferred from Blue Coat ProxySG to the Host Server is less than 2 Gigabytes (GB) (1.8 GB maximum). Regardless of the log collection method used (i.e., periodic or continuous), make sure that the network connection between Blue Coat ProxySG, the Host Server (if applicable), and the LogLogic Appliance is as fast and stable as possible. The maximum recommended Blue Coat ProxySG log collection rate, per LogLogic Appliance per day, should be less than 60 million records (or be approximately 21 GB in size). All LogLogic platform families (i.e., LX, ST, MX) support Blue Coat ProxySG log data. However, for better performance, LogLogic recommends using the LX 2010 platform. LogLogic recommends running Release 4.2 Hotfix 1 or later on the LogLogic Appliance. Optimized and Non-Optimized Report Fields Blue Coat ProxySG log data can be viewed within the LogLogic Appliance s Web Cache Activity report. For more information, see LogLogic Real-Time Reports on page 30. In order to improve reporting performance, LogLogic attempts to pre-summarize log data as it is collected on the Appliance. The log data is stored in a manner that helps optimize the speed at which the data can be retrieved during reporting. By default, LogLogic summarizes or optimizes specific fields of parsed data collected for the Web Cache Activity report. So, each report field (i.e., Advanced Option) is either optimized or non-optimized. As a result, reports that exclusively use optimized fields will run faster than reports that include non-optimized fields. Blue Coat ProxySG Log Configuration Guide 33

34 For example, if you ran a report that contained one non-optimized field and three optimized fields, that report s run time will be constrained by the non-optimized field. Therefore, the report will run slower than a report that contained only optimized fields. Table 2 on page 34 provides a listing of all the optimized and non-optimized fields available within the Web Cache Activity report. Note: When using periodic collection, it may take approximately one hour for every 8 Gigabytes of batched log data (i.e., approximately 25 million messages collected) to complete the automatic parsing and aggregation processes needed before report generation is possible. Table 2 Web Cache Activity Report - Advanced Options Advanced Option Optimized Non-Optimized Source Device Source User Source IP Source Host Domain Name Destination IP Destination Port Peer IP Peer Host Peer Status Method URL Cache Code Status Type Size Filter Category Filter Result User Agent Referred By X Protocol X Action X Status X Product Virus ID Virus Details User Group 34 Blue Coat ProxySG Log Configuration Guide

35 Caution: When you drill-down on the Web Cache Activity report's results there is no default sort-by selection. The drill-down results are generally in order by time. If you specify a sort-by selection for this report's drill-down, performance in generating the drill-down results is slower. The default timeout period for reports is 15 minutes. Some reports might require a run time that exceeds 15 minutes. If you run into this issue, consider the following workarounds: Shorten the Time Interval for the report Modify the Advanced Options to include fewer fields Change the default timeout period on the LogLogic Appliance: Caution: LogLogic does not recommend changing the default timeout period. However, if you do change the default, you should only use this command on the Appliance where you will run the report. 1. Log in to the LogLogic Appliance s Command Line Interface (CLI). 2. Type in the following command: mysql logappconfig -e 'update Settings set maxidledbquery=####;' Where the maxidledbquery value (i.e., ####) is the timeout period (in seconds) for reports. The default is 900 seconds (15 minutes). For example, if you wanted to change the timeout to 7200 seconds (2 hours), the command would be: mysql logappconfig -e 'update Settings set maxidledbquery=7200;' 3. To save the configuration change on the Appliance, type in the following command: /loglogic/bin/loadsettings Blue Coat ProxySG Log Configuration Guide 35

36 Blue Coat Best Practices Reporting Package LogLogic also provides a Blue Coat Best Practices Reporting Package that includes pre-defined custom reports that allow you to monitor data regarding user activity, policy violations, etc. Note: The Blue Coat Best Practices Reporting Package is not part of a Log Source Package (LSP). This reporting package is available upon request from LogLogic s Account team. To access Blue Coat Best Practices Reports: 1. Log in to the LogLogic Appliance. 2. From the navigation tree, select Custom Reports. 3. Select BlueCoat Best Practices. The following Custom Reports are available: Domains by Unique User Domains visited by unique users yesterday sorted by count Files Downloaded via the Web All Web-based downloads yesterday greater the 5 Kilobytes (KB) Files Uploaded via the Web All Web-based uploads in the past day Filter Results by Count Blue Coat filter results for the past day sorted by count Peer Servers and Status Yesterday's peer web servers providing data for cache servers and status Top Bandwidth Total number of bytes transferred yesterday Top Categories Visited Blue Coat filter categories visited yesterday sorted by count Top Domains Accessed Domains visited yesterday sorted by count Top Spyware Src IPs Spyware Source IP addresses sorted by the number of bytes transferred during the past day Top Src IPs by Bandwidth Source IP addresses sorted by the number of bytes transferred in the past day Top URLs Visited Visited URLs in the past day sorted by count Top Users by Bandwidth Top users by bandwidth for the past day Top Users by Page Views Top users by number of total page views for the past day Web Access to Applications Access to Web-based applications in the past day The performance time for each report depends upon the fields used and the amount of data being displayed. Table 3 on page 37 provides a breakdown of each report, the Advanced Options (i.e., fields) used within the report by default, and an estimated run time in minutes. The run time estimate in minutes is based upon log data containing approximately 35 million messages (or be approximately GB in size). Note: There is a direct correlation between longer report run times and reports that make use of non-optimized fields. For more information, see Optimized and Non-Optimized Report Fields on page 33 and Table 2 on page Blue Coat ProxySG Log Configuration Guide

37 Table 3 Best Practices Reports, Advanced Options, and Run Times Report Default Advanced Options Run Time (mins) Domains by Unique User Source Device, Source User, Domain Name, Count Files Downloaded via the Web Source Device, Source IP, Domain Name, Method, Status, Size, Count 8.21 Files Uploaded via the Web Source Device, Source IP, Method, Status, Size, Count 0.04 Filter Results by Count Source Device, Filter Result, Count 0.12 Peer Servers and Status Source Device, Peer IP, Peer Host, Peer Status, Size, Count Top Bandwidth Source Device, Size 0.01 Top Categories Visited Source Device, Filter Category, Count 0.13 Top Domains Accessed Source Device, Domain Name, Count 9.91 Top Spyware Src IPs Source Device, Source IP, Size, Filter Category, Count 0.02 Top Src IPs by Bandwidth Source Device, Source IP, Size, Count 0.04 Top URLs Visited Source Device, Domain Name, URL, Count Top Users by Bandwidth Source Device, Source User, Size 0.01 Top Users by Page Views Source Device, Source User, Count 0.01 Web Access to Applications Source Device, Source User, Domain Name, URL, Status, Count Note: The Advanced Options listed here do not represent all of the options available within a report. The options listed only represent the fields that are selected to be displayed by default. Blue Coat ProxySG Log Configuration Guide 37

38 38 Blue Coat ProxySG Log Configuration Guide

39 Appendix B Format Reference This appendix describes the log format and report supported by Blue Coat ProxySG. LogLogic Support for Blue Coat ProxySG Table 4 LogLogic Support for Blue Coat ProxySG Supported Log Format Agile Reports date time time-taken c-ip sc-status s-action sc-bytes cs-bytes cs-method cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-username cs-auth-group s-hierarchy s-supplier-name rs(content-type) cs(referer) cs(user-agent) sc-filter-result cs-categories x-virus-id s-ip Web Cache Activity Sample Log :42: CONNECT tcp / NONE "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-us; rv: ) Gecko/ Firefox/ " PROXIED "none" Field Description date - The date on which the activity occurred. time - The time, in coordinated universal time (UTC), at which the activity occurred. time-taken Time taken for transaction to complete in seconds. c-ip - The IP address of the client that made the request. sc-status - The HTTP status code. s-action What type of action did the appliance take to process this request. sc-bytes - The number of bytes that the server sent. cs-bytes - The number of bytes that the server received. cs-method - The requested action, for example, a GET method. Blue Coat ProxySG logs can be classified according to the values in the cs-method field. HEAD GET POST PUT DELETE TRACE OPTIONS cs-uri-scheme Scheme from the 'log' URL. cs-host - The host header name, if any. cs-uri-port Port from the 'log' URL. cs-uri-path Path from the 'log' URL. Does not include query. cs-uri-query The query, if any, that the client was trying to perform. A Universal Resource Identifier (URI) query is necessary only for dynamic pages. cs-username The name of the authenticated user who accessed your server. Anonymous users are indicated by a hyphen. Blue Coat ProxySG Log Configuration Guide 39

40 LogLogic Support for Blue Coat ProxySG -- continued Field Description (continued) cs-auth-group One group that an authenticated user belongs to. If a user belongs to multiple groups, the group logged is determined by the Group Log Order configuration specified in VPM. If Group Log Order is not specified, an arbitrary group is logged. Note that only groups referenced by policy are considered. s-hierarchy How and where the object was retrieved in the cache hierarchy.time-taken Time taken for transaction to complete in seconds. cs-supplier-name Hostname of the upstream host (not available for a cache hit). rs(content-type) - Response header: Content-Type. cs(referer) The site that the user last visited. This site provided a link to the current site. cs(user-agent) The browser type that the client used. sc-filter-result Content filtering result: Denied, Proxied, or Observed. cs-categories All content categories of the request URL x-virus-id Identifier of a virus if one was detected s-ip - The IP address of the server on which the log file entry was generated. 40 Blue Coat ProxySG Log Configuration Guide