Closed Environment Testing of ISP Level Internet Content Filters. Report to the Minister for Broadband, Communications and the Digital Economy

Transcription

1

2

3 Closed Environment Testing of ISP Level Internet Content Filters Report to the Minister for Broadband, Communications and the Digital Economy June 2008

4 Commonwealth of Australia 2008 This work is copyright. Apart from any use as permitted under the Copyright Act 1968, no part may be reproduced by any process without prior written permission from the Commonwealth. Requests and inquiries concerning reproduction and rights should be addressed to the Manager, Communications and Publishing, Australian Communications and Media Authority, PO Box Law Courts, Melbourne Vic Published by the Australian Communications and Media Authority Canberra Central Office Melbourne Central Office Sydney Central Office Purple Building, Benjamin Offices Level 44, Melbourne Central Tower Level 15, Tower 1 Darling Park Chan Street, Belconnen 360 Elizabeth Street, Melbourne 201 Sussex Street, Sydney PO Box 78, PO Box Law Courts PO Box Q500 Belconnen ACT 2616 Melbourne Vic 8010 Queen Victoria Building NSW 1230 Tel: Tel: Tel: , Fax: Fax: Fax: TTY: Australian Communications and Media Authority iv

5 Contents EXECUTIVE SUMMARY... 1 Introduction to the trial... 2 Project background... 3 Execution of the trial... 3 Results... 4 Conclusions... 5 CHAPTER 1: INTRODUCTION... 8 Terms of reference... 8 Previous technical studies How filtering technologies operate Identification techniques Blocking techniques Outline of the report CHAPTER 2: PROJECT BACKGROUND Appointment of test agency Selection of filter products Compilation of test data Selection of test site CHAPTER 3: EXECUTION OF THE TRIAL Performance Operation of an ISP s network Network performance metrics Test network and hardware... 27

6 Test methodology...29 Calculation of results...34 Effectiveness Test network and hardware...35 Test methodology...36 Calculation of results...37 Scope Adaptability CHAPTER 4: RESULTS Performance Effectiveness Scope Adaptability Summary CHAPTER 5: CONCLUSIONS Performance Effectiveness Scope Adaptability Current state-of-the-art APPENDICES Appendix A: The Ministerial Direction Appendix B: The tiered hierarchy of ISPs Appendix C: Types of network filter products not assessed Filter products targeting illegal content only Enterprise-level filter products Appendix D: Specifications of the hardware used Network Performance Test Accuracy Test Appendix E: Baseline network performance characteristics of the test network Appendix F: Individual product performance... 62

7 Alpha Beta Gamma Delta Theta Omega GLOSSARY BIBLIOGRAPHY... 78

8

9 Executive summary Executive summary This report has been prepared by the Australian Communications Media Authority (ACMA) in response to a ministerial direction received in June 2007 to conduct closed environment testing of internet service provider (ISP)-level internet content filters. The purpose of the trial was to assess the current maturity of commercial filtering products that are suitable for deployment by internet service providers. The direction is at Appendix A. A detailed assessment of the state-of-the-art of ISP-level filtering technology is at pages ACMA considers that, under the conditions created for the trial, the state of ISP-level filtering technology has significantly advanced, and stands in contrast with the state of this technology evidenced in the previous trial of filter products commissioned by NetAlert Ltd in The main indicators of the increasing maturity of ISP-level filtering technology are: the number of filter products that are specifically designed to be deployed by ISPs; availability of a number of filter products that produce moderate to nearly nil performance degradation; improvements in accuracy the products tested exhibited high levels of successful blocking and low levels of overblocking (that is, blocking access to content that is intended to be accessible); availability of the capability to offer different filtering options to ISPs customers or for customers themselves to customise filtering; and actual deployments of filter products by ISPs in other countries. The one area which showed little sign of advance was reflected by the absence in this trial, for the most part, of any capability of filtering content carried via non web protocols. The findings of this report and the assessment of the state-of-the-art of ISP-level filter products reflect testing in a controlled laboratory environment. In particular, the testing simulated a Tier 3 network (the lowest level of the ISP network hierarchy); different results might be observed in a real world Tier 3 network or in networks at Australian Communications and Media Authority 1

10 Executive summary higher levels in the ISP network hierarchy. This is due to variations in architecture, including hardware used, size and complexity of the network and traffic demands. Introduction to the trial The purpose of the trial was to assess the capability of available technology to filter illegal or inappropriate content at ISP-level and advances in filtering technology since the previous trial in The report commences with definitions of the four criteria against which the capability of ISP-level filtering technologies was assessed: performance whether the products degrade internet performance; effectiveness the extent to which the products correctly identify and block illegal content, content that may be inappropriate for minors and innocuous content; scope whether the products are capable of filtering non-web internet traffic; and adaptability whether the products can be customised to apply different levels of blocking according to the preferences of the user. ACMA was not asked, as part of the trial, to assess the capability of ISP-level filtering technologies that filter only illegal content. ACMA was also not asked to investigate the balance of costs and benefits associated with implementing ISP-level filtering, including: capital and operating costs associated with implementing filter products; costs associated with any upgrading of an ISP s network to address performance degradation associated with a particular filter product; and the nature and implications of the implementation of ISP-level filtering for ISPs customers. ACMA also did not assess other matters that may be of relevance to the efficacy of ISP-level filters in a real-world context, such as; the extent to which a filter can be circumvented; and the ease with which it is installed, deployed and implemented. Chapter 1 summarises the findings of previous technical studies of filtering for the Australian Government and concludes with an explanation of the different types of filtering techniques. Australian Communications and Media Authority 2

11 Executive summary Project background Chapter 2 describes the selection of six filter products for testing, following a public call for expressions of interest from filter vendors. The method of compiling the Category 1, 2 and 3 lists of URLs is described at pages In accordance with the ministerial direction, the trial was required to test the capability of filter products to distinguish between illegal, inappropriate and innocuous content. To test this capability, three lists of URLs were created as test data: 1. Category 1 was intended to test the extent to which the selected filter products blocked content on the ACMA prohibited content list. 2. Category 2 was intended to test the extent to which the selected filter products underblock, by allowing access to content that may be regarded as harmful or inappropriate for children but is not illegal. 3. Category 3 was intended to test the extent to which filter products overblock by blocking access to content that may be regarded as innocuous. Execution of the trial Chapter 3 describes how an isolated purpose-built network was established to test performance, simulating both the function of the internet as a source of content and the function of end users requesting content. The test network was analogous to a Tier 3 ISP. Details of the methodology used in testing performance are provided at pages Testing of the effect on network performance of each filter product involved measuring: baseline performance of the test network with no filter installed; performance of the test network with each filter connected, in turn, but with no active filtering occurring; and performance of the test network with each filter connected, in turn and actively filtering. Three indices representing the performance of each filter product were calculated from the results of these tests. Details of the methodology used in testing effectiveness are provided at pages Testing of effectiveness involved measuring: the effectiveness of a filter in blocking content corresponding to Categories 1 and 2 that is, content that was intended to be blocked; and the effectiveness of a filter in distinguishing content from Category 3 that is, content that was not intended to be blocked. Australian Communications and Media Authority 3

12 Executive summary Two indices representing the effectiveness of each filter product were calculated from the results of these tests. Chapter 3 concludes by describing the methodology used to assess the scope and adaptability, which involved an expert review of product documentation and interviews with suppliers of the products to identify specific features of each product. Results Chapter 4 sets out the measurements from the quantitative performance and effectiveness tests for the filter products and lists the capabilities of the filter products with for scope and adaptability. Details of the performance results are at pages For the performance test, the percentage results showing the degree of degradation introduced by a filter connected to the test network but not actively filtering (where a low figure indicates a lesser degree of performance degradation) were: below 30 per cent for all products; and below 10 per cent for five of the six products. The percentage results showing the degree of degradation introduced by a filter connected to the test network and actively filtering (where a low figure again indicates a lesser degree of performance degradation) were: nearly nil (two per cent) for one product; in the range 22 to 30 per cent for three products; and in excess of 75 per cent for two products. Details of the effectiveness results are at pages For the effectiveness test, the results showing the degree of success in blocking content corresponding to each of the URLs listed in the Category 1 and 2 lists (where figures fall in the range of 0 to 1 and a high figure indicates a greater degree of success in blocking content that was intended to be blocked) were: above 0.88 for all products; and 0.94 or above for three products. The results showing the degree of success in not blocking content corresponding to each of the URLs listed in the Category 3 list (where figures fall in the range of 0 to 1 and a low figure indicates a greater degree of success in not blocking content that was intended to be blocked) were: below 0.08 for all products; and below 0.03 for four products. Australian Communications and Media Authority 4

13 Executive summary Details of the scope results are at pages Details of the adaptability results are at pages Each of the filter products is able to block traffic entirely across a wide range of non-web protocols, such as instant messaging and peer-to-peer protocols. However, a capability to identify illegal content and content that may be regarded as inappropriate carried via such protocols was not found, excepting: two products that can identify particular types of content carried via one protocol; and one product that can identify particular types of content carried via one streaming media protocol. Chapter 4 concludes by reporting that all of the products allow the customisation of filtering policies for groups of users, for individual customers of an ISP and for individual users. Conclusions The specific findings for performance and effectiveness in this trial are substantively different to those of the previous trial. A comparison of the results for performance in this trial with those of the previous trial is at pages The previous trial reported that, when filters were connected to the test network and actively filtering, performance degradation ranged from 75 per cent to a very high 98 per cent between the best-andworst performing filter products. In the current trial, the corresponding performance degradation varied across a greater range from a very low two per cent to 87 per cent between the bestand-worst performing filter products. Network degradation as a percentage of baseline throughput 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Previous trial Current trial Although the performance of two of the six products tested in the current trial was relatively poor, one product generated almost no network degradation and the remaining three products exhibited low to moderate levels of degradation in network performance. The median network degradation of the tested filters significantly dropped indicating a significant improvement in network performance in the current trial compared with that of the Australian Communications and Media Authority 5

14 Executive summary previous trial. ACMA considers that this improvement in the performance of filters tested in the current trial compared with the previous trial represents a profound advance in ISP-level filtering technology. A comparison of the results for accuracy in this trial with those of the previous trial is at pages The previous trial reported a difference in the level of successful blocking (that is, the proportion of content that should have been blocked that was actually blocked) between the least and the most accurate filter products in the range 70 to 100 per cent. The corresponding levels measured in the current trial varied across a smaller range, between 88 and 97 per cent, with most achieving over 92 per cent. The median rate of successful blocking was improved from the previous trial. Percentage of Category 1 and 2 content successfully blocked 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Previous trial Current trial The previous trial reported a difference in the level of overblocking (that is, the proportion of content that was blocked that should not have been blocked) between the most and the least accurate filter products in the range six to 62 per cent. The corresponding levels measured in the current trial varied across a significantly smaller range between one and eight per cent, with most falling under three per cent. The median overblocking rate was significantly improved from the previous trial. Percentage of Category 3 content overblocked 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Previous trial Current trial Australian Communications and Media Authority 6

15 Executive summary An assessment of the ability of ISP-level filters to control non-web content is at pages Despite the general nature of advances in ISP-level filtering technology between the current trial and the previous trial, most filters are not presently able to identify illegal content and content that may be regarded as inappropriate that is carried via the majority of non-web protocols, although development work by filter vendors is underway in this area. This is despite developments in the use of internet technologies that have led to increased use of non-web protocols such as instant messaging and file-sharing. 1 1 See Chapter 2 of Australian Communications and Media Authority, Developments in Internet Filtering Technologies and other Measures for Promoting Online Safety, February Australian Communications and Media Authority 7

16 Chapter 1: Introduction Chapter 1: Introduction Overview Chapter 1 sets out the terms of reference for the trial and specifically describes the criteria against which the filter products tested in the trial were evaluated performance of filters in an ISP s network, effectiveness of the filters in blocking particular content, scope or range of internet content on which filters were able to operate and adaptability to the specific requirements of ISPs or their customers. The purposes served by evaluating ISP-level filtering products against these criteria are set out. The chapter goes on to outline the previous Australian Government technical studies on internet filtering and presents a summary of their respective findings and concludes by providing an overview of the contents of each of the following chapters. This report sets out the results of a study into the maturity of products for filtering internet content that are suitable for deployment by internet service providers (ISPs). The study was informed via a technical, laboratory-based trial of a sample of commercial filter products available to ISPs, conducted in the first half of The study was undertaken at the direction of the former Minister for Communications, Information Technology and the Arts for the purpose of assessing: the capacity of available technology suitable for deployment by ISPs to filter internet content that is illegal and inappropriate for minors which consumers may access through their internet connection; and the extent of advances made since a previous trial of ISP-based filter technologies that was carried out during Terms of reference The Protecting Australian Families Online Direction No. 1 of 2007 (Appendix A) was made under subsection 14(1) of the Australian Communications and Media Authority Act 2005 in relation to ACMA s functions under paragraph 8(1)(d) of that Act (dealing with, reporting to and advising the Minister in relation to matters affecting consumers of carriage services). The direction instructed ACMA to conduct a trial of one or more commercial filter products that provide the capability of filtering of internet content that is illegal and inappropriate for minors at the ISP-level and to report the findings of the trial to the Minister by 30 June Specifically, ACMA was directed to test the products against the following criteria: 1. Performance whether the products cause delays or otherwise degrade internet performance. 2. Effectiveness the extent to which the products distinguish illegal content, content that may be inappropriate for minors and innocuous content. 3. Scope whether the products are capable of filtering internet traffic other than web content, such as peer-to-peer file transfers, chat and instant messaging. Australian Communications and Media Authority 8

17 Chapter 1: Introduction 4. Adaptability whether the products are capable of being customised to apply different levels of blocking appropriate for children of different ages and to target different categories of content. The performance criterion is intended to quantitatively measure whether ISP-level filtering products, when deployed and operational within an ISP s network, adversely affect the performance of the network such that it would be necessary for an ISP to upgrade its network sooner than forecast in order to restore performance to a satisfactory level. The effectiveness criterion is intended to quantitatively measure the extent to which ISP-level filtering products satisfactorily perform their essential role that is, successfully identifying and preventing the delivery of illegal internet content and internet content that is inappropriate for children, while permitting innocuous internet content. The scope criterion is intended to indicate the ability of ISP-level filtering products to identify and block illegal and inappropriate internet content that is transmitted and delivered across the internet using non-web protocols (for example, instant messaging, file transfers) in addition to web content. The adaptability criterion is intended to indicate whether ISP-level filtering products can offer a range of filtering options that enable a filtered service supplied to an ISP s customers to be tailored to meet specific requirements of the ISP or of the customer, as opposed to a one-size fits all filtering solution. Accordingly, the assessment of ISP-level filtering products against these four criteria seeks to update knowledge about the overall maturity of ISP-level filters, not to identify a best buy product. There are some other matters that may also be relevant to the efficacy of a filter in a real-world context, but which were outside the terms of reference and not assessed by ACMA. These include the following. Filters may vary in their resistance to circumvention. Table 1 illustrates some common methods of how filters can be circumvented and how these methods may apply to both PC based filters and ISP level filters. Method PC based filters ISP level filters Administrator passwords Can be circumvented Cannot be circumvented Boot disks Can be circumvented Cannot be circumvented Anonymisers Translation software Search engine caching Mirrors Low probability of circumvention Moderate probability of circumvention Low probability of circumvention Low probability of circumvention Additional domain names Low probability of circumvention Table 1: Common methods of circumventing filters Unlike PC based filters, which are open, ISP level filters lie on the internal network of an ISP which is generally firewalled off from public access and are therefore less open to attack. Filters may also vary in the administrative complexity of installation, deployment and implementation. Factors such as the compatibility of existing hardware in the ISP s network in Australian Communications and Media Authority 9

18 Chapter 1: Introduction which the product is installed, the availability of skilled personnel and degree of change that such a device brings into a network make it impossible to assess in a closed trial of this nature. Previous technical studies This report is the latest in a series of Australian Government technical studies of internet content filtering, which are summarised below. BLOCKING CONTENT ON THE INTERNET: A TECHNICAL PERSPECTIVE, CSIRO, JUNE 1998 Commissioned by the National Office for the Information Economy, this report examined the technical aspects of the internet that allow particular content which has already been identified as illegal or offensive to be blocked, particularly by ISPs and content hosts. The report explored the technical issues associated with different methods of blocking content, as well as some nontechnical issues and evaluated the pros and cons of each method. The primary observations of the report were that: ISPs could offer differentiated services, including a walled-garden service that provides access to a limited subset of websites and a filtered internet access service; and international cooperation would be needed in order to deal with hosting of illegal (or offensive) content outside Australia. EFFECTIVENESS OF INTERNET FILTERING SOFTWARE PRODUCTS, CSIRO, SEPTEMBER 2001 Commissioned by NetAlert Ltd and the former Australian Broadcasting Authority, this report presented the findings of testing on 14 filter products for: ease of installation, configuration and use; ease of bypassing or disabling filter; ability to stop access to undesirable content; ability to avoid blocking desirable content; and ability to track access. The majority of the products tested were client-side desktop applications, but a number of serverside solutions suitable for enterprise environments were also tested. None of the products were specifically intended for deployment by ISPs. The primary observation of the report was that almost all the products were effective in blocking undesirable content, but all blocked some portion of desirable content. INTERNET CONTENT FILTERING, OVUM, APRIL 2003 Commissioned by the then Department of Communication, Information Technology and the Arts to evaluate the state of the art internet content filtering technologies, this study provided: technical advice on the emergence of new blocking or filtering methodologies since the commencement of the online content co-regulatory scheme set out in Schedule 5 to the Broadcasting Services Act 1992, in 2000; Australian Communications and Media Authority 10

19 Chapter 1: Introduction information about financial costs and administrative requirements associated with ISP-level filtering and the potential impacts on the performance and efficiency of the internet; and an overview of the application of filter technologies and other access management techniques in government administered schemes overseas. The study was undertaken using a combination of secondary and primary research, including interviewing leading content filtering vendors, ISPs in Australia and Singapore, government officials and interested parties involved in internet content filtering in countries covered in three case studies. No filter products were actually tested in the study. The major observations of the study were that no major technological developments had occurred since the commencement of the online content co-regulatory scheme in 2000 and that ISPs who had adopted filtering had seen a limited impact on throughput. It also noted, however, that the costs associated with such an implementation at the ISP-level made it unattractive for ISPs. A STUDY ON SERVER BASED INTERNET FILTERS: ACCURACY, BROADBAND PERFORMANCE DEGRADATION AND SOME EFFECTS ON THE USER EXPERIENCE, RMIT IT TESTLAB, APRIL 2005 Commissioned by NetAlert Ltd to provide a quantitative analysis of the performance impact of applying server based internet content filtering applications and appliances to an internet feed in both live and controlled environments, this technical trial provided information on: the extent of degradation of internet access speed and performance; the accuracy of filtering; and the effect of filtering on the user experience when using broadband internet services. RMIT IT TestLab encountered difficulties in finding products for the trial that were specifically designed for ISP level filtering; consequently, all the products tested were designed for enterpriselevel content filtering but had the capability of being deployed in an ISP s network. The primary findings of the trial were: a reduction of between 18 and 78 per cent in network performance when accessing the internet through a content filter; the filters performed significantly better when blocking pornography and other adult content but performed less well when blocking other types of content; of the filters tested, the most effective in terms of accuracy blocked 76 per cent of the URLs used in the testing and only one filter blocked 100 per cent of URLs on ACMA s prohibited content list; only one in six users noticed any degradation in network performance, but this degradation was regarded as minor and acceptable; and all users reported a measure of over-blocking. The present report updates the body of evidence about the state of the art of ISP-level filtering technologies in 2008, recognising both the possible developments in filtering technologies and the changes in internet use since the reports described above. Australian Communications and Media Authority 11

20 Chapter 1: Introduction How filtering technologies operate In order to put the nature of the products tested in the trial in context, the following is an overview of the ways in which different filtering technologies operate. Filter products perform two basic functions in order to limit users access to content they identify content that is to be excluded (or included) and then block (or allow) access to that material. Identification methods are common to most filter products, while blocking methods vary depending on the type of location a filter is deployed; for example, a home computer, ISP network or mobile phone network. Terminology describing different methods of filtering is imprecise and is rarely used among filter vendors in a standard manner, while several terms often exist to describe the same filtering methodologies. The following section provides a description of the most common filtering methods. IDENTIFICATION TECHNIQUES The two basic methods for identifying content to be filtered are: 1. Index-based filtering material is included in a list of either good or bad content. 2. Analysis-based filtering on examination of the content, it is found to meet a set of criteria intended to determine its acceptability. Australian Communications and Media Authority 12

21 Chapter 1: Introduction Index-based filtering is the process of permitting or blocking access to web pages on the basis of their inclusion on a list (or index) of web resources. Such filtering can be based on whitelists (exclusively permitting specific content while blocking all other content) or blacklists (exclusively denying specific content while permitting all other content). Indexes may be developed manually, by human searches and analysis of content, by category indexes (most commercial filter vendors provide a database of URLs classified by the nature of the content) or automatically by analysis-based filtering, as discussed next. Figure 1 illustrates the architecture of index-based filtering. Figure 1: Index-based filtering process Australian Communications and Media Authority 13

22 Chapter 1: Introduction Analysis-based filtering refers to the dynamic classification of content using computer software and has emerged in response to the shortcomings of index-based filtering that is, that the latter is only applicable to web pages that have previously been assessed. Such analysis may be based on key word, profile, image analysis, file type, link analysis, reputation and deep packet inspection among other criteria. Such classification may be in real-time or offline. Figure 2 illustrates the implementation of analysis-based filtering. Step 2: Request for content is forwarded to web server Web server hosting requested content Step 3: Web server returns content Internet Step 1: User requests content Filter Step 4: Filter assesses content User Step 5: If content is assessed as inappropriate, it content is blocked; otherwise it is delivered Figure 2: Analysis-based filtering process BLOCKING TECHNIQUES Common blocking techniques used are: packet filtering; DNS poisoning; caching web proxies; port blocking; pass-by filtering; and pass-through filtering. Packet filtering involves a router or other device determining whether to allow or deny the passage of a request for content by examining the headers 2 of the packets 3 as they pass through. Packet filtering examines the destination IP address in the header and determines whether that IP 2 Header refers to the information contained at the beginning of a packet, which contains information about the handling of the packet. Analogous to an envelope around a letter, header information includes, among other things, the destination and source IP addresses. 3 A packet is a formatted block of data that can be transmitted over a network. It includes both header information and the content to be delivered. Australian Communications and Media Authority 14