Seven Pillars of Carrier Grade Security in the AT&T Global IP/MPLS Network INTRODUCTION AT&T s legacy and expertise lies in the creation and maintenance of secure, reliable networks that are always on and available when you need them. This goal is as valid today for our Internet Protocol (IP) and MPLS (Multi Protocol Label Switching) networks as it was for traditional circuit switching network. And now that IP /MPLS networks are embedded in the critical processing of our business and government customers applications, we are more committed than ever to ensuring superior levels of carrier grade security for our customers especially those using IP. More specifically, in support of AT&T s Global IP/MPLS network, AT&T s Chief Security Office has developed a set of seven basic security protection methods or pillars, as we refer to them. These firm pillars maintain a constant security focus in all of our design, deployment, and operational processes around our IP/MPLS core. This brief article explains these security building blocks and principles that are inherent in our IP/MPLS backbone network. In particular, we provide an introduction to our security methodology rooted in what we call the Seven Pillars of Carrier Grade Security in the AT&T IP/MPLS Network : Separation, Automation, Monitoring, Control, Testing, Response, and Innovation. Pillar 1: Separation Customer traffic is separated using MPLS and the concept of Virtual Private Networks AT&T s MPLS based VPN services AT&T s Virtual Private Network (AVPN) service and its legacy MPLS VPN products, IP Enabled Frame Relay Services, Enhanced VPN Service (evpn), and MPLS Private Network Transport (PNT) are all designed to take advantage of the inherent separation strength of MPLS and Multi Protocol BGP (MP BGP)
AT&T's standards based implementation assures that data packets and customer specific routing information cannot leak" out of, or into, one customer s VPN to another. Or, from a customer's VPN to AT&T control traffic on our backbone. What this means is that customers need not be concerned with unauthorized disclosure or modification of their sensitive VPN traffic (or routing information) by other IP/MPLS network users. Several MPLS standards have been specifically published to prevent any type of unauthorized or illegitimate cross VPN sharing of customer routing or data traffic. Independent tests performed by many companies including Cisco, Miercom and AT&T repeatedly have confirmed this important security requirement. 1 Sadly, many backbone network providers suggest that by simply not allowing Internet peering links to directly attach to their private IP networks, they can mix together customer IP traffic using different less secure methods. This naive approach conveniently ignores the fact that the major source of many Intranet security violations to date has come not from direct attacks on a carrier's backbone infrastructure, but rather from customer networks and/or network management systems that attach via "backdoor" connections to the Internet. Eight four percent of enterprises and government agencies reported some type of security breaches in the last year (2009), according to a new survey by Computer Associates International. The survey also found that security breaches have increased 17 percent in the last three years, according to the survey released by the Islandia, N.Y., IT management software company on July 5. Customers of AT&T s MPLS based VPN services can expect that their own VPN network will have the following basic security characteristics derived directly from the strengths of MPLS and MPLS VPN related standards (e.g., RFC 4364): - Containment: Traffic (and routing information) sent between customer edge (CE) routers on the same VPN will always stays within that specific VPN no spillover or "leakage" can occur. - Isolation: No customer s VPN can in any way materially affect or influence the content or privacy of another customer s VPN Availability: Aside from the basic security related attributes of MPLS and MPLS VPNs, AT&T carefully engineers shared resources to meet the highest levels of availability and mitigates potential denial of service activities through additional methods such as access control lists, route filters, turning off unnecessary services, and other infrastructure hardening techniques. 1 Note: MPLS security testing referenced here was performed in the early 2000s; AT&T and Cisco testing are proprietary. The Miercom paper is found here: http://www.noida.stpi.in/mercommpls.pdf 2 http://www.eweek.com/c/a/security/study-security-breaches-afflict-most-enterprises-governments/
Simplicity: MPLS networks allow for simplified provisioning in both the customer and carrier domains (and hence can help to avoids security related configuration mistakes). First, MPLS VPNs are much simpler for customers to configure than legacy Layer 1 (e.g., private line), Layer 2 (e.g., Frame Relay or ATM) point to point solutions, or Layer 3 (e.g., IPSec VPNs). Second, MPLS VPNs allow for much more scalable service provider architectures, unlike some other VPN solutions (e.g., L2TP) based on ACLs and separating customer address space. A service provider network using access control lists or separate IP spaces as the primary method to create VPN separation has a very difficult task to manage. In this scenario, every new site or route that is added can potentially require a change on every other router in the network to ensure security. This is not a scalable solution and can lead to errors in configuration and potential security breaches. In summary, the most scalable MPLS based architectures allow the service provider to more cost effectively provide reliable, high performing services to a large number of enterprise customers without disrupting their existing customer base as organic growth occurs on the provider s network. The diagram below illustrates the AT&T Global MPLS Network Architecture.
This architecture incorporates several key components to ensure the security and reliability of the VPN customers. Security/Privacy In order to provide the highest level of security, separate "edge" routers are used in the AT&T network for VPN customers versus public Internet customers. The VPN edge routers are physically separate devices that only support private MPLS VPN customer connections. AT&T's MPLS VPN services are based on RFC 4364 (formerly RFC 2547) and provides privacy equivalent to Frame Relay or ATM Service according to most industry experts. AT&T's standards based MPLS network architecture provides: VPN route uniqueness and segregation using MP BGP attributes such as: Route Targets (RTs) which are used to control route distribution of customer specific VPN routes into their own dedicated route table or VRF. Customers are unaware of the RDs and RTs and VRFs associated with their VPN Route Distinguishers (RDs) which are appended to a customer's routes to help identify them as belonging to a specific VPN. Virtual Routing & Forwarding Tables (VRF) Each customer's VPN routes are stored in a separate and unique routing table. VPN membership and all network configurations are controlled by automated AT&T provisioning systems. Customer specific interfaces on each PE are automatically assigned to that particular customer s VPN. Several consistency checks are made in systems to ensure that any new connection or VPN is legitimate for that customer.
Reliability While the edges for VPN and Internet services are physically separate, the AT&T IP/MPLS core is shared across these services. To ensure security and reliability several architectural design steps have been taken: 1.Route Free Core: The core network only provides Label Transport over Label Switched Paths. The routers have no knowledge of any Internet or customer specific VPN routes and IP addresses. 2. Control Plane: No backbone routers are visible to the Internet or customer specific VPN and none can be reached from any external endpoint. 3. Data Plane: VPN and Internet traffic is carried across separate E LSPs so that different traffic or service types can be differentiated in the core. For example, if a large internet DOS type of event were to occur, VPN capacity is protected to ensure no adverse impact to a customer's private VPN traffic. Pillar 2: Automation Automated perimeter security tools protect AT&T s MPLS core Over the past several decades, AT&T has invented, patented, and perfected many automated tools and systems to manage and protect its telecommunications infrastructure and networks. AT&T has continued to invest in automated methods for provisioning and maintaining its global IP/MPLS network and services. This included the development of several patented IP and MPLS troubleshooting tools, and further automation surrounding detection of security anomalies. Seamless integration and automation has been one of our key strengths. AT&T also knows, however, that in the creation of any new network infrastructure or service, that some manual techniques must precede such automation. That is, before such critical tasks such as automated provisioning or change management can be integrated into a carrier backbone environment, manual processes must first be documented, tested and perfected. Then, and only then are the systems and tools developed to carry forward these proven methods and best practices. One of AT&T s key advantages in the inevitable convergence of global network telecommunications traffic to MPLS is that we have made significant progress in moving through the experience curve toward the use of automation. Stated more simply: We are way ahead of our competitors in the charge toward flawless use of automated management techniques.
AT&T has had many years of experience with MPLS, and was an early adopter, announcing its first MPLS based service in 1999. Since then, AT&T has continually rolled out new and enhanced MPLS based IP VPN services in support of enterprise customers. Today, AT&T is regarded by leading telecommunications analysts as having one of the most comprehensive VPN portfolios in the industry, including MPLS, IPSec and SSL based solutions. AT&T has put in place a number of specific security measures, supported by automation, to support its global IP/MPLS network infrastructure. These include: Filtering: AT&T uses standardized engineering rules and automated provisioning systems to manage infrastructure specific access control lists (ACLs) related to access to the network as well as control traffic across the network. Least Privilege: Infrastructure routers, and PE interfaces, are hardened by turningoff, or severely restricting, unnecessary protocols and ports. BGP Authentication: Border Gateway Protocol (BGP) authentication can be implemented on upon customer request for many services CE PE ebgp sessions. BGP authentication ensures that the BGP routes passed between two BGP speakers (routers running the BGP protocol) are authorized and have not been tampered with in any way Limits: per session and per VPN routing prefix limits, dampening, and other mechanisms are used on many services to limit either the rate, or total number of routing update transactions that can be processed by an AT&T edge router. Authentication: TACACS+, tokens, SSH and other mechanisms are used to control access for authorized AT&T employees to access infrastructure devices. Pillar 3: Monitoring IP traffic monitoring provides early warning of Internet worms, botnets, and denial-of-service. One critical component in the management of large scale network traffic is the generation and analysis of traffic flow data to detect trends and anomalies. Such exception based processing has become the basis for many new forms of intrusion detection. AT&T has been using this technique for a number of years to identify patterns of normal network behavior and to measure differences from observed patterns. AT&T has the most extensive commercially available infrastructure in the industry for detection of traffic anomalies that are indicative of denial of service attacks. As well, AT&T uses proprietary
technology to detect patterns that indicate worms, botnet command & control, and other anomalies. Since the late 1990 s, AT&T has used these technologies to identify clear network patterns of anomalous behavior leading up to the Slammer, Blaster, Nachi, and SoBig worms and viruses. This was accomplished through a proactive, 24/7 analysis of network flow data (data content is not necessary for such profile based security). In some cases, clearly recognizable spikes occurred days before large events. AT&T has continued to develop this technology that has helped detect alert customers promptly of changes in Conficker/Downadup worm behavior in late 2008 and into 2009. Detection of malicious botnet command and control as the botnets recruit new zombies allows AT&T to take a more preemptive approach to network security and attack prevention. This monitoring provides unique protection benefits for the MPLS network in two ways. First, it allows our security teams to take steps toward the appropriate filtering often well in advance of other providers. And second, by tasking the monitoring system to detect any probes aimed at the MPLS core address space, we ve invented a novel means for dramatically reducing risk in our core. Customers of AT&T s MPLS based services thus can enjoy the following benefits of our monitoring systems: Anomaly Detection: AT&T proactively monitors traffic for anomalies that provide evidence of worm and virus trends in real time. External Access: AT&T also alarms and monitors infrastructure elements for resource consumption and attacks. Analysis: The world class statisticians from AT&T Laboratories Research continue to make great strides in algorithms for security anomaly detection. By virtue of the MPLS network, AT&T has developed a suite of optional security services that complement and can be used in conjunction with MPLS based VPN services. Private Intranet Protect service is available to provide flow data analysis of an enterprise s own VPN environment. The service does not require any additional equipment to be deployed at your sites, which keeps costs down and reliability high. AT&T can provision systems in the core network to gather flow record data (no content), pass the flow data to analysis systems, and provide alerting and analysis through a web interface that is specific to an enterprise s network traffic. Network based Firewall is another service that can be used to safely access the Internet directly from an enterprise s MPLS VPN with packet inspection and IDS with options for, user authentication, IPS, web content filtering, email scanning, multiple DMZs hosting, and DDoS Defense services. All of these services are facilitated through MPLS capabilities. Security Network and Operations Center (S/NOC, SOC) functions as well as premise based solutions can also be provided to provide a full
complement of security enforcement and monitoring capabilities throughout an enterprise s network. Advanced security analysis and threat management is available through AT&T Security Event & Threat Analysis (SETA) providing prioritized alerting based on correlated analysis of logs and alerting from multiple network devices, device types, and applications. Pillar 4: Control AT&T enforces strict operational security controls in its MPLS core. 24/7 network operations strike at the heart of the basic value proposition AT&T offers its customers. We ve been in the business for over a century and while technologies and customer needs have changed dramatically through these years, one thing has remained constant in our service provision: Operational Excellence. Our original experiences moving massive volumes of customer data over our first high speed packet networks for Frame Relay and Asynchronous Transfer Mode (ATM) networks exemplifies this operations focus. When these technologies became popular in the early nineties, one common criticism was that the reliability associated with these networks could never approach the excellence achieved in circuit switched environments. The good news is that after our initial period of growth (and yes there were occasional operational errors during this period), we managed to achieve levels of reliability and availability in these networks that exceed that of our circuit switched networks. Our MPLS core is no different. When we chose to build our first IP network and then enable it with MPLS technology, we applied the same relentless focus on reliability and resiliency to that network, both in the U.S. and globally. Today our IP/MPLS network consistently achieves higher reliability than traditional TDM voice, which used to be considered the gold standard for reliability. We have achieved this high reliability in a manner that is simpler to provision, easier to operate, and more difficult to attack than any network technology we ve ever operated. So the basic elements of our operational excellence in support of MPLS security are as follows: Administrative Separation All MPLS network management traffic is isolated on a separate MPLS VPN, using loopback addresses provisioned on a separate MPLS VPN, from protected address space which is not advertised; all traffic to this protected address space is blocked at the edges of the network and is not visible from the rest of the network.
Processes: AT&T s operations follow mature Methods and Procedures (M&Ps) that are derived from decades of best practices in operating carrier networks. Root Cause Analysis: All incidents are subject to comprehensive Root Cause Analysis steps a process used to ensure process improvements through any operational policy violations. Pillar 5: Testing AT&T uses testing, audits, and reviews to ensure security compliance. Our Information Security team employs some of the best ethical hackers on the planet. These engineers are tasked with the constant chore of probing, testing, and trying to find weaknesses in our MPLS network. Occasionally they find an area in which improvements are necessary and steps are taken immediately to address their findings. In addition, our Information Security team works with both internal and external auditors to ensure that all operations and infrastructure teams follow the industry s best security requirements. This is an on going task that sweeps through all aspects of our infrastructure including the MPLS network. The mature AT&T Security Policy Requirements (ASPR) 3 stands as the basic guide for all these activities. Our processes also include the use of expert reviews and organizational approvals as socalled Security Gates in almost everything we do. Our design and development efforts, for instance, follow a corporate wide standard and documented methodology. The ASPR process mandates an expert security review such that newly developed processes will not even pass the first conceptualization step without approval from designated teams of security experts. The result of this discipline is the following for our MPLS customers: - Testing: AT&T conducts ongoing intrusion detection, audits and penetration testing against server complexes for network management, customer care and service support. Customer MPLS VPN s are created and configured by an automated provisioning system, and any changes or discrepancies in router configuration, from that in the backend provisioning database, will be detected by regular discords detection/reports. - Auditing: On going independent audits by independent, internal security teams are used to confirm compliance with the AT&T Security Policy Requirements. - Reviews: All processes have embedded controls that require expert security reviews. 3 AT&T Security Policy and Requirements establish the security controls necessary to protect computing and networking environments across all AT&T working environments.
Pillar 6: Response AT&T deploys proactive response teams trained in the details of MPLS. Security incident response at AT&T is performed using a tiered operations structure. AT&T utilizes a mature, global three tiered 24/7 security operations team that is centrally coordinated in the Global Network Operations Center in Bedminster, NJ. Expert Tier 3 security analysts support this structure as incidents are escalated using well defined security methods and procedures. At Tier 1, trained operations managers use mature monitoring tools to proactively identify conditions that might warrant response. A Tier 2 management interface oversees this activity and is used to tie together conditions that might be brewing in disparate locations. When the appropriate condition has been identified perhaps a spike or anomaly in traffic Tier 3 activity is initiated. From a security perspective, Tier 3 analysts are among the most senior and trained security engineers in the company. They make the real time determination as to whether our AT&T Computer Security Incident Response Team (CSIRT) must be initiated. The AT&T CSIRT is a mature 24/7 operational structures and set of processes in which experts from the AT&T Network Security team coordinate real time response activities with operations staff from the various parts of AT&T's business. The ACSIRT centrally manages and coordinates all response activities related to proactive mediation based on early indicators, as well as mitigation of any detected security problems We re proud that the current state of the practice for our ACSIRT involves proactive response to conditions often long before customer impacts can ever become visible. In fact, for several years AT&T has extended this proactive response to our customers through a service called AT&T Internet Protect. Thus, customers who subscribe to this service can rely on incident response protection from AT&T as follows: - Tiered Response: Incidents are dealt with via a mature tiered response infrastructure that includes senior security and operations experts. - Proactive Indicators: The AT&T Computer Security Incident Response Team acts routinely in a proactive manner on indicators that typically precede any customer visible problems. - Innovative Customer Notification Service: AT&T has extended this capability to customers through an innovative notification service that provides real time indicators of anomalous behavior or detected security incidents to clients on a 24/7 basis. - WAN Analysis: Private Intranet Protect (PiP) option allows, with permission, AT&T
the ability to perform analysis across customer MPLS VPNs and notify administrators when potentially harmful traffic patterns are detected. Pillar 7: Innovation AT&T funds the most extensive MPLS security research in the world. AT&T s research laboratory has evolved over the past two decades from a Bell Laboratories that was involved in a broad range of technologies to a more focused AT&T Laboratories. One of the most important issues of concern for this organization continues to be network operations. In fact, AT&T s laboratory is the only organization of its kind rooted in the excellence and tradition of Bell Laboratories, but guided by the day to day needs of our operations teams. And security is one of our research laboratory s key focus areas. MPLS, in particular, provides a landscape on which our researchers have tried to find new techniques for protecting our customer traffic and systems. Creative means for analyzing anomalies, algorithms for integrating control and data plane information, and new means for MPLS management and monitoring are among the many areas in which our researchers are actively working, publishing, and sharing in the community. As such, this research laboratory complements our development, engineering, and operations teams in a way that remains unique in our industry. No other service provider on the globe maintains the type of research commitment to networking and more specifically to MPLS than AT&T. This serves to underscore our commitment to excellence in this area and will ensure that the best available innovations are always embedded into the MPLS infrastructure we use to support customers. Conclusion Our Seven Pillars of Carrier Grade Security for MPLS result in a set of conclusions that we view as critical to our value proposition for our customers: - Security Equivalence: We are proud to report that AT&T s MPLS security is currently equivalent to the type of security provided on other technologies such as layer 2 services. This does not mean things are perfect but it does point to great advances made in the past few years. - Continued Improvement: AT&T has always dedicated itself to programs of continued improvements to security and will continue coming down the experience curve for
MPLS just as we did for Circuit Switched, Frame Relay, ATM, and Managed IP Networks. Appendix A includes some common security/reliability questions and the mitigation processes.
APPENDIX A Common Reliability Security Concerns Concern Privacy/Intrusion: Can someone break into my VPN? Denial of Service: Can an Internet Traffic Storm, such as a DDOS attack affect my network or performance? Intrusion/Reliability: Can bogus routes bring down an edge? Core Protection: Can the backbone network be compromised? Mitigation 1. MPLS VPN endpoints are provisioned with same privacy level as Frame Relay/ATM. 2. VPN Edges are separate from Internet Edges 3.No Customer routes are visible on the backbone, only labels on the backbone 4.Rigorous Security Procedures in place for Provisioning and Maintenance VPNs and router configurations are auto provisioned avoiding human touch error. Operational Support Systems Firewalled All access/changes are logged Automated Discord Checks Established incident response procedures Documented Deploying active intrusion detection 1. VPN Edges are separate from Internet Edges 2. Backbone is segmented so Internet and VPN traffic are in separate E LSPs. An Internet storm only impacts the Internet. 3. DDOS detection and pro active filtering even on Internet 1.VPN Edges are separate from Internet Edges 2.Each VPN is route limited (<20K routes) 3.Route filtering is done on Internet endpoints and peering points to filter out bogus routes 1. Routing elements in the core are not visible or reachable. Backbone is Internet and VPN route free. 2. ACLs used at edges to protect core elements 3. Route filtering (Anti spoofing) applied on all inbound access points 4. Limited management access to specific ports (Telnet, SNMP), all others turned off. Require encrypted authentication and auditing. 5. Management access to equipment uses a separate network. 6. Internal servers are firewall protected.
Core Protection: Can a POP be compromised? Monitoring: Is Security Monitored? Increasing Privacy: Can I increase my security/privacy level? 1.Guarded and hardened AT&T facilities 2.Access requires authentication through advanced security systems 3. All hardware is redundant 4. All POPs equipped with dual commercial power supplies with generator and battery backup 5. Protected by AT&T s Network Disaster Recovery Program 1.Elements monitored 7x24x365 by multiple NOCs 2.On site vendor support 3.All access monitored and logged 4.Customer alerts and notification 1. Customers can optionally add encryption services (e.g., IPSEC based) to compliment the MPLS VPN to further increase privacy out to the edge. 2. Enhanced Security Services offered: Firewall Services (Network and CPE based) Intrusion Detection Network Scanning services Private Intranet Protect Authentication and Directory Services Security Event & Threat Analysis Professional Services