Managed Enterprise Internet and Security Services NOMINATING CATEGORY: CYBER SECURITY INITIATIVES NOMINATOR: TONY ENCINIAS, CHIEF INFORMATION OFFICER COMMONWEALTH OF PENNSYLVANIA FINANCE BUILDING HARRISBURG, PA 17102 JUNE 2010 DECEMBER 2012
EXECUTIVE SUMMARY: The Commonwealth of Pennsylvania is a trusted steward of information serving over 13 million citizens and businesses, and supporting more than 80,000 employees across more than 40 state agencies. Citizens and business partners put their trust in state government to collect, store, transmit and protect a wide variety of Personally Identifiable Information (PII). Critical security services to protect this data were aging and decentralized, making them inefficient and costly to maintain. Between December 2010 and February 2011, the Commonwealth of Pennsylvania launched a new suite of centralized, high availability, fully managed enterprise Internet and security services. The transition consisted of moving the commonwealth from its old telecommunications provider and outdated network to a more robust, fully managed design with redundant security infrastructure and services. These centrally managed enterprise services include high availability firewalls, intrusion prevention systems, remote access services and a web content filtering solution. All security services are managed by fully staffed, 24x7x365 security operations centers (SOCs) and network operation centers (NOCs) that are fully redundant and resilient to any single point of failure. The transition to these services have provided commonwealth agencies with greater capabilities by providing high availability with robust audit and reporting capabilities, 24x7 handling of change requests and support issues, and highly skilled staff to detect, mitigate and report on security threats. The project helps to fulfill Governor Corbett s goal to cut administrative costs and to reduce state spending, as well as aligning with the strategic plan developed by the commonwealth s chief information officer to modernize IT Infrastructure and to provide more secure services to Pennsylvania citizens, businesses, and government employees. 2012-2015 IT Strategic Plan Key Strategic Concepts: Critical security services which were decentralized, disparate and costly to maintain are now provided by a fully managed, centralized enterprise internet and security solution. The overall solution has significantly strengthened the commonwealth s overall enterprise security posture while enabling the business. Significant cost savings on an agency-by-agency basis have been realized with the transition resulting in a total annual savings of $2,300,000 due to a decrease in 40 agency resources and IT infrastructure spend. Reduce Risk Leverage IT to Improve Service Delivery Protect information Enhance citizen services Better serve state agencies 2
BUSINESS PROBLEM: The commonwealth s enterprise web content filtering and VPN infrastructure were outdated and falling short of the business needs of state government agencies and customers. For many years, all forty commonwealth agencies were running and managing their own instances of web content filtering hardware and software. Many agencies were also reporting poor performance on old, outdated servers in need of hardware refresh. Allowing agencies to staff, maintain and administer their own instances of web content filtering solutions resulted in significant hardware requirements and inefficient staffing. The decentralized nature of these solutions also meant that no event correlation was possible across agency boundaries and Internet use reporting and data storage was maintained by each agency. This caused difficulty for human resources staff as they conducted investigations of online activity by employees during work time or using commonwealth IT assets. Additionally, because the legacy web content filtering solution was an old product (in place since 2006), it did not offer many of the new security protections capabilities and features designed to address the newer types of attacks occurring today and zero-day web threats. The move to an enterprise web content filtering solution was required to bring down overall costs, reduce the increased risk footprint in the commonwealth, and meet business requirements regarding HR investigations and user reporting and analysis. In addition, the enterprise VPN infrastructure maintained by the incumbent telecommunications provider was old, out of date, and had been in need of refresh for a considerable time. The platform vendor had also stopped providing updates to the technology because it was no longer supported. Additionally, the VPN solution did not support more current operating systems. Because agencies and their business partners required the use of these newer operating systems, the requirement to move to a new enterprise VPN was critical to enabling the business. SOLUTION APPROACH: The Office of Administration, Office for Information Technology (OA/OIT) oversees investments in and performance of all IT systems across the commonwealth, including enterprise-wide initiatives such as IT consolidation, shared services, IT support and all aspects of cyber-security, including enterprise security initiatives. The implementation and transition from the current services to the proposed Internet and security services solution relied on collaborative efforts. The approach was based 3
upon methodologies utilized by Verizon in support of past and existing state and federal government customers that had transitioned critical services to a fully managed Internet and security service solution. This included relocation and termination of new point of presence (PoP) demarcation facilities, full suites of integrated security appliances and services and the connections into the existing commonwealth network. The objectives of the transition were first to move the commonwealth s Internet traffic off of the current backbone to Verizon s data centers located in Pittsburgh and Philadelphia. Leveraging a primary/secondary site approach, both locations were built to consist of redundant Ethernet connections, one primary and one backup, to the Verizon IP backbone. While the commonwealth had a redundant and failover design with the old infrastructure, the new solution design would provide added redundancy and core failover between data centers. The transition also included moving agencies to the new centralized and fully managed web content filtering solution through a phased approach. Agencies had been accustomed to staffing, maintaining and administering their own instances of web content filtering solutions, leading to an array of hardware requirements and staffing resource constraints across 40 separate agencies. Furthermore, since these solutions were all independent, no event correlation was possible across agency boundaries and Internet use reporting and data storage was maintained by each agency. The amount of hardware that had to be maintained was extremely significant. The enterprise security service components included enterprise Internet, firewall, and intrusion prevention systems (IPS), enterprise remote access VPN, enterprise web content filtering solution and fully-staffed, 24/7 security operations centers (SOCs). The transition included a detailed project plan, which was meticulously followed along with a joint governance process. Collectively, these critical components encompassed a well-defined transition process, including pre-and post-transition testing, regular performance measurement and reporting, integrated change management, problem resolution processes, escalation procedures, recovery processes complete with defined roll-back plans and comprehensive dependency and mitigation checklists. The plan used a critical path methodology due to the significant impact and potential risk to the network if any component of the transition failed; completion of any paths identified in the transition plan were required to reach actual transition and failure of any one process would cause the respective path(s) to fail, thus jeopardizing the actual transition process. Success criteria for the transition were clearly defined with the goal of ensuring all Internet and security services were fully implemented, tested and operational. 4
With this project, the commonwealth successfully transitioned: 1) All state Internet traffic to two highly available, replicated, and geographically diverse data center facilities connected to the Verizon IP backbone. Each facility is capable of supporting 100% of traffic,. This robust solution provides redundancy and core failover between data centers. 2) The commonwealth s current security services to Verizon s fully managed security support services, including, 24x7x365 monitoring and event response and support for firewalls, intrusion prevention systems, web content filtering and remote access. This includes rapid response and reporting to the commonwealth from the provider s SOCs regarding security or security system health issues. Incidents that are detected by the managed security services team are analyzed and reported to the commonwealth Computer incident Response Team (CIRT team for action. 5
3) Existing decentralized web content filtering solutions to a new, fully managed web content filtering solution installed at each data center to provide redundant, highly available services. 4) The commonwealth s outdated remote access solution to a new enterprise remote access solution, with redundant VPN devices in both data center facilities. The VPN solution is a fully managed service. 5) All commonwealth s colocated and direct access business partners from the current solution to the new fully managed solution. SIGNIFICANCE: The transition to a centralized enterprise internet and security services solution to protect data and lower costs is a logical solution to realize improvements in government operations. It maps to the strategic objectives for OA/OIT: Improve the delivery of services to our customers through increased and improved online functionality while reducing cost of delivery. The solution provides for a single access point for incidents and governance for the Commonwealth resulting in an improved security posture for Pennsylvania s constituent data. The solution standardizes the security practices and governance of those practices across 40 disparate agencies. Reduce agency costs related to enterprise software by implementing core offerings as shared services. The costing model for the Internet and security services solution clearly identified where efficiencies could be gained and better services could be provided through a centralized model. BENEFITS: The financial benefits of successfully implementing the solution across the commonwealth have been significant. Each agency previously required a web filtering administrator at an average annual cost of $90,000 per year. The solution reduced the number of human resources required across all agencies by 40 resulting in annual personnel savings of $3,600,000. In addition, the annual license costs for the legacy software alone were $260,000. The 40 servers which had to be maintained by agencies 6
also had a collective cost of $240,000 resulting in total administrative and technical annual costs of $500,000. The cost for the new solution equaled $1,800,000 annually resulting in an overall cost savings of $2,300,000 annually. Overall, the key outcomes of the implementation have provided: High availability and high bandwidth services delivered through geographically diverse data centers and security operations centers. Redundant hardware with multiple site and device failure recovery capabilities within each datacenter Comprehensive logging, monitoring and reporting with robust policy management, platform management, change control, routine system patching, incident management, escalation and system management A simplified end-user VPN experience with integration into existing dual authentication solutions. Fully managed administration, including 24/7 monitoring and event response/support for firewalls, intrusion prevention system, web content filtering, and remote access solutions including support for end-user support calls. Custom monitoring and reporting capabilities with complete details and auditing of alerts, system availability statistics and graphs, system resource usage and policy modification events. Greater cost savings, including overall lower administration and architecture costs. A more robust security posture ensuring end users are prevented from intentionally or accidentally accessing sites that could damage the commonwealth s reputation or cause damage to the network. A scalable infrastructure for future growth to accommodate new business requirements with new services or enhancements to existing services when they are needed. SUMMARY: OA/OIT s desire to deliver improved telecommunication services cost effectively and to maximize its return on investment was in direct alignment with both the Keystone IT Plan introduced in 2007 and Governor Corbett s 2011-2012 Budget Address. The Commonwealth of Pennsylvania addressed a real problem of disparate systems, budgets, oversight and governance. The use of a centralized services facilitation model to drive ongoing enterprise-wide security services across over 40 agencies has provided for a more robust security posture, which better serves citizens and safeguards their data while maximizing taxpayer dollars. 7