Enterprise Data Center (EDC) SQL Server Database Services Rules of Engagement Application Management Team Version 1.0 October, 09, 2015 SECURITY WARNING The information contained herein is proprietary to the Commonwealth of Pennsylvania and must not be disclosed to un-authorized personnel. The recipient of this document, by its retention and use, agrees to protect the information contained herein. Readers are advised that this document may be subject to the terms of a non-disclosure agreement. DO NOT DISCLOSE ANY OF THIS INFORMATION WITHOUT OBTAINING PERMISSION FROM THE MANAGEMENT RESPONSIBLE FOR THIS DOCUMENT.
Version History Date Version Modified By / Approved By Section(s) Comment 10/09/2015 1.0 Michael Porter All Create new document for SQL Server Database Services ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 2 OF 20
Table of Contents 1 EDC OVERVIEW... 5 1.1 EDC OVERVIEW... 5 1.1.1 EDC Engagement Process... 5 1.1.2 EDC Deployment Process... 5 1.1.3 Commonwealth Application Certification and Accreditation (CA 2 )... 5 1.2 EDC ARCHITECTURE... 6 1.2.1 External DMZ Security Zone... 6 1.2.2 Internal Services Security Zone... 6 1.2.3 Internal DMZ Security Zone... 6 2 EDC INFRASTRUCTURE AND SERVICES... 7 2.1 PURPOSE / OVERVIEW... 7 2.2 EDC SERVERS... 7 2.2.1 Physical... 7 2.2.2 Virtual... 7 2.3 EDC STORAGE... 7 2.3.1 Options... 7 2.4 EDC RECOVERABILITY... 7 2.5 EDC ENVIRONMENTS... 7 2.6 EDC MONITORING... 8 2.7 EDC MAINTENANCE... 8 3 SQL SERVER DATABASE SERVICES... 9 3.1 PURPOSE / OVERVIEW... 9 3.1.1 SQL Server Licensing... 9 3.1.2 SQL DBA Support Services... 10 3.1.2.1 Architectural Services... 10 3.1.2.2 Build Services... 10 3.1.2.3 Operational Services... 10 3.1.2.4 Transition Services... 10 3.1.3 SQL Server platforms... 11 3.1.4 SQL Backup & Recovery... 11 3.1.4.1 Overview... 11 3.1.4.2 Backup... 11 3.1.4.3 Recovery... 11 3.1.4.4 Archive (Optional)... 12 3.1.5 SQL Server Security... 12 3.1.5.1 Authentication Modes... 12 3.1.5.2 Authentication... 12 3.2 MANAGED SERVICES... 13 3.2.1 Overview... 13 3.2.2 Shared Database Services... 14 3.2.2.1 Deployment Options... 14 3.2.2.2 Standards... 15 3.2.2.3 Service Usage... 15 3.2.2.4 Encryption... 15 3.2.2.5 Jobs... 15 3.2.3 Dedicated Database Services... 16 3.2.3.1 Deployment Options... 16 3.2.3.2 Standards... 16 3.2.3.3 Service Class... 16 3.2.4 Roles and Responsibilities... 16 3.2.5 Maintenance... 16 ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 3 OF 20
3.2.6 SQL Server Monitoring... 17 3.2.7 Incident Management... 18 3.2.8 Change Management... 18 3.3 MANAGED SERVICES LITE... 19 3.3.1 Overview... 19 3.3.2 Hosted Database Services... 19 3.3.2.1 Deployment Options... 19 3.3.2.2 Service Class... 19 4 APPENDIX A - ADDITIONAL RESOURCES AND REFERENCES... 20 4.1 SQL SERVER DATABASE RESOURCES... 20 4.2 EDC SQL SERVER DEPLOYMENT COSTS... 20 ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 4 OF 20
1 EDC Overview This section contains standard information that is included in all ROE documents. 1.1 EDC OVERVIEW The Commonwealth of Pennsylvania s Enterprise Data Center (EDC) provides Hosting Services for Agency Web-Based and Agency Specific applications. Its mission is to maintain a high level of security, availability, reliability, and management of the Commonwealth of Pennsylvania's mission critical web applications. Refer to Enterprise Data Center, for a full description of the EDC and all hosting and service offerings. 1.1.1 EDC Engagement Process If your agency is considering deploying applications in the EDC, examine the EDC web site to understand the EDC Services Portfolio, and then contact your Service Coordinator (SC). SCs are liaisons between agencies and the EDC. They answer preliminary questions and coordinate meetings with EDC personnel to ensure consistent communication on simple or complex projects. Refer to EDC Getting Started, for an overview of the benefits, services, and options for hosting your application at the CTC EDC. Refer to EDC Services Coordinator, to identify your agency Service Coordinator. 1.1.2 EDC Deployment Process The EDC follows a well-defined deployment process for all application deployments. Application development is performed at the agency or contractor location while the EDC houses both a staging and a production environment, which are mirror images of each other. This structured deployment and testing process ensures a stable application in production. Prior to entering the EDC, every new application is required to undergo a security assessment. Refer to Deploying in Managed Services to review MS deployment process documents Refer to Deploying in Managed Services Lite to review MSL deployment process documents. 1.1.3 Commonwealth Application Certification and Accreditation (CA 2 ) Refer to Commonwealth Policy ITB-SEC005 regarding "Commonwealth Application Certification and Accreditation" Click https://www.sqca.state.pa.us to initiate the Commonwealth Application Certification and Accreditation (CA 2 ) Process. ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 5 OF 20
1.2 EDC ARCHITECTURE The EDC Web Farm architecture is segmented into security zones that are isolated from each other via firewalls. The EDC Network contains the External DMZ security zone, the Internal Services security zone, and the Internal DMZ security zone. These three primary networks are either, physically or logically, connected to one another. MAN Internet Agency MAN site Agency Firewall ESF Intranet Firewall Perimeter Firewall Internal DMZ / Services Managed Services Lite Internal DMZ Managed Services Internal Services Managed Services Internal DMZ Co-location Internal Services Co-location External DMZ Managed Services External DMZ Co-location External DMZ Managed Services Lite ESF InterZone Firewall 1.2.1 External DMZ Security Zone The External DMZ security zone contains Internet-facing servers that are connected to the Enterprise DMZ. EDC-managed web servers (such as Managed Services) and Agency-managed servers (such as Co- Location servers) both exist in the External DMZ Security zone. Managed Services and Co-Location servers are on separate subnets secured by either firewalls or Access Control Lists (ACLs). 1.2.2 Internal Services Security Zone The Internal Services security zone contains Managed Services database servers and other application servers from which dynamic content is obtained by web servers. 1.2.3 Internal DMZ Security Zone The Internal DMZ security zone contains the Managed Web and application servers that need to be accessible only from the Commonwealth Metropolitan Area Network (MAN). This Security Zone also contains internal Co-Location databases and web and application servers that are isolated from the Managed Services servers. When EDC Domain Controllers intercommunicate in a security zone, all communications use standard RPC and do not require IPSEC encryption or authentication. Domain Controller-to-Domain Controller communications between security zones only use IPSEC with Authentication Headers (AH). Other host-to-ad Component communication in the Managed Services portion of the Enterprise Data Center does not require IPSEC. However, IPSEC is required for all communications between entities outside the Managed Services and EDC AD components. ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 6 OF 20
2 EDC Infrastructure and Services 2.1 PURPOSE / OVERVIEW This section focuses on EDC services that are related to deploying or hosting databases at the EDC 2.2 EDC SERVERS The EDC provides hosting support for physical and virtual servers 2.2.1 Physical Physical server support is limited to servers supported by EDC Server Operations. 2.2.2 Virtual Virtual database servers are provided from VMware s ESX 5.5 software running on multiple clusters at EDC and managed by a dedicated team. Virtual machines are less expensive and have proven to be more reliable that physical servers. VMware provides analysis tools that can determine if and how a current physical implementation can be virtualized. 2.3 EDC STORAGE The EDC provides enterprise class storage for physical and virtual servers 2.3.1 Options Tier0 Enterprise Databases Tier1 Appropriate for high activity applications Tier2 Appropriate for most applications 2.4 EDC RECOVERABILITY The EDC provides backup services at the host and database levels. An optional archive service is also available. 2.5 EDC ENVIRONMENTS The EDC provides 3 deployment environments. Each environment has its own patch schedule. EDC Staging environment is meant to be a mirror image of the Production environment to facilitate the analysis of the impact of changes and patches. Development Staging Production ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 7 OF 20
2.6 EDC MONITORING The EDC provides server and services level monitoring utilizing Microsoft s System Center Operations Manager (SCOM). SCOM alerts are available to MSL customers upon request. 2.7 EDC MAINTENANCE The EDC provides OS level patching on a scheduled basis, usually monthly. ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 8 OF 20
3 SQL Server Database Services 3.1 PURPOSE / OVERVIEW This section focuses on back-end Microsoft SQL Server database service. SQL Server is supported in the EDC in Managed Services (MS) as a fully supported DBaaS offering and in Managed Services Lite (MSL) as a hosted offering. Using this document, an agency will gain an understanding of how to deploy databases in the EDC within a supportable model that meets EDC standards. Physical and logical isolation make up the foundation of SQL Server security. EDC databases services are in a physically protected location (locked and conditioned Data Center). Databases are installed in a secure zone of the Commonwealth intranet and are never directly connected to the Internet. The EDC backs up all data regularly and stores copies in a secure off-site location. The SQL Servers will reside in the External Active Directory Forest, which trusts the internal CWOPA domain. This trust facilitates the Single Sign-On security model whereby user accounts in CWOPA can be used to grant access to the applications in the EDC. EDC Application Management Team (AMT) has full administrative access over the External Active Directory Forest. 3.1.1 SQL Server Licensing All servers running SQL Server in the EDC must be fully licensed for the software to be installed. All databases deployed in the Shared Database Services environment have the license costs included as part of their deployment charge. EDC offers the following licensing options for Dedicated and Hosted database servers: Agency provided license as documented in SERP (Agency owns the license and is responsible for the SA) Acquired by EDC, on behalf of the Agency, as part of the SERP (Agency owns the license and is responsible for the SA) EDC SQL Licensing Service - Provided by EDC, as an annual service charge, when deployed to the dedicated SQL ESX Cluster. The licensing service cost is embedded in the price of the Standard SQL Server VM. (EDC owns the license) Microsoft requires all processor cores be licensed for servers running SQL Server Services, including servers running Analysis Services, Integration Services and Reporting Services. There is a minimum requirement of 4 processor cores per OS (Servers deployed with the EDC SQL Licensing Service may be deployed with fewer than 4 processor cores). Microsoft also requires that in order for VMware VMotion to be utilized, the SQL Server licensing must include Software Assurance (SA). ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 9 OF 20
3.1.2 SQL DBA Support Services EDC may provide SQL DBA Support Services for MS and MSL customers. Service Engagement is via SERP. Build Services and Operational Services are included in Managed Services deployments. Service 3.1.2.1 Architectural Services Planning and design Capacity planning projections Disaster recovery design and implementation Consolidation planning 3.1.2.2 Build Services Database implementation Upgrade and patch updates Backup implementation Cluster implementation Replication setup Standby setup and mirroring 3.1.2.3 Operational Services Service Now and Incident handling Backup monitoring Authorization and Security Administration Storage management DDL Auditing Query and Memory tuning Backups, Restores and Refreshes 3.1.2.4 Transition Services Cross platform and RDBMS migration Virtualization Consolidation Managed Services (MS) Available Included Included Available Managed Services Lite (MSL) Available Available N/A Available ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 10 OF 20
3.1.3 SQL Server platforms EDC provides support for hosting SQL Server on the following platforms: Physical server - Database outages occur when the server is restarted. This solution is more costly and less available. Not recommended. VMware HA Database is located on a virtual server. Database outages occur when the server is restarted. Restart times are faster than with physical servers, generally less than 5 minutes from restart to steady state. Virtual servers are also isolated from hardware failure outages associated with physical servers. Clustering Database is located on a SQL Failover Cluster Instance (FCI) located in a Windows Cluster. Database outages occur when the FCI is moved between clustered servers. Failover times are generally less than 5 minutes from failover to steady state on new server. Main advantage in availability over VMware HA is during SQL patching, when during SQL patching the VMware HA instance is unavailable, the FCI instance is limited to a 5 minutes failover. SQL patching usually occurs at most once per year. On the negative side, SQL FCI's are configured with standby or underutilized servers which increase the cost of deployment. Always-On Always-On is a hybrid between SQL Mirroring and Windows Clustering and was first available in SQL Server 2012. Databases are synchronously replicated between database instances located on a Windows Cluster. Outages are limited to only those transactions which have not committed at the time of an instance failure. This translates to zero to a few seconds of outage. Always-On requires 2x the storage of Standalone or SQL FCI databases. As with SQL FCI, an additional server is required. Secondary databases can be set as read-only, separating reporting transactions from OLTP. 3.1.4 SQL Backup & Recovery 3.1.4.1 Overview Backup and Recovery provides the ability to restore data, whether from an accidental deletion in a table, a failed application upgrade or disaster scenario. RPO and RTO are key in determining backup and recoverability. For databases, they are defined as: RPO Recovery Point Objective - targeted period in which data might be lost from a database RTO Recovery Time Objective - targeted duration of time to restore a database 3.1.4.2 Backup Production Daily full, TLog @ 9am, 1pm, 5pm (weekdays) Non-Production Daily full, No TLog (Databases must be set to Simple recovery) Other On request. Requirement specified in the SERP 3.1.4.3 Recovery RPO in a non disaster event will be a point in time specified by the requestor for Production and the last full backup for Non-Production. In disaster situations, the RPO will be the last accessible full backup + last accessible TLog backup for Production and the last accessible full backup for Non-Production. RTO in a non disaster event will be the length of time required to restore the data plus 1 day (time required to retrieve the tape). ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 11 OF 20
3.1.4.4 Archive (Optional) RTO in a non disaster event will be the length of time required to restore the data plus 1 day Daily (Son) - Tapes are sent off-site using a 7-day off-site/on-site process Weekly (Father) - Saturday tape is sent off-site and held until the Monthly tape is sent off-site. Weekly tapes are therefore held off-site for up to 5 weeks. Weekly tapes are recycled every month (roughly 30 days). Monthly (Grandfather) - Full back up from the last Friday / Saturday of every month. Each month these tapes are sent off-site and are held for one year. After one year lapses the tape hold time expires and the tape is reused. (Example: October 2010 will be recalled the end of month October 2011 and reused.) Yearly (Archive Tape) - In December one set of tapes is generated on the last Friday / Saturday of the month, both for the Monthly and for the Yearly. Retention of the Yearly overrides the Monthly. These tapes are held off site for 7 years and are not put back into rotation. 3.1.5 SQL Server Security 3.1.5.1 Authentication Modes SQL Server Database provides two authentication modes for securing access to the server: Windows Authentication Mode and Mixed Mode. The EDC leverages Mixed Mode; in this case users can be authenticated by Windows or by SQL Server Authentication. Users who are authenticated by SQL Server have their username and password pairs maintained within SQL Server. SQL Server connecting in Mixed Mode relies on Windows to authenticate users if the client and server are capable of using NTLM, or Kerberos logon authentication protocols. If the client is unable to use a standard Windows logon, SQL Server requires a username and password pair, and compares this pair against those stored in its system tables. Connections that rely on username and password pairs are called non-trusted connections, or SQL connections. Application tiers that exist within a single Windows domain should always use Windows authentication when communicating with each other. Credentials are managed for you and are not transmitted over the network. You also avoid embedding user names and passwords in connection strings. Client and middle tier applications should connect to SQL Server using Windows Authentication whenever possible. When forced to use a less secure authentication mode, application designers should ensure that SQL Server credentials are handled with appropriate care. 3.1.5.2 Authentication Applications should connect to the database using a least privileged account. If you connect using Windows Authentication, the Windows account should be least privileged from an operating system perspective, and should have limited privileges and limited ability to access Windows resources. The corresponding SQL Server login in the database should be assigned only those permissions required by your application. If a user or group is only allowed to view particular information then they should have the permission to READ, not WRITE to the database. The service account under whose context the application runs should also have the least required permissions. DBO access is not recommended for service accounts. As a rule, the EDC restricts use of ALL server level roles (such as SA, DB_Creator, etc ). ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 12 OF 20
3.2 MANAGED SERVICES The EDC provides a fully managed environment to deploy databases or database servers and have them managed by a dedicated experienced staff providing 24x7 support. Databases are located in either a Shared or Dedicated database service depending on their need for isolation. 3.2.1 Overview Deploying in the EDC Managed Services environment provides customers with the following benefits: ITIL based service operation o Incident management o Request fulfillment o Event management o Access management o Problem management Higher Uptime / Availability o Always On (Optional) o Clustering (Optional) o VMware HA o Proactive monitoring Reduced Total Cost of Ownership (TCO) for database operations o Reduced administration o Reduced operational costs o Reduced data center costs Service Management o Increased reliability o Improved performance o Improved consistency 24 x 7 x 365 support ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 13 OF 20
3.2.2 Shared Database Services Databases located in Shared Database Services are deployed to instances shared by multiple agency databases. Colocation of databases greatly reduces the cost to host a database. EDC manages all instance level objects while the agencies are able to perform database level functions. 3.2.2.1 Deployment Options Deployment Options will be priced according to the resources and licensing required. 3.2.2.1.1 Standard Database Standalone virtual database servers located in the MS Internal DMZ: SQL Server 2005 EE (End of Support 4/13/2016) SQL Server 2008 R1 EE SQL Server 2012 EE SQL Server 2014 EE Standalone virtual database server located on the MAN: SQL Server 2008 R2 EE 3.2.2.1.2 High Availability MSCS AlwaysOn Availability Group hosted on clustered virtual servers located in the MS Internal DMZ SQL Server 2012 EE SQL Server 2014 EE 3.2.2.1.3 Reporting Services SQL Server Reporting Services are software load balanced using Cisco Host Solution Engine (HSE) SQL Server 2008 R2 Reporting Services EE SQL Server 2012 Reporting Services EE (Planned) SQL Server 2014 Reporting Services EE (Planned) ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 14 OF 20
3.2.2.2 Standards Database Naming 2 digit Agency Code + Application Name. Should be similar to Stargate Application Name. Login Naming Similar to Database Name 3 rd party software Not permitted on Shared Database Servers xp_cmdshell - Not enabled 3.2.2.3 Service Usage Standard - Defined as a database deployment which consumes no more than the average CPU (1hr/day) and memory as measured by Resource Governor. Resource Governor may be used to throttle connections which may compromise the performance of the database server. Premium Additional fee charged for resource (cpu/memory) usage beyond Standard. Databases utilizing more than 24 hours of CPU per day may be required to move to dedicated database services. 3.2.2.4 Encryption Transparent Data Encryption (TDE) - Provides encryption At Rest for databases with only a minor impact to performance (3-5%). TDE is available for all databases hosted in Shared Database Services, except SQL Server 2005, and is recommended for any database storing sensitive data. 3.2.2.5 Jobs SSL Encryption Available on request. A SQL Server Agent job is a scheduled series of executable steps. Jobs are typically used to automate administrative tasks performed against a Microsoft SQL Server database. A job can perform a wide range of activities including running Transact-SQL scripts, command line applications, and Microsoft ActiveX scripts. While SQL Server supports scheduled jobs, the EDC must evaluate requested Jobs and Job Steps in concert with an Agency to ensure planned Jobs fit within EDC Security requirements. Agencies must schedule jobs in concert with the EDC. Agencies will not have elevated permissions in the EDC SQL Environment. All jobs must be documented for EDC implementation. Agencies must provide instructions, and failure notification information. ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 15 OF 20
3.2.3 Dedicated Database Services 3.2.3.1 Deployment Options Databases located in Dedicated Database Services are deployed to dedicated (non-shared) instances of SQL Server. EDC supports deploying all versions of SQL Server supported by Microsoft. EDC provides full Managed Service Support for the instance and databases. 3.2.3.2 Standards Where possible, EDC standards should be followed Database Naming 2 digit Agency Code + Application Name. Should be similar to Stargate Application Name. Login Naming Similar to Database Name 3 rd party software As required xp_cmdshell As required 3.2.3.3 Service Class EDC supports the following server configurations: Standard Standard Database Server Premium HA or DR configuration 3.2.4 Roles and Responsibilities Server privileges Not allowed Instance Level Privileges Not allowed Instance Level Objects EDC must evaluate Objects, such as Linked Servers and Jobs, to ensure they fit within EDC Security model. DBO Allowed In order to expedite recovery, it is the Agency s responsibility to notify EDC ahead of any major update or change made to the database The Agency is required to notify EDC ahead of any expected spike in database usage 3.2.5 Maintenance All maintenance to the servers will be tracked via change management. Any maintenance which will affect availability will be accompanied by an agency notification and scheduled in an Enterprise maintenance window. Examples of maintenances affecting SQL Server: Monthly Windows patching Required SQL patches Resource allocation (CPU/Memory/Storage) ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 16 OF 20
3.2.6 SQL Server Monitoring EDC is constantly monitoring the Managed Services environment to ensure the highest level of availability and performance. This service highlights events that may indicate possible service outages or configuration problems, so the EDC can quickly take corrective or preventive actions. EDC utilizes the following tools: System Center Operations Manager (SCOM) Idera SQL Diagnostic Manager In-House developed procedures EDC monitors the following: SQL Server Cluster Availability o SQL Server clustered SQL service availability SQL Server Agent Availability o SQL Server Agent availability o Monitor failed SQL Server agent jobs MSDTC o SQL Server MSDTC errors Performance o Deadlocks and blocks o Excessive user connections o Pages, writes, and allocations o Active transactions o Average wait time o DBCC logical scan bytes per second o Full scans per second o Memory grants pending Reporting o Replication metrics o Backup device throughput o Lock performance o Log performance o SQL Server critical errors Security o Trusted and non-trusted security configurations o Collection of failed SQL Server login attempts Backups o Failed full database backups o Failed incremental database backups o Database device throughput o Successful database backups ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 17 OF 20
3.2.7 Incident Management All requests for service in the Managed Services database environment are to be submitted through Service Now. 3.2.8 Change Management All changes made in the Managed Services database environment are tracked through Service Now. ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 18 OF 20
3.3 MANAGED SERVICES LITE Managed Services Lite provides the agency with an environment where they have control over the applications and services. Databases are located in a Hosted database service. 3.3.1 Overview Deploying in the EDC Managed Services Lite environment provides customers with the following benefits: Access the deployment resources available at EDC Windows patching SCOM alerting EDC will install and configure SQL software and initiate backup process as part of the SERP process (Optional) 3.3.2 Hosted Database Services 3.3.2.1 Deployment Options Databases hosted in Hosted Database Services are deployed to dedicated (non-shared) instances of SQL Server. EDC supports deploying all versions of SQL Server supported by Microsoft. The database server is fully managed by the Agency. 3.3.2.2 Service Class EDC supports the following server configurations: Standard Standard Database Server Premium HA or DR configuration ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 19 OF 20
4 Appendix A - Additional Resources and References 4.1 SQL SERVER DATABASE RESOURCES There are numerous resources and papers available from the Microsoft SQL Server web site http://www.microsoft.com/sql 4.2 EDC SQL SERVER DEPLOYMENT COSTS Deployment costs will be provided through the SERP process. ESF SQL SERVER 2008 RULES OF ENGAGEMENT PAGE 20 OF 20