Information Technology @ Johns Hopkins Enterprise Technology Services (ETS) Systems Management Service Level Agreement February 2015 (rates effective FY16) Information Technology at Johns Hopkins (IT@JH) is a full service IT organization serving the Johns Hopkins Institutions. The organization is responsible for a wide range of centrally funded IT services that benefit the entire Johns Hopkins Institutions. In addition, the organization provides fee based services for departments and divisions. This document describes how departments, divisions, schools and projects can obtain server, data storage and related services. Contents Overview... 2 Service Offerings... 2 Service Offering Matrix... 2 How to Obtain Systems Management Services... 3 Johns Hopkins IT Policies, Standards and Guidelines... 3 Service Variations & Exceptions... 3 Service Offerings and Key Service Level Responsibilities... 3 Managed Services... 3 Cloud Server and Storage Infrastructure... 4 Agreement... 4 Infrastructure Services Detail... 5 Detailed Service Level Terms... 7 Help Desk / On call Teams... 7 Documentation Requirements... 7 Response & Recovery Times... 7 Backup & Restore Services... 7 Delegation of Technical Administrative Rights... 8 Administrator or Root Access, Remote Access... 8 Domain & Directory Services... 8 Security Scans, Systems Management... 8 Alerting & Monitoring... 9 Risk Management & Disaster Planning... 9 Change Management Practices... 9 Data Center Environment... 10 Licenses... 10 Legacy & Unsupported Systems... 10 Capabilities... 11 Service Offering... 11 In Depth Overview of Offered Services (need to refine)... 11 System Administrator involvement in application loads and upgrades... 11 Coordination of security, firewall and intrusion detection with network security... 11 Installation and Configuration of Server Hardware and OS... 11 Account Management... 12 Patches and Upgrades... 12 License Management... 12 3 rd Party Software... 12 Legacy Systems Support... 12 Appendix: Enterprise Web Services... 13 Appendix: Customer Managed (Hosted) Systems... 13 Server Virtualization Team... 14 Page 1
Storage / Data Protection Team... 14 Appendix: Storage Offering Details... 14 Appendix: Virtual Desktop Offering... 16 Appendix: Citrix XenApp Application Virtualization... 16 Appendix: Useful IT@JH contacts... 16 Appendix: Midrange Team... 17 Overview The most effective way to use IT@JH infrastructure services is by subscribing to a Managed Service offering. These services provide a complete systems management solution, adhering to all JH policies, guidelines and best practices. These services are especially critical when protected information or external access is required. The Enterprise Technology Services division of IT@JH manages IT services infrastructure. The groups involved in systems management are responsible for building and managing highly reliable, secure systems for the greater Johns Hopkins community, at the lowest possible cost. A highly trained staff of experienced professionals are available for support, consultation and proactive systems management. Many services are available through simple email transactions, and in some cases through a web based self service system. In other cases, a technical contact or manager assists in providing services and resources. Service Offerings IT@JH provides the following systems management services: Enterprise systems used by all of Johns Hopkins, managed to the highest levels of availability. Systems critical to the operations of a department, division or line of business. Managed Web Servers Hosted, or customer managed systems where IT@JH provides capacity and infrastructure and systems administration is provided by a department or division. Data Storage and data protection (backup/restore), either connected to host systems or directly accessible across the network. Support for custom systems requiring special networking, systems support, or other features not specifically identified in this document. Service Offering Matrix This document describes infrastructure fees billed on a monthly basis for a defined unit of measure. Monthly fees are charged to SAP cost centers and transferred to a Service Center budget. Usage reports are available as needed or on a monthly basis. Managed Services Rate Typical Uses Managed Virtual Server Management fee (in addition to resource fees) Managed Virtual Web Server (resource fees add l) Managed Citrix Virtual Server (in additional to resource fees) Resource Fees & Other Services Virtual Server Resource Hosting fees For customers who wish to utilize $200 per server per month IT@JH systems engineers for server management & security $300 per server per month Includes a managed web services package (see below for details) $200 per server per month (plus Citrix license fees) Charged in addition to Managed Service Fees $45.00 per VM per month base cost, plus $4.50 per virtual CPU (vcpu) per month $1.50 per GB of RAM (vram) per month $0.16 per GB Disk Storage per month For customers who need a Citrix server to publish applications to end users using any device Includes multi site data protection & Windows Server OS fees Enterprise Class Storage $0.16 per GB per month (XIV/3PAR) For virtual infrastructure, enterprise and critical applications Page 2
NAS Storage $0.04 per GB per month Network attached storage, includes replication and 6 weeks of snapshots. Two platforms are available based on usage requirements Managed File Services $0.25 per user per month (6GB user directory limits) End user home directories, no technical background required Hourly Services $110 per hour For any services outside the scope of included services Enterprise Unix Servers Vendor Priced Backend, large scale or clustered systems Physical Servers Vendor priced Used in cases where virtual servers do not meet resource requirements. Customer is responsible for all associated equipment costs Virtual Desktop $12 per user per month Provides a standard, secure, Windows 7 desktop, available from a wide range of devices How to Obtain Systems Management Services New customers who wish to initiate services with IT@JH should submit a request via email to: etsmanagedservices@lists.johnshopkins.edu This email inbox is monitored during business hours and you will receive a response within 1 business day. This email inbox is not for problems requiring quick response. For problems, contact the IT@JH help desk at 410 955 HELP. The appropriate oncall engineer will be contacted. Johns Hopkins IT Policies, Standards and Guidelines IT@JH adheres to all audit, compliance and regulatory policies relevant to the management of IT systems. The following link provides additional information related to these topics: http://www.it.johnshopkins.edu/policies/ Customers who manage their own virtual servers are required to adhere to the same set of policies and standards. Service Variations & Exceptions For each project or set of resources provided to a customer, a Service Level Addendum is used to document any variations, exceptions, exclusions and additions to the services outlined in this document. The SLA addendum will show itemized fees for various services. It is reviewed periodically with the customer and includes information on financial matters relating to any variations. Service Offerings and Key Service Level Responsibilities Managed Services are the most effective way to utilize IT@JH infrastructure services. The following section summarizes these services Depending on the level of service required, one or more groups will be responsible for delivering resources and services to a customer. Detailed service descriptions are provided in the appendices of this document. Managed Services IT@JH engineers work with the customer, vendor, and application administration team to optimally and securely configure enterprise & departmental class UNIX/Linux & Windows servers. IT@JH Managed Services include: Collaboratively define and configure a robust server environment that will utilize physical and/or virtual UNIX/Linux & Windows systems. Configure and install customized UNIX/Linux & Windows servers to meet the customer s specifications. Troubleshoot and resolve system related problems. Monitor system performance and tune the operating system to best support the application. Monitor hardware/software vendors for any required operating system patches or upgrades. Monitor hardware/software vendors for any required hardware or firmware upgrades. Monitor security advisories for operating system and infrastructure software, and take appropriate actions to safeguard resources. Page 3
Install security patches, upgrade software packages, and update system configuration to meet IT@JH best practices. Maintain operating system and supported software documentation. Backup management Define, configure, and install a disaster recovery environment if the application is added to the critical applications list Managed services are required for systems that meet both the following criteria: If the system requires external access from outside the Johns Hopkins network, and; If the system will contain any Restricted Data Cloud Server and Storage Infrastructure In most cases, managed services are hosted within a common infrastructure, consisting of virtual hosts, data storage, networking and data protection services. Where appropriate, these services are also available for customer managed, or hosted systems, as described in the Cloud Services appendix. Agreement Acceptance of this agreement indicates that the customer agrees to all IT@JH Policies and Standards, and all components of this document, including appendices found in linked documents and the Service Level Agreement Addendum that outlines specific customer costs and terms. Cost center information will be supplied on the SLA Addendum. Customer Name: Date: Customer Signature: IT@JH Representative: Date: IT@JH Signature: Page 4
Infrastructure Services Detail Web Servers Virtual Virtual Hosting Fully Managed Fully Managed Customer Managed Environmental multiple power feeds included included included available in MTW only cooling included included included fire protection included included included Network 1gbps standard connectivity included included included 10gbps network connectivity optional Optional Optional Available in hosted virtual environment dual network connections included included included single or multiple site load balancing included optional optional requires multiple servers Hardware funding for server hardware, racks included included included redundant power supplies included included included hardware support contracts included included included 7x24x365 coverage req d System Configuration Access to virtual management console included Provided by sys admin Included Windows, Unix, Linux install and config included Included customer provided multi factor sys admin authentication included Included Included Required by IT@JH policy patch and update management included included customer provided Required to use IT@JH standards basic application and web services config included included customer provided standard operating system build included included customer provided Provided as a template Active Directory / DNS management included included customer provided Required to use IT@JH standards Admin/Services account management included included customer provided database services install/config included optional customer provided may require additional fee custom operating system build optional optional customer provided may require additional fee SSL traffic encryption included optional customer provided may require additional fee Windows Operating system license cost included Included Included JH funded Microsoft contract OS and system software support contract included Included Customer provided may require additional fee Monitoring/Alerting CMDB enrollment included included customer task required for all systems inventory / monitoring agent included included customer task required for all systems standard alerting included included customer task required for all systems anti virus, periodic security scans included included customer task required for all systems pager / email alerts to specified contacts included included customer task requires configuration custom or application layer alerts included optional customer task requires configuration Mid range security/monitoring NA UpTime, QRADAR NA Additional Fee SAN Storage Features / Costs SAN connectivity included included included dedicated fiber channel network backup and restore services included included included 6 weeks data protection, 2 locations Enterprise grade virtual storage included Included Included Flash, SSD, RAID 10 disk solutions optional Optional optional Additional planning, funding required multi site data replication optional optional optional requires additional storage data snapshots optional optional optional requires additional storage Disaster recovery / disaster tolerance Off site data protection included Included Included multiple zone architecture optional Available Available configured using multiple servers support for multi site recovery & failover optional optional optional requires additional servers dedicated hardware for failover optional optional optional requires additional servers Personnel and Support named systems administrator included included customer provided Must adhere to all IT@JH policies Page 5
included included for virtual Customer is responsible for Windows/Linux 24x7x365 oncall support included infrastructure OS, security and management included IT@JH adheres to all institutional standards audit compliance included customer provided and best practices included IT@JH uses a formal change management change management process included customer provided process; customers are required to utilize the same process included Vendors are not provided admin access on an vendor coordination included customer provided ongoing basis documentation included included customer provided Use of CMDB required Page 6
Detailed Service Level Terms Help Desk / On call Teams The help desk is the primary contact for all customer issues regarding system problems and availability. The help desk maintains a directory of technical contacts to route issues to the appropriate person, 24 hours per day, 7 days per week, 365 days per year. Documentation Requirements IT@JH uses 3 systems to register and document servers and projects. These include: CMDB Configuration Management Database All systems must be registered in the CMDB. This tool is the primary record for tracking system configuration data. The CMDB consists of automatically collected data from system resident agents and manual data submitted by the project team or systems administrator. Change Management This tool tracks all changes made to a system over time, and contains an approval workflow. System changes requiring use of the change management systems are outlined later in this document. Incident Management This tool tracks all problems reported about the system and is managed by the Help Desk. Response & Recovery Times IT@JH uses a variety of tools and processes to prevent outages, monitor availability and performance and to restore data and functionality in case of problems. IT@JH teams provide 24x7x365 support coverage for production systems. If an problem occurs, call the help desk to get rapid access to the right team for your issue. The Johns Hopkins help desk will route calls to the appropriate team. Systems administrators will alert the help desk, application team and other technical teams as appropriate when a problem is discovered. Hours of daytime coverage by the primary system administrator or their backup are 8am 5pm After hours & weekend/holiday support is provided by on call support teams. Initial response to priority 1 and 2 problems is 30 minutes On site response during on call period, if required, is 2 hours Initial response to priority 3, or non critical problems is next half business day In case of delayed response, the Help Desk will escalate a problem to management Problem resolution efforts will continue around the clock until completed. Following is the typical approach to system recovery and restoration: Technical support staff first communicate with the help desk to, determining impact of a problem. Technical staff will then attempt to restore services and involve other technical staff as needed. In the event that a system requires a restore from backup, this determination is made within 8 hours. Restoration from backup is necessary if the system is not recoverable. Restore from backup will result in the loss of any data generated since the last backup. Once restored from backup, system access is returned to the customer or application team for verification, then to end users after confirmation is received. Backup & Restore Services Unless specifically directed, IT@JH maintains backups for each server it manages or hosts. Each server administrator is responsible for ensuring that backups are occurring. Reports are available to verify backup completion. When a system is restored from a backup copy, it will revert to the data and configuration present at the time of the backup. This may be the previous day or two days prior to outage, depending on the availability of a recent, successful backup. Backups occur each day, typically at night. Server performance may be lower during these times. Specialized backups are requested by working with an assigned systems administrator. Using multiple solutions to backup a Page 7
server can lead to performance and availability problems. This is a common issue with using database and server backups simultaneously. IT@JH retains 6 weeks of backup data, using disk based backups at two locations. Backup copies are used to restore a system to a recent copy, and are not intended as an archive solution. In the event of a server or database failure, the system is restored to the most recent successful backup. Restoring a server from a backup will result in the loss of any data added or changed since the last successful backup. Other data management tools, such as snapshots, replication or database transaction logging, are required to reduce the risk of data loss. The backup administrator is responsible for the overall availability of the backup system. Server administrators are responsible for Ensuring that backups are successful for each server, including the amount and type of data for each. Testing periodic restores, to ensure a server can be successfully returned to service after a failure. Delegation of Technical Administrative Rights To delegate control of customer managed systems, administrator access is provided to one or more of the following: Windows or Linux servers Active Directory Organizational Units, DNS subdomains vcenter folders and/or Resource Groups (for the purpose of adjusting server settings) vcenter datastores or storage pools (for the purpose of adjusting storage presented to individual servers) Customers are required to adhere to all Johns Hopkins IT policies and all IT@JH Standards and Guidelines, found at http://it.johnshopkins.edu/policies Remote access to a server by vendors or non IT@JH systems administrators is provided under a strict set of guidelines and communications protocols. Contact your systems administrator for the appropriate access process. Administrator or Root Access, Remote Access IT@JH adheres to all institutional policies regarding the confidentiality of data, including ephi (Protected Health Information) and all other protected information. Only authorized technical personnel and identified user communities receive access to servers. Application teams and customer representatives will not receive ongoing access to root, or administrative level rights. Domain & Directory Services IT@JH uses directory based management structures, including Active Directory and Domain Name Services (DNS). These structures provide a naming structure and various levels of security and access for server resources. Active Directory uses a common pool of user ID s that are used with access control lists (ACL s) for various server services and file systems. The creation of additional user accounts is restricted. Customers who manage servers are must obtain an Active Directory Organizational Unit to organize and manage their Windows systems, groups, service accounts and other required resources. Security Scans, Systems Management All servers managed or hosted in the IT@JH data centers are subject to scans for security vulnerabilities. Each systems administrator is responsible for updating servers with current patches, secure passwords, and best available security practices. Standards and guidelines for security practices and operating systems management are on the IT Policies web site (http://it.johnshopkins.edu/policies). A Configuration Manager Inventory agent is required for all Windows systems. Systems Center Endpoint Protection is required to reduce risk of malware infection. These and other monitoring and logging tools are available from the Enterprise Monitoring & Management Services Team. Systems Administrators will work with customers to determine the most appropriate method and schedule of patching. Standard Windows system patching options include Automated Patching of servers during test cycle recommended only for test servers. Fully automatic patching with automated reboot after test cycle used by most servers Page 8
Automated installation of patches with manual reboot used for clustered systems and other systems requiring additional attention. Scan only manual installation of patches. Used in rare cases where a system has specific software build dependencies. Use of this method must be approved by the technical area Director and the Project Sponsor at a Director level. The systems management team offers a variety of patch schedules designed to provide flexibility and minimize disruption IT@JH attempts to stay within the upgrade path provided by software vendors requiring testing and implementation of software upgrades for which the customer may receive no individual benefit. This practice provides an additional level of security for other systems in the data center environment. Alerting & Monitoring IT@JH uses monitoring and alerting systems to provide real time and historical information about the status of each server and the critical services it provides. IT@JH monitors the infrastructure technologies and the network for availability, performance, and security issues. Customers managing operating systems are responsible for monitoring those technologies, including security, operating system and any applications. Customers who manage systems are responsible for conducting network security vulnerability scans to obtain security reports. The EMMS Monitoring Guide, which covers CMDB, SCCM for Servers, Security Updates and Monitoring can be found at the following location: https://collaborate.johnshopkins.edu/sites/emms/scom/documents/emmsmonitoringguide.docx Risk Management & Disaster Planning Disaster recovery (DR) planning addresses the risk of data loss or outages exceeding 72 hours. Methods used to mitigate effects of longer outage require additional planning and engagement with the customer, application team and vendor. DR planning often requires additional costs. The IT@JH DR team is available for formal Disaster Recovery planning. Customers are responsible for preparing business continuity plans to mitigate risks related to system outages affecting normal business or clinical operations, or the loss of data. These downtime plans must also address the possibility of losing up to 48 hours of the most recent data, caused by reverting to a recent backup. Systems that require a higher degree of fault tolerance or data preservation are evaluated on a case by case basis. Risk of outage and data loss is present for all systems. Mitigation of risk is the responsibility of all parties. The architecture of an application or system will determine the level of risk. Use of technologies such as clustering, failover, multiple site recovery, data replication and redundancy are prevalent in modern IT systems. The infrastructure solutions provided by IT@JH allow for the creation of highly fault tolerant systems when designed properly. Change Management Practices Servers managed by IT@JH are categorized as being in one of 2 operating modes non production or production. Non production servers may be unavailable for at any time, with no guarantee of data integrity. A change management system tracks activities all types of activities on production systems. Use of the change management systems is required for customer and IT@JH managed systems. The types of activities subject to a change control include, but are not limited to: Turning the server off and on, or restarting the main server service. Starting and stopping any resident service. Installing any server based software. Changing access rights to any file system or application Modifying web, application and database server parameters Modifications to application software or databases Changing network device configurations between the server and the end users Any changes that may affect performance, security or stability of the system Any changes that may affecting an end user s experience with the system, particularly software changes that alter the functionality of an application residing on the server. Page 9
A change control record includes a description, potential impact, the type of change, contingency for failure, and backout procedures. In addition, changes include notification to the potentially affected users and require customer approval. The change management system uses a managerial and director approval process. Information about changes is available through the Service Manager system. Each Wednesday morning at 9am, changes are reported to the IT community via conference call and the change initiator or a designee must be present. Normal changes are discussed at the weekly change control meeting and require five business days of notice. These require manager and customer approval. Emergency changes occur when the system has partially failed and needs service, or when some type of critical failure is imminent. Emergency changes require approval by a director. Data Center Environment Servers managed by IT@JH are typically deployed in one of 2 data centers. The primary data center provides hosting for production systems and is configured with 2 zones for production failover and load balancing. Several miles away, a secondary data center provides recovery and testing functions. These centers provide a secure environment with restricted access to authorized personnel. The data centers are equipped with appropriate levels of air handling, power conditioning and centralized uninterruptable power supply (UPS). Servers are installed in racks, connected to the network by switched Ethernet network cards connected to switches, with a default 1 gigabit connectivity. Physical servers must be connected to at least two network connections and two different power feeds (Mt. Washington only). Critical equipment in the 1830 data center connects to two separate power feeds. Connectivity to storage area networks is provided by multiple fiber optic cables. The facilities are equipped with air handling, fire suppression and conditioned AC power. While rare, a data center may be subject to a power outage or other environmental problem. Scheduled maintenance outages to test power or for network and environmental changes occur on a regular basis. Some of these outages will require planned system downtime, to be communicated and scheduled in advance with customers and technical contacts. Licenses IT@JH maintains many site licenses for software used on servers to help reduce costs for customers. Microsoft products, including most client access licenses (CALs) and server licenses are included in the cost of shared services. For database software and application software, customers are typically responsible for these costs, as well as the cost of ongoing support and maintenance. Legacy & Unsupported Systems A legacy system is considered any set of software, hardware and technology that consists of one or more components (server, hardware, software, technology, etc) no longer considered current or supportable. A 3 rd party vendor or IT@JH support group may designate a system as legacy or out of support. Support for legacy systems is limited in terms of response time and problem resolution time. Specific support limitations for legacy systems are identified in an SLA Addendum. Page 10
Appendix: Enterprise Systems Management Capabilities Enterprise Systems Management hosts and manages enterprise and departmental servers on behalf of our customer base, spanning many applications and/or projects. These servers provide a wide range of services, including groupware, clinical, research, financial/administrative applications, gateway and application based web. A highly trained team of educated and experienced personnel are on staff and available for your support, consultation, proactive systems management as well as routine trouble ticket and/or problem response. Items below are additions and/or modifications to the Information Technology @ Johns Hopkins Enterprise Technologies Services Server & Storage Management Service Offerings Service Offering Enterprise Systems Management offers the following services: Server Management for departmental or enterprise level systems Project support that require windows based servers Technical liaison between other technical groups, vendors and IT teams Enterprise Systems Management offers a single support option for server management, with this option being complete and total management of the server. This service begins with the acquisition of the hardware (if necessary) through server and operating system build, production implementation and maintenance through the project/server lifecycle. The fee schedule is flexible depending on the nature of the server(s) or project that needs management. (fee schedule will reflect base service agreement pricing) There still may be the need for additional costs, like hardware or software purchases, licensing and other similarly related expenses common with server management though. All clients are encouraged to discuss their individual requirements with ESM prior to budgeting, allowing customizable solutions specifically architected to your needs. In Depth Overview of Offered Services This offering is a comprehensive server management solution for internal clients of the Johns Hopkins Institutions. Enterprise Systems Management assumes all server management responsibilities including the following: Hardware standards & specifications Implementation planning and system architecture Installation of server hardware into the appropriate physical environment Installation of operating system and current service packs Configuration of operating system according to industry and institutional best practices for the application being supported Adherence to institutional standards for installation and support of operating system Inclusion into the proper management entity (domain, AD forest) Server accounts creation, deletion and modification Automated updates with recommended patches, manual updates where required Notification of license purchase requirements and Johns Hopkins agreements for reduced rates Backup and restore of server data, including databases, configuration data and file systems 24x7x365 monitoring of appropriate server services, with interruption notification to support staff 24x7x365 human response to critical server outages resulting in a priority 1 problem, as defined by the JHMCIS Support Center. 2 hour on site response time for server outages Adherence to IT@Hopkins change control policies Disk space utilization planning, data use policies High level of coordination with vendor and application teams High degree of outage planning Disaster Recovery and Tolerance planning, coordination and testing. Emergency response using previously staged failover hardware and procedures Proactive capacity and performance planning System Administrator involvement in application loads and upgrades Coordination of security, firewall and intrusion detection with network security Installation and Configuration of Server Hardware and OS Page 11
Enterprise Systems Management uses a standard set of installation practices for each operating system it supports. These adhere to common industry best practices and are designed for supportability and high levels of security. Data and systems partitions are kept separate from one another and systems monitoring parameters are increased from the default levels. The service pack level is generally current with the manufacturer s latest release, with a 30 90 day lag time for internal testing and acceptance. Account Management Enterprise Systems Management provides all user account management for servers and users of server services. Maintenance of the central pool of user ID s and the information provided in this pool is the responsibility of Engineering Services Security. Users are provided access to resources by completing a security form obtainable from JHMCIS Security and having it approved by departmental management. Clients are encouraged to ensure that the change in status of persons with access to servers is communicated to JHMCIS security as soon as is practicable. Enterprise Systems Management will remove or change access privileges to these systems based on changes in status. All persons needing access to servers managed by Enterprise Systems Management will be required to submit confidentiality & systems access form, which indicates acceptance of all JHH and JHU policies regarding the use of Johns Hopkins owned computer resources. Patches and Upgrades Frequently vendors of operating systems and applications release upgrades, service packs or general software patches for all systems. It is the preference of Enterprise Systems Management to install all patches and service packs as quickly as possible, after a short period of testing. Although Enterprise Systems Management will install application patches, it is up to the application team to make the installation request and provide the patch installation software and any corresponding documentation. Enterprise Systems Management employs a technology that automatically distributes approved operating system patches to all servers post testing on a bi weekly basis. While Enterprise Systems Management will coordinate upgrades with the appropriate parties, including application specialists, vendors and clients, it is the responsibility of the application vendor to alert Enterprise Systems Management when specific patches and service packs are not compatible with server based software. In cases where a specific update is needed to address issues with a specific client s server, Enterprise Systems Management will install this product after discussions with the client, and the appropriate vendor representatives. Enterprise Systems Management attempts to stay within the upgrade path provided by software vendors and may require testing and implementation of software upgrades for which the client may receive no specific or individual benefit. This is done to ensure that the operating system revisions is supported by the vendor or to ensure that all servers are at the same revision level to keep management costs in check. License Management The client is responsible for the purchase and management of all application related licensing and in certain cases non customary licensing needed for complex or nonstandard architectures. Clients will generally pay for costs on a yearly basis and this should be factored in to the cost of each project. 3 rd Party Software Application groups that choose to use third party software which does not have a direct relationship towards the function of the Enterprise application itself will be the responsibility of the application team or vendor. ESM will offer support services for the intel SERVER hardware and operating system but will not be responsible for support, usage, training, licensing and associated costs with such third party software. Legacy Systems Support Enterprise Systems Management is not in a position to provide priority one support for legacy servers, hardware, software or any support for environments that run legacy application software, regardless of hardware status. ESM will make no guarantees on its ability to fix, repair, recover or limit downtime on a system deemed legacy. Page 12
Appendix: Enterprise Web Services (Hosted) Systems Enterprise Web Services hosts and manages web hosting servers on behalf of our customer base, spanning multiple web sites, applications and/or projects. A highly trained web system engineer team of industry certified, educated and experienced personnel are on staff and available for your support, consultation, and proactive systems management. Web Service engineers work with you, your vendor, and developers to optimally and securely configure web based application for internal and/or external access. Apache Tomcat, IIS, ColdFusion,.NET Provide MySQL and MSSQL backend Provide Reverse Caching Proxies for Security, Load Balancing, and Compression Manage SSL Certificates Free Domain Name Registration SSO/JHED Integration Troubleshoot and resolve system related problems Monitor vendor resources for any required operating system patches or upgrades Monitor vendor resources for any required hardware upgrades Monitor for file system intrusion (intrusion detection) Respond to monitoring alerts and client reported problems. During non business hours, support will be provided when either the hardware, operating system, or infrastructure software is unavailable or the ability to use these resources is severely degraded. Monitor security advisories for operating system and infrastructure software, and take appropriate actions to safeguard resources Implement security patches as needed System account management Document and submit change management requests for proper approval as required. Change Management is required for any change that may impact end users. Security patches, upgrade software packages, and update system configuration to meet IT@JH best practices Firmware upgrades as required Maintain operating system and supported software documentation System level housekeeping activities to ensure systems are operating at optimal levels Backup management Requesting firewall configuration Appendix: Customer Managed (Hosted) Systems Where appropriate, IT@JH provides a hosted service for infrastructure services. In these cases, the core systems management and operating system management responsibilities are provided by the customer organization, through customer supplied Windows and Linux Systems administrators and engineers. This document outlines the methods and practices used by IT@JH systems management personnel. For customermanaged systems, the Systems Administrator/Engineer provided by the customer is responsible for adhering to all of these methods and practices, as well as all relevant Johns Hopkins audit, compliance and regulatory policies. Customer using hosted services work with the Cloud Services team as a primary point of contact to other IT@JH teams. To request these services, email cloudrequests@johnshopkins.edu. Additional teams are responsible for incident management, change management, documentation standards, networking support, storage and virtualization hosting services and monitoring/alerting systems. On customer managed servers, the customer is responsible for operating system installation and configuration. Templates, or standard operating system builds are available for Windows. The customer maintains full responsibility for configuration, security an ongoing server management, including the procurement of 3 rd party support for operating systems, databases and other server based components and applications. Page 13
Server Virtualization Team This team is responsible for the server virtualization infrastructure including the following items: Virtual Host and cluster capacity planning and infrastructure architectural design Server virtualization configuration and resource pool creation and allocation Engineering and technical support of the virtual hypervisor OS Hardware standards, configuration and engineering technical support Automation of various engineering processes involving virtualization Host monitoring and alerting using vsphere Operations Manager Management of virtual environment network VLANs and security policies in coordination with Networking Virtual machine migration processes Coordination of all issues that require virtualization (i.e. VMware) support; internal and external Planning and maintenance of virtual infrastructure upgrades, including hypervisor upgrades and host firmware upgrades Management of virtualization administration consoles, security roles and management servers. All virtual infrastructure storage requests and configuration Communication and implementation of best practices in VM configuration and management in a virtual environment (VM Tools versions, Network configuration, Storage Configuration) Approval of all third party software for virtualization integration 24x7x365 oncall engineering support for hardware and virtual environment Research, development and implementation of new virtualization technologies and hardware solutions Storage / Data Protection Team This team is responsible for the following: Storage Network and Storage Array configuration and management Storage pool allocation Storage pool capacity management and monitoring Storage quota management Backup System management and configuration Overall management of the enterprise backup system including media servers, storage arrays, access control, and schedules Distribution of daily backup reports Education and information related to backup options and configuration Communication related to upgrades, incidents, best practices, and standards Provide console access so that users are able to self service Setup proper alerting and monitoring of the system Ensure that backup jobs are running and address any issues that are seen on a daily basis. Appendix: Storage Offering Details All current storage offerings are housed in managed data centers with redundant power and service contracts that provide 4 hour response time for disk and controller failures. The key differences in storage offerings are noted below. Enterprise Class Storage Enterprise class storage is our highest performing SAN storage option. This is typically used for databases, virtual servers, critical applications and other uses where performance is an important factor. The storage systems are typically using smaller high speed drives, fast internal connectivity and utilize disk cache and SSD to improve performance. Software on the arrays allow for different options for storage protection and redundancy such as snapshots, mirroring and replications. Examples of current hardware used for this option are IBM DS8800, XIV, 3PAR. NAS Storage Page 14
NAS storage is a storage option presented over the network. NAS storage is an enterprise class replacement for individual file servers. The NAS environment is built with capacity and growth in mind. NAS is used for departmental shared drives and bulk storage where performance requirements are low. Access is controlled by Active Directory security groups and subfolders can be restricted to only allow access to authorized users. Quota allocations can be purchased in 100GB increments. We maintain several snapshot policies to help protect your data. You may elect to protect your data for 6 weeks, 2 weeks, or 2 days. All snapshot space will count against quota. Snapshots can be viewed using windows previous version functionality. We currently have an EMC VNX and an EMC Isilon solution in place to support this environment. Home Directory Services Provides network file storage to IT support groups and individuals within the Johns Hopkins institutions. This provides a centrally stored, safe and reliable location for user files and frees the support staff from the time and cost of maintaining separate storage and backup systems. No technical background is required. 6GB of storage space is provided for each user Home Directory. Home Directories are secure and are only accessed by the user to which they are assigned. These directories are configured to automatically map at logon using the Home Directory path in the user s Active Directory profile. Discontinued Services Commodity Secondary Storage originally provided as a 2 nd or 3 rd copy without backup, this solution is no longer offered Backup Services (a la carte) some customers use the enterprise backup system from a customer managed data center. These services are integrated into virtual server and data center offerings, but no longer offered separately. Existing backup only customers will be able to continue to use these services, or may migrate to a virtual hosting environment Page 15
Appendix: Virtual Desktop Offering Cloud & Virtualization Services has a standard offering of Windows7 virtual desktops. Hopkins standard application software is pre loaded. This includes Microsoft Office 2013, Adobe Reader, IE10, Google Chrome and other utility software. It utilizes the standard Enterprise Client Image to ensure all included software is agreed upon by enterprise IT groups. Current specifications: 1 CPU 2 Gb RAM 40Gb Hard drive Appendix: Citrix XenApp Application Virtualization Cloud & Virtualization Services administers a large, disaster tolerant Citrix Farm. Applications can be made available to groups that are geographically diverse or groups that utilize different types of client operating systems. The application can be delivered to any operating system (including tablets and smartphones) via the Citrix Receiver. Most applications can be made available via Citrix. Occasionally, some modifications are required to emulate a standard endpoint installation. Requirements: Initially a request to Cloudrequests@jhmi.edu is required to instantiate the process. Software necessary to install the application Installation instructions Custom requirements (i.e. policy driven controls, ancillary hardware to be used with application, etc.) Security access process Appendix: Useful IT@JH contacts In most cases, the request process addresses indicated above are the most effective way to obtain services. However, for more specific situations, the following addresses are provided as a useful reference: Cloud & Virtualization Services Web Hosting Mid Range (Unix/Linux) CMDB, Security Updates, Monitoring Data Center Network Domain Name Services (DNS) Active Directory Network security/firewalls: Data Center Engineering ESSO, SafeNet, Web Single Sign on: VPN (Virtual Private Networking) SSL Certs Disaster Recovery Team cloudrequests@johnshopkins.edu webhosting@jhu.edu midrangets@jhmi.edu monitoring@jhmi.edu DCN@jhu.edu Hostmaster@jhmi.edu ad@jhmi.edu Network.Security@jhmi.edu DCE@jhmi.edu enterpriseauth@jhmi.edu vpnadmins@jhmi.edu pkiadminis@jhmi.edu IT_JH_DR@jhmi.edu Page 16
Appendix: Midrange Team Midrange Team SLA Addendum to the IT@JH ETS Service Offerings Unified SLA Updated: February 25, 2015 This document is intended to provide internal Johns Hopkins clients with information regarding the service and support capabilities of the Midrange System Team, as it relates to the management of servers in the Johns Hopkins networking environment. Items below are additions and/or modifications to the Information Technology @ Johns Hopkins Enterprise Technologies Services Server & Storage Management Service Offerings. Sections below are paired between the Midrange Addendum and the Unified SLA. Installation & Configuration of Server Operating System The Midrange team should be involved in the system design of new application environments that will utilize UNIX/Linux systems to insure that the servers integrate successfully into our existing environment and are capable of providing the level of support needed by the application. Installation of the operating system, server network configuration, and all system level tasks will be performed by the Midrange Team. Most operating system builds are custom builds by the Midrange team to meet specific needs of customer s applications. The Application admin team will provide all operating system and operating system related ancillary software requirements for their application. A test environment must be in place to support the application. This environment will allow for testing of operating system, configuration, and/or application updates without impacting the production environment. The Midrange Team does not provide installation, troubleshooting, or support for databases. Operating System Patching & Maintenance The Midrange team has adopted a conservative strategy for operating system patch management typical of large scale UNIX implementations. This insures the integrity of the system, high application availability, compatibility with application components and/or databases, proper patch testing, and compatibility with hardware microcode. Operating system patches and/or hardware microcode updates will be done on an as needed basis. This includes, but is not limited to: o An upgrade or patch to an application may require operating system or component patching. o The operating system vendor may drop support on an old version of the operating system resulting in an O/S upgrade. o The application vendor may have patches to resolve problems that include operating system related fixes. As security vulnerabilities become known security patches will be implemented accordingly. An agreed upon system maintenance window should be established. Change Management Practices Customer calls for application failures/outages/errors should go to the Help Desk and an appropriate incident should be opened and assigned to the Application Admin team. Initial triage of application problems will be completed by the application admin team. If they or the vendor determine that the problem is related to the operating system the ticket should be reassigned to Midrange and a Midrange Team member will work with them to continue diagnostics and resolution of the problem. Vendor problem tickets associated with an application will be opened with the application vendor by the Application Admin team. Operating system related vendor problem tickets will be opened and managed by the Midrange Team. Administrator or Root Access, Remote Access Page 17 In order to insure the integrity of the system, root access will be limited to the Midrange Team. During application installation, customization, and configuration (the application build process) the root password may be made available to the application admin team for an agreed upon time period.
In an emergency, when root access is needed quickly, the Midrange on call person is to be called to grant this access. All access to servers will be done via encrypted program (ssh). Managed Hardware A 7X24 hardware support contract must be in place providing hardware support for production servers. A lower level of coverage is acceptable for non production servers if the application owner is willing to accept the down time and risk. An operating system support contract must be completed providing 7X24 support for all production servers. Alerting & Monitoring UpTime is the standard monitoring package for all servers the Midrange team supports. Customers will need to purchase a license of UpTime for their servers and pay the annual maintenance fee. Application related alerts can be configured in the UpTime application and should be sent to the Application Team on call pager. Operating system based alerts will be sent to the Midrange team and the on call pager. Production UNIX servers will have a Tenable (security) scan run monthly. These scans will analyze the server to help prevent potential security holes. The systems engineer will work with the Application Admin team to resolve any outstanding security exposures. IBM s Security QRadar SIEM is used to collect and consolidate log source event data from production UNIX servers to distinguish real threats from false positives. IBM s AIX OS AIXPert and Linux s SELinux can be utilized to harden and secure system related access control policies. Page 18