Enabling Trusted Multi-Tenancy with Vblock Systems

Transcription

1 Enabling Trusted Multi-Tenancy with Vblock Systems Version 1.0 March 2015

2 THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." VCE MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OR MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright 2015 VCE Company, LLC. VCE believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. 2

3 Contents Introduction... 5 Document purpose... 5 Scope... 6 Audience... 7 Feedback... 7 Secure segmentation... 8 Tenancy boundaries... 8 Risks, exposures, and controls... 9 Tenant platform administrator Tenant tenant Platform administrator supplier Controls VCE trusted multi-tenancy design Segmentation options Trusted multi-tenancy technology overview Vblock System components Vblock System layers and secure segmentation Physical segmentation Design considerations Network layer separation Disjoint layer Dedicated port channels Storage separation Dedicated disk pools Dedicated disk array enclosures Compute separation Dedicated blades Dedicated Cisco UCS chassis Dedicated Cisco UCS domains Management and physical separation Logical segmentation Network separation Virtual local area networks Virtual Machine Fabric Extender Virtual Device Context Storage separation Virtual storage area networks

4 Zoning Virtual Data Mover LUN masking Virtual machine file system management Compute separation Cisco UCS service profiles Cisco UCS organizations Virtual local area networks Cisco UCS Virtual Interface Card Virtualization Cisco Nexus 1000V Switch VMware vsphere Cisco Cloud Services Router 1000V Additional considerations Security and compliance Network security Endpoint security Role-based access control (RBAC) Data-at-rest encryption Management models Data center management Tenant management Scaling Combined segmentation Combined segmentation design Combined segmentation design Conclusion Next steps

5 Introduction Organizations are consolidating as much infrastructure as possible into the simplest to manage, smallest possible footprint, such as a single VCE Vblock System. This consolidation frequently includes workloads that might have existed in physically discrete security boundaries supported by separate operational silos. This document explores variations on this consolidation scenario. Physically discrete infrastructure traditionally met organizational security objectives; however, it often led to inefficiencies in terms of underuse of resources and vendor sprawl, and adversely impact business agility in IT processes. Trusted multi-tenancy (TMT), as implemented using Vblock Systems, addresses the mechanics of creating secure boundaries between tenants in different trust zones. Individual tenants or workloads, application tiers, applications, DMZs, departments, partners, customers, missions, and so forth likely have business requirements mandating specific segmentation needs regarding administrative, network, compute, and data access. In some cases, parts of the management or infrastructure might be shared with other tenants. For example, a service provider (or perhaps enterprise IT) might handle all administration tasks with a common management framework, while other resources are largely dedicated to and managed by tenants (or departments). Meeting the differing segmentation needs of these diverse environments requires flexibility and design modularity. A range of capabilities is available to enable this modular trust model, providing controls supporting separation and protection at various levels of administration, communications, and access. The modular trusted multi-tenancy model is implemented using core Vblock System controls, along with optional technologies from partner products. In practice, similar products can be substituted for the optional products discussed in this document, although overall solution interoperability and/or security profile might be impacted. Document purpose This document examines controls both inherent to Vblock Systems and provided through partner products that enable secure or trusted multi-tenant deployments. As segmentation can mean different things, this document establishes segmentation boundaries and definitions, and explores risks and fundamental mitigation considerations. Particularly important is an understanding of two major risk classes: Lateral tenant influence -- resource leakage between tenants, either with information flow consequences and the possibility of further resource allocation tampering (for example, cracking cryptographic keys through monitoring CPU activity), or as a volume issue, impacting service availability, variants commonly known as the nosy neighbor and noisy neighbor problems. Vertical tenant influence -- using one layer within an architecture as a privileged jumping off point for an intruder to attack other tenants. 5

6 Applying segmentation to Vblock Systems requires an understanding of the Vblock System, its components, and their capabilities. Vblock Systems are highly adaptable platforms, but the baseline must be well understood for maximum effectiveness in deployment for multi-tenancy scenarios. This document examines some of the controls available in Vblock Systems networking, storage, compute, and management domains: Controls that provide separation that is physical, such as dedicated hard disks and dedicated servers Controls implemented at a logical level, such as virtual local area networks (VLAN), virtual storage area networks (VSAN), and virtual firewalls Common combinations of controls and how VCE has worked with customers to turn catalogs of capabilities into functional systems This document also explores topics having either a variety of potential solutions at the physical or logical levels or design considerations that apply through the architecture assembly, such as encryption and scaling. Scope This document assumes a business decision has been made to leverage one or more Vblock Systems to provide services to multiple tenants. There are many important considerations in designing information systems and security requirements to consider early in the system development process. Designing a shared infrastructure solution for multiple tenants can be a complex task. This document provides high-level guidance to help understand and address key security concerns for a multi-tenant Vblock Systems solution. VCE recommends customers consider a VCE professional services engagement to address out-of- scope areas such as: Scale: One tenant, tens, hundreds, thousands? How will things look over time? How important is flexibility? Tenant characterization: What do tenants need? Expect? What are their compliance requirements? Resource expectations? Bursting needs? Supplemental services: What additional functionality are tenants going to want, such as authentication, application delivery controller (ADC), name services, and so forth? Data protection (BC/DR): How do you plan to handle things going wrong? How do tenants expect you to handle things going wrong? And what is the geographic footprint of your operations? Can you make more money off of a more comprehensive approach? Deployment models: Are tenants integrated manually or will there be an infrastructure/platform/software/communications-as-a-service offering? Workspace management model: Following on with the deployment model, who manages what for whom through what interfaces? How much do tenants expect to be able to do for themselves? Consider areas like infrastructure auditing, which is traditionally awkward for service providers. Infrastructure management: Patching and release management can be especially challenging, particularly when coordinating with a large number of parties. VCE s Release Certification Matrices (RCM) de-risk technical aspects of this, but procedurally it is potentially a complicated topic in single tenant environments and more so in multi-tenant settings. 6

7 This document does not address basic security controls which are assumed to be applied as appropriate unless specifically relevant to trusted multi-tenancy. The document looks briefly at front-end aspects of multitenancy, such as portals and orchestration tools. As these products work as expected on Vblock Systems, the focus is on areas that might be new, such as multi-tenancy aspects of the core infrastructure. As this document focuses on the core infrastructure components, it also does not address software-defined networking (SDN) elements. Audience The intended audience for this paper is data center and security architects tasked with implementing multitenancy upon an infrastructure built using one or more Vblock Systems. This document does not provide detailed implementation guidance. Readers are assumed to have a reasonable understanding of Vblock Systems architecture (as documented in the various VCE Vblock System architecture overview guides and related product documentation.). Feedback To suggest documentation changes and provide feedback on this paper, send to docfeedback@vce.com. Include the title of this paper, the name of the topic to which your comment applies, and your feedback. 7

8 Secure segmentation Examining multi-tenancy security requirements typically involves looking at the relationships between all parties in the infrastructure, including tenants and platform administrators. This document uses that perspective to address both secure segmentation within Vblock Systems and the concepts of tenancy and potential threat vectors. Tenancy boundaries Infrastructure resources can be characterized by the following: How the resource (including applications) is managed, monitored, and used, and any constraints on resource management How associated data (including backups, snapshots, business continuity/disaster recovery data, and so forth) is stored and any constraints on storing or accessing data How associated applications are consumed, in what trust zones consumers are located, protections regulating consumption, and what trust or protective elements need to be provided locally or assumed/inherited from the infrastructure Multi-tenancy arises when the differences in characteristics for two or more blocks of infrastructure resources become too large to resolve through simple process. These resource blocks frequently referred to in frontend orchestration platform as containers become tenants. Think of each tenant as having some unique although potentially similar set of agreements or contracts with platform administrators and other entities relating to management, data, application access, and so forth, effectively regulating security, performance, and, possibly, scaling. Note: This document uses the concept of a contract similar to how it is used by software-defined networking vendors: as a mutually agreeable high-level policy directive used to shape specific implementation details between two parties. Although not necessarily a security concept, this document focuses on contract areas related to trusted multi-tenancy. The following are some examples of how blocks of resources might be designated as tenants based on characteristics: Different parties with financial responsibility for the resources; for example, different companies (traditional service provider scenario), different divisions in an enterprise Different categorizations within a compliance framework; for example, one block of resources hosts a PCI Cardholder Data Environment (CDE), while another block does not Different tiers within an n-tier application (dedicated tiers), or the more traditional legacy model of shared tiers for DMZ, application servers, databases, and so forth, each of which might have full set of compute, networking, storage, security, and, potentially, management, resources 8

9 Acting on behalf of the tenant, the data owner has the best perspective on the tenant s security requirements these requirements being the basis for a subset of the above-mentioned contracts. The data owner s responsibilities for championing the security interests of the tenant might be shared with or delegated to other entities, such as the corporate security organization in a private cloud multi-tenancy scenario. However, ultimate responsibility remains with the data owner for appropriate treatment of tenant resources. Data owners or their delegates are strongly encouraged to challenge assumptions and resulting contracts that seem overly complex or where it is unclear how the technology can support the contract. Risks, exposures, and controls The following risk-related terminology is used in this document: Term Risk Exposure Control Definition The probability that a threat agent will (act upon a threat vector to) exploit a vulnerability The impact, typically expressed financially The countermeasure or safeguard that eliminates, reduces, or transfers risk associated with one or more exposures A more detailed perspective on threat agent => safeguard cycle 9

10 Tenants specifically, the data owners have a responsibility to identify and take appropriate steps to protect valuable information. As this information is typically the reason the infrastructure exists, it makes sense to first examine the risks and exposures inherent to tenant contracts. Management security also has a significant impact on infrastructure security. Therefore, this document explores the risks and exposures inherent to management contracts that do not directly involve tenants. Note: Although the focus of this document is technical, there are areas where the appropriate path is not purely technical but rather an administrative or procedural control to better safeguard certain exposures. While VCE can advise in creating and implementing such controls, they are not addressed in this document. This section outlines multi-tenancy security expectations organized by relationship that loosely corresponds to key threats and threat agents or, in some cases, jumps right to countermeasures (typically for areas this document will not cover in more depth). However, without organization-specific risk factors and asset/data valuation, gauging the importance of specific risks and determining an appropriate investment strategy in compensating controls is an arbitrary exercise. VCE can work with you to help understand your risks in the context of Vblock Systems and to prioritize mitigation of those risks. Tenant platform administrator The following table describes, in the form of contracts, some security relationships and expectations between tenants and platform administrators, such as service providers or centralized IT organizations. Note: Use the contract tables in this section as a template when building a model for assessing specific multitenant deployments. By building out and adapting the contracts to specific deployments, the scoping details become easier to manage for compliance objectives such as PCI. The contracts template becomes a tool in the ongoing assessment process. Contract Provider mitigates against virtual machine escape Provider implements robust administrative and procedural controls for infrastructure Discussion The targeted threat is based on the idea that, having gained a foothold within a virtual machine, an attacker could perform a privileged attack upon the hypervisor and use it to access other resources, including those belonging to other tenants. There are trusted multi-tenancy architectures that remove a large portion of the virtual machine escape risk specifically in the multi-tenancy context simply by isolating tenants to their own hypervisors. Note that attacks over service networks and against storage would have different risk characteristics after a successful virtual machine escape, so the risk is not completely mitigated. Encompasses a range of threats; the specific concerns vary depending on the nature of the multi-tenant relationship, the assets being protected, and the applicable compliance frameworks. Key areas include: Physical security for shared infrastructure, management infrastructure, monitoring resources, business continuity and disaster recovery, and so forth. Background checks and appropriate hiring protections. Change management practices. Is the provider more fail open or fail closed? If something goes wrong how does it align with the tenant s business posture? Incident response plans. Such plans must include scenarios for malicious insiders. 10

11 Contract Provider mitigates against lateral attack (or leakage) via storage Provider mitigates against lateral attack (or leakage) via network beyond what be possible from outside Provider takes appropriate efforts to mitigate against signals intelligence attacks Discussion Storage sub-systems within Vblock Systems might include one or more highly available storage arrays connected via traditional fiber channel SAN switches. These storage subsystem(s) are heavily leveraged due to the availability, performance, and broad spectrum of data services offered by these solutions (for example, flash, auto-tiering, encryption, deduplication, replication, and so forth.). As a result, storage sub-systems and their associated management infrastructure are high-value targets for attackers and careful consideration of threat potential must be taken in the solution design. When properly hardened, storage systems are difficult to attack, even after virtual machine control is obtained using a datastore hosted on the SAN. Hosts within a private network (whether delivered within a traditional enterprise or a cloud) achieve efficient operations by relying on management networks for monitoring, backup networks (perhaps replication or similar networks) for high availability services, and potentially many special-purpose networks that might have large footprints within the data center. In a shared environment, an attacker might reasonably expect that these networks would exist and that one or more might bridge multiple tenants. The provider can mitigate risk by preventing traffic across any of the service networks between two tenants. In the past, simply sharing a host with an application performing frequent cryptographic activities (for example, SSL encrypts/decrypts) has been sufficient to extract the secret keys, based on changes in access to the CPU. Elasticity exposed can also be revealing, perhaps showing business prematurely won or lost. Tenant tenant The following table describes sample security relationships and expectations between tenants. Note that responsibility for implementing these expectations falls largely to the platform administrator. There are two clusters of scenarios to consider for tenant-tenant interactions: Traditional service provider environments, where each tenant is a unique paying customer Central IT environment, where a tenant is a more flexible concept Service provider scenarios Contract Traditional tenant cannot see evidence that other tenants exist Discussion Management tools, disk, and resource contention should not reveal any form of data about another tenant unless it is an explicit business model; for example, multiple tenants under the control of the same consumer. Special provider tenants such as service level agreement monitoring systems might have visibility requirements into tenant resources, but there must be tight constraints. The tenant should know what the provider has access to and under what circumstances. 11

12 Central IT scenarios Contract Requirements vary Discussion Unlike the provider model, tenants within the same company or organization may need to interact frequently. The degree of separation between tenants can vary widely, even within a single deployment. For example, a government entity might maintain a very high level of separation between missions while also treating the application tiers within a mission as subtenants with a lesser degree of isolation. Special tenants, such as central security management systems or network management systems, need privileged access along certain vectors. Data owners or their proxies should ensure such access is compatible with mission requirements. Platform administrator supplier The following table describes the security relationships and expectations between platform administrators and suppliers. Contract Provider secondary relationships are insulated from tenants Discussion The platform administrator likely contracts with suppliers having network, remote, or physical access to the infrastructure or to secondary infrastructure critical to its operation (for example, power, cooling, network uplink, certificates). The provider is responsible for implementing and maintaining the technical, administrative, and procedural controls that remove these risks. The tenant is responsible for assessing thoroughness against their requirements and ensuring that an appropriate validating entity can attest to implementation of the controls. Controls Controls are technical, physical, procedural, or administrative measures collectively intended to mitigate risks. In many cases, the alignment is not direct as a combination of controls might address a handful of identified risks. Additionally, controls from different categories might come together to reinforce one another, as in this example on operations hygiene: 1 Specific VCE technology choices (Release Certification Matrix (RCM), the related compliance testing capability in VCE Vision Intelligent Operations, and its RCM pre-positioning capabilities) reduce testing, clutter, web downloads, risky behavior, and potential bad hygiene on production management systems. 2 Processes encouraging testing outside of production, staging on transient systems, and so forth, reduce risk through behavior optimization. Ideally, it would be enough to tell people to do things properly but in practice, simultaneously making it easier for them to do the right thing yields superior results. From the security perspective, making it harder for them to do the wrong thing reduces risk and increases trust. 12

13 VCE trusted multi-tenancy design The VCE trusted multi-tenancy approach is based on the concepts that each organization is different with significant diversity in its threat environments and observed risk appetites. VCE uses a modular approach to multi-tenant security to help organizations identify the right path to their needs, balancing security, compliance, cost, business models, and other requirements to build a tailored solution using predictable building blocks that provide supportability and consistency. VCE Vblock Systems are manufactured blocks of data center capacity supported by a single telephone call. Trusted multi-tenancy adds layers of capability reflecting an organization s particular needs, but fundamentally remains the same set of platforms. Segmentation options Segmentation controls enforcing the divisions between tenants are critical to the trust aspect of trusted multitenancy. Various physical or logical mechanisms can enforce segmentation, making available pools of dedicated and/or shared resources. Due to the modular nature of trusted multi-tenancy, many organizations choose to combine options to build an overall solution. In fact, some of the individual technologies covered in this document such as disjoint layer 2, encompass elements of physical and logical segmentation. Native physical segmentation options include discrete Cisco UCS domains, dedicated compute nodes, and dedicated storage pools. While there are only so many practical ways to insert air between two physical components, the range of possible logical segmentation options is considerable, particularly when considering the ecosystem. Native logical segmentation options include Cisco UCS profiles, VMware vcenter clusters and resource pools, Virtual Data Movers and VSAN, Cisco UCS Virtual Machine Fabric Extender (VM-FEX), VLAN, and port channel controls. This document addresses each of these in more detail. The native capabilities of Vblock Systems are only the starting point as trusted multi-tenancy deployments often involve ecosystem partner and/or third-party activity to bring the platform to full operation. This document addresses some of the most frequently used segmentation controls introduced through the ecosystem, such as virtual firewalls. Compliance regimes or best practices such as data-at-rest encryption might require that other areas be addressed. These might have complex answers, depending on specific needs, hardware preferences, ecosystem partnerships, and so forth. It might be possible to address an organization s needs with native capabilities, but where that is not feasible, VCE can help provide a product or trusted partner to meet the needs. 13

14 Trusted multi-tenancy technology overview The Vblock System from VCE is the world s most advanced converged infrastructure one that optimizes infrastructure, lowers costs, secures the environment, simplifies management, speeds deployment, and promotes innovation. The Vblock System is designed as one architecture that spans the entire portfolio, includes best-in-class components, offers a single point of contact from initiation through support, and provides the industry s most robust range of configurations. VCE offers a converged infrastructure portfolio that includes a variety of Vblock Systems, including the Vblock System 200 family, Vblock System 300 family, Vblock System 500 family, and Vblock System 700 family. Each family has varying compute and storage platforms and scalability. As engineered products, these systems use common architectures. The question of how the architectures handle storage networking is very helpful when designing a trusted multi-tenancy solution. Note: VCE also supports larger architectures and SDN offerings from Cisco and VMware. As most controls implemented through these solutions focus on the upper orchestration level of the stack, these are not currently in scope for this document. Contact VCE for more information on multi-tenant solutions based on VCE Vscale Architecture, Cisco Application Centric Infrastructure (ACI), or VMware NSXv. Vblock System components The following table summarizes the components used in the larger Vblock Systems families. Optional components are listed in italics. Technology Vblock 300 family Vblock 500 family Vblock 700 family Network Cisco Nexus 9000 Series Switches Cisco Nexus 9000 Series Switches Cisco Nexus 9000 Series Switches Cisco Nexus 5000 Series Switches Cisco Nexus 5000 Series Switches Cisco Nexus 5000 Series Switches Cisco Nexus 3000 Series Switches Cisco Nexus 3000 Series Switches Cisco Nexus 3000 Series Switches Cisco Nexus 1000V Switch Cisco Nexus 1000V Switch Cisco Nexus 1000V Switch Cisco MDS Switches Cisco MDS Switches Cisco MDS Switches Storage EMC VNX EMC XtremeIO EMX VMAX EMC VPLEX EMC VPLEX EMC VPLEX EMC RecoverPoint EMC RecoverPoint EMC RecoverPoint 14

15 Technology Vblock 300 family Vblock 500 family Vblock 700 family Compute Cisco Unified Computing System (UCS) Cisco UCS 5000 Series Blade Server Chassis Cisco UCS B Series Blade Servers Cisco UCS 6000 Series Fabric Interconnect Cisco UCS Cisco UCS 5000 Series Blade Server Chassis Cisco UCS B Series Blade Servers Cisco UCS 6000 Series Fabric Interconnect Cisco UCS Cisco UCS 5000 Series Blade Server Chassis Cisco UCS B Series Blade Servers Cisco UCS 6000 Series Fabric Interconnect Virtualization VMware vsphere VMware vsphere VMware vsphere Management VMware vcenter Server VMware vcenter Server VMware vcenter Server Cisco UCS Manager Cisco UCS Manager Cisco UCS Manager EMC Unisphere Manager EMC Unisphere Manager EMC Unisphere Manager Cisco Data Center Network Manager Cisco Data Center Network Manager Cisco Data Center Network Manager VMware vcloud Director VMware vcloud Director VMware vcloud Director VMware vrealize Automation VMware vrealize Automation VMware vrealize Automation Cisco UCS Director Cisco UCS Director Cisco UCS Director Security Cisco Virtual Security Gateway Cisco Virtual Security Gateway Cisco Virtual Security Gateway VMware vcloud Networking and Security* VMware vcloud Networking and Security* VMware vcloud Networking and Security* Cisco Virtual Adaptive Security Appliance (ASA) Cisco Virtual ASA Cisco Virtual ASA *Included as a component of VMware vcloud Suite VCE ships Vblock Systems in both traditional and unified storage networking configurations. The following table highlights the differences between these architectures. The Cisco MDS switch provides more flexibility in trusted multi-tenancy scenarios. Architecture Features Traditional modular Uses Cisco MDS switches for SAN networking Advanced security features available Simplifies role administration for IP and SAN management Higher physical port capacity Unified modular Uses ports on existing Cisco Nexus switches for SAN networking Fewer devices, fewer ports to manage 15

16 Each Vblock System ships with an Advanced Management Platform (AMP) that provides the resources needed to configure and troubleshoot a Vblock System in the event of a problem. The AMP helps missioncritical resources rapidly return to operation. It can be enlarged and used to run a more general set of management workloads, but it is usually recommended that anything beyond VMware vcenter Server be allocated its own space on a Vblock System rather than be placed on an AMP. As the AMP hosts element managers, it is a security target and needs to be protected. Most AMP variants are locked down and cannot have additional tools installed, the exception being the high-availability (HA) AMP. However, many trusted multi-tenancy scenarios involve higher levels of interaction with AMP resources and the installation of additional security or monitoring tools that might need to be close to element managers, such as VMware vcenter. Therefore VCE strongly recommends using the HA AMP for Vblock Systems used for trusted multi-tenancy. Vblock System layers and secure segmentation VCE s multi-tenancy approach addresses secure isolation and segmentation at each layer of the Vblock System, as shown in the following table. Component Network Storage Compute and virtualization Management Description Multi-tenancy concerns can be addressed at multiple levels within the network infrastructure of the Vblock System. Various methods can enforce network separation, including disjoint layer 2, virtual and physical firewalls, and VLANs. By default, data within Vblock Systems is accessed through block-based mechanisms (storage area networks (SAN)). It can optionally be accessed through file-based network attached storage (NAS), such as CIFS or NFS. Segmentation can occur within the storage itself or in transit. Options for segmenting block storage include SAN zoning and LUN masking. Options for segmenting file-based storage include VLANs and Virtual Data Movers. Encryption is always an option as well. Multi-tenancy concerns are addressed at multiple levels within the Vblock System compute and virtualization infrastructure, including the Cisco Unified Computing System and the VMware vsphere Hypervisor. The Vblock System AMP provides a single management point, enabling organizations to monitor and manage the health, performance, and capacity of Vblock Systems. It provides fault isolation for management, eliminates resource overhead on the Vblock System, and provides a clear demarcation point for remote operations. The AMP hosts technologies such as VCE Vision Intelligent Operations, Cisco Data Center Network Manager, Cisco Nexus 1000V Virtual Supervisor Modules, EMC Unisphere, and the EMC Secure Remote Services storage monitoring system. VCE Vision Intelligent Operations is part of the management environment and provides essential secure operational hygiene functions such as backing up configurations, additional logging, discovery, and compliance reporting for firmware versions and hardening configurations. 16

17 Physical segmentation Traditionally, data center architectures contain dedicated stacks of infrastructure providing isolated environments for individual tenants, each with their own dedicated physical resources. The physical segmentation options available on Vblock Systems allow a shared infrastructure to provide much of the isolation of a traditional standalone architecture while continuing to leverage the benefits of converged infrastructure. Trusted multi-tenancy high-level design VCE's multi-tenancy approach empowers IT administrators to choose from among a range of cost effective mechanisms to separate the tenants within a Vblock System while the tenants share a common infrastructure. The VCE multi-tenant architecture for physical segmentation provides: Increased dedicated hardware resources for each tenant Increased path isolation Increased resource usage at lower cost Ability to meet organizational requirements for separating data when physical separation is mandated Different tenants on the same multi-tenant platform Design considerations While physical separation options exist at multiple layers within Vblock Systems, there are trade-offs between physical separation and effective resource usage. Organizations with tenants requiring physical separation between network, data, compute, and applications require dedicated underlying IT resources for those tenants, which might impact solution scaling and resource usage efficiency. Organizational objectives will drive the degree of separation between the compute, network, storage, and management infrastructure. 17

18 Choose a modified level of physical isolation based upon organizational policies and business objectives. Consider the following design decision points and the extent to which each is individually necessary to achieve overall segmentation objectives: Network layer separation Storage separation Compute separation Management Network layer separation Network layer separation is one of the first important steps to take to build a trusted multi-tenancy network. VCE's trusted multi-tenancy approach provides the same degree of tenant isolation as a dedicated infrastructure. The following controls are available for physical network separation: Disjoint layer 2 Dedicated port channel Disjoint layer 2 Organizations can consolidate workloads on a single converged infrastructure using disjoint layer 2 networks. Disjoint layer 2 provides separation between compute, storage, and network resources. The VCE trusted multi-tenancy approach enables traffic isolation at either the Cisco Nexus switches or Cisco UCS fabric interconnects, depending on an organization's traffic isolation and traffic management requirements. The simplest approach is to use the Cisco Nexus platform as the demarcation point for traffic isolation using trunk connections to forward traffic to the different tenant networks. Although the VLANs cross a shared set of connections, traffic is not forwarded between VLANs unless a device external to the Vblock System is configured. The Cisco Nexus platform switches have a number of reserved ports that can be used for connectivity to different networks, which does not affect Vblock System scalability. Another approach for traffic isolation close to the server is to configure trunk connections from the Cisco UCS to the tenant networks using discrete uplink connections per network. However, this approach requires that each additional network have connections at the Cisco UCS 6200 Series. This reduces both the number of available ports for Cisco UCS Blade Server Chassis connections and the number of servers the Vblock System can support. 18

19 Disjoint layer 2 network upstream Dedicated port channels Standard Vblock Systems have a single port channel for each chassis assigned to eight physical ports per chassis. Implementing the VCE trusted multi-tenancy approach, dedicated port channels can be assigned for each tenant, ensuring that the tenant's network traffic flows across one physical path as compared to a single port channel in the standard Vblock System network architecture. Storage separation The main objective of storage separation is to provide an isolated storage environment where no tenant can access another tenant's data. This principle includes but is not limited to separation of data at rest and separation of data in motion. The following controls are available for physical storage separation: Dedicated disk pools Dedicated disk array enclosures 19

20 Dedicated disk pools Although it is not possible to partition a storage array cache, it is possible to maintain separation within the overall storage array cache by using separate physical disk pools for different tenants or a dedicated VSAN for a tenant. The VCE trusted multi-tenancy approach addresses the concerns of secure data separation by providing a mechanism to isolate tenants at one or more layers of the infrastructure. For example, on the EMC VNX 5400 in a default Vblock System configuration, all disks are assigned to a single tenant and there are only two VSANs for redundancy. However, in the VCE trusted multi-tenancy approach, the disks can be split between tenants, each tenant having its own dedicated disks, VSAN, and physical ports on the controller. Dedicated disk array enclosures EMC storage arrays offer resource isolation and secure segmentation at the storage layer. An organization can choose the dedicated location of tenant data on the storage array down to the disk array enclosures. Compute separation In standard Vblock System solutions, compute resources are aggregated into a single layer cluster. All local users of the single tenant share resources, as the standard configuration has one tenant. Physical cables carry both network and storage traffic and connect to the fabric interconnects. VCE trusted multi-tenancy separation allows for each tenant to have dedicated compute resources. The following controls are available for physical compute separation: Dedicated blades Dedicated Cisco UCS chassis Dedicated Cisco UCS domains Dedicated blades Standard Vblock Systems configure all physical blades for a single tenant. With the VCE trusted multi-tenancy approach, dedicated physical blades can be assigned to each tenant. For example, the Cisco UCS chassis could be divided to serve four tenants, each of which could be assigned two blades per Cisco UCS chassis. Alternatively, each tenant could be assigned a dedicated physical port in which all traffic for that particular UCS blade resides on that port and cable. 20

21 Dedicated Cisco UCS chassis Organizations with higher bandwidth and compute requirements can achieve multi-tenancy by dedicating a complete Cisco UCS chassis to a single tenant. This approach completely isolates each tenant s traffic, even at chassis level, as every chassis carries traffic belonging to only one tenant. From blade level through input/output module, each chassis forwards and computes the traffic from the same tenant, resulting in a secure environment. Each dedicated chassis network and fibre channel traffic splits into separate network at the Cisco UCS fabric interconnect. Dedicated Cisco UCS domains Vblock Systems support multiple Cisco UCS domains. Although not required for secure operation, it is possible to isolate tenant processing and UCS administrative function to a completely separate fabric interconnect pair. In a standard solution design, upstream communications to a shared Cisco Nexus switch pair would continue to leverage logical separation. Management and physical separation Physical segmentation can introduce barriers to the flow of management traffic. It might be necessary to create additional instances of management tools, even when the tools are normally multi-tenant capable, due to the thoroughness of the separation. Technical necessity is not the only driver for management isolation. For many sites implementing physical segmentation elements, even when the management tools are capable of supporting multi-tenancy and the architecture is not an impediment to their use, it is not unusual for tenantfacing tools in particular to be implemented with discrete instances per tenant. This is less consistent in environments relying solely on logical segmentation. 21

22 Logical segmentation Logical segmentation refers to the effective separation and isolation of shared virtual compute, storage, and network resources between multiple tenants. Organizations can achieve segmentation as described in the Physical Segmentation section of this document; however, there are some shared components that cannot be physically partitioned or for which dedicating physical resources is not a viable business proposition. This section introduces strategies for logically separating these resources. VCE s multi-tenancy approach provides options for multiple control points to isolate tenant resources. When combined with a comprehensive security program, these controls provide an effective foundation for tenant protection. Although each tenant might have access to different amounts of network, compute, and storage resources in the shared infrastructure, a tenant sees only those resources allocated to them. The VCE multitenant architecture for logical segmentation provides: Shared resources to increase usage Service density and rapid elasticity; easier growth and scale using standard infrastructures Secure isolation of resources and data to meet tenant security requirements Reduced service costs and expenses Improved predictably around planning around capacity and workloads Service assurance and faster updates Appropriate data separation under most circumstances; use physical segmentation options for the remaining scenarios Consider the following design decision points when architecting multi-tenant segmentation: Network separation Storage separation Compute separation Virtualization Network separation Multi-tenancy concerns must be addressed at multiple levels within the Vblock System network infrastructure. Methods available to enforce network separation include security zoning and VLANs. Each tenant requires logical resource separation and/or path isolation at the network layer. VCE s trusted multi-tenancy approach provides the same degree of tenant isolation as a dedicated infrastructure, where each tenant receives appropriate separation, controls, and auditing. The following controls are available for logical network separation: VLAN 22

23 Virtual Machine Fabric Extender (VM-FEX) Virtual Device Context (VDC) Note: Virtual firewalls and other traditional network security controls are addressed later in this document. Virtual local area networks VLANs provide a layer 2 option to scale virtual machine connectivity, providing multi-tenant isolation and application tier separation. By default, a VLAN does not allow direct communications between virtual machines on different VLANs and all VLAN traffic must cross a router or firewall, which can be used to enforce connectivity policies. This provides a good degree of isolation between the virtual machines and allows for policy enforcement. It is a simple and common way to isolate network traffic across the layer 2 domains and shared links throughout the infrastructure. In general, Vblock Systems use two types of VLANs: VLAN Description Routed Includes management VLAN, virtual machine VLAN, and data VLAN Traffic passes through layer 2 trunks and is routed to the external network Internal Carries VMkernel traffic such as VMware vmotion, service console, network file system (NFS), and high availability Vblock Systems explicitly do not allow these VLANs across the trunks VCE s trusted multi-tenancy approach is to associate each tenant s resources with a different VLAN to ensure that the management, tenant, and Vblock System internal VLANs are isolated. VCE recommends always separating data and management VLANs. Since significant isolation can result in quick exhaustion of VLAN resources, the best practice is to allocate appropriately sized contiguous ranges with the flexibility to reallocate ranges if required. Virtual Machine Fabric Extender VM-FEX collapses virtual switches and physical networks into a single infrastructure. VM-FEX is a hardwarebased alternative to the Cisco Nexus 1000V switch that allows separation of traffic from different servers through network management domain separation from server or virtual machine management domain. VM-FEX allows a virtual machine to bypass the hypervisor and send network traffic directly to a vnic on the Cisco virtual interface card (VIC). All network switching and processing is redirected from the hypervisor to an external network switch, reducing overhead and freeing up the server CPU for application workloads. This is ideal for applications with high packet rates. VM-FEX does not support N-Port Virtualizer (NPV), which would allow a per-virtual machine virtual host bus adapter (vhba) and relies on VMFS for storage access. Instead, use VMware vsphere Raw Device Mode (RDM) with VM-FEX for a per-virtual machine vhba identity. 23

24 Organizations planning to implement a VM-FEX solution in VMDirectPath mode to bypass the hypervisor entirely should consider that they are limited to the number of devices that ESXi can support per host; only 8 VMDirectPath PCIe devices per host and 16 per virtual machine per VMware vsphere documentation. Virtual Device Context Use a Virtual Device Context (VDC) to logically separate an optional Cisco Nexus 7000 switch into multiple device contexts (virtual switches) to provide management separation, change domain isolation, and provide isolated VLAN and overlapping IP address support. A Virtual Device Context can contain its own unique and independent set of VLANs. Each Virtual Device Context can be assigned to its own physical ports, allowing for virtualization of the hardware data plane as well. The Vblock System 300, Vblock System 500, and Vblock System 700 families support Virtual Device Context. The Cisco Nexus 7000 switch natively supports four Virtual Device Contexts. A maximum of eight VDC can be supported with the Nexus 7000 Series Supervisor 2 Enhanced (N7K-SUP2E=) and the Cisco Nexus 7000 Incremental VDC license (N7K-VDC1-K9). Storage separation At times it is necessary to ensure that a specific dataset does not share spindles with any other dataset. This separation might be required between tenants or even within a single tenant s dataset for organizations using the same shared service requirements. With VCE s trusted multi-tenancy design approach, multiple features can be combined with standard security methods such as SAN zoning and Ethernet VLAN to separate, control, and manage storage resources among an infrastructure s tenants. The following logical controls are available for storage separation: VSAN Zoning Virtual Data Mover (VDM) Virtual Machine File System Management (VMFS) LUN masking Virtual storage area networks VSAN is a separation mechanism similar to the VLAN enabled by the Cisco MDS storage area network (SAN) implementation. VSANs enable logical separation of large groups of fabrics at no additional hardware cost. Fibre channel services are fully replicated for each new VSAN after setting up the VSAN profile to ensure any failures from fabric changes are limited to the impacted VSAN. Security is improved because the VSAN independence minimizes the total system's vulnerability. 24

25 VSAN physical topology example Zoning SAN zoning isolates resources on a per pwwn basis. Resources can communicate only with other resources in the same zone as the initiating HBA/pWWN. Incorporating zoning in a multi-tenant infrastructure restricts visibility and connectivity between devices connected to a common fibre channel SAN, preventing data leakage between zones. In Vblock Systems, Cisco MDS zoning occurs by default. Virtual Data Mover Virtual Data Mover (VDM) is an EMC VNX feature that allows CIFS servers and their associated environment to be grouped into virtual containers. Virtual Data Mover isolates CIFS and/or NFS servers to provide a higher level of security. VLAN and file system mounts for different home directories can be separated, isolating home directory databases and their associated users. Virtual Data Mover also allows the replication and movement of a CIFS and/or NFS environment into another local or remote data mover. LUN masking Logical Unit Number (LUN) masking is an authorization process that restricts storage LUN access to specific resources/hosts on a shared SAN. LUN masking is implemented mainly at host bus adapter (HBA) level but in Vblock Systems it can be implemented at the storage controller level. As the controller itself enforces the access policies to the device, it is more secure. LUN masking implemented with VSAN and fibre channel zoning extends the tenant data storage separation from the SAN switch ports to the physical disks and virtual media within the storage array. 25

26 Virtual machine file system management VMware uses a cluster file system called a virtual machine file system (VMFS). An ESXi host associates a VMFS volume that is made up of a larger logical unit. The virtual machine disk (VMDK) sub-directory in the VMFS volume stores each virtual machine directory. The VMFS volume locks those files to prevent updating by other ESXi servers. One VMDK directory is associated with a single virtual machine and multiple virtual machines cannot access the same VMDK directory within the VMFS volume, thereby isolating each tenant's VMDK, snapshots, and virtual machine files. VMFS reinforces the effects of the tenant Datastore isolation from the zoning mechanisms and LUN masking within the SAN at the file system level, serving to limit the effect of virtual machine-based exploits or inadvertent disk corruption. Compute separation Virtualization introduces new security concerns to the traditional data center where security policies are implemented at the physical level. When multiple logical servers belonging to a variety of tenants exist on a single physical server or single compute domain it must be possible to use logical mechanisms to isolate the logical servers belonging to each tenant. Secure separation at UCS blade level VCE s trusted multi-tenancy approach enables organizations to achieve secure separation at the compute layer by managing multi-tenancy concerns at multiple levels, including the CPU, the Cisco UCS server infrastructure, and VMware solution elements. 26