VMware vcloud Architecture Toolkit Operating a VMware vcloud

Transcription

1 VMware vclud Architecture Tlkit Versin Octber 2011

2 2011 VMware, Inc. All rights reserved. This prduct is prtected by U.S. and internatinal cpyright and intellectual prperty laws. This prduct is cvered by ne r mre patents listed at VMware is a registered trademark r trademark f VMware, Inc. in the United States and/r ther jurisdictins. All ther marks and names mentined herein may be trademarks f their respective cmpanies. VMware, Inc 3401 Hillview Ave Pal Alt, CA

3 Cntents 1. Overview Audience Scpe vclud Operatins Framewrk Organizing fr vclud Operatins vclud Center f Excellence Rle f vclud Center f Excellence in Standardizatin vclud Service Management vclud Operatins Management vclud Infrastructure Management vclud Service Management Service Catalg Management Service Level Management vclud Operatins Management Cnfiguratin Management Orchestratin Management Availability Management Cntinuity Management vclud Infrastructure Management Security and Cmpliance Management Capacity Management Perfrmance Management Mnitring Appendix A: vclud Directr Cell Mnitring Appendix B: Cmpliance Cnsideratins Appendix C: Capacity Planning Appendix D: Capacity Management Appendix E: Integrating with Existing Enterprise System Management Appendix F: Business Cntinuity

4 Appendix G: Upgrade Checklists List f Figures Figure 1. Clud Cmputing Layers... 8 Figure 2. vclud Operatins Framewrk... 9 Figure 3. vclud Operatins Framewrk Mapped t Service Layers Figure 4. vclud Center f Excellence Ecsystem Figure 5. Service Catalg Evlutin Figure 6. Example Organizatin with Public vclud IaaS and Private vclud PaaS/SaaS Layers Figure 7. Cnfiguratin Management Interrelatinships Figure 8. Architectural Example Drawing Figure 9. One Primary Functin per Server Figure 10. Lg Cllectin in the vclud Envirnment Figure 11. Architecture f vclud Cmpnents and Lg Cllectin Figure 12. Infrastructure Layers Figure 13. vclud Directr Extensin Overview Figure 14. vcenter Orchestratr as a vclud Directr Extensin Figure 15. vcenter Orchestratr AMQP Subscriptin Plicy Figure 16. Credential Management Wrkflw... 86

5 List f Tables Table 1. Public Catalg Benefits Table 2. Sample vapp Offerings Table 3. vclud vapp Requirements Checklist Table 4. MBeans Used t Mnitr vclud Cells Table 5. Audit Cncerns Within the vclud Table 6. vclud Cmpnent Lgs Table 7. Other Cmpnent Lgs Table 8. vsphere Hst Variables Table 9. Determining Redundancy Overhead Table 10. Netwrk Capacity Planning Items Table 11. Capacity Mnitring Metrics Table 12. Organizatin Virtual Datacenter Units f Cnsumptin Table 13. Recmmended Organizatin Virtual Datacenter Capacity Threshlds Table 14. Sample Organizatin Virtual Datacenter Resurce Allcatin Table 15. Organizatin Virtual Datacenter Trending Infrmatin Table 16. Organizatin Virtual Datacenter Capacity Trending Variables Table 17. Sample Organizatin Virtual Datacenter Trending Infrmatin Table 18. Apprve a vapp wrkflw Table 19. Recmmended Prtectin Plicies... 88

6

7 1. Overview prvides practical peratins-fcused, rganizatinal, prcess, and supprting technlgy cnsideratins and guidance based n the vclud Operatins Framewrk. The gal is t prvide custmers with the infrmatin they need t realize the benefits f prceeding alng the jurney f VMware vclud adptin and prviding Infrastructure-as-a-Service (IaaS) using a servicefcused, cmprehensive peratinal framewrk. Bth service prviders and enterprises can use the guidelines in this dcument, with sme variatins depending n pint f view. The dcuments, Architecting a VMware vclud,, and Cnsuming a VMware vclud are designed t wrk tgether thrughut the lifecycle f a VMware vclud cmputing implementatin with VMware technlgies. By using all three dcuments tgether, cmbined with a private r public service definitin, yu can gain a cmprehensive view f VMware vclud cmputing. Architecting a VMware vclud prvides best practices, design cnsideratins, and design patterns fr cnstructing a vclud envirnment frm its cnstituent cmpnents. includes best practices and cnsideratins fr perating and maintaining a vclud envirnment. It cvers the peple, prcess, and technlgy invlved in running a vclud envirnment. Cnsuming a VMware vclud cvers the varius cnsideratins fr the cnsumer when chsing t leverage vclud cmputing resurces. This dcument is nt a substitute fr VMware prduct dcumentatin, nr des it prvide detailed implementatin prcedures fr installing a vclud. 1.1 Audience This dcument is intended fr, but nt limited t, IT persnnel respnsible fr r invlved in the service, peratins, and infrastructure management f ne r mre vclud instances t deliver Infrastructure as a Service. It is assumed that the reader has knwledge f and familiarity with VMware vsphere cncepts. 1.2 Scpe This dcument fcuses n cnsideratins fr perating a vclud frm the perspectives f the rganizatinal structure, service management, peratins management, and infrastructure management.

8 2. Clud cmputing is an apprach t cmputing that leverages the efficient pling f n-demand, selfmanaged virtual infrastructures, cnsuming them as a service. The NIST standard defines three such service layers within a clud. Key clud cmputing principles, alng with the service layer paradigm are illustrated in Figure 1. Figure 1. Clud Cmputing Layers VMware defines the existing service layers as: Sftware as a Service (SaaS) Business-fcused services presented directly t users via a service catalg. Platfrm as a Service (PaaS) Technlgy-fcused services presented fr applicatin develpment and deplyment presented directly t applicatin develpers via a service catalg. Infrastructure as a Service (IaaS) Services prviding infrastructure cntainers fr varius uses in rder t prvide better agility, autmatin, and delivery f cmpnents. Additinal service layers will be added as ther services, such as Desktp as a Service, becme a reality. Cmpanies embark n the jurney t adpt clud cmputing t realize increased quality f service, business agility, and perating cst efficiency. Quality f service is achieved by prviding standardized service fferings with assciated availability levels and service management. Custmers can expect the prvisin f a reliable VMware vclud service with predictable service levels s that end users are prvided with the service they require in the manner they require within expected timeframes. T prvide standardized, repeatable service, IT must intrduce their wn peratinal efficiencies t make sure that they are in cntrl f the underlying infrastructure and applicatins, but are nt restricting their use by ver-managing it.

9 Business agility is achieved by changing the way that IT thinks abut managing services. Sme enduser cntrl must be prvided s that IT des nt becme a bttleneck when prvisining services, but IT must retain sme cntrl t prevent the envirnment frm becming unmanageable. T prvide business agility and speed t market, autmatin is key t the success f the vclud. Autmatin f time-cnsuming, errr-prne, and cmplex tasks is recmmended t prvide the reliable, rapid service expected frm the vclud. Increased cst efficiency can be realized by decreasing peratinal expenses. T achieve this, the current peratinal cst and burden f managing IT (apprximately 70% peratinal expenses and 30% capital expenses) must change, especially as IT becmes mre like a service prvider prviding Infrastructure as a Service. Reducing peratinal expenses can nly be accmplished by enhancing IT peratinal prcesses fr clud cmputing, implementing the tls t supprt and autmate the enhanced peratinal prcesses, and ptimizing the rganizatinal structure t mst effectively align with managing the clud cmputing infrastructure and IT services ffered. 2.1 vclud Operatins Framewrk The underlying VMware vclud Operatins Framewrk within which rganizatinal structure and critical prcesses can be defined is shwn in Figure 2. Figure 2. vclud Operatins Framewrk

10 The vclud Operatins Framewrk cnsists f the fllwing layers: vclud Business Management Addresses the surce f business drivers and requirements fr vclud services t be ffered, alng with management f the businesses r line f business cnsumers and its supprting prcesses. vclud Service Management Cnverts the business drivers and requirements int vclud service definitins, manages service develpment and transitin, creates and reprts n Service Level Agreements back t the business and its cnsumers, and manages the service catalg lifecycle. vclud Operatins Management Defines, develps, and delivers standardized IT Service Management prcesses and tls t supprt them, t manage the peratins and delivery f vclud services. vclud Infrastructure Management Architects, deplys, and manages the underlying vclud infrastructure upn which the vclud services are deplyed and delivered. These layers are required regardless f the specific vclud service layer being addressed. Figure 3. vclud Operatins Framewrk Mapped t Service Layers

11 3. Organizing fr vclud Operatins One f the truly transfrmative aspects f vclud cmputing is its impact n the IT rganizatin. By definitin, vclud cmputing prvides n-demand service delivery. As a result, the IT rganizatin has t becme service-driven. Delivering a vclud-based service impacts all three layers f the VMware vclud Operatins Framewrk: vclud Service Management, vclud Operatins Management, and vclud Infrastructure Management. The VMware apprach t addressing this impact is t use the vclud Center f Excellence (vcoe) mdel. 3.1 vclud Center f Excellence The vclud Center f Excellence mdel is an extensin f the VMware Center f Excellence mdel that has been used by many rganizatins f varius sizes t facilitate the adptin f VMware technlgy and t reduce the cmplexity f managing a VMware virtual infrastructure. The vclud Center f Excellence mdel defines crss-dmain vclud Infrastructure Management accuntability and respnsibility within team rles acrss an rganizatin. These team rles enable an rganizatin t cnsistently measure, accunt fr, and imprve the effectiveness f its vclud infrastructure management even if its IT Service Management rles and respnsibilities are distributed acrss multiple IT functinal areas. The vclud Center f Excellence mdel als invlves the practive inclusin f vclud Infrastructure Management champins wh represent related functinal teams that are critical t prviding vclud Infrastructure Management in supprt f vclud service fferings. The vclud Center f Excellence is a fcused virtual team f vclud Infrastructure Management specialists and an ecsystem f related functinal grups which, taken tgether, frm a vclud Center f Excellence extended team as shwn in Figure 4. It serves as the fcal pint fr all decisins invlving vclud Infrastructure Management, including infrastructure architecture, implementatin, and management. The team is tasked with creating, reviewing, and publishing vclud Infrastructure Management guidelines and dcumentatin, as well as effecting plicy and prcess change fr internal use. T reach an peratinally mature level, yu must establish a frmal vclud Center f Excellence with aligned rles and respnsibilities.

12 Figure 4. vclud Center f Excellence Ecsystem As shwn in Figure 4, the vclud Center f Excellence ecsystem includes the ther tw layers f the vclud Operatins Framewrk, vclud Service Management and vclud Operatins Management. It als includes the ther cre IT teams: Enterprise Architecture, Facilities, Infrastructure (netwrk, strage, servers), and the Security team. The rganizatinal structure f Enterprise Architecture, Facilities, Cre Infrastructure, and Security are nt addressed in this dcument. 3.2 Rle f vclud Center f Excellence in Standardizatin In a traditinal rganizatin, IT is driven by multiple business units. The business units cntrl IT funding and each business unit (BU) can enfrce separate plicies and prcedures fr their infrastructure. This apprach leads t disjinted architectures and a lack f standardizatin. IT grups that supprt such an envirnment struggle t achieve agreed service levels, leading t end-user frustratin, IT supprt inefficiencies, and pssibly even financial liabilities. The implementatin f a vclud changes this paradigm. A vclud is built as a shared resurce that requires enfrcement f cnsistent standards acrss the entire IT rganizatin. T define and enfrce these standards, all plicies and prcedures assciated with the vclud shuld be driven by the vclud Center f Excellence team rather than frm business units. This shift pses a significant challenge fr rganizatins wh try t mve int a vcludapprpriate perating mdel. The vclud Center f Excellence needs t negtiate with different business grups and rely n executive spnsrship and supprt during this transitin. Mre rigrus standards need t apply acrss the whle rganizatin. One recmmended apprach is t align with the rganizatins phased develpment apprach, adding a new vclud architecture review and signff by the vclud Center f Excellence during the analysis and design phase fr all new prjects. Other recmmendatins include running vclud-specific assessments n applicatins that are being cnsidered fr migratin t the vclud. Assessments determine gaps and set expectatins with business units n expected changes. The key t success is the ability t balance agility t meet business needs with stringent enfrcement f defined standards within the vclud.

13 3.2.1 Layers f Standardizatin The vclud is a shared resurce running n infrastructure supprted by the vclud Center f Excellence and cre infrastructure teams. As the vclud Center f Excellence sets standards fr the vclud, cre infrastructure teams may develp standards fr the infrastructure that supprts the vclud. Fr example, the strage team may create standards fr hw new lgical unit number (LUN) strage is presented fr vclud cnsumptin. This layer f abstractin allws the strage team t have the flexibility t chse the mst cst-effective SAN vendr and, if required, supprt a multi-vendr envirnment Measurement with Industry Benchmarks Clud technlgy is evlving at a rapid pace. After a vclud is established within an rganizatin a cntinuus imprvement cycle needs t be set up annually t make sure that the rganizatin s vclud is nt lacking any current industry standards r benchmarks. The vclud Center f Excellence is respnsible fr running this assessment and presenting the results, including recmmendatins fr remediatin, back t the leadership team. 3.3 vclud Service Management vclud Service Management is critical t prviding vclud service fferings and is the entry pint int IT frm vclud Business Management. The fllwing are rles and respnsibilities that are invlved with vclud Service Management: vclud Service Prtfli Management: Manages the prtfli f vclud services and wrks with rganizatinal leadership t develp the vclud service ffering strategy used t determine what services shuld be included in the verall prtfli. Practively identifies and defines ptential vclud service fferings based n demand infrmatin gathered frm vclud Business Management r ther surces such as requests cming in thrugh the Service Desk. vclud Service Owner: Respnsible fr verall delivery f their vclud service ffering. Prvides the required infrmatin t Service Catalg Management t crrectly set up the service catalg ffering. Wrks with Service Level Management t review Service Level Agreements and Operating Level Agreements t make sure that they are achievable. Als, negtiates updated Service Level Agreements and Operating Level Agreements as the service ffering is updated. Leads develpment and enhancement effrts and wrks with vclud Service Develpment Management n their vclud service ffering based n new requirements frm vclud Business Management. Liaises between IT Business Management and vclud Center f Excellence. vclud Service Develpment Management: Defines a vclud service ffering based n the requirements prvided by vclud Business Management after it s determined that a particular vclud Service Offering is t be included in the vclud Service Prtfli. This invlves translating vclud Business Management requirements int requirements that are used by vclud Infrastructure Management t create deplyment templates. Manages any additinal effrts required t ppulate deplyment templates, such as wrking with Applicatin Develpment Managers wh may prvide an applicatin fr a vclud service ffering.

14 Wrks with vclud Business Management and Financial Management t determine a price fr a vclud service ffering, and determine whether multiple prices are apprpriate if the vclud service ffering is prvided in multiple service tiers. Creates, cllects, and maintains any vclud service ffering develpment dcumentatin. vclud Service Level Management: Defines the Service Level Agreement (SLA) assciated with a vclud service ffering r a tier f service such as prvided by a particular prvider virtual datacenter. Makes sure that the service levels are met thrugh crrespnding Operating Level Agreements with vclud Operatins Management and vclud Infrastructure Management. Regularly mnitrs and reprts n service level attainment. vclud Service Catalg Management Manages the vclud service ffering catalg and makes sure that all f the infrmatin cntained in the catalg is accurate and up-t-date. These rles and respnsibilities can be satisfied by a single persn r multiple peple. The decisin t emply ne r multiple peple depends n the number f vclud service fferings. If the number f vclud service fferings is large enugh t justify multiple peple, r as the number f vclud service fferings grws and multiple peple are needed, the recmmended distributin is: A single vclud Service Prtfli Manager. A single vclud Service Catalg Manager. A single vclud Service Level Manager. Multiple vclud Service Owners, each respnsible fr vclud service develpment and wrking with ther teams t make sure that the agreed vclud service levels fr their vclud service ffering r suite f vclud service fferings are maintained. As key members f the vclud Center f Excellence ecsystem, they als act as vclud champins. In this rle, they interact clsely and regularly with the vclud Center f Excellence as well as champining vclud t the business and ther teams with which they interact in the rganizatin at large. 3.4 vclud Operatins Management T realize the perating expense savings benefit ffered by vclud cmputing, vclud Operatins Management must be respnsible fr develping and executing standardized, autmated prcesses and tls ptimized t efficiently manage the peratins and delivery f vclud service fferings. The key terms are standardized, autmated, ptimized, and service. Thugh vclud cmputing des nt require a radical rerganizatin f hw ITSM peratins are prvided, it des require each rle t have: An unwavering service-riented fcus fr peratins resurces (bth peple and tls), as ppsed t the traditinal technical infrastructure fcus. The gal f standardizing and autmating the prcesses required t execute their respnsibilities t the highest pssible degree. An understanding f the impact vclud cmputing has n their area f respnsibility, and respnsibility fr ptimizing their area fr vclud cmputing. These requirements shuld be directly related t each individual s perfrmance gals.

15 The areas within vclud Operatins Management impacted by r impactful n vclud include: Prvisining Management. Change Management. Capacity Management. Availability Management. Perfrmance Management. Orchestratin Management. Event Management. Incident and Prblem Management. Cnfiguratin and Asset Management. Cntinuity Management. Due t the increased fcus n autmatin, in additin t the traditinal IT Service Management areas, ne new rle and assciated set f respnsibilities shuld be included. That rle is Orchestratin Management (r Autmatin Management). Orchestratin Management is a crss-dmain rle within vclud Operatins and the persn in this rle is respnsible fr: Being the rchestratin expert within vclud Operatins. Wrking with Enterprise Architecture t develp the peratin s rchestratin strategy. Wrking with the ther vclud Operatins rles t design, develp, test, and deply their specific prcess autmatin wrkflws. Develping and maintaining prcess autmatin wrkflw dcumentatin. Wrking with Event Management and vclud Infrastructure Management t establish wrkflw mnitring and, t the extent pssible, autmate respnses t events. Prviding Level 3 prcess autmatin wrkflw incident reslutin. Like the vclud Service Level Management team, members f the vclud Operatins Management team are key members f the vclud Center f Excellence ecsystem and als act as vclud champins. In this rle, they interact clsely and regularly with the vclud Center f Excellence as well as champining vclud t the ther teams with which they interact in the rganizatin. 3.5 vclud Infrastructure Management vclud Infrastructure Management is respnsible fr architecture, deplyment, and peratins f the underlying vclud infrastructure, and gains the mst by rerganizing. Traditinal Infrastructure Management is siled by infrastructure dmain with very little crss-dmain interactin unless required fr a particular prject r deplyment. Virtualizatin prvided the mst recent and cmpelling pprtunity fr Infrastructure Management t break frm this traditinal apprach. If Infrastructure Management brke frm the traditinal apprach, adapting t vclud cmputing shuld be reasnably straightfrward. If they did nt, they must nw break with the traditinal apprach t successfully supprt vclud cmputing. This rerganizatin takes the frm f the vclud Center f Excellence cre team. Generally, the vclud Center f Excellence encmpasses respnsibility fr bth the VMware vsphere and vclud layers f the infrastructure. Fr the purpse f the fllwing descriptins, vclud encmpasses the vsphere infrastructure as well. The primary rles fr members f the vclud Center f Excellence cre team are described in the fllwing sectins.

16 3.5.1 Executive Spnsr Prvides clear messaging, leadership, and guidance t the entire IT rganizatin and affected rganizatins abut the vclud Center f Excellence. Drives the crss-dmain alignment required fr establishing a successful, functining vclud Center f Excellence extended team. This level f spnsrship is key t breaking dwn rganizatinal barriers and mandating integrated prcess design and implementatin acrss the affected rganizatins. Crss-dmain alignment and integrated prcess implementatin are abslutely required t sustain a vclud infrastructure at the level required t supprt vclud-based service fferings and assciated service levels vclud Center f Excellence Leader Prvides leadership and guidance t vclud Center f Excellence members. Has a direct line f cmmunicatin t the executive spnsrs. Has visibility int the planned vclud-based service ffering prtfli as well as any prtfli changes. Is respnsible and accuntable fr making sure that the vclud infrastructure can supprt and cntinue t supprt the vclud-based service fferings and service levels. Actively prmtes awareness f the impact the vclud infrastructure has n service ffering and service level supprt and delivery. Facilitates integratin f the vclud infrastructure int existing IT Service Management prcesses. Crdinates and assists with planning vclud infrastructure initiatives. Prvides guidance t change management fr changes related t the vclud infrastructure; may authrize lw risk, lw impact changes t the vclud infrastructure. Facilitates develpment and maintenance f vclud infrastructure capacity frecasts. Manages the acquisitin and installatin f vclud infrastructure cmpnents. Maintains relatinships with the vclud Center f Excellence extended team members and prvides subject matter expertise t team members as required. Is invlved in managing vendr relatinships fr vclud infrastructure cmpnents vclud Center f Excellence Architect Respnsible fr develpment and maintenance f vclud infrastructure architecture and design dcuments and blueprints. Wrks clsely with strage and netwrk grups t architect and design vclud infrastructure extensins. Wrks with enterprise architects t make sure that the vclud infrastructure architecture is aligned with cmpany architectural standards and strategies. Respnsible fr architecting and designing the vclud layer in supprt f the planned vclud-based service ffering prtfli and any prtfli changes. Respnsible fr wrking with the IT Security team t make sure any architecture r design decisins address security and cmpliance.

17 Respnsible fr architecting r designing slutins fr vclud infrastructure integratin pints with extended team systems. Prvides subject matter expertise t supprt design, build, cnfiguratin, and validatin prcesses. Maintains awareness f VMware sftware patches and their impact n the envirnment. Develps and maintains peratinal guidelines fr the maintenance and supprt f the vclud infrastructure. Mentrs vclud Center f Excellence cre and extended team members. Assists with the incident and prblem management prcesses t reslve issues related t vclud infrastructure. Develps sftware and hardware upgrade plans. Develps and maintains the availability plicy fr the vclud infrastructure in crdinatin with Availability Management and Service Level Management vclud Center f Excellence Analyst Respnsible fr the develpment and maintenance f the vclud infrastructure capacity frecast. Respnsible fr the day-t-day capacity and resurce management f the vclud infrastructure. Wrks with the IT Security team t make sure that the vclud infrastructure aligns with IT security and cmpliance plicies. Initiates requests fr new vclud infrastructure cmpnents. Assists with Incident and Prblem Management prcesses fr issues related t vclud infrastructure capacity and perfrmance. Assists with Change Management prcess as applied t the vclud infrastructure. Respnsible fr maintaining the vclud infrastructure Cnfiguratin Management data. Respnsible fr validating billing metering data cllected fr the vclud-based service fferings vclud Center f Excellence Administratr Installs and cnfigures vclud infrastructure cmpnents. Executes the validatin plan when deplying new infrastructure cmpnents. Wrks with vclud Center f Excellence extended team members t cnfigure vclud infrastructure cmpnents. Respnsible fr auditing vclud infrastructure cmpnent cnfiguratin cnsistency. Develps and maintains vsphere and vclud internal user access rles. Creates, cnfigures, and administers vclud prvider-related bjects. Wrks with the IT Security team t implement vclud-related security and cmpliance plicies. Wrks with Service Level Management t determine maintenance windws fr the vclud infrastructure. Prvides Tier 3 supprt f the vclud infrastructure.

18 Tests and installs vclud infrastructure patches. Cnfirms that the vclud infrastructure is crrectly instrumented fr mnitring and lgging purpses. Respnsible fr wrking with ther teams t implement any required vclud integratin with their systems. Wrks with Orchestratin Management t implement the vclud infrastructure-impacting wrkflws. These rles and respnsibilities require a unique set f skills s, at a minimum, each rle shuld be filled by a different persn. The number f peple in each rle, with the exceptin f the vclud Center f Excellence Leader, depends n the scale and scpe f the vclud infrastructure.

19 4. vclud Service Management 4.1 Service Catalg Management The purpse f a service catalg is t prvide a clearly defined set f services available t custmers fr cnsumptin in a vclud envirnment. Ideally, the service catalg is ffered frm a ne stp shp where a custmer can select the services they require with minimal interventin r manual activity. An initial vclud service catalg shuld align with the vclud prvider s gals, but shuld aim fr simplicity while at the same time integrating with capacity planning and cst transparency. Regular reviews f the service catalg shuld be perfrmed with adjustments made in line with any increased functinality prvided by future releases f VMware vclud Directr (VCD), vsphere, r any additinal supprting prducts vclud Service Catalg Cmpnents The service catalg fr the vclud supprted by VCD ffers a number f different service cmpnents t the end custmer. The cmbinatin f all f these cmpnents creates the service as a whle. At a minimum, the vclud prvider must ffer: Organizatin This is the cntainer fr the custmer s IaaS with attributes that hld basic, default service cnfiguratin infrmatin. Typically, nly ne rganizatin cntainer is purchased per custmer. Organizatin virtual datacenters These are the bundaries fr running the virtual machines within the IaaS service. They are cnfigured with sizing infrmatin depending n the custmers requirements, and have an apprpriate SLA assigned t them. A minimum f ne rganizatin virtual datacenter is required fr a custmer t ffer a service, but additinal rganizatin virtual datacenters can be requested if required. In additin t these cre vclud cmpnents, it is pssible fr the vclud prvider t establish a standard set f fferings within the vclud service catalg t prvide vapps (standardized grupings f precnfigured virtual machines) and media (installable sftware packages) t end custmers. These fferings are gruped int the fllwing types f VCD catalgs: Public catalgs These cntain vapps and media (fr installatin f sftware) that are ffered t the end custmer by the vclud prvider. Organizatin catalgs These als cntain vapps and media, but are nly available within an individual rganizatin, and can nly be shared with individuals within that particular custmer rganizatin. Organizatin catalgs are created, cntrlled, and wned by individuals within the custmer rganizatin.

20 The benefits assciated with ffering vapps and media via public catalgs as part f vclud prvider s verall service catalg are listed in Table 1. Table 1. Public Catalg Benefits Qualities Supprtability Agility Descriptin By ffering a discrete set f services, it is much simpler t prvide a reliable demand pipeline and, in turn, prvide the capacity t supprt that pipeline. By prviding standardized vapps, it becmes simpler t manage the verall envirnment. Standard fferings prvide fr the simple selectin f virtual machine cnfiguratins (vapps) and enable quick prvisining using vclud Directr vclud Service Catalg Evlutin T imprve the vclud service catalg prcess and t help realize vclud benefits as many service fferings as pssible shuld be made available directly t the end user with autmated prvisining. Typically, in the virtualizatin wrld, the initial prcess fr prcurement f virtual machines fllws the mdel that is applied t physical infrastructure. Thugh this wrks, it is nt the mst efficient mechanism fr prviding services, and vclud benefits cannt be fully realized unless this prcess is changed. A lgical representatin f the evlutin f the vclud service catalg frm this current state t the desired end state is illustrated in Figure 5.

21 Figure 5. Service Catalg Evlutin In the Service Catalg current state, when a new service is requested, a Service Request is raised that is used t select and prvisin an ffering frm the service catalg. This request includes nt nly the utility (vapp r rganizatin virtual datacenter) t be prvided t the custmer, but als includes the required service level (this is prvided by the virtual datacenter in which the vapp is t be prvisined as well as any built-in availability features within the vapp itself). After the service has been rdered, the end custmer must wait fr staff t fulfill the Service Request fr the virtual machines that will prvide the service t be prvisined. T satisfy the self-service, n-demand attribute f vclud cmputing, the custmer shuld be able t cnnect t a prtal, select the required service ffering, and have it autmatically prvisined. This remves the manual task f selecting frm the service catalg, and als remves the delay in the prvisining prcesses. This prcess is shwn as the vclud Service Catalg target state in Figure 5. VCD prvides the ability t manage these requests frm the service catalg. Fr vapps, an rganizatin administratr can determine wh within the rganizatin has rights t request and prvisin vapps, thus prviding end-t-end self-service. With VCD, nt nly can the end user select and prvisin the vapp, but the user can als specify in which rganizatin virtual datacenter it is deplyed. Because rganizatin virtual datacenters are assciated with prvider virtual datacenters, the end user is effectively selecting the service level they require. The evlutin t the target vclud service catalg can be accmplished as fllws: 1. Cntinue with the Service Request prcess until the vclud service catalg is available n the prtal. 2. Enable apprpriate IT staff t perfrm vclud service catalg requests with autmated prvisining, including required apprvals, n behalf f the end user. 3. Add the ability fr end users t access the vclud service catalg and request services that result in autmated prvisining, including required apprvals, f the crrespnding vapps.

22 4.1.3 Standardizatin f vclud Offerings int the Service Catalg Standardizatin f the service fferings is essential t achieving a scalable, cst efficient vclud envirnment. Typically, cmpute resurce-based service fferings (CPU, memry, and strage) are a baseline fr vclud cnsumptin and shuld be standardized as much as pssible regardless f whether they apply t rganizatin virtual datacenters r vapps (and their assciated virtual machines). Cmpute resurces fr rganizatin virtual datacenters available in the service catalg shuld be standardized int varius sizes. Additinally, the required cmpute resurce cnfiguratins vary depending n the selected VCD allcatin mdel (Allcatin Pl, Pay-as-Yu-G, r Reservatin Pl), because attributes such as CPU speed and CPU/memry guarantee vary. Cmbining these tw cmpnents means the service catalg culd ffer a number f differently sized rganizatin virtual datacenters fr each type f allcatin mdel. Similarly, t create a vapp catalg item (public r rganizatin), there shuld be as much standardizatin as pssible. Initially, frm a cmpute resurce pint f view, standard sized virtual machines shuld be created t enable a pick list f machines fr vapp creatin. These standardized virtual machines culd vary in resurce size fr CPU, memry and strage; fr example, Standard, Standard Plus, Advanced, Premium, and Premium Plus. As a vapp cmprises a number f individual virtual machines, the apprpriately sized virtual machines can be selected frm the pick list during the vapp catalg creatin prcess. In additin t the basic cmpute fferings f the virtual machines within the vapps, it is necessary t develp the service catalg t include vapp sftware cnfiguratins. These culd be basic grupings f cmpute resurces r culd be expanded ver time t ffer mre advanced services. Sample vapp fferings are shwn in Table 2. Table 2. Sample vapp Offerings vapp 2-Tier Standard Cmpute 3-Tier Standard Cmpute, Advanced Database 3-Server Standard Plus Cmpute (nt necessarily tiered) Cnfiguratin 1 x Standard RHEL Web virtual machine 1 x Standard Windws Server 2008 Applicatin virtual machine 1 x Standard RHEL Web virtual machine 1 x Standard RHEL Applicatin virtual machine 1 x Advanced MySQL Database virtual machine 3 x Standard Plus Windws Server 2008 Applicatin virtual machine

23 4.1.4 Establish Service Levels fr vclud Services in the Service Catalg T prvide an apprpriate level f service depending n the vclud custmers requirements, services shuld be further differentiated by their crrespnding service levels. These service levels can be defined by ffering availability and recverability attributes such as Recvery Time Objective (RTO), Recvery Pint Objective (RPO), and incident respnse times. These attributes can be applied t the different cmpnents within the service catalg. Within a vapp, it is pssible t design fr different service levels via the virtual machines cntained in the vapp. Fr example, a vapp culd cntain multiple Web servers t prvide resilience in the event f server failure, and thus a lwer RTO fr the service. As virtual datacenters prvide abstracted physical and virtual resurces, different service levels can be defined by using (r nt using) the underlying hardware technlgy (server capabilities, strage array technlgies, strage prtcls, replicatin, and s n) and virtualizatin technlgy (HA, DRS, VMware vsphere vmtin, and thers). Taken tgether, the vapps ffered and the capabilities f the virtual datacenters n which they can be deplyed make fr a pwerful and cmplete vclud service catalg. 4.2 Service Level Management Service Level Management defines the SLA assciated with a vclud service ffering r a tier f service, makes sure that the service levels are met thrugh crrespnding OLAs, and regularly mnitrs and reprts n service level attainment Definitin f Service IT services can be defined as a set f related activities r wrkflws that serve a defined business purpse, supprted by a cmbinatin f peple, prcess, and technlgy cmpnents. Generally, IT services are ffered t users thrugh a service catalg Service Types Service types include business user services and technlgy services Business User Services Services are generally directly cnsumed by end users and are available as part f the rganizatin s enterprise service catalg Technlgy Services Technlgy services are nt cnsumed directly by users, but enable infrastructure autmatin that enhances an IT rganizatin's ability t better supprt business needs Service Interrelatinships T ptimally prvide vclud business user services, all types f technlgy services need t be seamlessly integrated. Generally, this is accmplished using a wrkflw engine called the rchestratin layer. Invking a business user service may autmatically trigger ne r a cmbinatin f technlgy services. The rules gverning these wrkflws need t be precnfigured and preapprved fr cntrl. They are als needed t prvide an agreed t level f service t the business user. This agreed t level f service is knwn as a Service Level Agreement (SLA).

24 4.2.4 Definitin f Service Level Agreement A service level is a predetermined agreement between the service cnsumer and the service prvider that measures the quality and perfrmance f the available services. SLAs can be f multiple types, frm measuring pure server uptime t measuring respnse time fr technlgy cmpnents, prcess wrkflws, users, and s n. There are services running at every layer f the vclud stack, s service cnsumers may be business users r internal IT grups wh access the vclud primarily fr technlgy and infrastructure services. In cases where SLAs are established fr base technlgy services that are nt cnsumed directly by business users, but are needed t make sure that dwnstream peratins and infrastructure cmpnents supprt the business users SLAs, these agreements are referred t as Operatinal Level Agreements (OLAs) vclud Layers and SLAs A typical vclud cmputing envirnment cnsists f multiple layers (IaaS, PaaS, SaaS, and s n.). Based n business requirements, every custmer chses hw t implement the vclud stack. Optins include creating a private vclud, using a public vclud prvider, r creating a hybrid vclud mdel in which bth private and public vclud resurces are used. The enabler fr this flexibility is the ability f an rganizatin t guarantee the availability and perfrmance at every vclud layer. This is achieved by signing SLAs externally with service prviders, and fr a private vclud, creating SLAs and OLAs internally Example The fllwing is a use case example fr an rganizatin with an IaaS layer hsted by a public vclud prvider, but PaaS and SaaS layers are maintained internally. Figure 6. Example Organizatin with Public vclud IaaS and Private vclud PaaS/SaaS Layers

25 Nte The SLAs listed are fr illustratin purpses nly and are merely a subset f the ttal number f SLAs created within an rganizatin in such a case. SLAs fr this example are: IaaS Layer: Uptime/Availability SLA signed with the external vclud service prvider. Netwrk perfrmance SLA signed with the external service prvider. Request Fulfillment SLA Measure f respnse time fr prvisining and access cnfiguratin requests. Restre time SLA. PaaS Layer: Uptime/Availability SLA fr develpment envirnment. Uptime/Availability SLA fr critical develpment envirnment cmpnents. Restre time SLA fr develpment envirnment. SaaS Layer: Uptime/Availability SLA specific t an applicatin. Applicatin respnse time SLA Measure f hw the applicatin is perfrming fr the business users. Time t reslutin SLA Time t recver an applicatin in case f a failure. Given this example, the fllwing are sme key takeaways: SLAs are relevant at all levels within a vclud stack. SLAs are required t prvide efficiency and accuntability at every layer, fr bth external prviders and internal IT grups. Managing SLAs within every layer als helps islate systemic prblems and eliminates delays. SLAs can be between external vendrs r prviders f vclud services r between internal IT grups. An rganizatin can chse whether t implement a private, public, r hybrid clud. At every layer, SLAs give rganizatins this flexibility by guaranteeing availability and quality f service. There are interrelatinships between SLAs set up at different vclud layers. A change in quality f service r breach f an SLA at a lwer vclud layer may impact multiple SLAs in a higher vclud layer. In this example, if there is a breach f a perfrmance SLA that results in the external vclud prvider s inability t supprt OS perfrmance needs, this breach has a ripple effect at the SaaS layer, decreasing applicatin perfrmance and respnse time fr business users. SLAs need t be cntinuusly managed and evaluated t maintain quality f service within a vclud. Business needs are cntinuusly evlving, resulting in changing vclud business requirements. SLAs must be cntinuusly updated t reflect current business requirements. Cnsider the impact f adding anther 1000 users t a particular applicatin. Given this new demand the applicatin criticality is increased and the applicatin is classified as missin critical. This business change means that SLAs supprting the applicatin may need t be updated t prvide increased uptime and availability. This may lead t increased demands at the IaaS layer, s SLAs with the external IaaS prvider may have t be expanded.

26 4.2.6 vclud SLA Cnsideratins Sme example vclud SLA cnsideratins are: Uptime/Availability SLA: Business hurs Fr what timeframe des the SLA pertain? These are generally divided int tiers depending n business criticality (9 t 5, 24 by 7). Are maintenance windws (cnfiguratin changes, capacity changes, OS and applicatin patch management) included r excluded frm availability SLAs? Single versus multi-virtual machine vapps D multi-virtual machine vapps need t be treated as a single entity frm a SLA perspective? End User Respnse Time SLA This is generally fcused n end-t-end respnse time as perceived by the business user. This may require implementing remte simulatrs t measure and mnitr respnse time. Recvery (system, data) SLA What Recvery Time Objectives and Recvery Pint Objectives need t be met? Are backups required? Is high availability required? Is fault tlerance required within the management cluster? Is autmated disaster recvery failver required within certain time parameters? Privacy SLA (data security, access and cntrl): D data privacy requirements (encryptin, thers) exist? Are there regulatry requirements? Are specific rles and permissin grups required? Prvisining SLA Are there prvisining time requirements?

27 5. vclud Operatins Management vclud Operatins Management includes Cnfiguratin Management, Orchestratin Management, Availability Management, and Cntinuity Management. 5.1 Cnfiguratin Management Cnfiguratin Management is a key prcess fr realizing vclud peratinal benefits. Cnfiguratin Managers track cnfiguratin items using a cnfiguratin management system Cnfiguratin Management Definitin Cnfiguratin Management is the prcess respnsible fr defining and maintaining infrmatin abut cmpnents f an IT service these cmpnents are referred t as cnfiguratin items (CIs) and are managed end-t-end thrughut their lifecycle. The gal f cnfiguratin management is t understand the histrical, planned, and current state f cnfiguratin items, their interrelatinships, and their impact n an IT service Value f Cnfiguratin Management in a vclud T fully realize the benefits f vclud cmputing frm an peratins perspective, cnfiguratin management is the key. The cnfiguratin management prcess is administered in an rganizatin thrugh a set f tls and databases cllectively knwn as a Cnfiguratin Management System (CMS). The CMS acts as the definitive surce f recrd fr all cmpnents and their interrelatinships within a vclud envirnment. The CMS prvides the visibility IT needs t manage the multilayered vclud envirnments. Cnfiguratin management is the enabler fr critical vclud functins, frm autmated prvisining, discvery, and maintenance, t helping effectively plan and implement changes in the envirnment. Figure 7 shws interrelatinships between a Cnfiguratin Management System and ther critical IT Service Management prcesses.

28 Figure 7. Cnfiguratin Management Interrelatinships The fllwing sectins prvide a detailed review and explain hw cnfiguratin management supprts a vclud in an IT rganizatin Cnfiguratin Management in Relatin t Orchestratin and Prvisining in a vclud In the vclud envirnment users request new resurces (virtual machines, vapps) that are prvisined directly frm the service catalg. The enabler fr this autmated prvisining is the CMS. After a request is received frm a user via the service catalg the rchestratin engine interacts with the CMS. Fr example, t cnfirm available capacity, it determines cnfiguratin data fr prvisining as well as access and security restrictins, and cmmissins the new virtual machine. The CMS is then infrmed f this new virtual machine and begins its lifecycle management.

29 Cnfiguratin Management in a Multilayer Envirnment A vclud envirnment, as ppsed t a traditinal IT envirnment, is built f independent vclud layers. These layers can be built internally r created by the use f internal and external service prviders. T manage this new dynamic envirnment, the CMS rle becmes even mre critical. The CMS needs t have capabilities t expand and understand all f the internal and external cmpnents f an IT service. Therefre, a CMS needs integratin and recnciliatin abilities with multiple surces and needs t be service-centric t shw dependencies fr hw different cmpnents (internal r external) can impact an IT service Cnfiguratin Management Needs t Be Self-Aware A vclud envirnment is dynamic. The mbility f wrklads and resurces within the vclud makes tracking cmpnents a challenge. A CMS platfrm fr a vclud needs autmated discvery and mapping functins t make it self-aware. Tighter integratin with virtual vclud infrastructures allws fr real time views f a service t be created. This allws the IT rganizatin t rganize their vclud infrastructures better with mre predictive planning capabilities Cnfiguratin Management in relatin t Change Management A vclud has dynamic functins that require their wn set f change cnsideratins. These changes need t be preapprved by change management, and stred as cnfiguratin data within the CMS t be used by the rchestratin layer t execute. CMS als remains central t the change management prcess itself, prviding the visibility required by the change advisry bard t access the impact f a change in the envirnment Cnfiguratin Management in Relatin t a Service A CMS needs t be service-centric, with graphical views shwing IT service dependencies. The CMS shuld als be used t stre service-related infrmatin such as expected service levels and key stakehlder infrmatin. As IT rganizatins evlve t be mre service riented the CMS rle in relatin t service-related data increases. It will include nt nly cmpnent infrmatin, but infrmatin abut service dependencies and tls that enable utage simulatins t understand impacts. The CMS is respnsible fr predictive and trending analysis f nt nly technical cmpnents, but fr an IT service as a unit Cnfiguratin Management in Relatin t Physical vclud Infrastructure A vclud is cmprised f multiple service layers (SaaS, PaaS, IaaS, and pssibly thers). All f these vclud layers are supprted by a set f cre physical infrastructure cmpnents (server, strage, netwrk, and s n). As the vclud autmatically allcates resurces, the CMS needs t understand the impact f the changes t the physical cre infrastructure. The rle f the CMS within the capacity management functin is t prtect against ver-allcatin f resurces in rder t maintain stability and quality f service within a vclud vclud Cnfiguratin Management Cnsideratins As the vclud cntinues t evlve s des cnfiguratin management. Newer cnfiguratin management systems are emerging in the market. Organizatins that are mving t a vclud mdel need t cntinuusly reassess the capabilities f existing cnfiguratin management systems in relatin t the vclud. Opprtunities fr prcess imprvement and autmatin need t be reviewed during every assessment iteratin. The fllwing are sme cnsideratins t help evaluate CMS capabilities:

30 vclud-aware cnsideratins fr a CMS: Is it capable f mdeling lgical business services within a vclud? Des it understand bth physical and virtual cmpnents? Can it map dependencies between cmpnents? Can it aut-discver applicatins? Des it have an understanding f vclud data mdels? Fr example, prvider virtual data centers, rganizatin netwrks, vapps, and s n. Flexibility and cnfiguratin cnsideratins within a CMS: Des it have a dashbard that displays IT services and their cmpnents? Can it integrate with multiple data surces thrugh pluggable cnnectrs? Des it have a recnciliatin capability? Des is allw fr CI mapping? Des it allw fr dependency mapping? Can it manage the varius lifecycle states f vclud cmpnents? Fr example, Dev, UAT, Prd, EOL. End-t-end autmatin: Is it capable f integrating with rchestratin engines? Is it capable f integrating with vclud financial management and chargeback systems? Des it have built in cmmissin and decmmissin prcesses? Is it capable f integrating with Asset Management fr the allcatin and recvery f licenses? Typical vclud Cnfiguratin Items (CIs) The fllwing list is starting pint fr recmmended cnfiguratin items that shuld be represented in a CMS fr an Infrastructure as a Service vclud envirnment. VMware vcenter Server. vcenter Database. Datacenter. Cluster. Resurce pl/prvider virtual datacenter. Organizatin virtual datacenter. Organizatin netwrk. External netwrk. VMware ESXi hst. Virtual machine. Datastre.

31 Virtual distributed switch. Prt grup. vapp. Fr each cnfiguratin item there is an attribute r relatinship entry. Each CI has a CI ID, a CI type field, and a CI status field. 5.2 Orchestratin Management Orchestratin Management is respnsible fr gathering and understanding service rchestratin wrkflw requirements, managing their develpment, testing, and release, and interacting with the vce t integrate infrastructure-related autmatin wrkflws Orchestratin Management Definitin Orchestratin Management is the prcess respnsible fr gvernance and cntrl ver the rchestratin wrkflws and the resulting autmatin within the vclud. The gal f Orchestratin Management is t understand the impact f rchestratin wrkflws n an rganizatin s vclud, the apprvers f and thse wh benefit frm the rchestratin, and the interrelatins between rchestratin and traditinal IT service management prcesses Value f Orchestratin Management in a vclud Orchestratin abilities make a vclud dynamic. This key feature adds t vclud agility, elasticity, and selfhealing prperties. Alng with the benefits, elasticity als raises sme risks. A successful vclud implementatin must fcus n delivering cnsistent quality f services. Orchestratin Management adds the layer f cntrl required t achieve this cnsistency within a vclud. Anther aspect f cntrl is the ability t prtect and secure the vclud. Unwarranted actins within a vclud cannt be tlerated, s rchestratin wrkflws and actins need t be tightly cntrlled t prtect the vclud. The fllwing sectins prvide infrmatin abut hw rchestratin shuld be cntrlled in a vclud. Orchestratin is a relatively new feature, and as rganizatins mature in their management f vclud envirnments the rle f rchestratin management becmes mre and mre relevant Orchestratin Wrkflw Creatin Cntrl in a vclud Befre implementing rchestratin wrkflws within a vclud envirnment sme basic questins need t be answered: Wh apprved the rchestratin wrkflw? Why d we need it? What impact des this rchestratin wrkflw have n the vclud envirnment? Wh needs t be infrmed when this wrkflw is executed? These questins need t be answered fr all rchestratin wrkflws that are built int the vclud. VMware recmmends that tw separate teams be invlved during develpment f rchestratin wrkflws. The first team is the Orchestratin Management team that fcuses n business requirements gathering and business unit negtiatins. The secnd team is the vclud Center f Excellence team that fcuses n technical develpment f wrkflws. This prvides fr the implementatin f cnsistent standards acrss all rchestratin wrkflws within an rganizatin.

32 Develpment f rchestratin wrkflws is very cmplex. Orchestratin engages with multiple internal and external systems in a vclud envirnment, s a cmplete develpment lifecycle must be fllwed with dedicated supprt frm the applicatin and business teams. VMware recmmends that apprpriate testing be cmpleted at every stage f develpment, including unit, system, and integratin testing befre mving rchestratin wrkflws int prductin. As part f develpment testing, peratinal testing that includes perfrmance and scalability scenaris fr end-tend autmatin prcesses must als be cmpleted. In many cases, rchestratin wrkflws themselves may be able t withstand new lads, but external r dwnstream systems may experience a perfrmance impact. A clear rll-back prcedure must be established fr exceptins t prtect against impacting prductin functins Orchestratin Wrkflw Executin Cntrl in a vclud A vclud is a dynamic envirnment where cntinuus changes are made t imprve the quality f services that run n it. Orchestratin plays a key part in this agility, allwing fr autmated actins t be perfrmed as required by vclud. Orchestratin management fcuses n vclud impacts, and avids adding inflexibility in the envirnment. VMware recmmends that there be cntrl n the executin f rchestratin wrkflws develped fr vclud, with errr handling built int the wrkflws. If there are wrkflw executin issues, ntificatins need t be sent t the peratins team with apprpriate escalatins and tiering fr alerts Orchestratin Management in Relatin t Change Management Orchestratin leads t change in a vclud envirnment. As rchestratin becmes mre mature cmplex manual tasks are autmated. Wrkflws that lead t changes in business services that directly impact users need t be analyzed in detail befre implementatin. The Change Advisry Bard (CAB) needs t preapprve actins n prductin applicatins. Additinal cntrls may als be set, allwing fr ntificatin back t the CAB n executin f critical business that impacts rchestratin wrkflws. This must be dne in accrdance with an rganizatin's change cntrl plicies. Business impact shuld be the main driver fr discussin between the rchestratin team and CAB. Simple rchestratin actins that impact vclud internal backgrund peratins (fr example, capacity-related actins), but which d nt directly impact a business applicatin r service, shuld be allwed mre flexibility by the CAB and may nt need apprval Orchestratin Management in Relatin t Cnfiguratin Management Orchestratin can be used t prvisin new vapps within a vclud. Orchestratin needs t integrate with and prvide status n new r updated cnfiguratin items (CIs) t the Cnfiguratin Management System (CMS) t prvide cnsistency. Als, the CMS can trigger aut scaling actins fr vapps executed by an rchestratin wrkflw t prvide quality f service. Anther aspect f the relatinship between rchestratin and cnfiguratin management is the understanding f the physical layer that supprts the vclud envirnment. In mature implementatins, rchestratin can interact with the cnfiguratin management layer t understand gaps within the physical layer and remediate as needed t maintain envirnment stability (fr example, adding new strage capacity).

33 Orchestratin Management in Relatin t Security vclud-based services are fcused n business users, allwing them t request new services directly via the service catalg. Orchestratin is critical t such autmatin, and shuld have an API t cmmunicate with external systems. Orchestratin adds flexibility within a vclud. With flexibility cmes a requirement t add cntrls such that there are n security risks r expsure fr the rganizatin. Because the rchestratin wrkflws have access rights t multiple systems, the rchestratin wrkflw cde needs t be prtected. Encryptin cntrls such as Set Digital Rights management need t be enabled while mving wrkflw cde packages within servers. Als, access t the rchestratin servers must be limited. VMware recmmends that the vclud Center f Excellence exclusively cntrl and manage access n these servers Orchestratin Management in Relatin t Audit and Cmpliance As nted, rchestratin wrkflws enable vclud t be mre dynamic. Autmated actins enhance key vclud functins such as prvisining and self-service. Thugh enhanced autmatin is very beneficial, it pses a challenge t an rganizatin that is bund by tight audit, regulatry, and cmpliance rules. VMware recmmends that rchestratin engines running the rchestratin wrkflws are centralized within an rganizatin. Centralized errr handing and lgging is recmmended fr all wrkflws. Reprting features that checkpint all wrkflw actins must be enabled fr audit cmpliance. Centralized rchestratin engines als enhance an rganizatin s prblem management and rt-cause analysis capabilities. Sme f the recmmended rchestratin management principles cannt currently be fully autmated and require manual cnfiguratin actins based n individual client needs. VMware cntinues t imprve existing libraries and as vclud implementatins mature, mre packaged rchestratins with cntrl and gvernance features shuld be available fr clients t dwnlad. 5.3 Availability Management Availability Management fcuses n making sure that the level f availability prvided fr all vclud service fferings meets r exceeds the agreed service level requirements in a cst-effective manner. Managing availability within a vclud envirnment depends n VMware vclud Directr cmpnent availability as well as the resilience f the underlying infrastructure. VCD wrks transparently with VMware vcenter Server t prvisin and deply virtual machines n hsts. Therefre, it is imperative t architect redundancy and prtect the infrastructure cmpnents. Prvisined virtual machines can be prtected by VMware vsphere High Availability (HA). Virtual machines can als be prtected using backup tls within the guest OS r vstrage API (vstrage APIs fr Data Prtectin (VADP)-based) applicatins. See Sectin 5.4.2, Backup and Restre f vapps, fr additinal infrmatin. At this time, virtual machines prvisined by VCD cannt be prtected by VMware Fault Tlerance (FT) r VMware vcenter Site Recvery Manager (SRM).

34 5.3.1 Uptime SLAs VMware vclud cmpnents supprt a 99.9% uptime SLA ut-f-the-bx. This may be sufficient fr nncritical applicatins r applicatins that are inherently highly available. Fr vclud, uptime SLAs typically require verificatin that: End custmer wrklads are running. End custmer wrklads are accessible (via the vclud prtal and API, as well as thrugh remte access prtcls). In sme cases a prvider (either an external service prvider r internal IT) may want t increase the vclud uptime SLA. VMware can nly cntrl the resiliency f its vclud platfrm cmpnents and prvide recmmendatins t mitigate single pints f failure (SPOF) in the underlying infrastructure. A prvider can eliminate SPOF by prviding redundancy. Fr example: Redundant pwer surced frm multiple feeds, with multiple whips t racks, as well as sufficient backup battery and generatr capacity. Redundant netwrk cmpnents. Redundant strage cmpnents: Strage design needs t be able t handle the I/O lad as well. Custmer wrklads may nt be accessible under high disk latency, file lcks, and s frth. Strage design shuld als be tied t business cntinuity and disaster recvery plans, pssibly including array-level backups. Redundant server cmpnents (multiple independent pwer supplies, netwrk interface cards (NICs) and, if apprpriate, hst bus adaptrs (HBAs). Sufficient cmpute resurces fr a minimum f N+1 redundancy within a vsphere high availability cluster including sufficient capacity fr timely recvery. Redundant databases and management. Apprpriate change, incident, prblem and capacity management prcesses must als be well defined and enfrced t make sure that pr peratinal prcesses d nt result in unnecessary dwntime. In additin t a redundant infrastructure, emplyees r cntractrs respnsible fr perating and maintaining the envirnment and the supprting infrastructure must be adequately trained and skilled. Fr mre detailed infrmatin abut increasing vclud cmpnent resiliency, refer t the vclud Availability Cnsideratins sectin f Architecting a VMware vclud.

35 5.4 Cntinuity Management Cntinuity Management fr vclud fcuses n making sure that the vclud-based service fferings, as well as the infrastructure upn which they are hsted, can be resumed within an agreed timeframe in the case f a disruptin f service regardless f whether the utage is at the vapp level r an entire vclud envirnment instance. In this cntext, VMware defines tw cmpnents t Cntinuity Management: Disaster Recvery (strategic), and vapp Backup and Restre (tactical) Disaster Recvery Disaster Recvery (DR) fcuses n the recvery f systems and infrastructure after an incident that interrupts nrmal peratins. A disaster can be defined as partial r cmplete unavailability f resurces and services, including sftware, the virtualizatin layer, the vclud layer, and the wrklads running in the resurce grups. Different appraches and technlgies are supprted, but there are at least tw areas that require disaster recvery: the management cluster and cnsumer resurces. Different appraches and technlgies are supprted Management Cluster Disaster Recvery Gd practices at the infrastructure level lead t easier disaster recvery f the management cluster. This includes technlgies such as HA and DRS fr reactive and practive prtectin at the primary site. VMware vcenter Heartbeat can als be used t prtect vcenter Server, specifically, at the primary site. Fr multi-site prtectin f virtual machines, VMware vcenter Site Recvery Manager is a VMware slutin that wrks well fr this use case, because the management virtual machines are nt part f a vclud instance f any type (they run the vclud instances) vclud Cnsumer Resurces Disaster Recvery The vclud infrastructure can be failed ver t an alternate site, but vcenter Site Recvery Manager is nt supprted. Manual prcedures can be applied as lng as vapp metadata is saved, cnfiguratin infrmatin is matched between the primary site and the recvery site, and the dcumented steps are validated. Thugh SRM is vcenter Server-aware, SRM is nt vclud Directr-aware. Withut the cllabratin between vclud Directr and SRM, the underlying mechanisms that wrk t synchrnize virtual machines cannt wrk t keep vclud Directr in sync as well thus, the recvery f vclud Directr can be prblematic. Thugh it is pssible t architect a slutin where ne site s ttal envirnment (100% f the peratinal parameters f that site including IP addressing, start-up rder f dependent systems and the like) can be duplicated t anther site, it wuld be very difficult t implement and maintain.

36 5.4.2 Backup and Restre f vapps This sectin fcuses n handling f backup and restre prcedures fr the vapps that are deplyed int the vclud. Traditinal backup tls d nt capture the required metadata assciated with a vapp, such as wner, netwrk, and rganizatin. This results in recvery and restratin issues. Withut this data, recvery must include manual steps and cnfiguratin attributes t be manually reentered. Within a vclud envirnment, a vapp can be a single virtual machine r grup f virtual machines, treated as ne bject. Backup f vapps n islated netwrks must be supprted. Identifying inventries f individual rganizatins becmes challenging based n current methds that enumerate the backup items using vsphere, which uses Universally Unique Identifiers t differentiate bjects. vclud Directr uses bject identifiers. Fr backing up and restring vapps, VMware recmmends the use f VMware vsphere Strage APIs Data Prtectin based backup technlgies. This technlgy has n agents n guest perating systems, is centralized fr imprved manageability, and has a reduced dependency n backup windws. Guest-based backup slutins may nt wrk in a vclud because nt all virtual machines are accessible by netwrk. Als, virtual machines may have identical IP addresses and that can cause prblems. Therefre, backups f vclud vapps require a virtual machine-level apprach. Use the full name and cmputer name fields t specify realistic names that will help describe the virtual machines when deplying virtual machines (as part f a vapp). If this is nt dne, the generic infrmatin in these fields can make it difficult t specify individual virtual machines. vapps and virtual machines that are prvisined by vclud Directr have a large GUID template_name s that many virtual machines culd appear t be very similar, making it difficult fr a user r administratr t identify and ask fr a specific virtual machine t be restred VMware Slutins VMware Data Recvery is a vstrage APIs fr Data Prtectin-based slutin. Other vstrage APIs fr Data Prtectin-based backup technlgies are available frm third-party backup vendrs. Currently, due t the Universally Unique Identifier versus bject identifier issue, VMware Data Recvery cannt be used with VMware vclud Directr. There are a few requirements t address fr backup f vclud wrklads. VMware recmmends that clients validate the level f supprt prvided by the vendr t make sure client requirements are supprted. Table 3 prvides a list f vclud vapp requirements t ask yur vendr abut. Table 3. vclud vapp Requirements Checklist vapp Requirement vstrage API Data Prtectin integratin Detail vstrage API Data Prtectin prvides change-blck tracking capability t reduce backup windws. Integratin t enable backup f islated virtual machines and vapps. Integratin with vstrage API Data Prtectin t prvide LAN-free and server-free backups t supprt better cnslidatin ratins fr vclud and the underlying vsphere infrastructure. Use f the virtual machine Universally Unique Identifier versus virtual machine name will supprt multitenancy and avid ptential name space cnflicts.

37 vapp Requirement vclud Directr integratin vapp requirements Detail Interface supprt fr vclud prvider administratr teams. In the future, cnsumer (rganizatin administratr and users) access may be prvided by sme vendrs. Include vclud metadata fr the vapps. This includes temprary and permanent metadata per virtual machine r vapp. This is required t make sure that recvery f the virtual machine r vapp has all data required t supprt resurce requirements and SLAs. Prvide vapp granularity fr backups. Supprt backup f multitiered vapps (fr example, a Micrsft Exchange vapp that has multiple virtual machines included. Backup selectin f the Exchange vapp wuld pick up all the underlying virtual machines that are part f the main vapp).this capability is nt available tday, but is being develped by vendrs Challenges Challenges assciated with backing up and restring a vclud are: vapp naming pses cnflict issues between tenants. vapp metadata required fr recvery. Multi-bject vapp backup (prtectin grups fr multitiered vapps). Manual recvery steps in the vclud. Supprt fr backup f vapps n islated netwrks r with n netwrk cnnectivity. Enumeratin f vapps by rganizatin fr use by the rganizatin administratr. Enumeratin f vapps by rganizatin and prvider fr use by the rganizatin prvider. User initiated backup/recvery. Supprt f prvider (prvider administratr) and cnsumer (rganizatin administratr and user). Fr a mre detailed treatment f vclud Business Cntinuity, see Appendix F: Business Cntinuity.

38 6. vclud Infrastructure Management vclud Infrastructure Management includes Security and Cmpliance Management, Capacity Management, Perfrmance Management, and mnitring. 6.1 Security and Cmpliance Management The fllwing sectins describe Security Management fr vclud access management and lgging Access Management User Access Security The service prvider has t cnfigure a directry service fr vclud Directr. This is als true fr a private vclud. Authenticatin and authrizatin mechanisms built int vclud Directr prvide user access security fr vclud resurces. vclud Directr can be cnfigured t integrate with a directry service (LDAPv3) such as Active Directry, OpenLDAP, r Kerbers v5. Refer t the VMware vclud Directr Administratin Guide ( fr mre infrmatin abut hw t set up the naming services (LDAPv3), Active Directry, r OpenLDAP and Kerbers v5 integratin. User authrizatin within vclud Directr is cntrlled thrugh rle-based access cntrl (RBAC). Als refer t the latest VMware vclud Directr Administratr s Guide fr additinal infrmatin n permissins, rles, and default settings Lg Management Lgs shuld be available fr custmers and their prviders in a vclud fr numerus reasns, including: Regulatry Cmpliance Cllect lgs t make them available fr analysis, security review, and cmpliance requirements as described in Appendix B: Cmpliance Cnsideratins. Individual lgs can be used t satisfy specific cmpliance cntrls; fr example, a user access lg can be used t shw an audit trail fr user access success and failure. Custmer Requirements End custmers (tenants) can retrieve lgs that pertain t their envirnment t satisfy their wn requirements, many f which, such as cmpliance, will prbably be similar t prvider requirements. Operatinal Integrity Operatinal alerts shuld be defined s that specific lgs trigger ntificatins fr further remediatin. This is typically a backup alert, and is secndary t mnitring. Trubleshting Clsely related t peratinal integrity, trubleshting can be perfrmed using lgs. Fr example, VMware vshield Edge lgs can shw whether r nt a specific external cnnectin request is being passed thrugh the firewall r via Netwrk Address Translatin (NAT) by the firewall.

39 Lgging Architectural Cnsideratins Redundancy Many cmpnents rely n syslg fr lgging events. Syslg is a UDP-based prtcl that lacks delivery guarantees. T facilitate delivery: Verify that infrastructure cmpnents have physically and lgically redundant netwrk interfaces. Send lgs t mre than ne syslg target. If nly ne syslg target is pssible, VMware recmmends lgging t a lcal syslg daemn that is cnfigured t retransmit t tw remte syslg targets. VMware vclud Directr 1.5 supprts nly a single syslg target fr its activity lgs. Place lg receivers n DRS-enabled hsts if pssible s that vcenter can restart them in case f failure. Scalability vclud infrastructure cmpnents generate a relatively lw level f lgs fr prvider infrastructure. Custmer cmpnents, especially vshield Edge firewalls, can generate a very high vlume f lgs. Cllecting lgs n IOPS perfrmance is critical. Cllecting lgs n CPU perfrmance is negligible, but they are needed fr analysis. It is strngly recmmended that lgs be cllected t dedicated lg partitins n cllectin servers. Reprting: Lgs need t be available t custmers. Custmers shuld be able t dwnlad in raw frmat all vclud Directr and vshield Edge lgs that pertain t their rganizatins and netwrks. Lgs with custmer identifiers shuld be flagged r indexed fr retrieval. Custmer activity in vclud Directr generates lgs that are flagged with their rganizatin identifier. vshield Edge devices can be uniquely identified. vclud Directr 1.5 can deply vshield Edges with descriptive, unique names, such that Security Event and Incident Management (SEIM) prducts can definitively crrelate lg messages frm vshield Edge appliances t the rganizatin that generated them. Figure 8 shws an example lgging architecture. Figure 8. Architectural Example Drawing

40 Lgging as a Service Lgging as a service can be dne with custmer cllectin and frwarding t prvider servers fr analysis and reprting, r with custmer cllectin, reprting, and analysis in the custmer envirnment and prvider lgs frwarded t the custmer envirnment. Custmer cllectin frwarding t prvider: Prs: Lgs can be sent directly t cllectr even n custmer private IP space. Resurces can be allcated at the custmer level fr cllectin, allwing mre granular scaling f cllectin. Cns: Mre difficult t scale analysis; challenges crrelating custmer activity t strage cnsumptin. Cllectin ndes still required even thugh utilizatin will be lw. Mst f the resurce cnsumptin is n the strage and analysis side, s the resurces billed using the IaaS mdel will be minimal. Custmer cllectin, with prvider lgs frwarded int custmer envirnment: Prs: Distributed analysis relies n general vclud resurces and can scale. Custmer can emply their wn analysis tls t rganize and reprt n the data, r use a prvider-supprted package r appliance. Cns: Prvider needs duplicate cpy f infrastructure lgs fr prvider purpses. Transmissin f lgs t the custmer envirnment requires cnnectivity; either Internet r a prvider service netwrk and inbund traffic thrugh a firewall int the custmer envirnment, adding risks.

41 6.2 Capacity Management Capacity Management fcuses n prviding vclud capacity t meet bth existing and future needs in supprt f vclud service fferings. Management cluster sizing is fairly predictable, with the main variables being the number f vclud Directr cells and the size f the vclud Directr database. Initial sizing guidelines fr the management cluster are prvided in Architecting a VMware vclud. The vclud cnsumer resurces have unpredictable usage, and thus shuld be sized by making an estimate f the initial capacity required, and by emplying capacity management techniques. Capacity management techniques predict future usage needs based upn past usage trends Onging Capacity Management One f the key benefits f implementing a vclud is the ability fr service prvider custmers (equivalent t public vclud prviders r private vclud internal IT) t rapidly prvisin vapps int the vclud envirnment. The gal f capacity management is t make sure that sufficient capacity exists within the vclud infrastructure t meet the current and future needs f the service prvider custmers under nrmal circumstances. Sufficient reserve capacity must be maintained within the vclud infrastructure t prevent vapps frm cntending fr resurces, and thus ptentially breaching agreed services levels. As vapps are prvisined and cnsumed within the vclud infrastructure, available capacity is reduced and additinal capacity must be prcured and prvisined. Capacity Management prcesses shuld be instituted t make sure apprpriate resurces are available t supprt the service level requirements assciated with vapp prvisining and perfrmance. Prper capacity management als prevents cstly ver-prvisining f hardware resurces by balancing high resurce utilizatin with agreed-upn levels f perfrmance. As the vclud is cnsumed, additinal capacity must be added t the vclud cnsumer resurces t allw fr anticipated future demand while preserving sufficient headrm. T predict future capacity needs, analyze current capacity usage and trends t determine grwth rates as well as estimate future needs, largely cming frm new cnsumers and prjects. VMware vcenter CapacityIQ is a tl that can be used t mnitr and predict capacity usage and requirements. In a vclud envirnment, CapacityIQ can prvide details n capacity at the virtual machine and hst levels, as well as the prvider virtual datacenter-level (because, fllwing VMware best practices, a prvider virtual datacenter is equivalent t a vsphere cluster), but it currently des nt prvide insight at the rganizatin virtual datacenter level. Fr further infrmatin n capacity planning and management, refer t the latest VMware vcenter CapacityIQ Installatin Guide ( Als see Appendix C: Capacity Planning fr guidance n hw t manually calculate capacity requirements and frecast capacity.

42 6.3 Perfrmance Management A key IT paradigm shift enabled by vclud cmputing is that f delivering services t end users. Ultimately these are business user services; service fferings directly cnsumed by end users that als include technlgy services. Technlgy services are nt directly cnsumed by end users, but represent the underlying infrastructure cmpnents which, when cnsidered tgether, enable the cnsumptin f business user services. One f the key challenges at the vclud Operatins Management layer cncerns rerienting peratins prcesses arund sustaining the delivery f a business user service as a whle, as ppsed t fcusing n perating the underlying infrastructure cmpnents as discreet entities. This challenge manifests itself acrss several key peratins prcesses, nt the least f which is Perfrmance Management. It is n lnger sufficient t mnitr and manage the perfrmance f individual IT infrastructure cmpnents. With the adptin f vclud cmputing, IT must mnitr and manage the perfrmance f the business user service and, mre precisely, the business user s service frm the end-user s perspective. The fcus is n hw the service is perfrming fr the end user, as ppsed t cnsidering individually the netwrk perfrmance r server perfrmance upn which a particular applicatin is hsted. Perfrmance Management frm the service end-user perspective can have far reaching ramificatins. These might range frm deplying remte prbes that can simulate an end user s interactin with a service while tracking perfrmance metrics, t mre integrated perfrmance mnitring, filtering, and analytics slutins that understand what cmprises a business user service, mnitrs the perfrmance f the cmpnents in the cntext f the cmplete service, and can take autmated actin based n early warning (predictive) smart alerts. Alng with these mre sphisticated tls cme the peple, prcess, and prcess interrelatinship mdificatins that are required t efficiently and effectively cmplement them. Based n this service rientatin, VMware prvides the fllwing cnsideratins and guidance fr vclud Perfrmance Management frm a peple, prcess, and tls perspective Peple Perspective Within the Operatins Management layer, assign a Perfrmance Management wner wh has the fllwing respnsibilities: Overall respnsibility fr the perfrmance management prcess. Regular interactin with ther, related prcess wners. Establishing agreed t Perfrmance Management Key Perfrmance Indicatrs (KPIs). Tracking and reprting n KPIs. Wrking with the vclud Center f Excellence architect t design a Perfrmance Management mnitring slutin that is business user service-riented. Establish a training plan fr the tls t be used by the Perfrmance Management wner and ther designated individuals.

43 6.3.2 Prcess Perspective Define a business user service-fcused perfrmance management prcess. Include an interface with Service Level Management fr perfrmance reprting in supprt f Service Level Agreements. Include an interface with Capacity Management fr frward-lking capacity requirements. Include an interface with event, incident, and prblem management t prvide Tier 3 incident reslutin supprt Tl Perspective VMware recmmends the fllwing fr a vclud Perfrmance Management tl: Integrates with vclud Directr, the underlying vsphere infrastructure, and third-party mnitring tls fr a hlistic perfrmance view. Aggregates, crrelates, and presents perfrmance data in the cntext f business user service health and perfrmance. Simplifies management f alarms by using a cmbinatin f static and dynamic threshlds based n self-learning. Capable f predictive analysis resulting in alerts f impending perfrmance degradatin. Crrelates cause and effect fr perfrmance prblem reslutin. Prvides visibility and drills dwn frm business user service t cmpnent level. Thugh n tl is currently available in the industry that prvides all f these capabilities ut-f-the-bx, the VMware vcenter Operatins Enterprise management slutin, in cmbinatin with the vclud Directr adapter, prvides capabilities t assess the health f yur vclud infrastructure frm a perfrmance perspective. This is visualized using health scres, heat maps, and health trees at varius aggregatin levels frm prvider virtual datacenters and rganizatins t virtual machines. 6.4 Mnitring Mnitring the cmpnents f a vclud Directr implementatin is essential t the health f a vclud envirnment, and is necessary t maintain capacity and meet service level agreements. This sectin prvides recmmendatins n what systems and assciated bjects t mnitr, and readily available tls that can be used t extract health-related metrics. Details f specific limits r threshlds are nt identified here as they are available in the prduct dcumentatin. This dcument des nt attempt t prvide specifics fr setting up a mnitring slutin as varius service prviders and enterprises may have very different mnitring slutins in place t be integrated.

44 6.4.1 Management Cluster The best practices fr mnitring the management cluster cmpnents are the same as the best practices fr mnitring vsphere cmpnents. As part f this, a centralized mnitring tl such as VMware vfabric Hyperic HQ Enterprise can be used t mnitr the cre bjects (Oracle Server, SQL Server, Active Directry Server, DNS Server, Red Hat Enterprise Linux Server, and Windws Server) that are needed t run a vclud envirnment. A custmer can use SNMP and SMASH t mnitr the hsts n which the vclud Directr cells are installed and running, but the vclud Directr applicatin itself cannt be mnitred by SNMP r SMASH. Hwever, SNMP can be integrated frm vcenter. Alternatively, cells can be mnitred thrugh integratin with a third-party mnitring platfrm via JMX Beans. Nte that JMX Beans mnitring is nly the start. The vclud and vsphere APIs prvide a significant amunt f cmpnent, resurce, and activity metrics that can be used fr health and capacity management Clud Cnsumer Resurces and Wrklads The best practices fr mnitring the vclud cnsumer resurces and wrklads are the same as fr mnitring vsphere. Hwever, there are additinal vclud-specific cnsideratins fr VMware vshield Edge and vclud cnsumer wrklads vshield Edge vshield Edge appliances are self-cntained envirnments that are stateless in nature. There is a health check API call that can be made t a vshield Edge appliance t determine if it is functining crrectly. If the API returns negative, initiate a rebt f the vshield Edge device. At the time f rebt, cnfiguratin infrmatin is updated frm the VMware vshield Manager and the vshield Edge device cntinues t functin prperly vclud Cnsumer Wrklads It may be desirable t mnitr wrklads prvisined by vclud cnsumers. vclud Directr des nt prvide any built-in mnitring f wrklads fr availability r perfrmance. Several third-party slutins are available t mnitr vsphere resurces and wrklads running n vsphere; hwever, nt all f these slutins may wrk all f the time when vclud Directr is in use. Islated netwrking in vapps may prevent mnitring tls frm acquiring the perfrmance r availability infrmatin f a vapp. Furthermre, vapps may be prvisined and de-prvisined r pwer-cycled at any time by a vclud cnsumer and these actins may create false psitives in the mnitring envirnment. Until there are slutins in the market that are fully integrated with vclud Directr, it may be difficult t prvide detailed mnitring fr vclud cnsumer wrklads.

45 Appendix A: vclud Directr Cell Mnitring The fllwing table represents a subset f MBeans that can be used fr imprving the mnitring perfrmance f a vclud instance. Table 4. MBeans Used t Mnitr vclud Cells Lcal user sessins Mbean Descriptin cm.vmware.vclud.diagnstics.usersessins Lcal (cell) user sessin statistics Cardinality 1 Instance ID Attribute ttalsessins successfullgins failedlgins Glbal user sessins Mbean Descriptin n/a Descriptin Ttal number f sessins created n this cell Ttal number f successful lgins t this cell Ttal number f failed lgin requests t this cell cm.vmware.vclud.glbalusersessinstatistics List f active user sessins by rganizatin. Cardinality 1 Instance ID n/a Attribute Descriptin rganizatin Database ID f the rganizatin active Number f active sessins Open_Sessin Number f pen sessins Data access diagnstics Mbean cm.vmware.vclud.diagnstics.dataaccess Descriptin Lcal (cell) user sessin statistics Cardinality 1 Instance ID Cnversatin Attribute Descriptin lastaccessinf.bjecttype Object type f the last database bject accessed lastaccessinf.accesstime Time taken t access the last database bject accessed wrstaccessinf.bjecttype Object type f the wrst (slwest) database bject access wrstaccessinf.accesstime Time taken by the wrst (slwest) database bject access Database Cnnectin Pl

46 Mbean Descriptin Cardinality 1 Instance ID Attribute abandnedcnnectintimeut availablecnnectinscunt brrwedcnnectinscunt cnnectinharvestmaxcunt cnnectinharvesttriggercunt cnnectinplname cnnectinwaittimeut databasename datasurcename fastcnnectinfailverenabled inactivecnnectintimeut initialplsize lgintimeut maxcnnectinreusecunt maxidletime maxplsize maxstatements minplsize netwrkprtcl ONSCnfiguratin prtnumber SQLFrValidateCnnectin timeutcheckinterval timetlivecnnectintimeut URL user validatecnnectinonbrrw cm.vmware.vclud.datasurce.glbaldatasurce Statistics and cnfiguratin infrmatin abut the database cnnectin pl. This infrmatin is currently specific t the database JDBC driver being used (Oracle). Descriptin Database cnnectin database name (SID) Maximum number f cnnectins allwed in the pl Minimum number f cnnectins that will exist in the pl Netwrk prtcl used by JDBC driver Database cnnectin prt number Database cnnectin URL Database cnnectin username

47 VIM Operatins Mbean Descriptin Cardinality Instance ID Attribute ObjectType.MethdName.httpTime Presentatin API Methds Mbean Descriptin Cardinality Instance ID Attribute currentinvcatins ttalfailed ttalinvcatins executintime Jetty Mbean Descriptin Cardinality Instance ID Attribute Active cm.vmware.vclud.diagnstics.vlsioperatins Lcal (cell) user sessin statistics 1 per VIM end-pint (VC r hst agent) VIM end-pint URL Descriptin The ttal netwrk rund-trip time taken t make the MethdName call n bject f type ObjectType in the VIM endpint. cm.vmware.vclud.diagnstics.vlsioperatins Lcal (cell) user sessin statistics 1 per presentatin layer methd methd name Descriptin Currently active invcatins Ttal number f failed executins Ttal number f invcatins ver time Ttal time taken t execute cm.vmware.vclud.diagnstics.jetty Web server request statistics 2:1 fr REST API and 1 fr UI "UI Requests" fr UI, "REST API Requests" fr REST API Descriptin Number f Web requests currently being handled REST API

48 Mbean Descriptin Cardinality Instance ID Attribute currentinvcatins ttalfailed ttalinvcatins executintime Task Executin Mbean Descriptin Cardinality Instance ID Attribute currentinvcatins ttalfailed ttalinvcatins executintime Query Service (UI) Mbean Descriptin Cardinality Instance ID Attribute currentinvcatins ttalfailed ttalinvcatins executintime returneditems VC Task Manager cm.vmware.vclud.diagnstics.vlsioperatins Lcal (cell) user sessin statistics 1 per peratin stage/granularity: RundTrip, BasicLgin, Lgut, Authenticatin, SecurityFilter, CnversatinFilter, JAXRSServlet. RundTrip is the mst interesting, as it represents the verall REST API perfrmance. One f: RundTrip, BasicLgin, Lgut, Authenticatin, SecurityFilter, CnversatinFilter, JAXRSServlet Descriptin Currently active invcatins Ttal number f failed executins Ttal number f invcatins ver time Ttal time taken t execute cm.vmware.vclud.diagnstics.taskexecutinjbs Statistics abut lng running tasks 1 per task Name f task Descriptin Currently active invcatins Ttal number f failed executins Ttal number f invcatins ver time Ttal time taken t execute cm.vmware.vclud.diagnstics.queryservice Presentatin layer query service statistics 1 per query query name Descriptin Currently active invcatins Ttal number f failed executins Ttal number f invcatins ver time Ttal time taken t execute Number f items returned by successful query executins

49 Mbean Descriptin Cardinality 1 Instance ID Attribute successfultaskscunt failedtaskscunt waitfrtaskinvcatinscunt cmpletedwaitfrtaskscunt histricaltaskscunt vcretrievedtaskcmpletinscunt taskcmpletinmessagespublishedcunt taskcmpletinmessagesreceivedcunt success_elapsedtaskwaittime failed_elapsedtaskwaittime cm.vmware.vclud.diagnstics.vctasks VC task management statistics Descriptin ttal successful tasks ttal failed tasks ttal invcatins f VIM "wait fr task" ttal cmpleted task waits ttal histrical task updates received ttal task cmpletins received ttal task cmpletin messages published n message bus ttal task cmpletin messages received n message bus time elapsed fr successful tasks time elapsed fr failed tasks VIM Inventry Update Prcessing Object Update Statistics Mbean Descriptin Cardinality Instance ID Attribute ttalupdates ttalfailed executintime cm.vmware.vclud.diagnstics.viminventryupdates Inventry prcessing statistics 3: ne fr ObjectUpdate, PrpertyCllectr and UpdateSets respectively ObjectUpdate Descriptin Ttal number f bject updates received Ttal number f bject updates failed t be prcessed Time taken fr updates VIM Inventry Events

50 Mbean Descriptin Cardinality Instance ID Attribute ttalinvcatins ttalfailed executintime VC Object Validatins Mbean Descriptin Cardinality Instance ID Attribute ttalinvcatins executintime ttalitemsinqueue bjectsinqueue bjectbusyrequeuecunt ladvalidatinobjecttime duplicatesdiscarded VC Object Validatin Reactins Mbean Descriptin Cardinality Instance ID Attribute ttalreactinsfired requeuecunt ttalinvcatins executintime cm.vmware.vclud.diagnstics.viminventryevents VIM inventry event manager statistics. Tracks the frequency f cmmn vcenter events. 1 per flder per VC URL, 1MBean per event name Event name Descriptin Ttal number f VIM inventry events dispatched since that VCD cell started Ttal number f VIM inventry events that were failed t be handled Ttal time t handle VIM inventry events cm.vmware.vclud.diagnstics.vcvalidatin VC bject validatin statistics 1 glbal plus 1 per validatr null = glbal, validatr name = per validatr Descriptin Ttal number f validatin executins Ttal time spent in validatr Ttal items currently queued fr validatin (glbal) Ttal items currently queued fr validatin (per validatr) Ttal number f bjects re-queued fr validatin due t bject being busy Time taken t lad validatin bject Ttal number f discarded duplicate validatins cm.vmware.vclud.diagnstics.reactins validatin reactin statistics 1 glbal plus 1 per reactin null = glbal, reactin name = per reactin Descriptin Ttal number f reactin executins Ttal number f reactins re-queued due t bjects being busy Ttal number f executins f this reactin Ttal time spent in reactin

51 failedreactins bjectrequeuecunt VC cnnectins Mbean Descriptin Cardinality Instance ID Attribute Cnnected Cunt Discnnected Cunt Start Cunt UI Vim Recnnect Cunt ActiveMQ Mbean Descriptin Cardinality Instance ID Attribute lasthealthcheckdate messagerundtripduratinms ishealthy timedoutmessages senderrrs crruptedorbadechmessages Ttal number f failed reactins Number f times this reactin was re-queued due t bjects being busy cm.vmware.vclud.diagnstics.vimcnnectin Lcal (cell) user sessin statistics 1 per VC "VC-VcInstanceId" where VcInstanceId is an integer identifying the vcenter instance Descriptin Ttal successful cnnectins Ttal discnnectins Ttal number f times the VC listener was started Ttal number f times the VC was recnnected thrugh the UI cm.vmware.vclud.diagnstics.activemq Active MQ (message bus) statistics 1 glbal and 1 per peer vclud Directr cell (each cell ther than the current ne) "Glbal" = glbal statistics"t_cellname_cellprimaryip_celluuid"=per cell Descriptin Last time health check was perfrmed (date/time) Time taken fr an ech message t be sent and returned (ms) Health f cnnectin t peer cell in the case f the per-cell Mbean, verall message bus cnnectin health in the case f the glbal Mbean (true/false) Ttal number f ech messages fr which n reply was received within the timeut (cntrlled by the activemnitrcheckdelayms cnfig parameter, default 10 minutes) Ttal number f failed ech message sends (messages) Ttal number f crrupted/bad ech messages received (starts)_ (messages) Transfer Server Mbean cm.vmware.vclud.diagnstics.vlsioperatins

52 Descriptin Transfer server statistics Cardinality 1 Instance ID Attribute successfulputs failedputs successfuluplads acceptedquarantinedtransfersessins rejectedquarantinedtransfersessins expiredtransfersessins Descriptin Number f items successfully transferred (transfer items) Number f items that were failed t be transferred (transfer items) Number f successful uplad peratins (uplads) Number f quarantined transfers which were accepted (quarantined items) Number f quarantined transfers which were rejected (quarantined items) Number f transfer sessins which timed ut (transfer sessins)

53 Appendix B: Cmpliance Cnsideratins Cmpliance Audit cncepts such as segmentatin and mnitring applied t a vclud envirnment reveal new challenges. Elasticity may break ld segmentatin cntrls and the ability t islate sensitive data within a rapidly grwing envirnment. Rle-based access cntrls and virtual firewalls must als demnstrate cmpatibility with audit requirements fr segmentatin, including detailed audit trails and lgs. Can a prvider guarantee that an ffline image with sensitive data in memry is accessible nly by authrized users, and can a lg tell wh accessed it and when? Multiple admin-level rles are necessary fr vclud resurce management. The cmplexity f clud envirnments, cupled with new and different technlgy, requires careful audits t dcument and detail cmpliance. Table 5 lists cmmn audit cncerns within the vclud. Table 5. Audit Cncerns Within the vclud Cncern Hypervisr Segmentatin and islatin Different/multiple primary functins per hst Detail An additinal layer f technlgy is present in every vclud and may present an attack surface. It intrduces a layer between the traditinal prcessing envirnment and the physical layer, which brings a new level f cmmunicatin with layers abve and belw it. Any envirnment may expse sensitive data when nt cnfigured and mnitred prperly; physical and lgical islatin has always been an audit cncern. The ease and speed f change t a virtualized envirnment within clud cmputing, ften called elasticity, makes the setup and review f segmentatin cntrls even mre relevant t cmpliance thrugh islatin. The vclud envirnment can make mre efficient use f hardware, but it increases the prximity f infrmatin in transit and at rest. Sme cmpliance standards explicitly require ne primary functin per server (r virtual server), as illustrated in the fllwing figure. Figure 9. One Primary Functin per Server

54 Enfrcement f least privilege Machine state and migratin Data is much less permanent Immaturity f mnitring slutins in vclud envirnments In a vclud envirnment, remte netwrk access is the nly available path ffered t custmers t manage their envirnment. Instead f physical access audits fr equipment installatin and mdificatin, virtual system management sftware must be audited. The ability f systems t quickly change and mve within a vclud envirnment gives auditrs a need t track authrizatin and related change cntrls. Separate and islated netwrks shuld be used fr data migratin that is in the clear t avid expsure f sensitive infrmatin. Clud envirnments make extensive use f shrt-lived instances. Virtual machines may have a lifecycle far shrter than physical systems as they are easy t prvisin and repurpse. Systems als share data acrss large arrays in swap space. Permanence f data is als affected by envirnments that push as much strage as pssible thrugh high-speed memry t avid the latency f spinning disks. Custmers need audit trails and views unique t their wn use f the vclud envirnment, which als supprts incident respnse and investigatins. Prviders have t extend and develp lg management and mnitring slutins t meet regulatry and client requirements fr the vclud envirnment. Use Cases: Why Lgs Shuld be Available It is imprtant t mnitr and recrd events in rder t mitigate damage and prevent future attacks. An audit lg enables an rganizatin t verify cmpliance, detect vilatins, and initiate remediatin activities. It can help detect attempts, whether successful r nt, fr unauthrized access, infrmatin prbes, r disruptin. Lg Purpses Lgs are a fundatin f many cntrls used t achieve internal requirements as well as regulatry cmpliance. They are the technical slutin t track and recrd changes and incidents as they frm an audit trail. Lgs ffer the fllwing benefits: Cmpliance requirements Lgs are required fr all cmpliance regulatins t assist with cntrl auditing as well as breach review, analysis, and respnse. Specific types f lgs ften can be matched with specific cmpliance cntrls. Fr example, the authenticatin lg can demnstrate access cntrls allwed t nly authrized users. Custmer requirements End custmers can retrieve lgs that pertain t their envirnment in rder t meet their wn requirements. Operatinal integrity Operatinal alerts shuld be defined fr lgs t trigger ntificatins fr remediatin. This is frequently set up as a backup alert, secndary t mnitring. A strage array that ges ffline generates errr messages in the lgs, which can be used t alert administratrs. Trubleshting Clsely related t peratinal integrity, lgs are essential fr trubleshting. Fr example, the use f vshield Edge lgs can shw whether a specific external cnnectin request is being passed thrugh r NATted by the firewall.

55 Frequency f Review Lgs shuld be reviewed daily fr unauthrized r unusual and suspicius activity n all systems and especially thse that handle intrusin detectin, authenticatin and authrizatin. This requires review and verificatin f lgs t establish baselines f nrmal peratins, such as mnitring access and authrizatin (every lgin and lgut) frm the cnsle, netwrk, and remte access pints. Mre frequent and rutine lg analysis fr security ften helps give early identificatin f system cnfiguratin errrs, failures, and issues that can impact SLAs. Minimum Data Types The fllwing are the minimum set f data types required t adequately lg vclud envirnment activity fr regulatry cmpliance: User (including system accunt) access. Actin taken. Use f identificatin and authenticatin mechanisms. Start and stp f audit lgs. Creatin r deletin f system-level bjects. The audit trail entries recrded fr each event must include the fllwing details: Identificatin (ID). Type f event. Date and time. Success r failure. Originatin f event. ID f affected data r cmpnent. Retentin Daily review f lgs alne may nt be sufficient t detect incidents they als must be retained fr a perid cnsistent with effective use and legal regulatins. The laws fr lg retentin range frm ne year t mre than twenty. Therefre, lg archives shuld always be able t prvide at least ne year f histry, typically scheduled t match financial calendar cycles, and a minimum f three mnths available fr immediate respnse and review in case f an incident. Example Cmpliance Use Cases fr Lgs The fllwing use cases are a sample f events that benefit frm careful lgging and mnitring in the vclud envirnment. Other examples may include unauthrized services r prtcls, remte lgin success, and certificate changes. Shared accunts An investigatin is initiated t review netwrk utages and finds multiple instances f an Administratr accunt had lgged int critical servers befre failure. Shared accunts make it very difficult t trace fault t ne individual; it is impssible t determine frm the lgs n that system which persn was lgged int the user accunt that made the errr. Therefre, usage must be tied t an individual user ID and unique passwrd with crrect time t aid in investigatins. Systems als shuld be cnfigured t detect any and all use f generic IDs such as an administratr r rt accunt and trace them t unique identities.

56 User accunt changes A malicius user finds an un-patched flaw in an envirnment that allws elevatin f privileges. That user then uses system-level privileges t create a new bgus user bject frm which t launch further attacks. A user bject is, fr example, a Micrsft Widws Dmain r lcal user accunt. User bject lgs can be used t figure ut when a name was changed r an accunt added. This assists in detectin f actins withut authrizatin r users trying t hide attacks. Unauthrized sftware Malware r a new virtual machine instance in the vclud can be fund in system bject lgs. A system must track system bjects that are added, remved r mdified. This can be very helpful during installatin t mnitr system changes caused by sftware. VMware vclud Lg Surces fr Cmpliance Custmers shuld be able t retrieve lgs frm all areas that are relevant and unique t their rganizatin. Prgrammatic retrieval shuld be pssible, such as an API t allw fr autmated queries. Lg cllectin ndes must be added t a vclud envirnment, as illustrated in Figure 10. Figure 10. Lg Cllectin in the vclud Envirnment Lgs generated by VMware cmpnents must be maintained by the prvider, but als must be available t tenants. Tenants shuld be able t dwnlad in raw frmat all vclud Directr and vshield Edge lgs that pertain t their rganizatins and netwrks. Lgs with custmer identifiers shuld be flagged r indexed fr retrieval.

57 Figure 11 illustrates architecture f vclud cmpnents and lg cllectin. Figure 11. Architecture f vclud Cmpnents and Lg Cllectin

58 The fllwing table lists the lgs t which the vclud tenant must have access. Table 6. vclud Cmpnent Lgs VMware Cmpnent Prvider Lgs Tenant Lgs VMware vclud Directr vcenter Server vsphere Server (ESXi) Chargeback Manager vcenter Orchestratr vshield Manager vshield Edge Other cmpnents als generate lgs in the vclud envirnment that must be maintained by the prvider, but direct tenant access is nt required. Table 7. Other Cmpnent Lgs Other Cmpnent Prvider Lgs Tenant Lgs vclud Directr DB (Oracle) vcenter Database vcenter Chargeback Database Micrsft SQL Server Linux (VCD) Windws System Lgs (CBM, vco, vcenter Server)

59 Lgs in the vclud datacenter envirnment can further be categrized int the fllwing lgical business layers: vclud Applicatin Represents the external interface with which the enterprise administratrs f the vclud interact. These administratrs are authenticated and authrized at this layer, and have n (direct r indirect) access t the underlying infrastructure. They interact nly with the Business Orchestratin Layer. Business Orchestratin Represents bth vclud cnfiguratin entities and the gvernance plicies that cntrl the vclud deplyment: Service catalg Presents the different service levels available and their cnfiguratin elements. Service design Represents the service level and specific cnfiguratin elements alng with any defined plicies. Cnfiguratin Management Database (CMDB) Represents the system f recrd, which may be federated with an enterprise CMDB. Service prvisin Represents the final cnfiguratin specificatin. Service Orchestratin Represents the prvisining lgic fr the vclud infrastructure. This layer cnsists f an rchestratin directr system, and autmatin elements fr netwrk, strage, security, and server/cmpute vcenter Server, VMware vclud Directr (vclud Directr), vcenter Orchestratr. Infrastructure Layer Represents the physical and virtual cmpute, netwrk, strage, hypervisr, security, and management cmpnents vsphere Server (ESXi), vshield Manager, and vshield Edge.

60 Figure 12. Infrastructure Layers The abstractin f these fur layers and their security cntrls helps illustrate audit and cmpliance requirements fr prper authenticatin and segregatin. Fr example, vclud prvider administratr accunts shuld be maintained in a central repsitry integrated with tw-factr authenticatin. Different tiers f vclud deplyments (prvider virtual datacenters) wuld be made available t enterprise users.

61 vclud Directr Diagnstic and Audit Lgs VMware vclud Directr includes the fllwing types f lgs: Audit lgs that are maintained in the database, and ptinally, in a syslg server. Diagnstic lgs that are maintained in each vclud Directr cell s lg directry. The VMware vclud Directr system audit lg is maintained in the Oracle database and can be mnitred thrugh the Web UI. Each rganizatin administratr and the system administratr have a view int the lg scped t their specific area f cntrl. A mre cmprehensive view f the audit lg (and lngterm persistence) is achieved thrugh the use f remte syslg (described belw). Lg management prducts are available frm a variety f vendrs and pen surce prjects. Audit events are nt the nly event types. Diagnstic lgs cntain infrmatin abut system peratin events and are stred as files in the lcal file system f each cell s perating system. Diagnstic lgs can be useful fr prblem reslutin, but are nt intended t preserve a trail f system interactins fr audit. Each VMware vclud Directr cell creates several diagnstic lg files, as described in the Viewing the vclud Directr Lgs sectin f the VMware vclud Directr s Administratin Guide fr the latest versin f VCD ( Audit lgs recrd significant actins, including lgin and lgut. A syslg server can be set up during installatin as detailed in the vclud Directr Installatin and Cnfiguratin Guide (( Exprting the lgs t a syslg server is required fr cmpliance due t multiple reasns: Database lgs are nt retained after 90 days, but lgs transmitted via syslg can be retained as lng as desired. It allws audit lgs frm all cells t be viewed tgether in a central lcatin at the same time. It prtects the audit lgs frm lss n the lcal system due t failure, a lack f disk space, cmprmise, and s n. It supprts frensics peratins in the face f prblems like thse listed abve. It is the methd by which many lg management and Security Infrmatin and Event Management (SIEM) systems will integrate with vclud Directr. This enables: Crrelatin f events and activities acrss vclud Directr, vshield, vsphere, and even the physical hardware layers f the stack. Integratin f vclud security peratins with the rest f the vclud prvider s r enterprise s security peratins, cutting acrss physical, virtual, and vclud infrastructures. Lgging t a remte system, instead f the system the cell is deplyed n, prvides data integrity by inhibiting tampering. Even if the cell is cmprmised it des nt necessarily enable access t r alteratin f the audit lg.

62 Appendix C: Capacity Planning Capacity frecasting prvides an efficient way t acquire the apprpriate amunt f physical resurces t supprt the increased demand fr the vclud. This allws fr the grwth f vclud t be planned and included in the service prviders budgetary prcess, and reduces the likelihd f panic buying, which generally increases csts dramatically and undermines standardizatin effrts. Capacity Planning als reduces the likelihd f last minute surprises, such as a lack f available space r pwer t supprt the new vclud infrastructure cmpnents. Frm a vclud perspective, capacity management is simplified by the existence f the prvider virtual datacenter and rganizatin virtual datacenter cnstructs, but ptentially mre cmplicated by the additin f three mdels f cnsumptin: Pay-As-Yu-G, Allcatin Pl (cmmitted), and Reservatin Pl (dedicated). Finally, all f these capacity management aspects, within a vclud cntext, must address bth the vclud (service prvider) administratr and the end-custmer (rganizatin) administratr perspectives. Sizing fr the wrklad resurce grup clusters can be difficult t predict because the prvider is nt in charge f what the cnsumer may run. The prvider is als nt aware f existing usage statistics fr virtual machines that are run in the vclud. The fllwing infrmatin shuld assist in initial sizing f the vclud envirnment and is based n infrmatin frm the Service Definitin fr a Private VMware vclud. This infrmatin is prvided in the frm f examples. VMware recmmends that yu engage with yur lcal VMware representative fr detailed sizing f yur envirnment. vclud Administratr (Service Prvider) Perspective The primary capacity management cncerns f the vclud administratr are: Capacity management f prvider virtual datacenters and the service fferings backed by each prvider virtual datacenter. Netwrk capacity management (netwrk bandwidth capacity management is beynd the scpe f this dcument). Capacity frecasting. Capacity mnitring and establishing triggers. The VMware vclud slutin makes extensive use f reservatins. As such, previus appraches t capacity management used in vsphere are nt as applicable t a vclud. Fr example, CPU and memry ver-cmmitment cannt be applied as extensively as it was in a multitenant envirnment. Unlike managing capacity fr vsphere, in a vclud, the virtual machine is n lnger the basis fr resurce cnsumptin frm a service prvider perspective. The rganizatin virtual datacenter is the basis fr resurce cnsumptin in a vclud. Capacity management is further impacted by the intrductin f multiple cnsumptin mdels in the vclud mdel. Each mdel requires its wn capacity management apprach. As a result, this appendix prvides guidance fr capacity management frm a service prvider vclud administratr perspective as it applies t each f the cnsumptin mdels: Pay-As-Yu-G, Allcatin Pl, and Reservatin Pl.

63 Regardless f the particular cnsumptin mdel applied in a prvider virtual datacenter, the cmmn starting pint f vclud capacity management is t calculate the ttal amunt f CPU and memry resurces available fr cnsumptin. Because the underlying infrastructure prvisining unit f a prvider virtual datacenter is a ESXi hst, the first step is t determine the ttal CPU and memry at the vsphere hst level. The fllwing table shws the key vsphere hst variables needed t calculate capacity, alng with example values. Table 8. vsphere Hst Variables Item Variable Value Units Prcessr Sckets N 2 scket,1 Prcessr Cres N 4 cres,1 Prcessr Speed S 2.4 prc,1 Hst Memry M 64 hst,1 integer integer GHz GB Calculating the ttal memry available is straightfrward. It is the ttal amunt f RAM fr the vsphere hst. Ttal CPU resurces are calculated using the fllwing frmula: P = N hst scket N cres S prc Using the example values frm the table, the ttal CPU resurce is equal t 19.2 GHz. After the vsphere hst capacity mdel has been defined, the next step is t determine the prvider virtual datacenter (vsphere cluster) capacity. Determining the prvider virtual datacenter capacity is critical as vclud capacity management shuld be perfrmed at the prvider virtual datacenter level, nt the vsphere hst level. When cnsidering vclud prvider virtual datacenter capacity, an additinal step is required t make sure that redundancy has been accunted fr. The prvider virtual datacenter cluster redundancy may vary depending upn service levels ffered. Fr the fllwing example, we assume N+2 cluster redundancy. This means that the prvider virtual datacenter can absrb up t tw vsphere hst failures and cntinue t supprt all hsted virtual machines at the same level f perfrmance. T accmplish this, there must be capacity available n the remaining vsphere hsts t take ver all wrklads. Based n a requirement fr prvider virtual datacenter cluster redundancy, the verall number f memry and CPU cnsumptin units fr the prvider virtual datacenter (cluster) must be reduced. T determine the redundancy verhead, the number f vsphere hsts in the cluster and the desired number f redundant vsphere hsts need t be cnsidered. This is described in Table 9.

64 Table 9. Determining Redundancy Overhead Redundancy Variables N N R ndes redundant redundancy,ha Descriptin Represents the number f ndes in a cluster. Represents the minimum number f redundant ndes. Represents a targeted rati f redundancy as indicated by a real number greater than ne. This rati (such as 1.10) indicates that there is a ten percent verhead cmmitted t availability. Fr example, a 10 nde prvider virtual datacenter with a 1.10 redundancy rati wuld require 11 ndes t deliver the apprpriate capacity. Nte that this level f redundancy may vary depending n the class f service ffering being delivered n that prvider virtual datacenter. Redundancy variables can be determined with the equatin belw. Calculating Redundancy Rati frm Minimal Level f Redundancy N ndes + N N ndes redundant = R redundancy Fr example, the level f redundancy is calculated belw fr a cluster size f ten ndes cntaining tw redundant ndes. N ndes + N N ndes redundant = = 1.25 = R 8 redundancy After the rati f redundancy is calculated, the number f units f cnsumptin per prvider virtual datacenter can be determined using the fllwing equatin: CPU resurces per Cluster N CPU, cluster = N hsts, cluster R P CPU, hst redundancy, HA Fr ur example where: P CPU, hst = This results in: N GHz CPU, cluster = = GHz The number f memry units f cnsumptin is calculated in the fllwing equatin.

65 Fr ur example where: N mem, hst = 64 This results in: N GB N M hsts, cluster mem, hst mem, cluster = = = GB We have nw established that ur example prvider virtual datacenter has GHz f available CPU and 409.6GB f available memry, taking a vsphere cluster redundancy f N+2 int accunt. Next we lk at sme guidance fr capacity management as it applies t each f the cnsumptin mdels. Pay-As-Yu-G Mdel When an rganizatin virtual datacenter is created in the Pay-As-Yu-G mdel, a resurce pl is instantiated with expandable reservatins. As such, the custmer rganizatin virtual datacenters cntained n that prvider virtual datacenter can grw t cnsume all f the available prvider virtual datacenter resurces. While this culd be true in any vsphere envirnment, the added challenge in a vclud is the use f reservatins at the vapp level. When an rganizatin virtual datacenter is created ut f a prvider virtual datacenter using the Pay-As-Yu-G cnsumptin mdel, a %guarantee is cnfigured fr CPU and memry. This is applied t each vapp r virtual machine within a vapp. Fr example, if the service prvider cnfigures the rganizatin virtual datacenter with a 50% guarantee fr CPU and 75% guarantee fr memry, then the custmer creates a virtual machine cnsuming 1 vcpu f 1GHz and 1GB f memry, a reservatin fr that virtual machine will be set at 50% f 1GHz, r 0.5 GHz and 75% f 1GB, r 0.75GB f memry. Because there is n way f knwing hw a custmer will define their virtual machine templates in their private custmer catalgs, cupled with the fact that rganizatin virtual datacenters can expand n demand, VMware recmmends the fllwing: Calculate the ttal available CPU and memry resurces (less an amunt reserved fr glbal catalg templates), adjusted by the cluster redundancy rati, at the prvider virtual datacenter level. Establish a CPU and Memry %RESERVED threshld at the prvider virtual datacenter level. Establish the %RESERVED fr the prvider virtual datacenter at a number in the 60% range initially. As the ttal amunt f reserved CPU r reserved memry appraches the %RESERVED threshld, d nt deply new rganizatin virtual datacenters in that prvider virtual datacenter withut adding additinal resurces. If the crrespnding vsphere cluster has reached its maximum pint f expansin, a new prvider virtual datacenter shuld be deplyed and any new rganizatin virtual datacenter s shuld be assigned t the new prvider virtual datacenter. In this way there is 40% f expansin capacity fr the existing rganizatin virtual datacenters in the case where the prvider virtual datacenter has reached its maximum pint f expansin. CPU and memry ver-cmmitment can be applied, and if s, the %RESERVED value shuld be set lwer than if n ver-cmmitment is applied due t the unpredictability f the virtual machine sizes being deplyed (and hence reservatins being established). Mnitr the %RESERVED n a regular basis and adjust the value accrding t histrical usage as well as prject demand.

66 Allcatin Pl Mdel When an rganizatin virtual datacenter is created in the Allcatin Pl mdel, a nn-expandable resurce pl is instantiated with a %guaranteed value fr CPU and memry that was specified. Using a %guaranteed value f 75%, this means if an rganizatin virtual datacenter is created specifying 100GHz f CPU and 100GB f memry, a resurce pl is created fr that rganizatin virtual datacenter with a reservatin f 75GHz and limit f 100GHz fr CPU and a reservatin f 75GB with a limit f 100GB fr memry. The additinal 25%, in this example, is nt guaranteed and can be accessed nly if it s available acrss the prvider virtual datacenter. In ther wrds, the 25% can be ver-cmmitted by the prvider at the prvider virtual datacenter level and therefre may nt be available depending n hw all f the rganizatin virtual datacenters in that prvider virtual datacenter are using it. At the virtual machine level, when a virtual machine is deplyed, it is instantiated with n CPU reservatin but with a memry reservatin equal t the virtual machine s memry allcatin multiplied by the %guaranteed. Despite the fact that n CPU reservatin is set at the virtual machine level, the ttal amunt f CPU allcated acrss all virtual machines in that rganizatin virtual datacenter is still subject t the verall CPU reservatin f the rganizatin virtual datacenter established by the %guarantee value. Based n this use f reservatins in the Allcatin Pl mdel, VMware recmmends the fllwing: Calculate the ttal available CPU and memry resurces (less an amunt reserved fr glbal catalg templates), adjusted by the cluster redundancy rati, at the prvider virtual datacenter level. Determine hw much resurce, at the prvider virtual datacenter level, yu want t make available fr expanding rganizatin virtual datacenters that are deplyed t that prvider virtual datacenter. Establish a CPU and Memry %RESERVED (guaranteed, nt allcated) threshld at the prvider virtual datacenter level based n the %guaranteed less the amunt reserved fr grwth. The remaining unreserved resurces are available t all rganizatin virtual datacenters fr bursting. As the ttal amunt f reserved CPU r reserved memry appraches the %RESERVED threshld, d nt deply new rganizatin virtual datacenters in that prvider virtual datacenter withut adding additinal resurces. If the crrespnding vsphere cluster has reached its maximum pint f expansin, a new prvider virtual datacenter shuld be deplyed and any new rganizatin virtual datacenters shuld be assigned t the new prvider virtual datacenter. This gives sme predetermined amunt f capacity available fr expanding the existing rganizatin virtual datacenters in the case where the prvider virtual datacenter has reached its maximum pint f expansin. CPU and memry ver-cmmitment can be applied, but it shuld be based nly n the amunt f unreserved resurces at the prvider virtual datacenter level, allwing fr ver-cmmitting the resurces available fr rganizatin virtual datacenter bursting. Mnitr the %RESERVED n a regular basis and adjust the value accrding t histrical usage as well as prject demand.

67 Reservatin Pl Mdel When an rganizatin virtual datacenter is created in the Reservatin Pl mdel, a nn-expandable resurce pl is instantiated with the reservatin and limit values equivalent t the amunt f resurces allcated. This means if an rganizatin virtual datacenter is created allcating 100GHz f CPU and 100GB f memry, a reservatin pl is created fr that rganizatin virtual datacenter with a reservatin and limit f 100GHz fr CPU and a reservatin and limit f 100GB fr memry. At the virtual machine level, when a virtual machine is deplyed, it is instantiated with n reservatin r limit fr either CPU r memry. Based n this use f reservatins in the Reservatin Pl mdel, VMware recmmends the fllwing: Calculate the ttal available CPU and memry resurces (less an amunt reserved fr glbal catalg templates), adjusted by the cluster redundancy rati, at the prvider virtual datacenter level. Determine hw much resurce, at the prvider virtual datacenter level, yu want t make available fr expanding rganizatin virtual datacenters that are deplyed t that prvider virtual datacenter. Establish a CPU and Memry %RESERVED threshld at the prvider virtual datacenter level equivalent t the capacity f the underlying vsphere cluster, taking int accunt HA redundancy. As the ttal amunt f reserved CPU r reserved memry appraches the %RESERVED threshld, d nt deply new rganizatin virtual datacenters in that prvider virtual datacenter withut adding additinal resurces. If the crrespnding vsphere cluster has reached its maximum pint f expansin, a new prvider virtual datacenter shuld be deplyed and any new rganizatin virtual datacenters shuld be assigned t the new prvider virtual datacenter. In this way there is sme predetermined amunt f capacity available fr expanding the existing rganizatin virtual datacenters in the case where the prvider virtual datacenter has reached its maximum pint f expansin. N ver-cmmitment can be applied t the prvider virtual datacenter in the Reservatin Pl mdel due t the reservatin being at the resurce pl level. Mnitr the %RESERVED n a regular basis and adjust the value accrding t histrical usage as well as prject demand. Strage VMware vclud Directr uses a largest available capacity algrithm fr deplying virtual machines t datastres. Strage capacity must be managed n bth an individual datastre basis as well as in the aggregate fr a prvider virtual datacenter. In additin t cnsidering VMware strage allcatin best practices, manage capacity at the datastre level using the largest virtual machine strage cnfiguratin, in terms f units f cnsumptin, ffered in the service catalg when determining the amunt f spare capacity t reserve. Fr example, if using 1TB datastres (100 strage units f cnsumptin based n a 10GB unit f cnsumptin) and the largest virtual machine strage cnfiguratin is 6 strage units f cnsumptin (60GB), then applying the VMware best practice f apprximately 80% datastre utilizatin wuld imply managing t 82 strage units f cnsumptin. This wuld result in 82% datastre utilizatin and reserve capacity equivalent t three f the largest virtual machines ffered in the service catalg in terms f strage.

68 Netwrk Capacity Planning A vclud als brings netwrk capacity planning t the frefrnt. Prviders must cnsider IP address, VLAN, and ephemeral prt capacity. The fllwing table describes what must be managed frm a capacity perspective and its impact. Table 10. Netwrk Capacity Planning Items Item t Manage Impact IP addresses Available IP addresses t be assigned in supprt f a dedicated external netwrk fr an rganizatin, such as fr Internet access r hardware-based firewall rules. Need t track IP addresses assigned t specific rganizatins t determine what is available fr a shared external rganizatin netwrk. VLANs VLANs available fr VLAN-backed pl assignment, if required. VLANs available fr vclud Directr Netwrk Islatin transprt netwrks, ne per vclud Directr Netwrk Islatin pl. Expandable static prt bindings Default vclud Directr netwrk pl type. Overall number f static prts expands in increments f ten as needed. Nte that unused but allcated static prt bindings d nt increase the ttal number f static prt bindings available.

69 Appendix D: Capacity Management vclud-specific Capacity Frecasting (Demand Management) Capacity frecasting cnsists f determining hw many rganizatin virtual datacenters are expected t be prvisined during a specific time perid. Capacity prvisining is cncerned with determining when vclud infrastructure cmpnents must be purchased in rder t maintain capacity. Frm a financial budget perspective, the prcurement f the vclud infrastructure requires mre planning and understanding f custmer future requirements. VMware recmmends perfrming tw frecasting functins ver time. Capacity Trending Using histrical rganizatin virtual datacenter capacity and utilizatin data, it is pssible t predict future capacity requirements. Demand Pipeline Understanding future custmer requirements via the sales pipeline prvides the necessary infrmatin t understand future capacity requirements, as well as knwledge f marketing/business develpment functins bringing new service fferings t market. Initially, n histrical utilizatin metrics are available, and thus it is nt pssible t perfrm capacity trending fr sme perid f time. During this initial perid, a gd understanding f the custmer demand pipeline needs t be established. Over time, this pipeline can be cmbined with trending analysis t mre accurately predict capacity requirements. The custmer demand pipeline must be established in cnjunctin with the service prvider s sales teams, r lines f business (LOB) if a private clud, s future vclud capacity requirements can be determined. This demand pipeline must cntain infrmatin f all knwn new custmers, expansin f existing custmer rganizatin virtual datacenters, prjected sizing metrics, plus any new service fferings that are in develpment. The frecasting plan must fit bth the budgetary cycle and the prcurement and prvisining timeframes. Fr example, if a quarterly budgetary cycle exists, and the prcurement and prvisining timeframe is ne mnth, it is necessary t have a pipeline f at least fur mnths t make sure all requests in the pipeline can be fulfilled. Over time, capacity trending can be used t assist with the frecasting f rganizatin virtual datacenter prvisining needs. It uses histrical infrmatin t determine trends and validates the rganizatin virtual datacenter frecast based n demand pipeline data. Capacity Mnitring and Establishing Triggers The metrics listed in Table 11shuld be carefully mnitred t warn f appraching r exceeding cnsumptin threshlds. These metrics shuld be measured against each vclud prvider virtual datacenter and fr each rganizatin virtual datacenter within each prvider virtual datacenter. T mnitr fr threshld breaches, and pssible subsequent vilatin f service level cmmitments t the vclud cnsumer, the apprpriate tls and triggers are needed fr prper ntificatin.

70 Table 11. Capacity Mnitring Metrics Attribute %RESERVED CPU Mnitred per Prvider virtual datacenter, rganizatin virtual datacenter Nte Fr the Pay-As-Yu-G allcatin mdel this is the aggregatin f reservatins values fr the cntained virtual machines. %RESERVED Memry CPU utilizatin Memry utilizatin Datastre utilizatin Transfer stre utilizatin Netwrk IP addresses available Netwrk IP addresses cnsumed Netwrk VLANs available Netwrk ephemeral prts cnsumed Prvider virtual datacenter, rganizatin virtual datacenter Prvider virtual datacenter, rganizatin virtual datacenter Prvider virtual datacenter, rganizatin virtual datacenter Prvider virtual datacenter vclud vclud Organizatin vclud vnetwrk Distributed Switch If threshlds are exceeded, the grup respnsible fr capacity management f the vclud shuld be ntified t add additinal capacity. Take int accunt the time required t add the physical cmpnents necessary t increase the capacity f a prvider virtual datacenter. A vclud-aware capacity management tl shuld be deplyed. Whichever tl is chsen, the capacity mdel can be used t frecast new prvider virtual datacenter capacity utilizatin as well as nging capacity management f existing prvider virtual datacenters. It shuld als accunt fr expansin triggers based n prvisining timeframes. After the ttal amunt f available resurces has been calculated fr a prvider virtual datacenter, n adjustments t that prvider virtual datacenter (such as adding r remving hsts) shuld be made withut updating the calculated value. This mdel may be altered if lng-term CPU and memry reservatins are nt at the levels fr which they were designed. An increase in the resurces allcated t an rganizatin virtual datacenter can affect the remaining capacity f a full prvider virtual datacenter. Full prvider virtual datacenters shuld be mnitred n a weekly basis. The resurce cnsumptin f virtual machines within an rganizatin virtual datacenter shuld be reviewed fr trends that indicate the resurces purchased fr that rganizatin virtual datacenter are insufficient. vcenter CapacityIQ, thugh nt vclud Directr aware, can be used t prvide insight int prvider virtual datacenter utilizatin and trends.

71 Capacity Management Manual Prcesses Prvider Virtual Datacenter The fllwing vclud administratr capacity management activities include peridic planning activities supprted by day-t-day peratinal activities. Peridic cntinuus imprvement activities are critical t extracting the mst value frm yur vclud infrastructure. Planning activities (initially mnthly, then quarterly): Determining usable capacity by prvider virtual datacenter and rganizatin virtual datacenter (taking int accunt vsphere verhead). Reviewing current utilizatin. Reviewing prvisining timeframes fr new prvider virtual datacenter cmpnents (hsts, netwrk, strage). Frecasting utilizatin grwth ver the cming perid (preferably based n the actual pipeline, validated with histrical trending). Planning fr prcurement and implementatin f additinal capacity ver the cming perid, including bills f materials and budgets. Reviewing capacity alert threshld levels and setting alerts fr capacity warnings. Operatinal activities (daily): Mnitring fr alerts. Investigating perfrmance issues t determine whether capacity is the rt cause. Initiating and managing the prcurement and prvisining f additinal prvider virtual datacenter capacity. Cntinuus imprvement activities (quarterly/yearly): Cmparing capacity mdel utilizatin levels t bserved levels and tuning mdel t drive greater utilizatin withut sacrificing reliability. Optimizing prvisining timeframes (shrtening them and making them mre predictable).

72 End-Custmer (Organizatin) Administratr Perspective The primary capacity management cncern f the rganizatin administratr is capacity management f the rganizatin s rganizatin virtual datacenters. VMware recmmends that all rganizatins establish a capacity management prcess based n a standard unit f cnsumptin. The recmmended base Unit f Cnsumptin fr each resurce imprtant t capacity management frm an rganizatin administratr perspective is shwn in Table 12. Table 12. Organizatin Virtual Datacenter Units f Cnsumptin Attribute Variable Value vcpu P 1 GHz UC Memry M 1GB UC Strage D 10GB UC Taking this apprach enables mre efficient capacity management because the vapp cmpnent virtual machine resurce allcatins are predefined in the service catalg, resulting in vclud infrastructure resurce cnsumptin being mre accurately predicted. Each rganizatin will be prvided with a finite quantity f resurces (in the cases f the Allcatin Pl and Reservatin Pl cnsumptin mdels) frm ne r mre prvider virtual datacenters in the frm f rganizatin virtual datacenters. This means that as the rganizatin cnsumes the rganizatin virtual datacenter resurces, a trigger pint needs t be defined t prmpt actins t be taken t expand the rganizatin virtual datacenter. First, the resurce cnsumptin limits fr an rganizatin s rganizatin virtual datacenters need t be defined, with these limits defining when actin needs t be taken t remve the ptential capacity issue.

73 Table 13. Recmmended Organizatin Virtual Datacenter Capacity Threshlds Attribute Variable Limit Descriptin rganizatin virtual datacenter CPU Peak Utilizatin rganizatin virtual datacenter Memry Allcatin Limit C 80% CPULimit C 80% memlimit The limit fr allcating CPU resurces within the rganizatin virtual datacenter befre expansin is required. This value varies depending n the cnsumptin mdel used. Frm an rganizatin virtual datacenter perspective, reservatin values shuld be cnsidered equal t the amunt f CPU allcated as reservatin values are nt available t the rganizatin administratr. The limit fr allcating memry resurces within the rganizatin virtual datacenter befre expansin is required. This value varies depending n the cnsumptin mdel used. Frm an rganizatin virtual datacenter perspective, reservatin values shuld be cnsidered equal t the amunt f memry allcated as reservatin values are nt available t the rganizatin administratr. The CPU and memry resurces vary depending n the size f the cntracted rganizatin virtual datacenter. Table 14 prvides an example f the resurces needed t calculate the rganizatin virtual datacenter s capacity. Table 14. Sample Organizatin Virtual Datacenter Resurce Allcatin Item Variable Value Units Ttal rganizatin virtual datacenter vcpu Units f Cnsumptin rganizatin virtual datacenter Memry Allcatin in Units f Cnsumptin S 50 rgvirtual datacenter M 64 rgvirtual datacenter GHz GB

74 The number f capacity units available within this rganizatin virtual datacenter is fund using the fllwing equatins. Determining rganizatin virtual datacenter memry units f cnsumptin M UC, rgvdc C = memlimit M M UC rgvdc Based n the infrmatin frm the abve tables, the ttal memry unit f cnsumptin fr the rganizatin virtual datacenter is calculated as shwn belw. M C = M memlimit rgvdc UC, rgvdc M = = UC 1 This results in 51.2 memry units f cnsumptin fr the sample rganizatin virtual datacenter. GB Determining rganizatin virtual datacenter CPU units f cnsumptin P UC, rgvdc S = rgvdc P C UC CPULimit Based n the infrmatin frm the abve tables, the CPU units f cnsumptin per rganizatin virtual datacenter are calculated as shwn belw. P S = S P rgvdc CPULimit UC, rgvdc = = 40 UC 1 GHz This results in 40 CPU units f cnsumptin fr this sample rganizatin virtual datacenter. Organizatin Virtual Datacenter-Specific Capacity Frecasting Capacity frecasting cnsists f determining hw many virtual machines are expected t be deplyed during a specific time perid f the rganizatin s chsing. The time perid used fr the virtual machine frecast shuld crrespnd t the budgetary prcess. Capacity prvisining is cncerned with determining when an rganizatin virtual datacenter must be expanded in rder t maintain capacity. VMware recmmends that rganizatins perfrm tw frecasting functins ver time. Capacity Trending Using histrical virtual machine capacity and utilizatin data, it is pssible t predict future capacity requirements. Capacity Pipeline Understanding future end-user virtual machine resurce requirements, via IT and LOB prjects, prvides the necessary infrmatin t understand future capacity requirements. Over time, capacity trending can be used t assist with the frecasting f virtual machine prvisining needs. It uses histrical infrmatin t determine trends and validates the virtual machine frecast based n pipeline data.

75 Capacity prvisining depends n determining the pint f expansin fr the rganizatin virtual datacenter. This is based n determining a pint f resurce cnsumptin at which the prcess f prcuring and expanding the rganizatin virtual datacenter must begin s that reserve capacity is nt exhausted befre the additinal capacity is available. In the vclud cntext, this can be cnsidered t be dependent upn the time it takes t prcess the purchase request fr additinal rganizatin virtual datacenter resurces. Prvisining time can be assumed t be zer but depends upn specific cntractual agreements with the service prvider. The fllwing are recmmended steps t perfrm capacity trending and t determine a pint f rganizatin virtual datacenter expansin. Regularly Cllect Organizatin Virtual Datacenter Cnsumptin Infrmatin The primary issue with the trending f rganizatin virtual datacenter cnsumptin is identifying the pint f recrd fr all new virtual machines. This can then be used t determine the capacity trends and therefre determine the verall need fr purchasing additinal rganizatin virtual datacenter capacity. T establish the pint f recrd fr new virtual machines, the items listed in Table 15 shuld be tracked, ideally in a Cnfiguratin Management r Capacity Planning Database as virtual machine attributes. Table 15. Organizatin Virtual Datacenter Trending Infrmatin Variable Name Descriptin Units rgvirtual datacenter Organizatin virtual datacenter This is the rganizatin virtual datacenter in which the virtual machine resides. Identifier D build Build Date This is the date the virtual machine is built. Date N UC,cpu CPU Units f Cnsumptin This is the number f CPU units f cnsumptin allcated t the virtual machine. CPU Units f Cnsumptin N UC,mem Memry Units f Cnsumptin This is the number f memry units f cnsumptin allcated t the virtual machine. Memry Units f Cnsumptin N VGB Strage This is the amunt f strage (GB) allcated t the virtual machine. GB

76 Determine Trending Variables With the infrmatin recrded as described in Table 15 it is pssible t determine the rate f rganizatin virtual datacenter cnsumptin. Table 16. Organizatin Virtual Datacenter Capacity Trending Variables Variable Name Descriptin Units T Time This is the time between pints f bservatin. Weeks N cpuuc New CPU Units This is the ttal number f CPU units f cnsumptin required fr the frecasted virtual machines. CPU Units f Cnsumptin N memuc New Memry Units This is the ttal number f memry units f cnsumptin required fr the frecasted virtual machines. Memry Units f Cnsumptin N VGB New Strage (GB) This is the ttal amunt f strage required fr the frecasted virtual machines. GB T purchase Organizatin Virtual Datacenter Expansin Purchase Time The amunt f time t prcure additinal rganizatin virtual datacenter resurces. Weeks Determining the Trended Grwth Rate N N N cpuuc memuc VGB = = N cpuuc T N N = T VGB memuc T

77 Determining the Trend It is imprtant t understand that the rate f increase dictates hw far in advance additinal rganizatin virtual datacenter resurces need t be purchased. The fllwing table presents a sample virtual machine frecast fr a quarter alng with sample time t purchase value. Table 17. Sample Organizatin Virtual Datacenter Trending Infrmatin Attribute Value ΔN 12 cpuuc ΔN 12 memuc ΔN T VGB purchase 360GB N 320 cpuuc,cluster N 717 memuc,cluster 2 weeks In this example, NcpuUC,free and NmemUC,free represents the number f free resurces within an rganizatin virtual datacenter at which pint additinal rganizatin virtual datacenter resurces shuld be rdered. T determine the trigger pint fr rdering use the fllwing equatin if n pipeline data exists. Determining Trigger Pint fr Ordering Capacity Using Trends N = N T UC, free CU purchase Fr example, frm the data prvided belw, ne wuld calculate the needed free cnsumptin units as listed in the fllwing equatin, r 24 units. N cpuuc, free = N cpuuc Tpurchase = 12 2 = 24GHz N memuc free = N memuc Tpurchase = GB, = Fr strage, in this example, the trigger pint is calculated at 720GB: ( T ) = GB NVGB, free = NVGB purchase = 720 Determine the Autmatic Pint f Expansin Based n the example abve, additinal rganizatin virtual datacenter resurces wuld need t be rdered when the available units f CPU r memry fall t 24GHz r 24GB respectively, r when strage capacity falls t 720GB. The additinal capacity needs t be n rder when described r the capacity will nt be available in time t meet demand. Currently there are n tls available t assist in rganizatin virtual datacenter capacity management. Hwever, it is pssible t develp scripts t gather pertinent infrmatin using languages such as PwerCLI.

78 Capacity Management Manual Prcesses Organizatin Virtual Datacenter The fllwing rganizatin administratr capacity management activities include peridic planning activities supprted by day-t-day peratinal activities. Peridic cntinuus imprvement activities are critical t extracting the mst value frm yur vclud. Planning activities (initially mnthly, then quarterly): Determining usable capacity by rganizatin virtual datacenter. Reviewing current utilizatin (and perfrmance, where pssible). Reviewing purchasing timeframes fr expanding an rganizatin virtual datacenter. Frecasting utilizatin grwth ver the cming perid (preferably based n actual pipeline validated by histrical trending). Reviewing capacity alert threshld levels and setting alerts fr capacity warnings. Operatinal activities (daily): Mnitring fr alerts. Investigating perfrmance issues t determine whether capacity is the rt cause. Initiating and managing the prcurement and prvisining f additinal capacity. Cntinuus imprvement activities (quarterly/yearly): Cmparing capacity mdel utilizatin levels t bserved levels and tuning mdel t drive greater utilizatin withut sacrificing reliability.

79 Appendix E: Integrating with Existing Enterprise System Management There are several mechanisms available fr integrating vclud with existing enterprise system management tls. These range frm the VCD ntificatin capabilities intrduced in VCD 1.5 t vcenter Orchestratr, the vclud API, and, fr prviders, the VIX API. This appendix addresses the VCD ntificatin capability, using vcenter Orchestratr, and the VIX API. Fr mre infrmatin abut the vclud API, see the vclud API Specificatin ( and the vclud API Prgramming Guide ( vclud Directr Ntificatins and Blcking Tasks Messages vclud Directr 1.5 supprts ntificatins and blcking tasks features that allw it t interperate with applicatins, extending its capabilities. Figure 13. vclud Directr Extensin Overview Message Publicatin The system administratr can cnfigure vclud Directr t enable the publicatin f messages fr all event ntificatins and/r fr specific blcking tasks: The ntificatins are published upn user-initiated events (fr example, creatin, deplyment and deletin f a vapp) as well as system-initiated events (fr example, vapp lease expiratin) cntaining the new state f the crrespnding vclud Directr entity. The blcking tasks suspend lng running peratins started as a task befre publishing messages and wait until a system administratr takes actin. The message publicatin is enabled bth fr peratins started in the vclud Directr GUI r vclud API and can be acted upn by using either interface. The ntificatin messages are published t an Advanced Message Queuing Prtcl (AMQP) exchange (AMQP versin supprted by RabbitMQ versin 2.0 and abve). Ruting The AMQP brker uses ruting as an effective way t filter vclud directr ntificatin messages and dispatch them t different queues fr ne r multiple extensins. The exchange rutes ntificatins t its bund queues accrding t their queue ruting key and exchange type. The vclud ntificatin messages ruting key has the fllwing syntax frmat: <peratinsuccess>.<entityuuid>.<rguuid>.<useruuid>.<subtype1>.<subtype2>... <subtypen>.[taskname]

80 Extensin An extensin is a script r an applicatin with the fllwing capabilities: Subscribe t an AMQP queue fr receiving new messages. Triage the received messages. Prcess messages int peratins (internal r external calls). Call vclud Directr API back fr getting mre infrmatin n the bjects invlved in an peratin and taking actin n blcked task. Subscribe t an AMQP queue Subscribing t queues invlves declaring a queue, binding with a ruting key, and then subscribing t the declared queue. The queue ruting key supprts the * and # wildcard characters t match a single segment and zer r mre segments. Fr example true.*.*.*. cm.vmware.vclud.event.vm.create r true.#.cm.vmware.vclud.event.vm.create will rute a ntificatin t the queue with this binding key every time any user frm any rganizatin successfully creates a virtual machine). Declaring asserts the existence f the bject. If it des nt exist, it will create it. Triage the cnsumed messages When a message is cnsumed, the extensin can use the message header that cntains all the ruting cmpnents t further filter and act upn. Fr example, sme ntificatins may be ignred. Separate the ntificatins messages frm the blcking tasks because the blcking tasks must be handled differently. Handling the ntificatin messages The ntificatin messages cntain the peratin triggering the event; the bject type, and identifiers and names fr rganizatin, user, and bject. These can be used as markers fr applicatins such as audit lgging, Change Management, and Incident Management. If the applicatin cannt crrelate the IDs t present the bjects prperties in an end-user cnsumable frm, the extensin applicatin has t call back the vclud API t extract these. Use ntificatin messages t start an peratin that must fllw anther ne. Fr example, enabling the public IPs f a vapp in a lad balancer. Handling blcking tasks messages Blcking tasks messages have similar identifier with the bject being the blcking task. The blcking task references: Its parent task The suspended task referencing the bject and the task parameters attributes it was set with in the riginal request. TaskOwner The bject n which the task perates. The actins that can be taken n this blcking task (resume, abrt fail, updateprgress). Receiving and acting upn n the blckings task is accmplished with the vclud directr API callbacks. System admin privileges are required t perfrm these peratins.

81 Abrting a task returns a success status. It shuld be dne nly: If the requested vapp went thrugh autmatic apprval lgic and was disapprved. T replace an peratin t be carried ut by anther ne. Fr example, start a pre-prvisined vapp instead f prvisining a vapp. When it is required that parameters fr a requested task be replaced. Fr example, when determining a specific virtual datacenter fr a vapp based n placement lgic Nte When calling the same peratin as the ne that triggered the ntificatin ruting and filtering must be prperly cnfigured t avid creating a lp. A task shuld be failed when the peratin ccurring befre the task is determined t fail. An example is an peratin required befre running the task failed. Fr example, CMDB was nt reachable. The task must be resumed fr peratins that must cmplete befre the next task starts. Examples include: OVF user infrmatin must be added t a vapp befre adding a vapp t catalg. Requested vapp ges thrugh autmatic apprval lgic and was apprved befre being added t vclud. Change request must recrd the bject state in CMDB system befre making change. Task prgress shuld be updated t avid having the task time ut, r t lg a status message t the end user. Blcking Tasks and Ntificatins Use Case This sectin cvers the messages published during the use case: App Authr adds a vapp frm catalg. Ntificatins and blcking task fr Instantiate vapp frm vapp Template are enabled. Ntificatin message: vapp creatin requested. (true.#.cm.vmware.vclud.event.vapp.create_request - # is used as a placehlder) Ntificatin message: VM creatin requested a scaffld bject is created and resurces are lcked ( true.#.cm.vmware.vclud.event.vm.create_request) Ntificatin message: A task t instantiate a vapp is created. ( true.#.cm.vmware.vclud.event.task.create.vdcinstantiatevapp) Blcking tasks message: vapp instantiatin has been blcked. ( true.#.cm.vmware.vclud.event.blckingtask.create.vdcinstantiatevapp) vclud Directr User Interface shws the task as Pending prcessing Case 1: System admin calls abrt n the blcked task. Blcking tasks message: The blcking task has been abrted. (true.#.cm.vmware.vclud.event.blckingtask.abrt.vdcinstantiatevapp) Ntificatin message: The vapp is mdified as per the next peratin. ( true.#.cm.vmware.vclud.event.vapp.mdify) Ntificatin message: The scaffld bject is deleted. Resurces are unlcked. ( true.#.cm.vmware.vclud.event.vm.delete) Ntificatin message: The vapp instantiatin is abrted. ( true.#.cm.vmware.vclud.event.task.abrt.vdcinstantiatevapp)

82 The newly created bject is n lnger displayed frm vclud Directr user interface. The task can be seen in Lgs/Tasks. Case 2: System admin fails the blcked task. Blcking tasks message: The blcking task has been failed. (true.#.cm.vmware.vclud.event.blckingtask.fail.vdcinstantiatevapp) Ntificatin message: The VM is nt created ( false.#.cm.vmware.vclud.event.vm.create) Ntificatin message: The vapp instantiatin task has been failed ( true.#.cm.vmware.vclud.event.task.fail.vdcinstantiatevapp) vclud Directr User Interface shws the task is having an errr n bject grid and in Lgs/Tasks. Case 3: System admin resumes the task. Blcking tasks message: The blcking task has been resumed. (true.#.cm.vmware.vclud.event.blckingtask.resume.vdcinstantiatevapp) Ntificatin message: The vapp is instantiated. ( true.#.cm.vmware.vclud.event.task.start.vdcinstantiatevapp) Ntificatin message: The vapp is created. ( true.#.cm.vmware.vclud.event.vapp.create) Ntificatin message: The VM is created. ( true.#.cm.vmware.vclud.event.vm.create) Case 3a: The vapp instantiatin is successful r abrted. ( true.#.cm.vmware.vclud.event.task.cmplete.vdcinstantiatevapp) Case 3b: The vapp instantiatin fails. ( false.#.cm.vmware.vclud.event.task.cmplete.vdcinstantiatevapp) Using vcenter Orchestratr as a vclud Directr Extensin VMware vcenter Orchestratr fully supprts cnsumptin f blcked tasks and ntificatins messages, callbacks, and calls t external systems via the vclud Directr, AMQP, and ther prduct plug-ins. Figure 14. vcenter Orchestratr as a vclud Directr Extensin The AMQP plug-ins cmes with wrkflws, and requires a ne time setup. 1. Add a brker Add an AMQP brker with prviding hstname and credentials. 2. Declare an exchange Declare an exchange fr the cnfigured brker. 3. Declare a queue Declare a queue. 4. Bind Bind a queue t an exchange by prviding a ruting key. 5. Subscribe t queues Enables message updates n new messages.

83 This cnfiguratin is saved and reladed autmatically when the vcenter Orchestratr server is restarted. The plug-in supprts adding a plicy element f type subscriptin having an nmessage trigger event. A plicy can be set up t start a wrkflw that prcesses new messages. Wrkflws are prvided t triage and prcess the message t utput vclud Directr bjects. These can prvide all f the infrmatin necessary fr audit purpses and fr designing custm lgic befre calling external systems. External systems are called using specific vcenter Orchestratr plug-in adapters such as vclud Directr, vcenter, Update Manager, Active Directry r generic plug-ins adapters such as REST, SOAP, XML, SSH, and JDBC. Blcked tasks bjects can then be abrted, resumed, r failed by calling vclud Directr Wrkflws. vcenter Orchestratr as an Extensin Example This sectin shws a simple example leveraging the blcked tasks as a trigger mechanism fr starting extensin wrkflws using different vcenter Orchestratr plug-ins. As a prerequisite, a subscriptin t an AMQP queue, bund t the exchange used by vclud directr, was created using the wrkflws listed in the previus sectin. As part f this, the ruting key is set t filter n vapp creatin (#.blckingtask.create.vdcinstantiatevapp). Next an Apprve new vapp plicy is created t listen n new messages. It is set t start the Apprve a vapp wrkflw. Figure 15. vcenter Orchestratr AMQP Subscriptin Plicy

84 The Apprve a vapp wrkflw is designed as shwn in Table 18. Table 18. Apprve a vapp wrkflw Wrkflw Descriptin Plug-in in use Imprtant infrmatin is extracted frm the subscriptin message such as the name f the vapp requester and the scaffld bject f the vapp being requested. The detailed prperties f the requested vapp are gathered. The vapp requester s manager name and is fund in Active Directry, an is sent t apprve the vapp. It cntains all the details gathered befre. AMQP vclud Directr Active Directry and Mail The wrkflw is stpped until the apprver fllws the link in his , authenticates using his Active Directry credential, and apprves r rejects the vapp. Depending n if the vapp was apprved r nt, the abrted task is resumed r abrted. An message is sent t the requester. vclud Directr and Mail VIX API The VMware VIX API enables autmatin f virtual machine peratins, and libraries are available fr C, Perl, and COM. Prgrams r scripts making use f the VIX API are referred t as VIX clients. Cmmn use cases fr VIX API virtual machine peratins include: Perfrming pwer peratins (start, stp, suspend, resume) n a virtual machine. Perfrming VMware Tls installatin (sme manual interventin may be required). Resetting passwrds. Killing system prcesses. Cleaning up temprary lg files. Installing/cnfiguring sftware inside the guest perating system. Cpying files t r frm the guest perating system.

85 If perfrming peratins that can affect the file system r execute prgrams within a guest perating system, the VIX client must authenticate with the guest perating system. The VIX client prvides a username and passwrd that can be authenticated as a valid user accunt by the guest perating system. VIX clients may run prgrams r scripts within a guest perating system. This capability can be used t install sftware, run maintenance tasks, and trigger actins based n cmplex event prcessing. When installing sftware using the VIX API, having the ability t install the sftware in an unattended and/r scripted fashin will simplify the prcess. Because VIX API virtual machine peratins use VMware Tls as the cmmunicatin path t the guest perating system, an available netwrk cnnectin is nt required. This allws VIX clients t run prgrams r scripts and perfrm ther cnfiguratin tasks befre a netwrk cnnectin is made available by the guest perating system. Private vclud and managed services prviders ften require agent-based sftware t be installed and cnfigured in the guest perating system f virtual machines. Public vclud prviders having additinal value-add capabilities may als require agent-based sftware and/r the ability t perfrm custmizatin f virtual machine guest perating system cnfiguratin elements. Agent-based sftware examples include: Backup/restre. Perfrmance mnitring. Virus scanners. Custmizatin f sftware within a virtual machine may be pssible thrugh scripts r prgrams executed within the guest perating system. A VIX client can execute these scripts r prgrams using cmmand line arguments t pass values t the script r prgram. As an example, cnsider the public vclud prvider that: Prvides NAS strage as a value-add service. Has a prtal that allws cnfiguratin and prvisining f the strage fr cnsumptin by client virtual machines running in the vclud. Autmatically cnfigures the guest perating system t munt the strage and makes the munt cnsistent acrss rebts. In large envirnments where cmplex events are ccurring in systems linked by infrastructure services r applicatin cmpnents, it may be necessary t have a centralized wrkflw system that can trigger tasks within virtual machine guest perating systems. vcenter Orchestratr has a VIX plugin that extends the wrkflw capabilities in vcenter Orchestratr all the way t the guest perating system within a virtual machine.

86 Appendix F: Business Cntinuity Backup and restre f the entire vclud infrastructure invlves the crdinatin f numerus cmpnents. Cnsider what is necessary t recver frm a service disruptin. What cmpnents are mst critical and cmplex t restre? What types f failures wuld be the mst catastrphic? The biggest threat t data lss is nt hardware failure, but peple accidentally deleting r incrrectly cnfiguring their vapps. vapp Backup/Restre Currently, mst backup prducts lack integratin with vclud Directr. Withut visibility int the vapp metadata stred in the vclud Directr database, recvery invlves manual steps t restre data and reestablish cnfiguratin attributes. Sme f the cnfiguratin attributes include the wner, netwrk, and rganizatin, and can be manually cnfigured r re-assigned thrugh the vclud API. The fllwing sectins walk thrugh hw a vclud backup prduct wuld backup and restre a vapp in the vclud envirnment. Use the fllwing high-level prcedure t back up and restre a vclud vapp: 1. Manage credentials. 2. Prtect vapps and create backup jbs. 3. Execute backup jb. 4. Recver vapp t new r verwrite existing. Manage Credentials Withut credentials, n systems are accessible. Because vclud Directr and vsphere cmpnents have separate sets f credentials, the backup prduct either requests the user t enter bth sets f credentials at runtime r harvests the credentials fr later use. Figure 16. Credential Management Wrkflw

87 Prtect vapps and Create Backup Jbs With valid credentials, the backup prduct can cnnect t vclud Directr and vsphere cmpnents, extract the data hierarchy, and list the UUIDs f the vapps available fr backup. Use the vclud API r the vclud Directr Web cnsle t perfrm this task. Then, find the lcatin f the virtual machines t backup. If using REST cde, the lgic is t: 1. Start at the tp level f the inventry by getting a list f the vcenter Servers that are attached t vclud Directr and all f the rganizatins. 2. Build a map f the vcenter Servers keyed n their ID fr easy lkup later. 3. Brwse t the apprpriate level. When brwsing t an rganizatin virtual datacenter, all the vapps in that rganizatin virtual datacenter are visible, as well as all the datastres accessible t the rganizatin virtual datacenter (thrugh the parent prvider virtual datacenter). When brwsing t a vapp, all virtual machines in that vapp are visible. 4. The data captured by the end f the prcess shuld be: Organizatin. Organizatin virtual datacenters. Datastres. vapp netwrk cnfiguratin (vapp netwrks, rganizatin netwrks, and NAT, firewall, and DHCP settings). Virtual machines belnging t that vapp. Fr each virtual machine, retrieve the same virtual machine prperties needed t perfrm vsphere backups (managed bject reference, netwrk, descriptin, thers). Execute Backup Jbs After lcating the virtual machines t back up, the backup prduct can execute the backup jb using the apprpriate infrmatin. The APIs used are the vclud API, vsphere API, and the VMware Disk Develpment Kit (VDDK). VDDK is a subset f the VMware APIs fr Data Prtectin (VADP). Mst custmers are n lnger using agent-based backups, pting fr mre efficient and tightly integrated prducts that leverage VADP. Agent-based backups can be used in a vclud envirnment t vercme sme f the challenges psed by vapp netwrks. Recvery Prir t recvery, place the vapp in maintenance mde t prhibit end users frm perfrming peratins that change the state f the vapp. After recvering the vapp, make the vapp available by exiting maintenance mde. T restre vapps t a previus state, shut dwn the vapp and use the backup prduct t verwrite existing virtual disk files in the vapp.

88 Recvery f a deleted vapp requires re-imprting virtual machines int vclud Directr as fllws: 1. Imprt the first virtual machine int a new vapp, thereby creating the vapp. 2. Imprt the rest f the virtual machines belnging t the vapp. 3. Cnfigure each virtual machine with the apprpriate prperties (rganizatin virtual datacenter, the newly restred name, vapp netwrk, and s n.) 4. After all virtual machines have been imprted, validate that the crrect prperties are in place (netwrk cnnectins, wnership). Infrastructure Backup/Restre Synchrnize the backup f all vclud infrastructure cmpnents. There are multiple ways f achieving this by using snapshts, VADP, r ther backup tls. Quiesce all databases at the same time befre taking snapshts r creating backups. A database ut f sync can cause a recvery nightmare. Table 19 cvers the recmmended prtectin plicies. Table 19. Recmmended Prtectin Plicies Type Descriptin Data Prtectin Plicy vclud Directr installatin files Infrastructure Static infrmatin cnsists f prduct binaries fr each cell. VM snapsht Frequency Once vclud Directr lg files Infrastructure Dynamic infrmatin generated by each cell. Lcated in $VCLOUD_HOME/lgs. Multicell installatins wuld use a syslg server t centralize lg files. File level backup Frequency peridic vclud Directr cnfiguratin file Infrastructure Dynamic infrmatin fr each cell. File is $VCLOUD_HOME/etc/glbal.prperties. File level backup Frequency n change, peridic vclud Directr VC Prxy vclud Directr Cnsle Prxy Infrastructure Stateless Nne Infrastructure Stateless Nne vclud Directr Database Server Infrastructure Dynamic infrmatin shared by all cells. The database instance may be shared with ther applicatins. vclud database schema level backup Frequency peridic vcenter Server installatin files Infrastructure Static infrmatin cnsists f prduct binaries, and cnfiguratin files. See Backup vcenter Chargeback database and cnfiguratin files ( VM snapsht Frequency Once

89 vcenter Server Lg Files Infrastructure Dynamic generatin generated by each vcenter Server. File level backup Frequency peridic vcenter Database Server Infrastructure Dynamic infrmatin shared by all cells. There may be multiple database servers in a multi-vc cnfiguratin. vcenter database schema level backup Frequency peridic vclud Organizatins Cntent Dynamic infrmatin virtual datacenter, netwrks, vapps, virtual machines, users, catalgs. vclud REST API Frequency peridic vclud Prvider Resurces Cntent Prvider virtual datacenters, prvider netwrks, netwrk pls. vclud REST API Frequency peridic Orchestratr Applicatin database Orchestratin Cntains the wrkflw engine library (wrkflws, actins, plicy templates, cnfiguratin elements, resurce elements, web views) and the wrkflw engine current state (wrkflws status, events). Very frequently Orchestratr Plug-ins databases Orchestratr Applicatin and plug-ins cnfiguratin Orchestratin Cntains plug-ins database bjects. Very frequently Orchestratin Cntains the cnfiguratin. Upn cnfiguratin change Orchestratr Applicatin and plug-ins Orchestratin Cntains the vcenter Orchestratr Server applicatin. Upn applicatin r plug-ins upgrade Orchestratr Applicatin lgs Orchestratin Cntains the vcenter Orchestratr Server lgs. Very frequently

90 Appendix G: Upgrade Checklists These checklists cver the upgrade f vclud Directr and assciated cmpnents. Review all applicable prduct dcumentatin fr a detailed upgrade prcess. Phase 1 Upgrade vclud Directr Cells Verify perating system, database, and ther cmpnent cmpatibility with target vclud Directr versin. See the nline VMware Cmpatibility Guide ( Obtain updated vclud Directr installatin package. Backup vclud Directr cnfiguratin and respnse files. Perfrm backup f vclud Directr database and vcenter database(s). If multiple cells exist, use cell management tl t quiesce and shutdwn services n each server (see the vclud Directr Installatin and Cnfiguratin Guide). Upgrade vclud Directr sftware n all servers, but d nt start the services yet. See the vclud Directr Installatin and Cnfiguratin Guide fr recmmendatins n minimizing the interruptin f vclud Directr prtal service. Upgrade the vclud Directr database with scripts included in vclud Directr 1.5 installatin. Restart the vclud Directr services n upgraded vclud Directr servers. Cautin If Chargeback is in use, upgrade t Chargeback r later befre cntinuing in rder t minimize disruptin f metering service. Versins prir t Chargeback cannt cllect data frm vclud Directr 1.5. Nte Fr details, refer t the vclud Directr Installatin and Cnfiguratin Guide ( Upgrade vshield Manager and Edge Devices Obtain vshield Manager update package. D nt deply a new appliance. Perfrm upgrade f vshield Manager servers. Update vshield Manager authenticatin settings within the vclud Directr prtal fr each cnfigured vcenter and vshield Manager t utilize directry-based service accunts with apprpriate permissins within vcenter Reset rganizatin and vapp netwrks within the vclud Directr prtal t redeply the updated vshield Edge devices. Nte Fr details, refer t the vshield Administratin Guide (

91 Upgrade Validatin Verify vclud Directr versin n each cell. Within vclud Directr prtal, cnfirm that vcenter and hsts are available. Verify versin f vshield Manager. Verify versin f each deplyed vshield Edge device. If in use, verify that lad balancer accurately detects status f all cells. Validate service availability thrugh access t vclud Directr rganizatin prtals. Validate usage metering cllectin within Chargeback. Nte Phase 2 Refer t the vshield Administratin Guide fr mre details Upgrade vcenter Server Verify perating system, database, and ther cmpnent cmpatibility with target vcenter versin. Perfrm backup f vcenter Server cnfiguratin files. Backup vcenter database using a methd apprpriate fr cnfigured databases. Disable the vcenter server within the vclud Directr system prtal. Perfrm upgrade installatin f vcenter Server. Enable the vcenter server within vclud Directr system prtal. Install VMware Update Manager and register with vcenter Server. vcenter Upgrade Validatin Validate vcenter versin and availability status within the vclud Directr system prtal. Validate usage metering cllectin within Chargeback. Phase 3 Upgrade Hsts Backup hst cnfiguratins. Place hst in maintenance mde, and cnfirm that vclud Directr detects that hst is unavailable. Perfrm upgrade t ESXi 5, remving any incmpatible third-party packages that may be installed. Recnnect upgraded hst within vcenter t upgrade vcenter agents. Disable maintenance mde.

92 Hst Upgrade Validatin Within the vclud Directr prtal refresh status t verify that new agents are installed and hsts are listed as available. Verify detected ESXi versin within vclud Directr system prtal. Phase 4 Additinal Upgrades Upgrade all hsts that are cnnected t datastres and vsphere Distributed Switches. Upgrade VMFS Datastres t VMFS-5. Upgrade vsphere Distributed Switches. Mdify prvider virtual datacenters t supprt virtual hardware versin 8, if desired. Mdify rganizatin virtual datacenters t enable fast prvisining, if desired.