Operations Guide: Support and Maintenance i
Table of Contents Introduction... 1 Support... 2 Structure... 2 Support Tools... 5 Testing and Change Control... 8 Testing Phases... 8 Change Implementation Process... 12 Ongoing Operations... 13 XenDesktop Operations... 13 XenApp Operations... 17 PVS Operations... 20 Appendix... 25 Appendix A XenApp Delegated Rights... 25 Appendix B XenDesktop Delegated Rights... 27 Appendix C Provisioning Services Delegated Rights... 27 Appendix D XenServer Delegated Rights... 27 Appendix E Sample Call Script/Questionnaire... 28 ii
Introduction When implementing Citrix environments, support and maintenance aspects for new farms often get overlooked. Effectively maintaining a Citrix environment necessitates reliable systems be in place to ensure smooth day to day operations. This document covers main duties involved in maintaining of Citrix infrastructures. This white paper covers the following 3 sections: Support When problems arise, technical support often is the first point of contact for issue resolution. This section addresses the proper staffing, organization, training, and tools utilized in effective support organizations. Testing and Change Control Regular upgrades are required to ensure a farm environment is up to date. Change management processes are critical to ensure improvements are properly approved, tested, and validated by appropriate parties. This section covers the proper processes that ensure changes in production environments are deliberate, proven, and accountable. Ongoing Operations Maintenance, issue prevention and resolution are core responsibilities in running a Citrix infrastructure. When the responsibilities and assignments are structured properly, friction is kept to a minimum, reducing issues and their resolution times. This section discusses routine operations that Citrix environments require for optimal performance. 1
Support Well trained support members with a clear organizational structure are critical for an effective support team. Since technical support is the first place many users go, thought needs to be given to its structure, training, and tools utilized by support members. Structure Multi-tier support levels have been found to be the most effective means in addressing user support issues. Low criticality, low complexity or frequently occurring issues should be managed and resolved at the lower tiers. High criticality and complex issues are escalated to more experienced architects or infrastructure owners. The diagram below outlines a common multi-tier support structure: The first level should resolve 75% of all issues encountered. These issues include routine problems requiring limited knowledge of the Citrix environment. Issues are quickly resolved and may also be automated, like password resets. When non-routine problems exceeding the abilities of level one s are encountered, they are then escalated to level two. Information on 2
the end user s problem and attempted troubleshooting steps are documented at the first level allowing level two technicians to immediately begin addressing the problem. Level two technicians should handle only 20% of the support tickets and are highly knowledgeable on the Citrix environments. Should a complex issue arise that exceeds level two s abilities it is escalated to the third, and final active support tier level three. Level three issues are complicated and often mission critical requiring expert knowledge of the Citrix environment. Tier three support tickets should be uncommon, amount to no more than 5% of all support issues, and be resolved as soon as possible. The final tier, level four, is not involved in active support of a production environment. It is focus solely on the strategic improvements for enterprise environment, testing new technologies, planning migrations, and other high level changes. Should support discover an issue is related to an application or underlying infrastructure the ticket is handed to the appropriate team for troubleshooting. If a program bug is discovered the issue is then re-escalated and a ticket is established with the program s vendor. In parallel with the active support structure, self-service portals are available for end users to assist themselves with basic troubleshooting tasks. This assists in resolving frequently encountered issues and helps prevent the creation of new support tickets. Table 1 below covers the levels and expectations in additional detail: 3
Support Level Description Responsibilities Skill Set Level 1 (Call Entry) Provide first-line support of reported issues. Initially, servicing support messages and phone calls. This level needs to perform initial issue analysis, problem definition, ticket routing, and simple issue resolution. Additionally can handle requests for application access or support with configuring plugins. Perform issue definition, initial analysis and basic issue resolution Perform initial troubleshooting to determine the nature of the issue Collect basic user and session information Create ticket and log all troubleshooting steps performed Resolve basic Citrix related issues, connectivity problems and application related issues using existing Wiki articles Escalate issue to Tier-2 if it requires advanced skills or elevated permissions If the issue is related to particular applications or other technologies and not the Citrix infrastructure, escalate the issue to the appropriate application or technology support units If the issue is deemed important enough, escalate directly to Tier-3 Generate requests for additional issue resolution guides as necessary Tier-1 support personnel should be provided with basic training on Citrix XenApp, Citrix XenDesktop and supporting technologies. This can include internal training from subject matter experts or formal training. The training provided should focus on the following topics: High-Level overview of the XenApp and XenDesktop implementation Using Citrix Desktop Director to manage user sessions Troubleshooting Citrix XenApp and XenDesktop sessions Managing AD group policies user profiles Troubleshooting methodology In addition, regular training should be provided to the Tier-1 team members on the latest troubleshooting recommendations from the Tier-2 and Tier-3 teams as well as details on any relevant changes to the environment. This will help ensure a base knowledge level amongst the team and consistent customer service. 1
Support Level Description Responsibilities Skill Set Level 2 (Production Support Engineer) Primarily supporting day-to-day operations of virtual desktops environment, may include proactive monitoring and management. In addition, this role would also perform advanced troubleshooting and utilize available monitoring / advanced troubleshooting tools. Assist with resolving issues escalated by Level One Support. Perform intermediate issue analysis and resolution Identify root cause of issues Respond to server alerts and system outages Review vendor knowledge base articles Respond to out-of-hours helpdesk calls Respond to critical monitoring alerts Generate internal Wiki articles and issue resolution scripts and maintain Tier-1 troubleshooting workflows Perform basic server maintenance and operational procedures Manage user profiles and data Escalate ticket to Tier-3 or appropriate technology owner if advanced skills or elevated permissions are required Generate requests for additional issue resolution scripts and Wiki articles as necessary All members of the Tier-2 team should achieve the Citrix Certified Administrator (CCA) certification for both XenApp and XenDesktop. In addition, basic and advanced training on Windows concepts will be essential for team members who do not have desktop or server support experience. Finally, on-the-job training along with close integration with Tier-3 administrators is essential as the Tier-2 roles are formalized and responsibilities are handed over from Tier-3 to Tier-2. In addition, regular training should be delivered to the Tier-2 team to maintain competencies on technology and architectural updates. This will ensure a baseline knowledge level across the team. 2
Support Level Description Responsibilities Skill Set Level 3 (Build Engineer) Central point for implementing, administering and maintaining Citrix desktop and application virtualization infrastructure. This role focuses on deploying new use cases and leading lifecycle management initiatives. Generally, one Build Engineer could focus on one use-case at a time. For example, three new concurrent use cases would require three Build Engineers. Escalates issues to software vendor specific Technical Support and notifies Level 4 about this issue. Perform advanced issue analysis and resolution Perform maintenance and environment upgrades Addressing high severity issues and service outages Manage the Citrix environment Oversee and lead administrative tasks at the Tier-2 level Manage network and storage infrastructure as it relates to the Citrix environment Review periodic reports of server health, resource usage, user experience, and overall environment performance Review vendor knowledge base articles and newly released updates Perform policy-level changes and make Active Directory updates Review change control requests that impact the Citrix environment Perform advanced server and infrastructure maintenance Review Wiki articles and issue resolution scripts for accuracy, compliance, and feasibility Create Wiki articles and issue resolution scripts to address Tier-2 requests Tier-3 support team members with a minimum of three years experience implementing and supporting XenApp, XenDesktop, Provisioning Services and Windows operating systems. The Tier-3 team should have enterprise experience with industry recognized certifications. The Tier-3 administrators should complete the Citrix Certified Enterprise Engineer certification track as this will prepare them to proactively manage the user community and the Citrix solution. Citrix Certified Integration Architect (CCIA) level certifications would be optimal at this level. 3
Support Level Description Responsibilities Skill Set Level 4 (Architect) Vendor Support Self Service The Level 4 team has minimal exposure to administrative tasks but focuses on translating business requirements into technical architectures, designing the infrastructure or planning migrations. Not involved in day-to-day support. Provide technical leadership for upcoming projects Lead design updates and architecture revisions Address high severity issues and service outages Oversee technology integration workflows Review periodic reports of server health, resource usage, user experience, and overall environment performance to determine next steps and upgrade paths Review change control requests which impact the Citrix environment Review frequently recurring helpdesk issues Ensure technical specifications continue to meet business needs Update design documentation A qualified Tier-4 resource should have a minimum of five to six years of experience implementing, supporting, and serving in a technology architect role for a XenApp and/or XenDesktop environment as well as additional administrative experience with integrated technologies such as application and profile management solutions. The ideal candidate will have served in such a capacity at two or more environments for purposes of product exposure and in at least one environment of over 1,200 concurrent users. The Tier-4 resource should have enterprise experience with industry recognized certifications. A Citrix Certified Integration Architect (CCIA) certification or comparable training and experience should be a prerequisite of the role. Training or practical knowledge on XenCenter, AD Group Policies and the utilized hypervisor should be considered essential. Vendor assistance may be necessary should defects in a program be discovered. At this stage level 3 engineers need to establish a support tickets with the appropriate vendor to assist with finding a solution. A self-service portal is utilized for noncritical tasks such as application access, permissions, etc. The portal can range from a simple FAQ page to a fully automated process requiring no human interaction. The purpose of the self-service portal is to add an additional touch point for end users to address basic issues, preventing the creation of new support tickets. 4
Support Tools Providing the appropriate tools for the aforementioned support levels is a key aspect in order to achieve a high First-Time-Fix-Rate and a low Time-to-Resolution. The table below highlights several tools available for supporting Citrix environments. Tools\Processes Tools Ticket Management System Call Scripts Shadowing Tool Knowledge Base Citrix Administrative Consoles Details Used to document customer information and issues. Help Desk should also document date and time of occurrence, the application and module, steps to repeat error, obtains screenshots as needed. The first contact help desk personnel should have documented scripts or triage guides to ensure that all relevant data is captured while the user is on the phone. This practice also assists in proper triage and allows the next support tier to perform research off-line and review logs prior to customer contact. Shadowing users is a useful way for troubleshooting new issues. This allows for support technicians and administrators to remotely observe a user s actions. Documentation should be created and maintained in a knowledge base or library of known issues. Articles should be searchable for quick recovery. Knowledge bases help quickly resolve issues upon reoccurrences and reduce the need to perform future research. The incident management team will have access to the Citrix Administrative Consoles for focused troubleshooting. Citrix recommends that the Help Desk also have same level of access. The initial support tier will primarily focus on resetting connections and obtaining data for escalation. The diagram below outlines the recommended tools for every support level: Tool Desktop Director Desktop Studio Description Desktop Director provides an overview of XenDesktop hosted desktops and XenApp sessions. It enables support teams to perform basic maintenance tasks and to monitor and troubleshoot system issues. Desktop Studio enables administrators to perform configuration as well as Product Support Level XD XA PVS XS L1 L2 L3 X X X X X X X 5
Tool AppCenter HDX Monitor Provisioning Services Console XenCenter License Administratio n Console Description maintenance tasks for a XenDesktop site and associated virtual desktops. AppCenter provides support teams the ability to monitor and troubleshoot user sessions as well as server systems within a XenApp farm. In addition AppCenter enables administrators to perform configuration and maintenance tasks for a XenApp farm. HDX Monitor is a tool to validate the operation of the Citrix ICA/HDX stack of a user session. Hereby HDX Monitor provides information about client capabilities, network performance / activity, session settings and many more items. The Provisioning Services Console enables administrators to perform configuration and maintenance tasks for a Provisioning Services farm. XenCenter enables administrators to perform configuration and maintenance tasks for a XenServer Resource Pool. The License Administration Console is the interface required to manage license files and a license server or appliance. Product Support Level XD XA PVS XS L1 L2 L3 X X X X X X X X X X X X X X X X X Please refer to the following appendices for details on how delegated rights could be configured for the support staff Appendix A (XenApp) Appendix B (XenDesktop) Appendix C (Provisioning Services) In addition to the aforementioned Citrix support tools, every support organization should leverage the following instruments: 6
Ticket Management System: It is vital for support organizations to implement a Ticket Management System for tracking and escalating support calls (in between the levels) as well as to document incident and customer related information. Knowledge Base / Wiki: Ever support organization should document and maintain a knowledge base or library of known issues. The articles should be searchable for quick recovery. Knowledge bases help quickly resolve issues upon reoccurrences and reduce the need to perform future research. Call Script: The first contact help desk personnel should have documented scripts or triage guides to ensure that all relevant data is captured while the user is on the phone. This practice also assists in proper triage and allows the next support tier to perform research off-line and review logs prior to customer contact. Please refer to Appendix E Sample Call Script / Questionnaire for a sample call script. Staffing Requirements This section provides a guideline for the number of support staff required for several sizes of Citrix XenApp or XenDesktop enterprises. This information is primarily based on Citrix experience in deploying enterprise environments, taking into account the most typical design requirements. Additional refinement may be necessary to optimize the support of the environment based on specific business requirements. XenApp centric organization Farm or Load Managed Groups #Servers #Users Help Desk Admins Architects 1 <100 <3000 1 CCA 1-2 CCEE 1 CCIA 2-5 <200 <4000 3 CCA 3-4 CCEE 1 CCIA >6 >200 >5000 3 CCEE 5+ CCEE 2 CCIA XenDesktop centric organization XenDesktop Sites #Desktops #vdisks Help Desk Admins Architects 1 <500 1-2 1 CCA 1-2 CCEE 1 CCIA 2-3 1000-5000 3-5 3 CCA 3-4 CCEE 1 CCIA >3 >5000 5+ 3 CCEE 5+ CCEE 2 CCIA 7
Testing and Change Control Regular updates and maintenance is an everyday part of Citrix environments. Standard processes must be closely followed to ensure modifications do not inadvertently damage a production environment. This includes maintaining dedicated infrastructure where modifications are tested prior to production and processes that ensure changes are deliberate and accountable. Testing Phases Since changes to Citrix infrastructure can impact thousands of XenApp and XenDesktop users, multi-phase testing is critical for the reliability of the environment. As such, Citrix recommends implementing the following testing strategy: Development. The development infrastructure exists outside of the production network. Typically it consists of short-lived virtual machines, whose configuration should match the configuration in production as close as possible. The purpose of the development phase is to provide change requestors a non-production environment to perform proof of concepts, determine integration requirements and perform iterative testing as part of a discovery phase to making changes. Proposed changes should be documented to they can be applied in the test phase. Test. The test environment is a standalone 1:1 copy of the production infrastructure and is used to confirm that the proposed changes can be easily repeated prior to the pre-production staging environment. The changes made should follow exactly from the documentation provided from discoveries made in the development stages. If testing fails within the testing stage, the architect must determine the severity of failure and determine whether minor updates to documentation is sufficient or a full development cycle is needed. Pre-Production. The staging environment can be co-located with the production environment as the changes should be tightly managed. The goal of staging is to implement the proposed changes with little risk or uncertainty. It is expected that any changes made to the staging infrastructure have been tested and documented for repeatability. There should not be any iterations or adjustments required within this phase. For updates, the starting point for all systems in the staging environment should be the current production status. However for new deployments, the architect will determine the appropriate start point; whether an existing system or newly created. Suring this phase and within this environment User Acceptance Testing (UAT) should be performed. 8
Production. The production environment is the fully redundant and scalable solution for normal usage of the XenDesktop and XenApp environments. There should be minimal changes to the environment. Infrastructure There is a need for four dedicated levels of testing that ensures changes do not harm a production farm. An isolated development environment should be used initially to try any new changes, application installations, and updates. Conflicts resulting from these changes have no impact in this environment and can be resolved before they affect any infrastructure. Once proven, safe changes are then applied to a larger test environment. The test environment should mimic production as closely as reasonably possible. This includes all patches, applications, and policies be similar so that testing is as realistic as possible. Once changes prove stable in the testing environment it is recommended the changes not be initially rolled out to production all at once. Updates should be first applied to a targeted pre-production subset of users to again validate and approve of changes made. This way if any problems go undetected in the test environment the problems affect only a small subset of users. When validated in pre-production changes can be applied to the production farm. Change Management Processes Standardized processes that manage application lifecycles through installation, updating, and end of life are necessary to ensure consistent and accountable performance of a farm. The change control process should be closely followed starting with a change request. For any changes to be made to a production environment a change request form must first be filled out detailing changes requested, reasons for the change, and intended timeframes for the action. This is then reviewed and edited if required by a change manager and then advisory board. When the change request has gone through the entire change approval process it is then given to a change implementer who then stages the change for testing, and finally conducts the implementation in production. The table and steps below further detail the process: 9
10
Procedure 1. The Request for Change (RfC) form is completed by any person requesting a change 2. After appropriate manager approvals have been acquired, the RfC is forwarded to the appropriate Change Manager(s). 3. The Change Manager validates the RfC for completeness and logs the RfC information into the Change Control Log for tracking. Incomplete CR s are returned to the requestor for update and re-submission. 4. The Change Manager assesses the impact of the change in conjunction with subject matter experts and/or managers of the teams associated/affected by this change. 5. The Change Manager works with the associated/affected teams as well as the change requestor in order to confirm the priority, category and type of the change as well as the proposed back-out plan. 6. If the process succeeds the RfC is forwarded to the Change Advisory Board for approval. If the process does not succeed the change control log is updated with the current status as well as the reason of the rejection and the RfC is send back to the requestor. 7. The Change Advisory Board reviews and validates the change in detail, and discusses and evaluates purpose, reasons, impact, cost and benefits. Each Board member represents their department and provides guidance on the change requests. The Change Advisory Board also reviews multiple requests to coordinate implementations and package requests into a single release schedule. 8. Upon approval the change is send back to the Change Manager to schedule the change for implementation into the staging environment. 9. The change is implemented and test are conducted. The results are sent back to the Change Manager. 10. If the staging implementation and testing was successful the change is scheduled for production implementation. In case the staging phase was not successful another staging iteration will be conducted. 11. The change is rolled out to production. 12. The Change Manager reviews the implementation and finally updates the Change Control Log. 13. On a periodic basis, the Change Manager reviews the Change Control Log to analyze for trends on type, frequency and size of changes and forwards the results to the Change Advisory Board for review. In the instance of an emergency the processes may be expedited. Should an issue be declared an emergency a change request form is still filled out and delivered to the appropriate change management representative. When approved the requested change is immediately 11
implemented and the advisory board notified. Due to the criticality of time in an emergency timeframes are not required. Change Implementation Process The process to implement changes in Citrix environments need to be highly structured. Standardization helps ensure reliable results while reducing risks of unexpected results to production environments. Update requests undergo a centralized approval and then must go through an engineering and tracking process as they are introduced into environment. The process begins at a development environment and progresses through testing, preproduction, and finally full production. Potential updates must first undergo an assessment to determine compatibility and functionality in a development environment. This environment provides a rudimentary setup where changes are initially attempted to observed potential conflicts or performance issues. These results provide feedback on areas to be further tested while not risking more significant environments. Once proven and approved in the development environment further tests are conducted in a testing environment. This is larger environment that models what is used in production. Having an arrangement that closely matches production ensures testing results are accurate. All changes to this environment are document for repeatability in the production environment, future reference, and for change control. After rigorous testing in the test environment changes must be re-approved and then reduplicated to a pre-production subset of users in the production environment. This selection of users provide additional feedback on their experiences and ensure the changes perform as expected. Should no issues be encountered by the pre-production users updates are then written off by appropriate parties and applied to the rest of the farm. The below diagram highlights the key sections in the change implementation process. 12
Ongoing Operations It is best practice to keep operating systems, applications, and Citrix components all up to date with applicable hotfixes and updates to ensure stability, performance, and security of a farm. In general Citrix recommends that Hotfix Rollup Packs and Service Packs always be installed, general hotfixes be installed on an as needed basis only, and that all changes follow standardized change management processes for any changes (i.e. development, testing, pre-production, and production). For more information regarding general hotfix maintenance refer to (CTX132799). Monitoring of a new environment enables administrators to address issues proactively. By having an in depth understanding of current and expected behavior of the various components, administrators are better equipped to discover an issue before it impacts the user community. Make sure to define the monitoring activities that administrators will perform on a periodic basis, as well as the tools that will be used. In addition, determine which reports will be made available to management. A variety of built-in and customizable reports exist within the Citrix products so that administrators can track trends, effectively plan capacity, and address auditing requirements. In addition, troubleshooting is typically minimized when administrators fully utilize the tools integrated into the Platinum edition, as well as other third party tools. It is recommended that support staff and administrators involved in implementation review the product documents, articles, and guides to support, manage and administer the environment. All of these documents are available from the Citrix Product Documentation Library (http://support.citrix.com/proddocs/index.jsp) or on the Citrix Knowledge Base (http://support.citrix.com). The remaining sections in this section identify the maintenance tasks (daily, weekly, yearly and preventative) required to support each component of the XenDesktop Platinum edition. XenDesktop Operations This section covers the regular maintenance and upkeep involved in a XenDesktop environment. Overview Several tasks need to be performed to ensure a stable, scalable Citrix XenDesktop platform. The team responsible for supporting and managing the Citrix XenDesktop infrastructure should perform regular tasks that should be incorporated into the existing standard operating procedures. These are usually broken down into daily, weekly or yearly tasks. 13
Periodic Tasks Daily Periodic Tasks The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenDesktop administrators on a daily basis: Daily Tasks Citrix XenDesktop Checking of the event logs on all Citrix XenDesktop (Desktop Delivery Event log checking Controller) servers including Windows System, Application and Error logs. Checking of the Citrix XenDesktop alerts in Desktop Studio needs to be Citrix XenDesktop Alerts performed at least once per day to ensure no issues have occurred with the Citrix XenDesktop environment. Check Citrix XenDesktop logs Test Virtual Desktop Access Checking of Citrix XenDesktop specific application logs including the Pool Management Logs. To assist with troubleshooting efforts, refer to Comprehensive List of Event Log Messages for XenDesktop on http://support.citrix.com/article/ctx119978 Simulate a connection to ensure access to Desktop Groups (Pooled and Assigned) is available to users. Weekly Periodic Tasks The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenDesktop administrators on a weekly basis: Weekly Tasks Citrix XenDesktop Review Access Check the status of the Citrix XenDesktop Desktop Groups, Idle Pool count Management Console and connectivity to backend hosting infrastructure. Review the latest Citrix hotfixes and ascertain whether the XenDesktop Hotfixes and Patches Controller and virtual desktops require them. Review the local installation of printer drivers on the virtual desktops, if Printer Drivers required for auto-creation. Review Citrix Knowledge Review incidents logged for any common issues related to Citrix XenDesktop Base for latest incidents on http://support.citrix.com/ Backup Citrix XenDesktop Perform full-data backup of the Citrix XenDesktop Datastore (could be a daily Database task due to size of the transaction log) Backup User Data Backup user-related file data. Yearly Periodic Tasks The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenDesktop administrators on a yearly basis: Yearly Tasks Citrix XenDesktop Perform restore test Conduct backup tests to validate the actual restore process is functioning correctly. Citrix Policy Assessment Review Citrix Policies and determine whether new policies are required and existing policies need to be updated. 14
Yearly Tasks Citrix XenDesktop Software Upgrade Review and asses the requirement for new Citrix XenDesktop software releases/versions. Capacity Assessment Perform capacity assessment of the Citrix XenDesktop environment. Preventative Maintenance As a part of your preventative maintenance routine for your Citrix XenDesktop environment, Citrix Xnapshot for XenDesktop should be run on regular three month intervals. This tool collects a wealth of information about the underlying system and its configuration, such as BIOS information, registry information, device drivers, Windows services, installed hotfixes, Citrix binaries, and Citrix XenDesktop farm information. Citrix recommends that the following key Microsoft Windows services be monitored on all of the Citrix XenDesktop (Desktop Delivery Controller) servers to ensure reliability. Citrix XenDesktop Services to be Monitored Citrix Desktop Delivery Controller Service Citrix Diagnostic Facility COM Server Citrix Management Server Citrix MFCOM Service Citrix Pool Management Service Citrix Services Manager Citrix XTE Server IPSEC Services Remote Procedure Call (RPC) Server Windows Management Instrumentation Driver Extensions Workstation Updating MCS Images MCS images require regular maintenance throughout the image s lifecycle. It is standard procedure to first take a snapshot of a master image prior to any updates. New changes are made to this snapshot and it will become the new master image. The new image, following standard change management policies, is fully tested in an isolated test environment. When the new image successfully passes testing and is proven stable the image is rolled to preproduction subset in of production users. When a new image is shown to be reliable in preproduction it can then be rolled out to the rest of the farm. Below is a diagram and steps further detailing this process. 15