Deploying CloudPortal Services Manager 11.x for High Availability and Disaster Recovery

CloudPortal Services Manager Deploying CloudPortal Services Manager 11.x for High Availability and Disaster Recovery Last Updated: Mar 23, 2015

Contents Introduction... 3 Databases... 3 Pre-requisite... 3 Installation... 3 Adding CPSM Databases to the Availability Group... 4 Login Replication... 5 Recover from the database failure... 5 Provisioning Engine... 6 Pre-requisites... 6 Installation... 6 Directory Web Service... 10 Web Portal and API... 11 HAad Services for App Orchestration... 12 Reporting... 13 Other Web Services... 13 Deploy CPSM with App Orchestration HAad Service in multi-datacentre... 14 Multi-Datacentre diagram... 14

Introduction CloudPortal Services Manager consists of multiple components: Core Components: Databases Provisioning Engine Directory Web Service Web Portal & API Reporting Optional Components: Service-specific web services or tools (e.g. XenDesktop Web Service) This document provides a basic guidance for deploying these components to support highly availability and the solution for disaster recovery based on Windows Server 2012, 2012R2, and SQL Server 2012. Databases CloudPortal Services Manager uses DNS Alias (CNAME) that points to the SQL server instance name in connection strings, this simplifies the potential future database move from one SQL server cluster/group to another. Pre-requisite SQL Server 2012 AlwaysOn Database Availability Group is configured correctly, enable SQL Server and Windows Authentication mode on each instance and operational following the Microsoft guidance. Installation For CPSM to support the SQL server AlwaysOn in a single subnet, first make the CORTEXSQL DNS alias point to the Listener name as part of the preparation for the initial deployment. Follow the normal process to start the CloudPortal Services Manager system database installation: http://support.citrix.com/proddocs/topic/ccps-115/ccps-install-database-create.html. If the listener configured for the AlwaysOn Availability group is on a port other than 1433, the correct port number must be specified during the installation step of Create System Databases :

Once the installer completes Create System Databases, verify that the two CPSM system databases OLM and OLMReports are restored on the primary replica. Adding CPSM Databases to the Availability Group Follow the steps below to add the CPSM databases OLM, OLMReports, and OLMReporting (Reporting database) to the database availability group. 1. Logon to the SQL server that hosts the primary replica and start SQL Server Management Studio. 2. For each of the 3 databases, change the recovery model from Simple to Full, and do a full backup. 3. In Object Explorer, browse and expand the Availability Groups. 4. Right-click the relevant group, and then click Add Database. 5. On the Select Databases page, all databases that are eligible to become the primary database for the new availability group are listed in the table. The CPSM databases should be shown as Meets requirements. Use the checkboxes to select the 3 CPSM databases and click Next. Alternatively these databases can be selected and added individually. 6. On Select Initial Data Synchronization page, accept the default Full option, and Next. 7. If the Validation page displays the results of six checks as successful, click Next to continue. If any test fails, action must be taken to correct the error items and re-run the validation. 8. On the Summary page, verify the configuration of the replica, and then click Finish.

9. And it s suggested to set the Availability Mode to Synchronous commit for each replicas on Availability Group since asynchronous mode may cause the data loss while failure happen Login Replication CloudPortal Services Manager SQL logins are not automatically replicated in the availability group so that need to be manually created on the secondary replicas. Follow the Method 3 in http://support.microsoft.com/en-us/kb/918992 Connect to secondary replicas by Management Studio, create the account for CortexProp, OLMUser,OLMReportsUser and OLMReportingUser, the statement will like: CREATE LOGIN [CortexProp] WITH PASSWORD = 0x02000B50F2F545B3F50C45069BCBFF1598A482E6E4448859D2FEA6C2C43FFAE3CB805E1F7FE7C C9F6BF7357358B1FDCEFCAC6865327AEAD3452B9D62718516B09ACEA4354278 HASHED, SID = 0x282407CECF437D46A8EEEBC0605F3C7E, DEFAULT_DATABASE = [master], CHECK_POLICY = ON, CHECK_EXPIRATION = OFF Where PASSWORD and SID will be the actual one from the output Recover from the database failure If the hardware/network failure happened on the main datacenter(node 01 and Node 02), Node 03 in the remote datacenter for example, below is the steps suggested: 1. Administrator connect to the SQL instance of Node 03(remote site) and perform a forced failover, right click on the availability group Failover, select the new primary replica Node 03, or it may automatically failover to Node 03 depend on your settings, now it will become the primary replica to provide the database services 2. Once the Node 01 and Node 02 comes back online and re-establish the communication with WSFC cluster, administrator manually resume the database, Resume an Availability Database (SQL Server) To resume a secondary database 1. In Object Explorer, connect to the server instance that hosts the availability replica on which you want to resume a database, and expand the server tree. 2. Expand the AlwaysOn High Availability node and the Availability Groups node. 3. Expand the availability group. 4. Expand the Availability Databases node, right-click the database, and click Resume Data Movement. 5. In the Resume Data Movement dialog box, click OK. 3. Administrator changes the new primary replica to synchronous-commit mode, it will enable resumed secondary databases to become SYNCHRONIZED,

After this step, Node 03 s database will overwrite the one in Node 01 and Node 02,skip this step if you want to keep Node 01 s database 4. Administrator perform a manual failover to original primary replica(node 01) Provisioning Engine CloudPortal Services Manager provisioning engine is dependent on Microsoft Message Queuing, for high availability requirement, MSMQ needs to be clustered, and so as the CPSM provisioning engine. Pre-requisites The provisioning server cluster (Windows Server 2012 Failover Cluster) is created, and in addition, all servers must be able to see a shared storage device (i.e. a SAN drive) and be able to take ownership of it. Shared storage is not a requirement for Windows Server 2012 clusters but is a requirement for some Microsoft services, in this case Microsoft Message Queuing, and port 1801 for Message Queue should not be blocked by the firewall, since Web portal will deliver the message to provision engine in the cluster by HTTP, now it can only support IPv4, so make sure IPv4 is your referred protocol, refer to http://support.microsoft.com/en-us/kb/929852 to know how to disable/enable IPv6 on Web Portal servers Installation Install and configure CPSM Provisioning role on each of the cluster nodes using the CPSM v11.x installer: http://support.citrix.com/proddocs/topic/ccps-115/ccps-install-config-roles-gui.html. If you configure the Provisioning role on the secondary nodes with the same service accounts, make sure the password matches the ones specified for the same accounts when configuring on the primary node, and the same Encryption Service can be contacted to retrieve the encrypted key, If the service cannot be contacted, the Configuration Tool prompts you to import the encrypted key using a key file. To generate the key file, see Generate and export keyfiles for the Encryption Service Follow the steps below to configure the cluster: 1. On the cluster node, open Failover Cluster Manager. 2. Expand Cluster and right click on Roles and select Configure Roles. 3. Click Next and select Message Queuing and click on Next.

4. Enter the name that the clients will use to access this cluster role. Click Next. 5. Select the shared disk drive name to assign to the cluster role, and Next.

6. Click Finish when High availability was successfully configured for the role message and summary are displayed. 7. Right-click on the newly added role above, and select Add Resource > Generic Service. 8. Select Citrix Queue Monitor Service from the list, click Next, follow the wizard and finish. 9. Right-click the new resource Citrix Queue Monitor Service, and select Properties. On Dependencies tab, add MSMQ-{your new cluster role} as the dependency, and click Apply. 10. Select General tab, check Use Network Name for computer name, and OK. This step must be performed after step 9, otherwise an error would occur.

11. On each server node, stop Citrix Queue Monitor Service, and open Registry. 12. On each server node, Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CortexQueueMonitor, replace the value off the key DependOnService with MSMQ$MSMQ- {YourClusterRoleName}. 13. On each provisioning server, navigate to the Provisioning Engine folder and edit the CortexQueueMonitor.exe.config file to point the various queues (request/bulk request etc) to the cluster name, for example: <add key= Message Queue value=.\private$\cortexrequest /> would become <add key= Message Queue value= CPSMCluMSMQ\Private$\cortexrequest /> 14. From Failover Cluster Manager, Bring Online Citrix Queue Monitor Service. 15. Logon to CloudPortal Services Manager portal as a Service Provider Administrator, go to Configuration > System Manager > Locations, and expand the relevant location. 16. Change the server name or IP address of the following queue paths to the Message Queuing cluster role name or IP (not the Windows server cluster name) and Save. Primary Queue Path Bulk Queue Path Usage Data Queue Path

17. When the preceding changes are complete, restart the CortexDotNet website(s) (through IIS) and the provisioning engine(s) (through the Failover Cluster Manager) to reflect the new changes 18. To view the message queue status, requests, and journals for monitoring and troubleshooting purposes, you can no longer use the local Message Queueing on each of the nodes, instead, on the host node, from Failover Cluster Manager, select Message Queuing cluster role, and open Manage Message Queuing 19. If you want to troubleshooting the Message Queue, you can enable the End2End even log 1. Open Event Viewer on the host node, navigate to Applications and Services Logs->Microsoft->Windows->MSMQ, right click on End2End, click Enable Log 2. Any sent/received message can be reviewed from End2End panel Directory Web Service The directory web service is typically located on the same server as the Provisioning Engine and listens on port 8095, after installed directory web service on each servers, load the global configuration file and configure it from the Configuration Tool, When the CPSM provisioning

server is clustered, the Directory Web Service is also installed on all cluster nodes (refer to Provisioning Engine section). Follow the steps below to configure CPSM to achieve Directory Web Service high availability. 1. Logon to CPSM portal as a service provider administrator. 2. Navigate to Configuration > System Manager > Servers. Click Refresh Server List, the provisioning server cluster name should appear in servers list. 3. If you are configuring for a remote location domain and the DNS of the server names may not be resolvable in the primary location domain, expand the server cluster, and enter IP address in Alias field, and Save. 4. Click Server Roles link on the left or got to Configuration > System Manager > Server Roles, expand the provisioning cluster, tick Directory under Server Connection Components and Save. 5. Click Server Connections link or go to Configuration > System Manager > Server Connections, expand the existing entry for the Directory role, select the provisioning cluster name from the Server dropdown list, and click Save. 6. Test the connection via the icon on the right. It should go green if valid and there no firewall blocking issues. Directory Web Service can also be load balanced, in this case the configuration should use the load balanced VIP address instead of the cluster name/address. Web Portal and API The web portal (interface) and API are under the same standard.net framework 4.0 website in IIS which listens on ports 80 and/or 443. To enable high availability, it is recommended to run two (or more) nodes in a standard load-balancing setup. Sticky sessions are required. Installation steps are as below: 1. Install and configure Web server role on all load balanced servers by the same configuration file: http://support.citrix.com/proddocs/topic/ccps-115/ccps-installconfig-roles-gui.html. If you are adding additional web servers to an existing deployment with functional services, it is recommended to skip the Service Package Import (deselect the services and properties) or select Ignore to the properties of all enabled services.

2. The following items of CPSM web on the first/primary web server must be replicated to all load balanced web servers. It is recommended to replicate all the files in CortexDotNet and CortexAPI sub sites (except the web.config files specific to the local sites) from the first/primary web server to all the rest of the web servers. Images for branding. Stylesheets for branding. Any custom downloads. Any custom DLLs or pages Web.config configuration changes IIS Security and authentication changes And be sure to keep above files synced if you have any configuration changes manully 3. Add the same URL (internal and/or external) host headers to Cortex Management site on all web servers in IIS Manager, by default it s CortexWeb 4. Add one DNS records with the IP address of load balanced VIP, and point CORTEXWEB to it 5. Recycle CortexMgmt application pool via IIS Manager on all web servers. Steps to configure load balance on Netscaler for Web Portal and API: 1. Login to Netscaler, Traffic Management->Servers, add the Web Portal servers, Specify the name and IP address 2. Go to Traffic Management->Services to configure the services, add services definition for https load balancing without SSL offload, in the Protocol, choose SSL_BRIDGE and Choose https as Monitors, for each servers 3. Go to Traffic Management->Virtual Servers, add one virtual server with Protocol SSL_BRIDGE, set Persistence to SOURCEIP to make sure the transaction is in one same session and activate the services 4. Add one DNS records for the virtual server, and point CortexWeb DNS aliases to this virtual server 5. If failure happen, restart the browser to open the web console, at this time, another live Web portal server will provide the service HAad Services for App Orchestration To provide high availability of HAad services for App Orchestration, at least 2 AO configuration servers should be installed and configured firstly, then 1. Install and configure HAad service http://support.citrix.com/proddocs/topic/ccps- 115/ccps-install-haad.html on each configuration server 2. Load balancing the AO configuration servers by Netscaler, follow the similar steps Steps to configure load balance on Netscaler for Web Portal mentioned before 3. Navigate to Configuration > System Manager > Servers. Click Refresh Server List, if VIP name does not exist on the list, click Add a Server link, enter the VIP name in the Server, FQDN in the Alias, the FQDN should match the common name in SSL certificate, or connection may fail, and click Add Server.

4. Click Server Roles link on the left or got to Configuration > System Manager > Server Roles, expand newly created VIP Server placeholder, tick the Hosted Apps And Desktops role under Server Connection Components and click Save. 5. Click Server Connections link or go to Configuration > System Manager > Server Connections, New Connections, select Hosted Apps And Desktops from server role dropdown list and choose the newly created server, Save 6. Test the connection via the icon on the right. It should go green if valid and there no firewall blocking issues. Reporting The high availability of CloudPortal Services Manager Reporting role is dependent on the SQL Reporting Services HA configurations. The MS SQL Reporting Services achieves HA via a scale-out deployment so that they share the same report server database: https://msdn.microsoft.com/en-us/library/bb522745.aspx. The data source DB (OLMReporting) of CPSM Reports can be added to AlwaysOn availability group (refer to Adding CPSM Databases to the Availability Group section), in this case when installing Reporting role via CPSM v11.x installer, the listener name and port should be specified for the Reporting SQL server. Other Web Services Similar to the Directory Web Service, other service integration related CPSM web services like Exchange, Lync, XenDesktop, XenApp, and IIS web services, etc. can be deployed to multiple servers for high availability: 1. Install and configure the web service on all HA servers using the installer: http://support.citrix.com/proddocs/topic/ccps-115/ccps-services-deploy.html. 2. Logon to CPSM portal as a service provider administrator to update the web service call configurations. 3. Navigate to Configuration > System Manager > Servers. Click Refresh Server List, if the cluster or VIP name does not exist on the list, click Add a Server link, enter the VIP name and click Add Server. 4. If you are configuring for a remote location domain and the DNS of the server names may not be resolvable in the primary location domain, expand the server name you have just added, enter the IP address in Alias field, and Save. 5. Click Server Roles link on the left or got to Configuration > System Manager > Server Roles, expand newly created VIP Server placeholder, tick the appropriate role under Server Connection Components and click Save. 6. Click Server Connections link or go to Configuration > System Manager > Server Connections, expand the existing entry for the web server or connection role to be

updated, select the newly created VIP Server name from the Server dropdown list, and click Save. 7. Test the connection via the icon on the right. It should go green if valid and there no firewall blocking issues. Deploy CPSM with App Orchestration HAad Service in multidatacentre In above sections, we have described how to setup SQL AlwaysOn availability group for CPSM, add Provision Engine and Directory Web service in the cluster, configure load balance for Web Portal and API and HAad Service for App Orchestration, all these deployment can reside on the same datacentre to provide high availability, duplicated environment can be deployed on the remote/secondary datacentre if disaster happen on one datacentre, another datacentre can function well to provide the services also To deploy CPSM in the multi-datacentre 1. Database accessibility is guaranteed by the SQL AlwaysOn availability group, SQL replicas are deployed in main and secondary datacentre, and join the same availability group, all the applications connect to the same database by the same connection string, to make sure the data is synced timely, set the Availability Mode to Synchronous commit for each replicas on Availability Group 2. Deploy the Provision Engine servers and Directory Web Service in the same Failover Cluster on multi-datacenter, and set the server in main datacenter as Owner Node which will provide the service at normal time, if failure happen, it will switch to secondary datacentre s node automatically 3. Deploy Web Portal and API role on multi-datacentre, configure it into one load balance group, set the LB Method to Least Response Time to make sure it s main datacentre s server provide the service at normal time 4. If App Orchestration is deployed in the multi-datacentre and want to enable HAad service, configure AO configuration servers in the same load balance group, then configure it on CPSM, this article will not cover how to configure AO in multi-datacentre Multi-Datacentre diagram Main Datacenter Secondary Datacenter Load balancer Provision Engine AO Configuration Servers

In above diagram, we deployed 2 servers for each roles in the main Datacenter, it can provide high availability within the same datacenter, if all the servers are down in one datacenter, load balancer and Failover cluster can switch it to secondary datacentre also without customer s awareness