This document contains information for planning Secret Server architecture and resource allocation within your environment. Read through or use one of the following links to skip ahead to the relevant section. Secret Server Architecture Secret Server Sizing Distributed Engine Sizing Additional Background Secret Server System Requirements Below are the minimum software requirements for Secret Server. Requirement Versions supported Notes Microsoft operating system Windows Server 2012 R2 Windows Server 2012 Windows Server 2008 R2 Windows Server 2008 / 2008 SP2* Windows Server Standard or higher supported. Small Business Server (SBS) is not supported. The Essentials edition is not supported. The Core (GUI-less) role is only supported for Server 2012 and Server 2012 R2. *Secret Server version 8.8 will be the last version to support Windows Server 2008 / 2008 SP2. Microsoft SQL Server Microsoft Internet Information Services (IIS) SQL Server 2005 or greater (all versions up to 2014) IIS 7 IIS 8 IIS 8.5 Microsoft.NET Framework.NET Framework 4.5.2.NET Framework 4.5.1.NET Framework 4.6* Express edition or higher supported. This is part of the Windows Server operating system. Both 32-bit and 64-bit versions are supported. However, some features of Secret Server require 64-bit to operate. *.NET Framework 4.6 is supported in version 8.5 and above. Page 1
Secret Server Architecture The following scenarios outline commonly deployed implementations of Secret Server. For more details information about Disaster Recovery options and configuring High Availability, see the Disaster Recovery guide. Basic Configurations Single Site, Single Server Microsoft SQL Server Express, Secret Server Free, Professional Edition or higher Production Backup Application & Database Backups Web Application SQL Database Network / Local Fileshare This is an example of a basic Secret Server installation. Secret Server and Microsoft SQL Server Express are installed on a single Windows Server. Automatic or manual application and database backup procedures can be used to stored backups on another server or fileshare. If the Windows Server is virtualized, leveraging strategies such as making scheduled snapshots or having a hot/cold site may add additional layers of redundancy. Page 2
Single Site, Active-Passive Microsoft SQL Server Standard, Secret Server Professional Edition or higher Production Backup Application & Database Backups Web Server (passive) Primary End User Network / Local Fileshare Microsoft AlwaysOn or DB Mirroring (optional) Primary SQL Database SQL Database Full Server Backups or Clones (optional) In this example, there are more than one front-end web servers, but only one active node. A web server node being active means that end users will hit this site when browsing to Secret Server at any given time. This server handles background processes (such as Remote Password Changing and Heartbeat) as well, making it the Primary server in addition to the fact that it is active. The passive web server also points to the backend SQL database, but informs users browsing to it that they must use the active node to use Secret Server. In the event that the active node is unavailable, the Primary status can be transferred to the passive node, and users can resume using the application. There can be more than one passive server node (no limit), depending on the needs of the organization. Page 3
High Availability Configurations Single Site, Active-Active Microsoft SQL Server Standard or higher, Secret Server Enterprise Plus Edition Production Backup Web Server (passive) Primary End User Application & Database Backups User-managed Backup Option (e.g. server cloning) Network / Local Fileshare Primary SQL Database Microsoft AlwaysOn or DB Mirroring (optional) SQL Database Load Balancer Full Server Backups In this example, there are more than one front-end web servers, and more than one active node. Allowing users to use Secret Server through more than one active node simultaneously requires enabling clustering within the application. Only one server handles background processes (such as Remote Password Changing and Heartbeat), meaning that one of the active nodes will be designated as the Primary server at any given time (this can be changed manually, if necessary, in the application). In the event that the Primary active node becomes unavailable, the Primary status will be transferred to one of the other active nodes and users can continue using the application without interruption. There can be more than one active and passive server nodes (no limit), depending on the needs of the organization. Page 4
Multi-Site, Active-Active Microsoft SQL Server Enterprise, Secret Server Enterprise Plus Edition This is an example of an implementation that would be used by a larger enterprise. Not only are there are more than one front-end web servers, more than one active node, and SQL AlwaysOn in use there is a setup of this configuration at each of two sites. Users can browse to the application through one load balancer at the production site, and if anything happens on the production site that cannot be handled by the Active-Active setup at that location, administrators can have users directed to the load balancer at the second site and an active server node there will become Primary. Still, only one server handles background processes, but it s recommended to offload Remote Password Changing, Heartbeat, Discovery, etc. to Distributed Engines. This means that regardless of which server is Primary, Distributed Engines will be able to retrieve workload tasks from Secret Server and connect to managed endpoints per usual. The diagram for this configuration is on the next page. For a higher-resolution version of the diagram, go to: https://updates.thycotic.net/links.ashx?referencearchitecture Page 5
Production Datacenter On-premises Network Web Server (passive) Primary Distributed Engine Managed Endpoints End User AlwaysOn Availability Group Load Balancer Legend Active Connection Microsoft SQL AlwaysOn Inactive Connection User HTTPS traffic Disaster Recovery Site Cloud VPC AlwaysOn Availability Group Load Balancer Distributed Engine Managed Endpoints Web Server (passive) Page 6
Secret Server Sizing Determining the resources needed for the core Secret Server application involves evaluating how much usage the application will receive and which features plan to be leveraged. See High Availability for details about the number of servers required per configuration. Hardware Requirements For an installation of Secret Server that will handle core tasks such as Remote Password Changing and Heartbeat: Size of Installation # Secrets # Users Processor RAM Database server Disk space Web application server Small 1,000 1-10 Dual-core 1.6 GHz or higher 2 GB 500 MB Medium 1,000-10,000 10-100 Dual-core 2 GHz or higher 4 GB 1 GB + 10 MB per user, per year 500 MB Large 10,000+ 100+ Quad-core 2 GHz or higher 8 GB 2 GB To improve performance in larger environments where the Discovery and/or Session Recording features will be used heavily, it is also recommended to scale up resources. If you are running Discovery and/or Session Recording: Size of Installation # Secrets # Users Processor RAM Database server Disk space Web application server Heavy Use 10,000+ 100+ Quad-core 2 GHz or higher 16 GB 2 GB + 10 MB per user, per year 500 MB + 1TB Shared or Local Drive Page 7
Other Considerations Virtualized Environments Secret Server will operate in a virtualized environment as long as it is configured on the Windows operating system. Hypervisors Secret Server has been run on include: VMware, Hyper-V, VirtualBox, Xen, AWS and Azure cloud environments. Server Type Do NOT install Secret Server on a domain controller. This is a Microsoft limitation. ASP.NET does not operate reliably when installed on a domain controller. Do NOT install Secret Server on a server running SharePoint. Application Security For maximum security, the application should be installed on a dedicated system. Secret Server can be run on the same machine as other applications (Secret Server will require sufficient RAM and CPU to operate normally), but these should at least be applications of the same level of security/sensitivity so access to these systems can be restricted. While all sensitive data in Secret Server is either securely hashed or encrypted, it is a security best practice to limit any opportunities for foul play. Additional Disk Space If you intend to use Session Recording, additional disk space will be needed for the database to store the recorded videos. This is based off of frequency of usage, the applications recorded, and codec configured with Secret Server. See Configuring Session Recording (KB) for more information. Test Results The following example is based off of a test instance with 120,000 Secrets. The database was 1.6 GB. The machine was a Windows 7 instance Intel i7 2.67 GHz processor 6GB of RAM Page 8
Searching performance is primarily driven by the number of Secrets a user has access to, so in the above example a user searching all folders with View access to all Secrets could see search times of 4-6 seconds. Meanwhile, a user with access to 6,000 Secrets in the same instance will see search queries return in 1-2 seconds. Search times can vary based on data; the fastest search will be for a distinct value in a smaller set of Secrets, the longer searches will be for a generic value in a larger set of Secrets. For example, the search for a Secret named "PRODSRV03\LocalAdmin" will return much faster if the search value is "PRODSRV03" and done in a specific folder than if the search is "Admin" and done at the root level. To increase performance, consider putting the database and web application on separate servers, which will reduce resource contention. Another option is to set up front-end clustering behind a load balancer using the Enterprise Plus edition, which will help scale out the work done on the web server. Distributed Engine Sizing Distributed Engine is the replacement for Secret Server Agent, designed for scalability for the enterprise. Released in version 8.9.000000, Distributed Engine supports Heartbeat, Password Changing, and Discovery. The Engine is a Windows Service which does the actual work of discovery, password changing, and heartbeat. Each Engine belongs to a Site. The Site can be thought of as a bucket of work items for a particular network area. Each Engine can only be assigned to a single Site but each Site can have multiple Engines assigned to it, significantly increasing throughput. For IT admins, phrases like designed for scalability or significantly increasing throughput are exciting and usually lead to more questions. Such as, At what point would adding more engines increase throughput? or How many endpoints make a customer small, medium or massive?, just to name a few. Continue reading for the findings of our internal testing to learn how to properly size Engines used in various enterprise environments. For more information on Sites and Engines see the Distributed Engine Guide. Licensing Licenses for Distributed Engine come in two forms: site licenses and engines per site licenses. Site Licenses An installation of Thycotic Secret Server has 1 site which is typically the local network. This site is supported without any additional licenses. Additional sites for remote networks such as a DMZ, partner network or lab network will require additional site licenses. Talk to your Page 9
account manager for pricing. (Existing customers prior to the 8.9.000000 are grandfathered into unlimited sites to replace their existing agents.) Engines Per Site Licenses An installation of Thycotic Secret Server allows one engine to be used per site. Customers should first test their performance with a single engine as this is already fast and may meet requirements. For large environments that need increased throughput, additional engines can be added to a site. This requires additional licenses. When an engine per site license is added to Thycotic Secret Server then that number of engines can be used at all sites. Talk to your account manager for pricing. Hardware Requirements Distributed Engine, MemoryMQ, and RabbitMQ require a Dual-core 2GHz or higher and 4 GB of RAM. This is regardless of installation size. An Engine, Site Connector, and the Secret Server website can all operate on the same machine, but we recommend a separate server for each. Sizing A summary of recommended engines per environment are below: Size of Enterprise # Endpoints # Engines Reason Small 2,000 1 Medium 25,000 2 High availability Large 50,000 2 High availability, Performance Massive 100,000 3 High availability, Performance Below is the estimate of how long it would take to run a Discovery scan for a number of endpoints, and how many engines should be added to reach the desired scan time: Page 10
Size of Enterprise # Endpoints Time to discover with 1 engine (rounded to nearest 8 hours) Desired Time Number of additional engines needed (speed/rate improvement) Small 2,000 < 1 hour Less than 1 hour N/A Medium 25,000 8 hours Less than 4 hours With 1 additional Engine, about 4.5 hours. With 2 additional Engines, about 3.5 hours. Large 50,000 16 hours Less than 1 day N/A Massive 100,000 32 hours Less than 1 day With 1 additional Engine, about 18 hours. With 2 additional Engines, about 14 hours. Test Results Thycotic ran a local account scan on our lab enterprise environment (see Table 2 below) using Remote Password Changing. In an enterprise environment, there will be numerous non-discovery activities underway at the same time - password changing, Heartbeat, etc. # Endpoints # Engines Time to discover # Computers/second Description of environment 859 1 16 min 9 secs 859 2 8 min 46 secs 0.89 1038 accounts discovered Local Account Windows discovery only, using RPC 1.5 Same 859 3 7 min 2.04 Same Page 11
Doing a local Windows account Discovery scan, with one Engine, in an Enterprise environment: CPU usage < 5% on average, spiked briefly up to 25% or so (of a 2.4 Ghz 2-core virtual CPU) Memory usage < 200 MB Network usage maxed out at 150k/sec, usually far less. High Availability Customers requiring high availability for Distributed Engine need to deploy at least two engines per site to each site requiring high availability. This will require Engines Per Site licenses. If one engine fails, the other engine will continue working. This has the added benefit of increased throughput, because both engines will be performing work during normal operation. Monitoring of engines can be done using standard infrastructure monitoring solutions - the engines are just Windows Services and are easily monitored. Events such as Engine down and All engines down for Site are on the product roadmap and will be incorporated into Thycotic Secret Server in 2016. Additional Background Many factors come into play when dealing with speed, including network latency, processing power, the number of computers that are bad entries in AD (in the case of Distributed Engine), and more. The metrics reported may vary widely from environment to environment. The recommendations in this document are a best estimate for achieving good performance while consuming least resources possible. Page 12