Secret Server SQL Server Replication

Transcription

1 Table of Contents SQL Server Replication with Thycotic Secret Server... 2 Overview... 2 SQL Server Replication... 2 Benefits of Replication... 2 Secret Server and SQL Replication... 4 Secret Server Web Cluster with SQL Server Replication... 5 Leveraging a subscriber when the publisher is offline... 6 Secret Server Engine... 7 Secret Server Replication Settings... 7 Publications... 7 Tracking level and Resolvers... 8 Conflict Auditing... 9 Operational Latency... 9 Article Settings... 9 Compensate for Errors Identity Range Variations Installing and Configuring SQL Server Replication Installation Troubleshooting the Install Removing SQL Server Replication On each subscriber: On the publisher: Managing SQL Server Replication Web Server Nodes Upgrading Secret Server Other Information about SQL Server Replication Last updated: June 28, 2016 Page 1

2 SQL Server Replication with Thycotic Secret Server OVERVIEW SQL Server Replication is a set of technologies for copying and distributing data and database objects from one database to another and then synchronizing between databases to maintain consistency. SQL SERVER REPLICATION Benefits of Replication Enabling SQL Server Replication allows a database and application to be hosted closer together and this allows for the mitigation of network latency and outages. Decrease application server to database network latency Resolve issues with unreliable network connectivity Allows for distribution of workload in a scale out fashion Works across large distances In a typical web-clustered version of Secret Server, all application servers would access a centralized database. In the event of a network outage, any users on affected application servers would not be able to use Secret Server until network access was restored. In addition, poorly performing networks can introduce latency that may decrease the responsiveness of Secret Server. This technology enables additional options when designing the network topology behind Secret Server that can help alleviate these issues. Page 2

3 High Availability SQL Server Replication should not be considered an option for high availability, but it can be coupled with other technologies like SQL Server AlwaysOn Availability Groups to provide high availability. Any architecture should be reviewed and designed with your database group. Architecture The SQL Server Replication technologies do all the work of ensuring data consistency between each database, and Secret Server is designed to work well with this technology. When SQL Server Replication is enabled on a specified database several system tables, views, stored procedures, and SQL Server jobs are added to the database schema. These tables store information about the data replication, and the procedures contain most of the code needed to perform the synchronization between databases. Data Synchronization This is the process through which SQL Server Replication integrates changes from each database node. These changes include data changes as well as schema changes. Each subscribing database node has a synchronization interval that defines when it will synchronize any changes between the main publication node and itself. Secret Server has been tested using pull based subscriptions where a scheduled job on each SQL Server subscriber runs at a specified interval to trigger the synchronization. Data Conflicts Enabling SQL Server Replication introduces the possibility of data conflicts occurring in the Secret Server environment. This can happen when two people in different regions attempt to update the same set of data. Due to the disconnected nature of the technology, the system is unaware of this conflict until a data synchronization occurs. During synchronization, SQL Server attempts to resolve any conflicts based on a defined set of parameters. Secret Server provides a setup script to help define optimal parameters for each article (table, view, stored procedure) in the Secret Server database. SQL Server Replication Monitor and Conflict Viewer SQL Server provides tools within SQL Server Management Studio that allow viewing of synchronization status, data conflicts, and publications. In the event that a data conflict is not automatically resolved, the Conflict Viewer tool can be used to resolve the conflict and pick which set of data should be kept. It is possible that manual intervention could be required to resolve the conflict in the base tables on one of the nodes; otherwise, the conflict may still occur after the next synchronization. For more information: Replication Monitor Conflict Viewer and Interactive Resolver Page 3

4 Tracking level SQL Server Replication tracks changes to each node by either row or column. Row Level Tracking Column Level Tracking Conflicts can occur when any change to any column is made on the same row even if it is not the same column. Conflicts only occur when the same columns are updated on a row. For more information: Row Level / Column Level Tracking Conflict Resolvers SQL Server Replication uses resolvers to determine the outcome of data conflicts between two database nodes. For example, the same row or column was updated on different nodes. These can be as simple as the primary node always wins, last change by date wins, and many others. For more information: Advanced Merge Replication Conflict Detection and Resolution Specify the Conflict Tracking and Resolution Level for Merge Articles Secret Server and SQL Replication There are many architectures for how SQL Replication can be setup. Determining the correct configuration requires proper planning with a good understanding of how SQL Replication works and the intended goals of leveraging this technology. Here are a few examples. Page 4

5 Secret Server Web Cluster with SQL Server Replication Some key points about designing the right architecture: Multiple web application servers can connect to the same database server. Secret Server has the concept of a Primary Web Node. This server runs the background processes necessary for Secret Server to operate properly and perform tasks such as heartbeat and password changing. The Primary node will always connect to a Publisher database. If the Primary application server is unavailable, then another application server will elect This is just one example and itself as primary. This application server must be connected each implementation will to the publisher database and if none are available then vary based on the specific certain activities such as Secret Heartbeat and Remote needs of each organization. Password Changing will be offline until such an application server becomes available. Secret Server Engines are an effective means to distribute workload to different networks or sites. Each engine must call back to a web application node that connects to a publisher database. This diagram shows the Publisher in an AlwaysOn Availability Group, but this is just an example. Depending on the needs of the organization, AlwaysOn could run on both the Publisher and the Subscribers or on neither. Many other High Availability options could be leveraged alongside SQL Replication. Page 5

6 Leveraging a subscriber when the publisher is offline It is important to note that there is a delay in data synchronization between database nodes. This is based on how frequently SQL Replication is configured to synchronize. One way to mitigate this data latency would be to have every user access a node connected to the publisher. This means that every user would be accessing the same database. In the event of a network outage, a load balancer could be configured to fail over to the subscriber nodes. The data on the subscriber would be as up to date as the last synchronization and when network connectivity is restored all data activity will be synchronized again. Obviously, if the issues of network latency or disconnects occur too frequently this may not be a workable solution, but is given as another example architecture. Page 6

7 Secret Server Engine Secret Server Engines can be distributed to different network segments, but must still be able to call back to a Secret Server Web Application Server that is connected to a Publisher database node. This helps to ensure that Heartbeats and Password Changes occur with the most up to date dataset. Secret Server Replication Settings Keep in mind that this is a disconnected technology so there are certain decisions that will need to be made when setting it up. How often should synchronization occur? Should any resolvers override the provided defaults? Are there any operational considerations based on the type of Secret data? Publications We recommend setting up two publications for Secret Server: 1. Events These are things that need to occur on a timely schedule, such as updating Secrets. The Events publication should be set to synchronize every minute. This means that if a Secret were created in one region it would not appear in another region for one minute. By default, this publication is called SSPubEvents. 2. Logs Information from each audit log is available on the server on which it occurs, but we do not recommend setting log synchronization to less than hour. If this happens too frequently then the database could end up creating deadlocks by constantly updating large sets of data. Page 7

8 Once the logs merge the events will appear in the order in which they occur and show on which server they took place. By default, this publication is called SSPubLogs. Tracking level and Resolvers Using the default Secret Server implementation, most conflicts are resolved by taking the change made on the publication server as the winner (using the Publisher Wins resolver). It is assumed that the publication server is most likely the main server in an environment and therefore most likely the decider in a case of a conflict. With this in mind it is recommended to do functions such as configuration changes or Secret template definition changes on the publication node. It is also worth noting that this is really only an issue if two people update the same data on two different servers before a synchronization occurs. There are some exceptions to the publisher always wins rule: Data Element Resolver / Tracking Level Example Secret Fields Last Update Wins If two people change a password on a Secret to a new password then the last person to make the change will win. Secret Access Request Approval Last Update Wins If two people are in a group that is requested to approve access then the last person to approve / not approve access would set that approval. File Attachment Last Update Wins tbconfiguration Column Level Configuration changes are and must be resolved by table column, not the default is row level where the entire row is resolved. This allows different configuration options to be set on different servers and not conflict with each other, but changing the same option would create a conflict. Always remember that the conflict is only resolved when synchronization occurs. This means that if there is a synchronization window of 24 hours there could be very different data on different servers for that entire 24-hour period. The risk of data conflicts rises the larger the synchronization window. The default recommended synchronization is every minute for most items and every hour for system logs. For a complete listing of recommended settings navigate to the SQL Replication Administration page in an installed version of Secret Server. This list can be accessed from ADMIN > Nodes > SQL Server Replication > Show Articles. Page 8

9 Conflict Auditing When conflicts occur during synchronization Secret Server will write an event to the system log that it has detected these conflicts. Depending on the type of conflict more specific information may be written to the audit log for a specific data entity like a Secret or Folder. Notification of the conflict will be written accordingly to the system log, Secret audit, Folder audit, and other logs. To view complete details of the conflict the SQL Conflict Viewer tool needs to be used to review the conflict by right clicking on the publication and choosing to View Conflicts. As an example, if the same field was changed on two different servers, resulting in a conflict, then the Secret Server audit log will indicate the field was changed twice and then denote that there was a data conflict. Operational Latency Consideration should be given to how Secret Server data is setup and managed. This is most easily described by an example where a manager uses a Secret Server node in Europe and their employee works on a node in Australia. Due to network latency, the organization has changed the default event log synchronization interval from every minute to every five minutes. The employee requests access to a Secret in Australia, but due to the synchronization window, the manager is not alerted until five minutes later since the manager is using the server in Europe. This issue can be mitigated by having operational groups work on data for their region or decreasing the synchronization window. Options to manage data latency: Distribution of duties Only update Secret templates on the main node. Distribution of data Only edit Secret data by region. For example, create a folder for Europe and one for Australia to segment the data. Only edit items in each folder when in the appropriate region. These should only be considered if data conflicts occur frequently. Article Settings When setting up the SQL Replication Publication certain articles need settings other than the default settings. The recommended settings can be found within Secret Server by accessing the SQL Server Replication page located at Admin > Nodes > SQL Server Replication > Show Articles. This page can also be used to generate a setup script for both the Publication and Subscribers that uses these default settings. A Distributor will need to be created before running these scripts. For more information: Configuring Distribution Page 9

10 Compensate for Errors When conflicts occur, an article that has compensate_for_errors set to true will automatically try to resolve the conflict. When false, a SQL Server administrator will need to use the SQL Server Conflict Viewer to review and resolve conflicts. Identity Range SQL Server Replication manages table identities by assigning ranges to the publisher and subscribers. Certain tables (logging / auditing) require larger identity ranges to be assigned. New ranges are only assigned when a data synchronization occurs. For more information: Replicate Identity Columns Variations How SQL Server Replication is setup can vary greatly and there may be reasons to not use the standard setup. Consult with your database group for approaches that may work well in your environment. The architecture diagrams contained within this document are just a few high level examples. Installing and Configuring SQL Server Replication INSTALLATION There are a multitude of options on how SQL Server Replication can be configured. At a high level, these are the steps to setup Secret Server in a SQL Server Replicated environment. 1. Install Secret Server 2. Enable Secret Server Clustering. For more information, see: Setting up Clustering Clustering Secret Server with Load Balancer Clustering FAQ Setup a SQL Server Replication Distributor a. This can run on the same database as the publication database or this can run on a separate dedicated distribution server. Page 10

11 b. This can be done from the SQL Server Management Studio by right clicking on the Replication node. c. Review what settings are appropriate for the distributor setup with your on-premise database group or with a database consultant. 4. Download the SQL Publication Script a. In Secret Server select ADMIN > Nodes > SQL Server Replication > Get SQL Publication Script b. Review this script and update the variables according to your environment c. Advanced users can use the article list on this page to configure SQL Replication differently than this script to suit your environment. 5. Run this script on the Secret Server Database. A DBA with administrative privileges will be needed to run this. Please consult with your on-premise database group or review your configuration with a database consultant. 6. Create snapshots for each publication a. Open SQL Server Management Studio b. Right click on each publication under Replication > Local Publications c. Select View Snapshot Agent Status d. Click Start 7. Download the SQL Subscriber Script a. In Secret Server select ADMIN > Nodes > SQL Server Replication > Get SQL Subscriber Script b. Review this script and update the variables to match your environment 8. Create a new database on the database server you intend to be your subscriber. You will need to make sure the script uses this database and machine name. Set up permissions for the user or network account that Secret Server will use to connect to this database. 9. Run the subscriber script on the Publication database first and then on the subscriber database. If your variables are set properly it will execute the appropriate part of the script. 10. Expand the SQL Server Jobs on the subscriber and you should see two jobs named for each Publication. Right click these jobs to start them. After they complete, your subscriber database should have replicated the schema objects from the publication. 11. Switch Secret Server nodes over to subscriber databases. The primary node must remain on the Publisher database, as must any node that an engine calls back to but all other nodes can be reconfigured to use subscriber databases. Which nodes to switch will depend on your specific needs as described in previous sections. To switch an existing node to a subscriber database, log into that node and go the DbConnectionReset.aspx page by entering that page name in the URL field of your browser (http[s]://[yoursecretservernamehere]/dbconnectionreset.aspx). Step through the wizard, entering the name of the new server and database when prompted. After completing this step, recycle the node s application pool. Page 11

12 TROUBLESHOOTING THE INSTALL Replication Setup Scripts Fail Make sure that the SQL Server Replication feature has been enabled before running the script. Check the error messages in the script output. Make sure you set all of the variables in the top of the script correctly (publisher server and database names, subscriber server and database names, etc.). SQL Replication Job Fails To see the error message it is easiest to right click on the job and choose to view history. This error can help to indicate the actual problem. REMOVING SQL SERVER REPLICATION Certain operations such as upgrading Secret Server, adding or removing DoubleLocks from secrets, and enabling or disabling HSM cannot be performed while SQL Server Replication is enabled. In order to perform these operations you must remove SQL Server Replication. When you are done with the action that required you to remove replication you can install and configure it again by repeating the previous instructions. You only need to remove replication from the Publisher and all Subscribers. The Replication Distributer does not need to be removed. These are the steps to remove SQL Server Replication from your publisher and subscriber databases. On each subscriber: 1. Stop the websites of all nodes using the subscriber database (TIP: You can see what database each node is using from ADMIN > Server Nodes). 2. In SQL Server Management Studio go to Replication, right-click Local Subscriptions and choose Generate Scripts... a. In the Generate SQL Script dialog check "Subscriptions in the following data sources" and select the Secret Server subscription database(s). b. Select "To drop or disable the components." c. Click Generate Script > Open in new query window. d. Click Close to close the dialog once the script is created. e. The script contains sections to be run on the subscriber and the publisher. Run the portions commented to be run on the subscriber on the subscriber. f. Copy the script to a query window on the publisher server. Run the portions commented to be run on the publisher. Perform any maintenance actions needed on the subscriber database and nodes. On the publisher: 1. Stop the websites of all nodes using the publisher database. Page 12

13 2. On the publisher, go to Replication, right-click Local Publications and choose Generate Scripts... a. In the Generate SQL Script dialog check "Publications in the following data sources" and select the Secret Server database. b. Uncheck "Distributor Properties." c. Select "To drop or disable the components." d. Click Generate Script > Open in new query window. e. Click Close to close the dialog once the script is created. f. Run the script. Perform any maintenance actions needed on the subscriber database and nodes. After all maintenance tasks are done restore replication as described in the previous section, Installing and Configuring SQL Server Replication. Managing SQL Server Replication Once replication is setup and working certain considerations should be given to managing it along with conflicts that can occur. The recommended settings for the publication have been tested to limit conflicts, but they can still occur. Here are some scenarios you might encounter: Conflict resolved automatically Conflict unable to be automatically resolved Some data stops replicating SQL Replication Synchronization Times SQL Server was able to determine how to resolve the conflict. Secret Server will log / audit this conflict. To see the specific details for the conflict use the SQL Server Conflict Viewer in SQL Server Management Studio. A user with SQL Server access will need to open the conflict in SQL Server Management Studio. Access these by right clicking on the publication and choosing to view conflicts. A table could become blocked if there are conflicts so other data may continue to synchronize, but one table may block until the conflict is reviewed. If data in one region / database node is different than another after a synchronization there could be conflicts that need to be reviewed and resolved. The status of each publication and subscribing server along with the last time of synchronization can be located in Secret Server by selecting ADMIN > Nodes > SQL Server Replication. WEB SERVER NODES The Web Server Nodes page now includes a column that lists the database server name and database name. This column also indicates whether the database is the publisher or a subscriber. Page 13

14 If you click the SQL Server Replication button, you can see more information about your SQL Server Replication. This page pulls from the replication data SQL Server manages and lists each subscription, the last time that subscription has synchronized, and the status of that synchronization. The same information is available within SQL Server Management Studio, but this page gathers all of the subscription information together in one place. As mentioned earlier, the Get SQL Publication Script and Get SQL Subscriber Script buttons will download replication script templates that you can use to set up replication for Secret Server. You can fill in the variables at the top of each script to match your environment and run them as-is or modify them further if you need to customize the default scripts. The Show Articles button will list each article in Secret Server that should be included in replication along with the recommended settings for SQL Server Replication. These are recommended settings and do not show the current state of replication on the publisher. UPGRADING SECRET SERVER New versions of Secret Server may issue schema changes including indexes, column changes, and views. In some cases, SQL Merge Replication will not automatically replicate these schema changes. For this reason, we recommend removing any publications and subscriptions targeting the Secret Server Page 14

15 database, redirecting users to the web server at the primary site before performing any upgrade, then recreating the publication and subscriptions from new versions of the replication scripts. In the following scenario, there are Secret Server web servers installed at a site in Australia, the US, and the UK. The UK is the Publisher node and Australia and US nodes are the Subscribers. 1. Redirect all users to the UK Secret Server URL. 2. Stop IIS at the Australia and US sites. 3. Manually force a synchronization between the Australia subscriber and the UK: a. Open SQL Server Management Studio and connect to the Australia subscriber database server. b. Go to Replication > Local Subscriptions. c. Right-click on one of the Secret Server subscriptions (if you used the scripts provided by Secret Server they are called SSPubLogs and SSPubEvents) and select View Synchronization Status. d. Click the Start button to force a synchronization. e. Repeat steps (c) and (d) for all Secret Server subscriptions. 4. Repeat step (3) for the US subscriber database. 5. Resolve any replication conflicts. a. Open SQL Server Management Studio and connect to the UK publisher database server. b. Go to Replication > Local Publications. c. Right-click on one of the Secret Server publications (if you used the scripts provided by Secret Server they are called SSPubLogs and SSPubEvents) and select View Conflicts. d. Examine and resolve any unresolved conflicts. 6. Generate the script to remove replication on the subscriber databases as specified above and run it to remove replication. 7. Generate the script to remove replication on the publisher database as specified above and run it to remove replication. 8. Restart IIS on the Secret Server web server in the UK. 9. Run the Secret Server web upgrade wizard. 10. Copy the website application directory to the web servers in Australia and the US. 11. Use the Scripts generated from the Secret Server UI to recreate replication on the publisher. 12. Push a snapshot to the Australia and US subscriber databases. 13. Recreate replication at the subscribers. 14. Restart the jobs on the subscriber databases as described above. 15. Start IIS on the Australia and US web servers. 16. Change redirection rules for Australia and US users so they access the local web server as normal. Page 15

16 Other Information about SQL Server Replication Upgrading Secret Server with SQL Replication SQL Server Replication (MS Books Online) Snapshot Replication Page 16