RDS Migration Tool Customer FAQ Updated 7/23/2015

Transcription

1 RDS Migration Tool Customer FAQ Updated 7/23/2015 Amazon Web Services is now offering the Amazon RDS Migration Tool a powerful utility for migrating data with minimal downtime from on-premise and EC2-based databases to RDS, Amazon Redshift and Amazon Aurora databases. The RDS Migration Tool supports not only like-to-like migrations, e.g. Oracle-to-Oracle, but also migrations between different database platforms, e.g. SQL Server-to-MySQL. It can capture changes on the source during the data migration and apply these changes to the target to simplify the switch-over to the new database environment. To start using the RDS Migration Tool, you will be granted access to the tool s Amazon Machine Image (AMI). Create an EC2 migration instance using this AMI as the template. This migration instance will serve as the central point that connects your source and target databases, manages the migration process, as well as moves the data. We recommend that you create your target database (whether in RDS or on EC2) in a Virtual Private Cloud (VPC) as this will provide better protection for your data. We also recommend that you launch the migration instance in the same VPC. By default, EC2 instances that you launch into a VPC cannot communicate with your on-premise network. You can enable access to your on-premise network from your VPC by configuring a Virtual Private Network (VPN) connection to your VPC or by using DirectConnect, as described here. General Information about Signup and Support: 1. How do I sign up for the RDS Migration Tool? Please review the RDS Migration Tool Customer FAQ (this document) and decide if the tool is right for you. Get in touch with your Amazon contact (typically an account manager, a solutions architect, or a business development manager) expressing your interest. 2. What are the next steps after I sign up for the RDS Migration Tool? Amazon will provide the RDS Migration Tool to you as an Amazon Machine Image (AMI). We will notify you by that your account was granted AMI access. When you get this , use the AWS Management Console to locate the AMI we enabled for you. a. Login to the AWS Management Console b. Click on EC2 in the top menu c. Select "Launch Instance" d. Click on the "My AMIs" tab e. Under "Ownership", select the "Shared with me" check box f. Press the "Select" button next to the RDS Migration Tool AMI g. After launching the EC2 instance follow the Quick Start Guide and the User Guide located on the Desktop of your new instance Review the screenshot below for the location of the AMI. 1

2 3. What documentation is available for RDS Migration Tool users? You will have access to the RDS Migration Tool Customer FAQ (this document). Additionally, review the following documentation: Quick Start Guide - Recommendations for the first steps after launching the migration instance. Available on the AMI and also for download. User Guide - Product technical documentation available on the AMI and also for download. Demo Video Video recording of the RDS Migration Tool setup process available for download. 4. How do I connect to the RDS Migration Console after I launch the AMI? The RDS Migration Console is the graphical user interface of the RDS Migration Tool. If you would like to connect to the RDS Migration Console locally on the EC2 migration instance, find the "Amazon RDS Migration Console" program shortcut under Start Menu > RDS Migration Tool folder or type in the browser. If you want to connect to the RDS Migration Console from your workstation, do the following: a. Add a rule to the security group you used with your EC2 migration instance to enable HTTPS (port 443) access b. Navigate to DNS of the EC2 migration instance>/amazonrdsmigrationconsole Enter Administrator as the username and use the Windows password of your EC2 migration instance. When you connect, you will get a warning message pointing to a self-signed certificate. Ignore this warning. Data between your browser and the RDS Migration Console will still be encrypted. 5. What information do you need to sign me up? Please provide the following information to your Amazon contact: - your name and - the AWS account number you will be using for launching the migration instance - your AWS forum user name (optional). There is a private forum available where you can discuss the product and query the AWS experts Please take note of the RDS Migration Tool Additional Terms and Conditions. You may not use the tool unless you agree to the terms and conditions. 6. How do I submit feedback or ask for help? Questions and support requests can be logged via the Database Migration Forum at If you do not have access to the forum, please speak to your Amazon contact such as a Technical Account Manager or Solutions Architect. 7. Will I be billed for the usage of the RDS Migration Tool? The use of the RDS Migration Tool software is free, however, the tool requires the use of Amazon EC2, EBS and other AWS services, and customers will be charged normal AWS fees for the migration instances they create based on the AMI. 8. What are the major features and benefits of the RDS Migration Tool? The RDS Migration Tool provides the following features and benefits: Support for transactional change data capture (CDC) and application, with low performance impact on the source Support for heterogeneous migration (e.g. Oracle-to-MySQL) Support for homogeneous migration (e.g. Oracle-to-Oracle) 2

3 Data transfer optimizations for migrating entire database tables Light-weight column mapping & transformations Ability to select individual tables and columns and filter data rows for migration Reliable delivery and recovery Intuitive user experience that simplifies the steps required to migrate to AWS Monitoring and control functions with dashboard, metrics and alerts No need to deploy agents on the source to capture changes Scheduling of migration tasks 9. Which data sources and targets does the RDS Migration Tool support? We currently support the following database sources: Oracle, RDS Oracle SQL Server, RDS SQL Server (RDS SQL Server is supported in bulk extract mode; the CDC mode is not yet supported) MySQL, RDS MySQL, Amazon Aurora (using the same access mechanisms as RDS MySQL) PostgreSQL, RDS PostgreSQL (RDS PostgreSQL is supported in bulk extract mode; the CDC mode is not yet supported) ODBC We support the following database targets: Oracle, RDS Oracle SQL Server, RDS SQL Server MySQL, RDS MySQL, Amazon Aurora (being MySQL compatible) PostgreSQL, RDS PostgreSQL Amazon Redshift Note: Databases without the RDS or Amazon prefix represent on-premise and EC2 installations. For more detailed info on supported source and target database versions please see Appendix A. 10. Can I use the RDS Migration Tool for migrating data from on-premise databases to on-premise databases? No. The use of the RDS Migration Tool for migrating data between on-premise databases is not permitted. Please review the Additional Terms and Conditions for using the RDS Migration Tool. You may not use the RDS Migration Tool unless you agree to these terms. 11. How will you release updates to the RDS Migration Tool? We will release updates to the tool by creating a new version of the RDS Migration Tool AMI. You will be notified by that a new version of the AMI is available. 3

4 RDS Migration Tool Quick Start Guide Creating and setting up your EC2 migration instance 1. Start the EC2 instance launch wizard and create a new migration Instance using the RDS Migration Tool AMI. Instructions for finding the AMI are provided in the top section of the Customer FAQ. 2. Pick an appropriate size for your EC2 migration instance Please review the performance considerations in this FAQ for sizing your migration instance. We recommend at least an XLarge (M3 or R3) instance for most migrations, and 4XLarge and bigger instances for migration tasks with a high level of parallelism and volume of data. 3. Login to your EC2 migration instance using the Remote Desktop Connection client. 4. Install the drivers necessary for accessing your source and target databases. To connect to source or target databases the appropriate database client software packages need to be installed: Oracle - Oracle Instant Client. Go to and look for the package called Instant Client Package - Basic: All files required to run OCI, OCCI, and JDBC-OCI applications, the 1 st package in the list. Review additional installation instructions located here. MySQLand Aurora - MySQL ODBC Client. Go to select the 64-bit Windows Installer, and choose the the "Typical" installation type SQL Server - SQL Server Native Client. Go to expand the Install Instructions node and look for the Microsoft SQL Server 2012 Native Client x64 package Redshift - Amazon Redshift ODBC Driver. Go to and download the 64-bit installer AmazonRedshiftODBC64.msi. Review additional installation instructions located here. PostgreSQL - PostgreSQL ODBC Client and PGAdmin. Download the PostgreSQL ODBC driver from Review additional installation instructions for PGAdmin located here. After the installer finishes successfully stop and start the RDS Migration Services. To stop the RDS Migration Services (including the RDS Migration Server, RDS Migration Console Server, and RDS Migration Transfer Service), click on the program shortcut Stop RDS Migration Server located on both the Desktop and the Start Menu. Then click on the program shortcut Start RDS Migration Server located on both the Desktop and the Start Menu. 5. Open the RDS Migration Console, the graphical UI of the RDS Migration Tool To connect to the RDS Migration Console locally on the EC2 migration instance, find the "Amazon RDS Migration Console" program shortcut in the Start Menu > RDS Migration Tool folder or type in the browser. Your EC2 migration instance is now set up and you can now create new migration tasks in the RDS Migration Console. 4

5 Data Migration FAQ 1. What is the basic flow of migrating a database using the RDS Migration Tool? During a typical database migration you will create a target database, migrate the database schema, setup the data migration process, initiate the full load and a subsequent change data capture and apply, and conclude with a swithcover of your production environment to the new database once the target database caught up with the source database. 2. How can I create a target database instance? If you are migrating databases to RDS, use the RDS Management Console to launch a new instance. The RDS Migration Tool does not yet provide the capability to create new database instances. If you are migrating to Redshift, use the Redshift Management Console. 3. How do I use the RDS Migration Tool to migrate my database schema? To quickly migrate a database to your target instance you can rely on the Basic Schema Copy feature of the the RDS Migration Tool. Basic Schema Copy is a built-in feature and will automatically create tables and primary keys in the target instance if the target does not contain similarly-named tables of the source. Basic Schema Copy is great for doing a test migration, or when you are migrating databases heterogeneously e.g. Oracle to MySQL or SQL Server to Oracle. Basic Schema Copy will not migrate secondary indexes, foreign keys or stored procedures. When you need to use a more customizable schema migration process (e.g. when you move your production database and need to move your stored procedures and secondary database objects), review the recommendations in the Schema Migration section. 4. How do I use the RDS Migration Tool to migrate my data? Open the RDS Migration Console in your browser. First, setup the source and target databases. Then, create a new migration task and reference the newly defined source and target databases. You can fine-tune the migration task by choosing the tables you would like to migrate. You can also define data transformations between source tables and target tables. Start the migration task and monitor its progress. 5. How do I monitor my migration task and decide when to switch over my databases? In the RDS Migration Console look at the queue of tables and queue of changes. When the RDS Migration Tool indicates that the queue of changes is nearing the zero mark, prepare your production application servers to switch-over to the new target database. Disable connections to the source database to prevent new changes in the source. Wait until the change queue has been drained and switch your applications to refer to the new target database. Schema Migration 6. How do I move my entire database schema including secondary database objects and stored procedures to the target instance? The RDS Migration Tool provides the ability to create tables and primary keys in your target database (Basic Schema Copy). Use the Basic Schema Copy to quickly test the data migration functionality. To migrate the full schema of your production database, use the schema export tools native to the source engine: 5

6 For SQL Server use the SQL Server Management Studio's Import and Export Wizard. SQL Server Management Studio can be downloaded as part of the free Express Edition ( For Oracle use the SQL Developer Database Export tool or script the export using the dbms_metadata package. Oracle SQL Developer can be downloaded from For MySQL use MySQL Workbench' Migration Wizard. MySQL Workbench can be downloaded from For PostgreSQL, use the 32-bit PGAdmin tool which can be downloaded from Export just the schema; you do not have to export the data as the RDS Migration Tool will take care of that. 7. How do I minimize the duration and maximize the throughput of my data migration process? If you have a large data set (>500GB), you can optimize your data migration by dividing the schema script into two parts. Part 1 should contain the table definitions and primary keys and can be used for creating a target schema optimized for quick data load. Part 2 should contain all other schema objects, including secondary indexes, foreign keys, views, stored procedures, triggers etc. It can be applied at the end of data migration. Your should consider splitting your schema script into two parts even if you have a smaller data set. The Full Load operation is done on a table-by-table basis. Foreign Keys need to either be disabled during the Full Load or created after the Full Load in order to avoid errors. 8. Are schema changes captured and applied? The RDS Migration Tool is able to capture and apply changes to tables and columns, but not to other objects such as indexes, stored procedures, or triggers. A list of supported schema change operations can be reviewed in the User Guide chapter Amazon RDS Migration Tool Supported DDL Statements. If you would like to enable schema change capture and apply, review the section Change Processing in chapter Task Settings of the User Guide. However, we recommend to put a hold on schema changes while performing a migration. 9. Why are uniqueness constraints and secondary indexes not copied during the Basic Schema Copy? For data load efficiency Basic Schema Copy will only migrate your primary key indexes. Creating secondary indexes and additional constraints would slow down the full data load, however, primary keys are required in order for change capture and apply to be efficient once the full load is complete. Review the steps for migrating your database using the RDS Migration Tool described above. Performance Tuning 10. Which instance size should I choose for my EC2 migration instance? While the AMI launch wizard will allow you to select a t2.micro instance type, we recommend at least an XLarge (M3 or R3) instance for most migrations, and 4XLarge and bigger instances for migration tasks with a high level of parallelism and volume of data. Remember that you can start with an XLarge instance while you are setting up data sources and migration tasks, and scale it to a larger size just before the actual data migration. 6

7 Migrating to or from SQL Server 11. What is the connection string syntax in RDS Migration Tool for SQL Server sources and targets? Copy the endpoint specified for the instance in the RDS Management Console. It will be similar to: testdb.<instanceid>.us-west-2.rds.amazonaws.com:1433 Replace the colon : with a comma, and enter this as the connection string: testdb. <instanceid>.us-west-2.rds.amazonaws.com, Why are Identity columns converted to regular columns when I use Basic Schema Copy? Tables that include identity columns will need to be pre-created on a SQL Server target in order to preserve the identity attribute. Migrating to or from Oracle 13. What is the connection string syntax in RDS Migration Tool for Oracle sources and targets? Copy the endpoint specified for the instance in the RDS Management Console. It will be similar to: testdb. <instanceid>.us-west-2.rds.amazonaws.com:1521 Add the Database SID at the end of the connection string: testdb. <instanceid>.us-west-2.rds.amazonaws.com:1521/<sid> 14. What level of Supplemental Logging should be enabled in RDS Oracle to support CDC? The level of logging is dependent upon the design of the tables in question. Please refer to Configuring an Oracle Database as an RDS Migration Tool Source in Chapter 5 of the RDS Migration Tool User Guide. Once the appropriate level of logging is determined, configure the RDS instance as required following these instructions. 15. Do you have additional installation instructions for the Oracle Instance Client? After you downloaded the Instant Client from html, create a directory C:\Oracle on the EC2 migration instance, extract the instantclient-basicwindows.x zip archive and copy the directory instantclient_12_1 to C:\Oracle. From a Windows Explorer left plane, right click on "This PC" and select properties. In the System Window's left side click on "Advanced system settings". In the System Properties dialog click on the "Environment Variables" button at the bottom. Find the Path variable in the bottom window, select it and click on Edit. Add ";C:\Oracle\instantclient_12_1" to the end of the variable s value and click OK. Add a new system variable named TNS_ADMIN with a value of "C:\Oracle\instantclient_12_1". The Oracle Instant Client should now be installed. Migrating to or from PostgreSQL 16. Do you have additional installation instructions for PostgreSQL? 7

8 In addition to downloading the PostgreSQL ODBC driver, download PGAdmin (32-bit version) from Also download the Microsoft VC Redistributable required for PGAdmin (x86 Edition: vcredist_x86.exe) from Make sure you download the 32-bit versions of PGAdmin and VC Redistributable, otherwise you could see errors when trying to insert data into PostgreSQL targets. Extract the archives for the PostgreSQL ODBC Driver, PGAdmin, and the Microsoft VC++ redistributable. After extracting the files run the installers. From a Windows Explorer left plane, right click on "This PC" and select properties. In the System Window's left side click on "Advanced system settings". In the System Properties dialog click on the "Environment Variables" button at the bottom. Find the Path variable in the bottom window, select it and click on Edit. Add the following ";C:\Program Files\psqlODBC\0903\bin;C:\Program Files (x86)\pgadmin III\1.20" to the end of the variable s value and click OK. The PostgreSQL client should now be installed. Migrating to Redshift 17. Is there anything I need to pay attention to when migrating data to Redshift? When defining a Redshift target in the RDS Migration Tool, set the ODBC driver to Amazon Redshift (x64) in the Advanced tab of the Redshift database configuration window. The Redshift database, the S3 bucket used for uploading data into Redshift, and the Migration Tool EC2 Instance all need to be located in the same region. The User Guide and RDS Migration Console Online Documentation refer to the RDS Migration Server and the RDS Migration Transfer Service AMI (EC2 Instance) as two distinct machines. However, the RDS Migration Tool AMI contains both components preinstalled. When setting up a connection to Redshift you will be asked for a Transfer service IP Address. Use "localhost" for this field. You will also be required to enter a Transfer service Password. You will need to run the RDS Migration Transfer Service Configuration utility from the Desktop to either generate a random password or set your custom password. Other questions 18. What do I do if I receive Failed to load library when attempting to setup a source or target? Check first that your database connection information is correct. Then, try to re-install the database drivers as described in the Quick Start Guide above. 19. What security configuration should I use? The security for the RDS Migration Tool server can be configured just like any other EC2 instance. We suggests using a VPN or DirectConnect to enable a secure connection between your corporate network and the EC2 migration instance. Use VPC settings to configure Security Groups restricting inbound/outbound ports to those needed for database and console connectivity as well as the relevant IP addresses making connections. 20. How do I connect to the RDS Migration Console remotely from my workstation? If you want to connect to the RDS Migration Console from your workstation, do the following: 8

9 a. Add a rule to the security group you used with your EC2 instance to enable HTTPS (port 443) access b. Navigate to DNS of the EC2 migration instance>/amazonrdsmigrationconsole Enter Administrator as the username and use the Windows password of your EC2 migration instance. When you connect, you will get a warning message pointing to a self-signed certificate. Ignore this warning. Data between your browser and the RDS Migration Console will still be encrypted. 21. What do I do if I receive You are not authorized to use the Amazon RDS Migration Console when attempting to connect to the console? The first time the AMI is launched, the RDS Migration Services must be restarted for the security changes to take effect. 9

10 Appendix A Supported Database Versions Endpoint Oracle Source Database Database Versions Editions 10g, 11g or 12c Enterprise, Standard, Standard One Comments Oracle Target (EC2) Oracle Target (RDS) 10g, 11g or 12c 11g or 12c Enterprise, Standard, Standard One SQL Server Source 2005, 2008, 2008R2, 2012 and 2014 SQL Server Target (EC2) SQL Server Target (RDS) 2005, 2008, 2008R2, 2012 and R2, 2012 MySQL Source 5.5 and 5.6 MySQL Target 5.5 and 5.6 Amazon Aurora Target Amazon Redshift Target PostgreSQL Source 9.4 Enterprise, Standard, Workgroup, Developer Enterprise, Standard, Workgroup, Developer Workgroup, Developer editions are not supported as RDS targets Amazon Aurora is supported as a MySQL-compatible data source PostgreSQL Target 9.3 and up ODBC Source Based on ODBC

11 Appendix B Data Restrictions Certain types of data cannot be migrated using the RDS Migration Tool. These are documented below. Endpoint Oracle SQL Server Source MySQL Source Limitations BFILE, ROWID, REF, UROWID, Nested Table, User-defined data types CURSOR, SQL_VARIANT, TABLE, GEOGRAPHY, GEOMETRY, HIERARCHYID CURVE, GEOMETRY, GEOMETRYCOLLECTION, LINE, LINEARRING, LINESTRING, MULTICURVE, MULTILINESTRING, MULTIPOINT, MULTIPOLYGON, MULTISURFACE, POINT, POLYGON, SURFACE Please see the Limitations section for your engine in the User Guide for further details. 11