DOAG 2015 Database Data Replication With Oracle GoldenGate Robert Bialek Principal Consultant Partner

DOAG 2015 Database Data Replication With Oracle GoldenGate Robert Bialek Principal Consultant Partner BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Who Am I Principal Consultant, Partner and Trainer at Trivadis GmbH in Munich robert.bialek@trivadis.com Focus: Oracle Database High Availability Database Architecture/Internals Backup/Recovery Troubleshooting/Performance Tuning Linux Administration Trainer for the following Trivadis courses Oracle Grid Infrastructure (O-GRINF) Oracle Real Application Cluster (O-RAC) Oracle Data Guard (O-DG) 2 16.06.2015 Data Replication With Oracle GoldenGate

Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interactive operation of your IT systems. 3 16.06.2015 Data Replication With Oracle GoldenGate

With over 600 specialists and IT experts in your region. COPENHAGEN 14 Trivadis branches and more than 600 employees HAMBURG 200 Service Level Agreements Over 4,000 training participants Research and development budget: CHF 5.0 / EUR 4 million DÜSSELDORF FRANKFURT Financially self-supporting and sustainably profitable Experience from more than 1,900 projects per year at over 800 customers BASEL STUTTGART FREIBURG BRUGG ZURICH MUNICH VIENNA BERN LAUSANNE GENEVA 4 16.06.2015 Data Replication With Oracle GoldenGate

Technology on its own won't help you. You need to know how to use it properly. 5 16.06.2015 Data Replication With Oracle GoldenGate

Agenda 1. Introduction 2. Oracle GoldenGate Architecture 3. Oracle GoldenGate Topologies 4. Oracle GoldenGate Setup 5. Core Messages 6 16.06.2015 Data Replication With Oracle GoldenGate

7 16.06.2015 Data Replication With Oracle GoldenGate Introduction

What is Oracle GoldenGate? Oracle strategic Middleware software product used for data extraction and replication Use-cases: data/service high-availabillity, zero-downtime upgrades/migrations, easy fallback (minimize risk) Supports asynchronous data transfer (DML, DDL) heterogeneous sources and targets (databases, operating systems) different extraction, propagation and apply methods different topologies, mappings, transformations, filtering, error handling, Source as well as the target environment requires a license Includes (EE): Oracle Active Data Guard and Oracle Global Data Services (12.1.0.1+) 8 16.06.2015 Data Replication With Oracle GoldenGate

Why Oracle GoldenGate? Oracle Streams is deprecated in the Oracle 12c version (but still supported!) https://docs.oracle.com/database/121/upgrd/deprecated.htm According to the Oracle GoldenGate Statement of Direction http://www.oracle.com/technetwork/middleware/goldengate/overview/statement-ofdirection-gg-132120.pdf Oracle Streams continues to be supported but will not be enhanced to support any new database features in future releases. This means that Streams will continue to support exactly what it supports today (11.2) in future releases. It also means that any new database features, data types, or capabilities will not be supported by Streams. Beginning with the version 12.1.0.2, the non-cdb architecture is also deprecated As of now, Streams is not supported with the new CDB architecture 9 16.06.2015 Data Replication With Oracle GoldenGate

Oracle GoldenGate Architecture 10 16.06.2015 Data Replication With Oracle GoldenGate

Optional Oracle GoldenGate Architecture: Big Picture Oracle GoldenGate Instance Oracle GoldenGate Instance Source DB Optional Primary (Passive) Extract Manager (VIP) MONITOR Manager (VIP) MONITOR Optional Primary (Alias) Extract Target DB Primary (Regular) Extract Collector Replicat Local Trail Secondary Extract Data Pump Remote Trail Static Configuration Runtime Configuration Runtime Configuration Static Configuration 11 16.06.2015 Data Replication With Oracle GoldenGate

Primary Extract Group: Capture Source DB Primary Extract can run in either Classic Capture mode (all database versions) Integrated Capture mode (full support: 11.2.0.3+) Primary (Regular) Extract Primary (Passive) Extract Extracts and optionally filters/transforms the transactions from the source database Only for configured objects Many Extracts per source database can operate at the same time Only committed transactions (CSN) will be propagated to the target system (in contrast to Oracle Streams ) Optionally written to a local trail (recommended) Without a local trail, the primary Extract send the data to the target system Not committed transactions will be discarded from memory or dirtmp directory 12 16.06.2015 Data Replication With Oracle GoldenGate

Primary Extract Group: Classic Capture Classic Capture mode Traditional mode, reads directly Oracle database redo logs Able to mine redo logs in ASM (two methods) Redo Logs All database versions supported Source DB With some restrictions. For example no support for: BASIC, OLTP, Exadata Hybrid Columnar compression PDML on RAC database CDB architecture! DDL, TDE support no out-of-the box support Primary Extract OS System Calls Local Trail Secondary Extract Data Pump Not transparent to RAC configuration changes (#threads) 13 16.06.2015 Data Replication With Oracle GoldenGate

Primary Extract Group: Integrated Capture Integrated Capture mode Interacts with the database logmining server Required to capture data from a CDB architecture Fully transparent with RAC, TDE, ASM Full support for databases 11.2.0.3+ Source database 11.2.0.4+ does not require supplementary objects for DDL support Supports more complex data types than classic capture Integrated capture deployments Local or a downstream deployment The best choice for most deployments Logmaining Server LCR LCR Primary Extract Local Trail Redo Logs Secondary Extract Data Pump Source DB 14 16.06.2015 Data Replication With Oracle GoldenGate

Replicat Process: Apply The Replicat process can run in either Classic Replicat mode (nonintegrated) - all database versions Coordinated Replicat mode (nonintegrated) - all database versions, GoldenGate 12c required Integrated Replicat mode database versions 11.2.0.4+ Target DB Replicat The Replicat process reads the remote trail, reconstructs the SQL and applies them to the target database Optionally filters/transforms the data Remote Trail The application of the data can be optionally delayed (DEFERAPPLYINTERVAL parameter) 15 16.06.2015 Data Replication With Oracle GoldenGate

Replicat Group: Classic/Coordinated Mode Classic Replicat Constructs SQL statements from the trails and applies the SQL through OCI Supported for all database versions Transactions are applied sequentially (single-threaded) Replicat SQL OCI Remote Trail Target DB Coordinated Replicat Transactions are applied in parallel (multi-threaded) There is 1 coordinator plus many apply threads Supported for all database versions Remote Trail Coordinator Thread 1 SQL OCI Target DB Thread 2 Thread n 16 16.06.2015 Data Replication With Oracle GoldenGate

Replicat Group: Integrated Mode Integrated Replicat Supported for target database versions 11.2.0.4+ Replicat LCR Database Inbound Server Target DB Constructs LCRs and transmits DMLs to the database inbound server Remote Trail SQL OCI Some changes (e.g. DDL) are not supported directly by the inbound server In those cases the Replicat process disconnects from the inbound server and executes the transaction in direct apply mode through OCI With many transactions applied in direct apply mode it might be necessary to switch to nonintegrated Replicat 17 16.06.2015 Data Replication With Oracle GoldenGate

Capture/Apply: CDB & Pluggable Databases Extract must operate in integrated mode Extract must connect to the CDB$ROOT container as a common user (c##) One Extract can capture from one or many PDBs to a single trail common_user_prefix (12.1.0.2) Replicat can connect and apply only to one PDB (one Apply/PDB, connect using service) Source DB CDB CDB$ROOT PDB1 PDB2 Target DB CDB CDB$ROOT PDB1 PDB2 Primary Extracts INTEGRATED Data Pumps Collectors Replicats 18 16.06.2015 Data Replication With Oracle GoldenGate

Oracle GoldenGate Topologies 19 16.06.2015 Data Replication With Oracle GoldenGate

Oracle GoldenGate Topologies Active-Passive Data replication only in one direction Only one set of primary Extract, Data Pump and Replicat groups Active/Standby Additional inactive primary Extract, Data Pump and Replicat groups Switchover: Stop one set of processes Start the inactive one 20 16.06.2015 Data Replication With Oracle GoldenGate

Oracle GoldenGate Topologies Active-Active Data replication in both directions Two active sets of primary Extract, Data Pump and Replicat groups With some restrictions: e.g.: DDL-support restricted to two databases with identical metadata Additionally broadcasting, cascading, consolidation is possible 21 16.06.2015 Data Replication With Oracle GoldenGate

Oracle GoldenGate Setup 22 16.06.2015 Data Replication With Oracle GoldenGate

Oracle GoldenGate: Setup Flow 1 2 3 Architecture Software Installation/ Design Patches Database Preparation 6 5 Primary Extract, Data 4 Register/Start Primary Pump, Replicat Extract, Data Pump Configuration Files Manager Configuration 7 8 9 Target Database Register/Start Replicat Instantiation Cluster Integration / INIT Scripts n Tuning/Customizing/ Maintenance/ 23 16.06.2015 Data Replication With Oracle GoldenGate

Supplemental Logging: Source 3 Database Preparation Activate minimal database level supplemental logging on the source database To add row chaining information to the redo stream SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; Add schema or table level supplemental logging on the source database Schema level logging is required for DDL support For all schema tables ADD SCHEMATRANDATA <schema> [ALLCOLS] Activates unconditional supplemental logging for PK and conditional for unique/foreign keys (scheduling columns) all existing an future tables For a table Depending on the use-case, different parameters are necessary ADD TRANDATA <schema>.<table> [COLS ( ) ALLCOLS] 24 16.06.2015 Data Replication With Oracle GoldenGate

Manager Configuration: Target/Source 4 Manager Configuration Manager configuration (tune the values according to your needs) <GG_HOME>/dirprm/mgr.prm PORT 7809 unreserved and unrestricted DYNAMICPORTLIST 7810-7910 AUTOSTART ER * AUTORESTART ER *, RETRIES 4, WAITMINUTES 4, RESETMINUTES 60 PURGEOLDEXTRACTS <...>/dirdat/*ext*/*t*, USECHECKPOINTS, & MINKEEPHOURS 4 LAGREPORTMINUTES 30 LAGINFOMINUTES 5 LAGCRITICALMINUTES 10 25 16.06.2015 Data Replication With Oracle GoldenGate

Primary Extract Configuration 5 Primary Extract, Data Pump, Replicat Configuration Files Create the configuration file for the primary Extract (source) EXTRACT app1ext1 SETENV (ORACLE_HOME = "/u00/app/oracle/product/11.2.0.4") USERIDALIAS ggadm_app1_11204 You need GoldeGate LOGALLSUPCOLS Credential Store UPDATERECORDFORMAT COMPACT DDL INCLUDE MAPPED EXCLUDE OBJNAME APP1.T4 & EXCLUDE OBJNAME APP1.T5 ENCRYPTTRAIL AES256 EXTTRAIL <...>/dirdat/app1ext1/lt SEQUENCE APP1.*; TABLE APP1.*; A Master-Key Wallet is necessary 26 16.06.2015 Data Replication With Oracle GoldenGate

Secondary Extract/Replicat Configuration Files 5 Primary Extract, Data Pump, Replicat Configuration Files Create the configuration file for the Data Pump Extract (source) Not necessary if the trail is already EXTRACT app1pmp1 encrypted USERIDALIAS ggadm_app1_11204 RMTHOST ggmgr-cl121.trivadis.com, MGRPORT 7809,ENCRYPT AES256,& COMPRESS, COMPRESSTHRESHOLD 1200 RMTTRAIL <...>/dirdat/app1rep1/rt SEQUENCE APP1.*; TABLE APP1.*; Create the configuration file for the Replicat group (target) REPLICAT app1rep1 SETENV (ORACLE_HOME = "/u00/app/oracle/product/12.1.0.2") DBOPTIONS INTEGRATEDPARAMS(parallelism 6) USERIDALIAS ggadm_app1_12102 DDLOPTIONS REPORT PDB ASSUMETARGETDEFS Mapping: non-cdb CDB MAP APP1.*, TARGET APP1.APP1.*; 27 16.06.2015 Data Replication With Oracle GoldenGate

Primary/Secondary Extract Group 6 Register/Start Primary Extract, Data Pump Register the primary Extract with the database (integrated) GoldeGate Credential DBLOGIN USERIDALIAS gg_app1_11204 Store REGISTER EXTRACT app1ext1 DATABASE [CONTAINER (APP1)] Extract APP1EXT1 successfully registered with database at SCN 3630440. Create the primary Extract group (integrated) ADD EXTRACT app1ext1, INTEGRATED TRANLOG, BEGIN NOW Add a local trail an associate it with the primary Extract ADD EXTTRAIL <...>/dirdat/app1ext1/lt, EXTRACT app1ext1 Add the Data Pump Extract ADD EXTRACT app1pmp1, EXTTRAILSOURCE <...>/dirdat/app1ext1/lt 28 16.06.2015 Data Replication With Oracle GoldenGate

Secondary Extract/Replicat Group 8 Register/Start Replicat Add a remote trail and associate it with the secondary Extract ADD RMTTRAIL <...>/dirdat/app1rep1/rt, EXTRACT app1pmp1 Add the Replicat group (integrated) ADD REPLICAT app1rep1, INTEGRATED, EXTTRAIL <...>/dirdat/app1rep1/rt Start the Replicat on the target with the correct CSN GGSCI> START REPLICAT app1rep1, AFTERCSN 3693221 29 16.06.2015 Data Replication With Oracle GoldenGate

Oracle GoldenGate: Demo Environment Source DB: 2-nodes RAC 11.2.0.4 Schema: APP1 GoldenGate version: 12.1.2.1.2 Active/Passive data replication Target DB: CDB 2-nodes RAC 12.1.0.2 PDB: APP1, Schema: APP1 GoldenGate version: 12.1.2.1.2 Active/Passive data replication CDB$ROOT APP1 APP2 APP3 Schema: APP1 Primary Extract INTEGRATED Local Trail RemoteTrail Replicat INTEGRATED Secondary Extract Data Pump Collector 30 16.06.2015 Data Replication With Oracle GoldenGate

31 16.06.2015 Data Replication With Oracle GoldenGate Core Messages

Core Messages GoldenGate is a very powerful replication software framework For most deployments the integrated Extract/Replicat is the best choice Supplemental logging generates more undo and redo In some cases you need even to supplementally log all columns! Script all the configuration steps Very useful if you need to re-setup the whole environment You don t need to replace your current Streams installation ASAP by Oracle GoldenGate, but Streams is deprecated in the version 12c and will not be enhanced in future releases There is a framework to ease the migration from Streams to GoldenGate (streams2ogg) As of now Streams is not supported with CDB architecture For new projects it might be a better choice (licensing)! 32 16.06.2015 Data Replication With Oracle GoldenGate

Questions and answers Robert Bialek Principal Consultant Partner Tel. +49 89 99275930 robert.bialek@trivadis.com 33 16.06.2015 Data Replication With Oracle GoldenGate