Oracle 12c Multitenant and Encryption in Real Life Christian Pfundtner
Christian Pfundtner, DB Masters GmbH Over 20 years of Oracle Database OCA, OCP, OCE, OCM, ACE Our Credo: Databases are our world 4 OCMs working for DB Masters Over 100 customers
Contents Features covered here: Oracle 12c Multitenant Architecture Overview Oracle 12c Database Security: Database Vault Overview Oracle 12c Database Security: Encryption Overview Oracle 12c Data Guard Architecture Overview Customer Project using this functionalities in real life Problems, pitfalls and solutions
Oracle 12c Multitenant Architecture Overview With the Multitenant Architecture you are able to consolidate several independent databases into ONE database. Within each of this so called pluggable databases applications can be managed independently of each others, database maintenance is mostly done globally
Oracle 12c Multitenant Architecture Overview Benefits Less instances to be managed, thus less management overhead Better usage of available resources (CPU, memory,...) Easier for performance tuning Oracle Resource Manager helps ensuring that no single application can use up all resources Isolation of applications is comparable to that of the single instance architecture
Oracle 12c Multitenant Architecture Overview Important functionalities for our customers Easy to create a COPY of an existing PDB for several purposes (application tests, training, freezing of current status of a PDB) CDB is treated as ONE database for Data Guard The PDB$SEED is a template for new, empty PDBs and is always open read only
Oracle 12c Database Security: Database Vault Overview Database Vault is just adding an additional, more flexible layer of privileges Database Vault is not new it has been introduced in Oracle 10g New in Oracle 12c is: Database Vault is no longer an easy to turn off functionality as it has been in Oracle 10g / 11g. It has to be turned on and off from within the database by a designated security administrator According to Oracle, no need to turn off DB Vault while patching
Oracle 12c Database Security: Database Vault Overview As soon as Database Vault is turned on, being a DBA does no longer mean to be able to see any data at all A designated security administrator defines who is allowed to see what data in what context (time, environment)
Oracle 12c Database Security: Encryption Overview Transparent Data Encryption Encryption Transparent Data Encryption ensures that all data is stored encrypted in the database files even if a database backup gets lost/stolen, data is still save (as long as nobody has been able to steal the database wallet too) Encryption keys are stored in Oracle wallets Wallets may be embedded within a hardware security module
Oracle 12c Database Security: Encryption Overview Oracle Wallet All required encryption keys are stored in the wallet As long as the wallet is not opened, accessing the encrypted data is not possible Wallets may be opened manually Or automatically by using the auto login functionality (in this case the wallet will be opened in read only mode)
Oracle 12c Data Guard Architecture Overview Data Guard (Standby Database) is an approved and stable technology enabling you to have a current copy of a database abroad to ensure disaster tolerance. With version 11g Oracle introduced the Active Data Guard: You may open the physical standby database for queries while it still receives redo from the primary
The customer project - Requirements Ensuring data security is top priority Nobody should be able to read data from other organizations Data must be encrypted everywhere (network and disk) Copy/cloning of databases happens quite frequently Environment has to be disaster tolerant Software development is ongoing At least new releases every 1-2 weeks Requirement to step back / compare to an older version
The customer project everything sounds so easy Multitenant was chosen for the capability of fast and easy cloning of PDBs (currently over 500) less management overhead Database Vault and encryption to ensure nobody is able to access data he/she should not have access to Data Guard to provide disaster protection Only about 20 CDBs Well, it really sounds so easy...
Real life begins Starting up the instance When starting/restarting the instance for any reason the Oracle wallet has to be opened before somebody starts working, but: If you decide to open the Oracle Wallet manually, this means that Either the DBA knows the secret password (which he definitely should not know) Or the security administrator prepares the Wallet to use autologin Sounds easy: go for the autologin wallet, right?
Real life begins Starting up the instance But... With autologin the wallet is opened read only! This means: all actions requiring to add/modify a key in the wallet are inhibited Since every new PDB requires its own encryption key within the wallet: Copy/Cloning of a PDB is not possible
Real life begins Starting up the instance Workaround Customer accepted that a restart of an instance has to be done by DBA and Security Administrator in cooperation using manual opened wallets. Pitfall In case of point in time recovery of a PDB or even only a single tablespace auto login for the wallet is required else it fails! Workaround: Create the autologin file just before recovery is started and remove the file as soon as it is completed
Real life begins Data Guard & Multitenant Data Guard in combination with multitenant database option means to have CDB and all PDBs protected at once Easier to set up and to manage So when creating a new PDB Data Guard takes care to mirror this action on the standby side (as long as force logging is enabled and standby_file_management=auto)
Real life begins Data Guard & Multitenant But... Sometimes Data Guard forgets providing all database files on standby site (which is a bug).
Real life begins Data Guard & Multitenant Workaround / Solution When using Active Data Guard, Data Guard Broker recognizes missing data files and corrects the issue Luckily the customer had the license already!
Real life begins Data Guard & Multitenant & Security To enable Data Guard doing recovery, the Oracle Wallet is needed to be available and open on the disaster site too.
Real life begins Data Guard & Multitenant & Security But... Oracle does not provide any functionality to transfer the wallet automatically to the disaster site à this has to be done manually be the DBA / Security Administrator In fact, every time there is a change to the wallet, it has to be copied again. Copying/Cloning a PDB is such a change to the wallet The wallet has to be transferred immediately otherwise the copied/cloned PDB remains unrecoverable
Real life begins Data Guard & Multitenant & Security Workaround / solution Implementing a Workflow for copy/cloning a PDB Stop Data Guard Apply Close the wallet on disaster site Create the new PDB Transfer the wallet to the disaster site Open the wallet on disaster site restart Data Guard Apply
Real life begins Database Vault & Multitenant To use Database Vault within an multitenant environment you have to: Configure and enable Database Vault on CDB Level Configure and enable Database Vault on each PDB where it is required Enable and disable Database Vault means also: To restart the database instance as last step (Do you remember the issue concerning wallets when restarting an instance?)
Real life begins Database Vault & Multitenant But... According to the customer s rules disabling Database Vault is only allowed in emergency situations along with a security administrator while access to the database is prevented by an other user (stop listener, disable user logon to server) Since several Multitenant Databases have up to 70 PDBs this means a lot of connects to PDB and password typing Additionally, there are PDBs mounted only containing older application releases that are kept until the functionality of the new release proves to be as expected.
Real life begins Database Vault & Multitenant Workaround / Solution Write a script which Ensures all PDBs get opened Prompt for the passwords of SYS DV_OWNER Oracle Wallet Password Walks through all PDBs and enables (or disables) DBV Restarts the instance at the end and opens wallet again for all PDBs
Real life begins Database Vault and PSU Upgrade to 12.1.0.2.3 According to the PSU documentation and Oracle ACS engineer Database Vault can stay enabled during the apply. In the upgrade documentation we found following Information for major upgrades: https://docs.oracle.com/database/121/dvadm/dvdisabl.htm#dvadm012 You may need to disable Oracle Database Vault to perform upgrade tasks or correct erroneous configurations. You can reenable Oracle Database Vault after you complete the corrective tasks.
The real life begins Database Vault and PSU Upgrade to 12.1.0.2.3 But... In fact, the upgrade with enabled Database Vault failed for all multitenant databases at customer site (having database vault enabled) The database got into some kind of deadlock situation, because: It was only possible to open the pluggable databases in upgrade or restricted mode Disabling Database Vault is not possible in upgrade or restricted mode...
Real life begins Database Vault and PSU Upgrade to 12.1.0.2.3 The recommended solution : Restore the database to just before the upgrade started, disable DBV and do the upgrade again. Sounds easy, but... The PDB$SEED Database is a read only database meaning that if RMAN BACKUP OPTIMIZATION is turned on (as recommended) it will backup the PDB$SEED only once. If crosscheck does not find a backup any more it changes its status only to expired and RMAN will not backup the PDB$SEED any more again if the backup is only expired. Say good bye to the backup of the PDB$SEED (e.g. if your backup software overwrites tapes) Restore was only possible for some of the databases!
Real life begins Database Vault and PSU Upgrade to 12.1.0.2.3 What we did: Restore the multitenent database to some minutes just before the upgrade started Eliminate the PDB$SEED database Open the CDB without PDB$SEED Create a new CDB with PDB$SEED Open all PDBs read only Unplug all PDBs from the damaged CDB and re- plug them into the new CDB, take care of the Oracle Wallet Disable DB Vault and restart the upgrade
Real life begins Database Vault and PSU Upgrade to 12.1.0.2.3 Workaround / Solution Turn off RMAN Backup Optimization and add some lines to the RMAN backup script doing additional CROSSCHECK and DELETE OBSOLTE to ensure PDB$SEED gets backed up Exact workflow of how upgrades have to be done in future - including Verifying if Data Guard is current and will be turned off before upgrading Turn off DB Vault Doing an additional RMAN Backup of CDB and PDB$SEED...
And the project is still going on.. Were this all issues we found? No, only those having the most impact. In fact, we where able to Find a workaround or solution the customer could live with In some cases the solution is a workflow you have to follow The customer has gone life with that environment, and their most valuable - the data is really very save!
Conclusion Each feature itself is working quite well, but there are some issues if you combine them with the very first edition they were introduced. There were a lot of fixes with 12.1.0.2 but still not all of our problems are solved. Can we recommend such a solution? Yes! But you should do a lot of testing of each and every usage scenario to ensure you know what you might experience! Less than 20 CDBs with over 500 PDBs much less administrative work Of course we would be happy supporting you!
Q & A