1
<Insert Picture Here> Database Technologies for Archiving Kevin Jernigan, Senior Director Product Management Advanced Compression, EHCC, DBFS, SecureFiles, ILM, Database Smart Flash Cache, Total Recall, Resource Manager, Direct NFS Client, CQN, IOT, Database Performance
Database Technologies for Archiving: Agenda Unstructured and semi-structured data SecureFiles DBFS DBFS HSM Structured / relational data Compression Partitioning Information Lifecycle Management 3
Enterprise Data Audio, Video Image Data SQL Data XML Data Yearly Data Growth Data Growth Structured 15-20% Semi & Unstructured 50-100% * Gartner & IDC Estimates Documents Vector, Spatial Emails, IM In a typical enterprise, Structured Data is ~20% Semi & Unstructured Data represents the other 80% Data growth is happening across the board! 4
Managing Information Organizations need to efficiently and securely manage all data Structured Semi- Structured Unstructured XML PDF File systems: simplicity and performance for files Relational data in (Oracle) RDBMS Oracle Database supports ANSI-standard LOBs Performance is a concern for many users 5
Files Belong with Relational Data Many enterprise applications manipulate both files and relational data Rich user experience, compliance, business integration This split compromises security, robustness, and management Disjoint security and auditing models Changes cannot be made atomically Backup and recovery are fragmented Search across relational data and files is difficult Space management is complicated Separate interfaces and protocols Application architecture more complex Two data managers for one application is one too many! 6
Oracle SecureFiles Consolidated Secure Management of Data SecureFiles is designed to break the performance barrier keeping file data out of databases Introduced with Oracle Database 11g Release 1 (4.5 years ago) Similar to LOBs but much faster, and with more capabilities Transparent encryption (with Advanced Security Option) Compression, deduplication (with Advanced Compression Option) Extends the security, reliability, and scalability of database to files Superset of LOB interfaces allows easy migration from LOBs Enables consolidation of file data with associated relational data Single security model Single view of data Single management of data SecureFiles is 100% backwards compatible with ANSI SQL 92 LOB interfaces 7
High Performance SecureFiles Vs. BasicFiles Writes: New Space Writes: Reused Space Read Performance MB/s 300 250 200 150 100 50 0 0.01 0.1 1 10 100 File Size (MB) MB/s 300 250 200 150 100 50 0 0.01 0.1 1 10 100 SecureFiles File Size (MB) MB/s BasicFiles 300 250 200 150 100 50 0 SQLPlus file test: Concurrent Reads/Writes, OCI, 4 streams Adding files using new disk space: Up to 2x faster Adding files reusing space: Up to 22x faster Reads up to 3x faster 0.01 0.1 1 10 100 File Size (MB) 8
High Performance SecureFiles Vs. NFS MB/s 100 10 1 0.1 File Read Performance 1 10 100 1000 10000 File Size (KB) SecureFiles MB/s 100 10 1 0.1 0.01 File Write Performance NFS 1 10 100 1000 10000 File Size (KB) SQLPlus file test, single stream, single host Using SecureFiles is faster across the board 2x-3x faster for Queries, 6x for Inserts Tests run using both SecureFiles and NFS/ext3 in metadata journaling only (default for NFS) Filesystem-like performance 9
Business Files in Database Applications Database applications store business data files as LOB columns 11g SecureFiles LOBs makes files in the database as fast as files in file systems Also provides transactional consistency with database data Advanced file management capabilities Compression, Encryption, Deduplication Relational Columns ID# Name Address Contract Photo 23 Lucas.. Customers Table Files / LOBs But Database LOBs are difficult to access from file-based tools 10
Files in the Database Reinvented Files DB Oracle Database 11g reinvents files in the database SecureFiles provides fast and powerful file storage Removes performance barrier to storing files in the database DBFS provides simple file system interface to files stored in the database Enables existing file based tools to access database files Familiar access through pathnames, directories, links Files kept in a dedicated file store, or existing application tables Oracle Database 11gR2 Storing business data files inside the database is now simpler, faster, and more robust than storing them outside 11
Database-Enabling File-Based Tools DBFS allows access to files in the db using file system interfaces File operations translated into SQL operations Directories and path names are derived from key columns in tables Enables access by existing file-based tools DBFS Client Acrobat Reader Network DBFS Server in DB Path Names, Directories /.. /Customers /Contract /Lucas /Photo Customers Table SQL Access ID# Name Address Contract Photo 23 Lucas.. Select Contract from Customers where Name = Lucas 12
DBFS SecureFiles Store DBFS also implements stand alone file systems in the database Directory information stored in tables Files stored in SecureFiles LOBs Used for operational application files such as ETL files, reports, etc. that are not in application tables Provides unified data and file backup, DR, management DBFS Client ETL Tool Network DBFS Server in DB /.. /ETL /Monday.dat DBFS SecureFiles Store Inode# Owner File Path 3768 Lucas /ETL/Monday.dat 13
DBFS SecureFiles Store Capabilities Powerful Configuration Options Full data logging, or meta-data only logging Cache, or direct read from disk Partitioning Compression File level de-duplication Encryption Total Recall Retain all file versions historical query, or regulatory compliance Snapshots Can create file system snapshots at user-selected past point in time Uses Flashback Query to create snapshots Can be used to recover deleted files or old file versions 14
DBFS HSM Store A DBFS HSM store allows archiving files to tape Application migrates older files to HSM store (e.g. old invoices) A DBFS Link replaces the LOB, LOB reads on links are transparent A LOB can be easily migrated back to the database for updates HSM store has disk staging area for storing recently accessed files Seldom accessed files are migrated to tape, brought back on reference Order# Customer Year Invoice /HSM 1234 Lucas 2003 /Old_Invoices /Invoice_1234 Sales Table DBFS Link /HSM/Old_Invoices/Invoice_1234 HSM Disk Staging Area Size is Configurable /.. LRU Tape 15
DBFS Store API DBFS interface is DBFS Store API PL/SQL & SQL interface called directly by applications inside the database PL/SQL interface has all operations SQL view for read only access Same interface called remotely by clients on other hosts Strong support for storage of metadata associated with files Transactional file system operations E.g. Multiple files can be created atomically in a DB transaction DBFS Store API Create Operations Create files, directories, links Delete Operations Delete files, directories, links Get/Put Operations Read and write LOB and attributes of existing file paths Rename Operation Directory Operations List, Search Locking Operations Create Snapshot at point in time using consistent read 16
Building your own file system integrated with your application You can now build your own file system inside the Oracle db Create new value by creating a file system interface Database developers can now write a robust file system: No need to be a OS kernel developer or debug kernel crashes Create a file system interface to data stored in relational tables: Like a file system view for an existing database application Write a file system in Java, SQL/PL/SQL Or any other language using callouts Build your application-integrated file system in less than a day Writing a basic read only file system view on the LOBs that are currently in your database application in 60 seconds! 17
A file system: 100 lines of PL/SQL create table tbfst( key varchar2(256) primary key check (instr(key, '/') = 0), data blob) tablespace users lob(data) store as securefile (tablespace users); grant select on tbfst to dbfs_role; grant insert on tbfst to dbfs_role; grant delete on tbfst to dbfs_role; grant update on tbfst to dbfs_role; function list( store_name in varchar2, path in varchar2, filter in varchar2, recurse in integer, ctx in dbms_dbfs_content_context_t) return dbms_dbfs_content_list_items_t pipelined is begin for rws in (select * from sys.tbfst) loop pipe row(dbms_dbfs_content_list_item_t( '/' rws.key, rws.key, dbms_dbfs_content.type_file)); end loop; end; procedure getpath( store_name in varchar2, path in varchar2, properties in out nocopy dbms_dbfs_content_properties_t, amount in out number, offset in number, buffer out nocopy raw, prop_flags in integer, ctx in dbms_dbfs_content_context_t) is content blob; guid number; begin if (path = '/') then raise dbms_dbfs_content.unsupported_operation; end if; end; select t.data into content from sys.tbfst t where ('/' t.key) = path; select ora_hash(path) into guid from dual; dbms_lob.read(content, amount, offset, buffer); properties := dbms_dbfs_content_properties_t( dbms_dbfs_content_property_t( 'std:length', to_char(dbms_lob.getlength(content)), dbms_types.typecode_number), dbms_dbfs_content_property_t( 'std:guid', to_char(guid), dbms_types.typecode_number)); 18
DBFS Performance Tested a mix of file reads and file writes Scaled from 1 to 4 Oracle Database instances on an Exadata ½-rack Ran steady-state test for 12 hours Achieved sustained throughput of 10TB per hour 19
DBFS Performance 20
DBFS Performance 21
Rich Capabilities Inherited from DB DBFS Capability Compression, Deduplication, Encryption Crash Tolerance Mirroring, Striping, Online Add Storage Disaster Recovery, Readable Remote Mirror Consistent Backup Multi-Node Scalability, Transparent Failover Impromptu Snapshots Restore to Point in Time Retention / Compliance Network Security Provided By SecureFiles Atomic transactions, Logging ASM Data Guard RMAN, Hot backup RAC Consistent Read Flashback, Media Recovery Total Recall SSL 22
Enterprise Data Audio, Video Image Data SQL Data XML Data Yearly Data Growth Data Growth Structured 15-20% Semi & Unstructured 50-100% * Gartner & IDC Estimates Documents Vector, Spatial Emails, IM In a typical enterprise, Structured Data is ~20% Semi & Unstructured Data represents the other 80% Data growth is happening across the board! 23
Information Lifecycle Management Partitioning and online archival Keep as much data as possible available online and store the data in the most appropriate manner Find the balance between data importance, storage performance, storage reliability, and storage form Data storage cost is driven by two factors Cost of storage Storage footprint of data on disk Key enabling technologies Partitioning Compression 24
Information Lifecycle Management Online data lifecycle without ILM Data Lifecycle Active Active Less Active Less Historical Active Data Lifecycle Archive Archive High Performance Storage Tier Offline Archive 25
Information Lifecycle Management Matching online data life-cycle to storage Data Lifecycle Active Active Less Active Less Historical Active Data Lifecycle Archive Archive High Performance Storage Tier Low Cost Storage Tier Online Archive Storage Tier Offline Archive 26
Information Lifecycle Management Matching online data life-cycle to storage Data Lifecycle Hyper Active Active Active Less Historical Active Data Lifecycle Archive Less Active Historical Archive SSD/Flash Storage Tier High Performance Storage Tier Low Cost Storage Tier Online Archive Storage Tier Offline Archive 27
Oracle s ILM Solution Step 1 Partition tables into smaller chunks Increases performance Queries execute faster against smaller partitions DBA operations like data loads, backup, recovery etc, also benefit Increases availability 24*7 access to critical data Enables simpler management Divides large tables into smaller, more manageable chunks Enables storage cost optimization Individual partitions can be assigned to specific storage areas Reduces the cost of retaining data Transparent to applications applications are unchanged Partitioning Option Oracle Database Enterprise Edition Active Less Active Historical 28
Oracle s ILM Solution Step 2 Compress data to maximize storage usage Compress Large Application Tables OLTP and DW Compress All Data Types Structured and unstructured data types Compress Backup Data Faster RMAN compression Data Pump compression Typical Compression of 2-4X Cascade storage savings throughout data center Partitioning Option Advanced Compression Oracle Database Enterprise Edition Active Up to 4X compression Less Active Historical Transparent to applications Applications are unchanged 29
Oracle s ILM Solution for Exadata/DB Machine Step 2 Compress data to maximize storage usage Data stored by column and then compressed Useful for data that is bulk loaded or moved Query mode for data warehousing Partitioning Option Advanced Compression Exadata HCC Oracle Database Enterprise Edition Typical 10X compression ratios Scans improve accordingly 50X Up To Archival mode for old data Typical 15-50X compression ratios Transparent to applications Applications are unchanged Hyper Active To10X Up Compression Active Less Active Compression Historical Choice of SATA or SAS drives 30
Oracle s ILM Solution Step 3 - Less storage and better performance Partitioning & Compression 10 TB of user data Requires 10 TB of IO 4 TB with compression 400 GB with partition pruning Improved query response times 31
Oracle s ILM Solution for Exadata/DB Machine Step 3 - Less storage and better performance Partitioning & Exadata HCC 10 TB of user data Requires 10 TB of IO 20 GB with Storage Indexes 1 TB with compression 5 GB with Smart Scans 100 GB with partition pruning Sub-second Response times 32
Oracle s ILM solution Step 4 - Delivers significant cost savings Storage Tier Single Tier Multiple Tiers Multiple Tiers compressed HP Exadata HC High Performance (3TB) $216,000 High Performance (200 GB) $14,400 $14,400 $10,400 $3,000 Low Cost (800 GB) $11,200 $4,480 $4,160 $1,200 Online Archive (2000 GB) $14,000 $5,600 $6,900 $2,000 4X 9X 10X 34X Total $216,000 $39,600 $24,480 $21,460 $6,200 Appropriate usage of storage tiers significantly reduces total cost of ownership Compression reduces TCO even more Transparent to applications 33
Oracle s ILM Solution Step 5 Security and Compliance Monitor database activity Transparently track data changes Ensure efficient, tamper-resistant storage of archives Real-time access to historical data Enforce access control Limit powers of privileged users enforce separation of duties Enforce who, where, when, and how using rules and factors Protect application data by preventing application by-pass Encrypt and mask sensitive data Remove sensitive data from non-production databases Sensitive data never leaves the database Advanced Security Partitioning Advanced Option Compression Exadata HCC Oracle Database Enterprise Edition Active Less Active Historical 34
Oracle s ILM Solution Key Benefits ILM is a Data Management strategy designed to: Reduce Cost of Retaining Data Comply with Legal, Regulatory, and Business Mandates Available for any database from 9i ILM is a set of policies and techniques for managing data Managing data is Oracle s core competency Some new technologies, but mostly an application of existing data management capabilities Keeping data in the database preserves security and auditing models archiving data doesn t 35 35
Random Points DBFS HSM uses tar to store files on tape or cloud QFS is certified for Oracle Database, including RAC RAC==Real Application Clusters, which means QFS can be used as a clustered / shared file system Oracle Database does block-level checksums and other corruption prevention and detection 36
37