Session: Archiving DB2 comes to the rescue (twice) Steve Thomas CA Technologies. Tuesday Nov 18th 10:00 Platform: z/os

Transcription

1 Session: Archiving DB2 comes to the rescue (twice) Steve Thomas CA Technologies Tuesday Nov 18th 10:00 Platform: z/os 1

2 Agenda Why Archive data? How have DB2 customers archived data up to now Transparent Archiving in DB2 11 for z/os Analytics Accelerator - High Performance Storage Saver 2

3 Introduction Requirement to Archive data has always existed Certainly back in 1985 when I became a DBA! Inevitably gets omitted from phase 1 of any new system Pressure to provide immediate business benefit Usually never quite makes it back to the top of the list Provides little immediate value in the eyes of the business Better to spend any available Development $$ elsewhere Complex and expensive to retrofit archiving Particularly if you need to be able to recall archived data Almost always requires application changes 3

4 What are the Benefits? Performance Tables will be smaller Indexes may have fewer levels Reduces number of duplicate index entries (especially NPIs) All this results in fewer Getpages and index entries to process Can also lower number of Application to Database interactions Improved Utility Operations Less data to process Particularly helps reduce Sorting and index rebuild costs Potential for reduced Storage costs Store archived data on lower cost media Peak performance for archived data usually not necessary 4

5 How has Archiving been implemented? Application logic Insert into Archive table before Deleting data Triggers Ensure if row gets Deleted it is stored in an Archive table Partitioning Use date based Range keys and avoid reading older data Log based Technology Scan Log for Deletes and insert deleted data into an Archive System time Temporal data - DB2 10 for z/os Archiving was not the primary design point for temporal data 3rd Party tools 5

6 None of these provides a good solution Generally require application changes Usually limited access to the Archived data Reduced flexibility for the live application NPI costs for Partition based solutions CPU cost of post-processing log data for Deletes Inflexibility in making Schema Changes And more 6

7 2 New Solutions available from IBM DB2 11 for z/os Transparent Archiving feature Provides a row based Archiving solution Similar in many ways to System Time Temporal Included in the base product Potential to implement without Application Changes High Performance Storage Saver - HPSS Requires an Analytics Accelerator Provides a Partition or Table based Archiving solution More restrictions but good for some types of data Supports DB2 10 for z/os as well as DB2 11 for z/os First implemented in Accelerator 3.1 Improved in 4.1 and now a much more viable option 7

8 DB2 11 for z/os Transparent Archiving Broadly similar to DB2 System Time Temporal feature Uses a two Table solution No requirement for time period columns Available in NFM Mode Uses an Archive table to store archived rows Archival of Data can be Manual or System Managed Access to Archive data independent of the Application Controlled by a new Global Variable New Bind option allows additional control at Package level No code changes are required to retrieve Archive data Supports Triggers, Functions and Stored Procedures 8

9 High Level Overview Create a second (Archive) table based on the primary Identical column structure Issue new ALTER command to link the two together The original table is now an Archive Enabled Table Archive your data Manually - typically using REORG DISCARD & LOAD Or System Managed Simply issue a DELETE statement Applications can access Archived data transparently Global variable controls whether result set includes Archived data Packages can be made (in)sensitive to this variable No Application Changes are required 9

10 Creating an Archive Enabled Table Create a second table based on your primary Table Column structure must be identical PTF may allow Archive Date column to be added (tbc) ALTER TABLE table ENABLE ARCHIVE USE archive-table Only allowed by ALTER TABLE not CREATE Restrictions on Archive Enabled Table: Cannot be MQT No Column Mask or Row Permission Cannot be involved in a Temporal relationship Restrictions on Archive Table: Long list of restrictions see Manual for full details Examples - No RI, Aux Tables and Temporal data 10

11 Impact of Enabling Archiving on a Table Packages are invalidated ALTER TABLE ADD COLUMN is allowed The new column is automatically added to the Archive Table No other column Alterations are allowed Not allowed to ROTATE partitions, CLONE table etc. Dynamic Scrollable Cursors are not allowed 11

12 Automated Archiving Global Variable SYSIBMADM.MOVE_TO_ARCHIVE Scope of the Global Variable is Session Y or E enables Automatic Archiving N (Default) disables Rows subject to an SQL DELETE are moved to archive Warning: Rows will still be deleted if MOVE_TO_ARCHIVE = N Other DML modification statements may be disallowed MOVE_TO_ARCHIVE = Y prevents INSERT, UPDATE & MERGE New SQLCODE with Reason Code 2 MOVE_TO_ARCHIVE = E allows these No need for Insert privileges on Archive Table Only need DELETE on the Archive Enabled table 12

13 Manual Archiving Up to you how to implement this Most common method: REORG with DISCARD LOAD RESUME into the Archive Table LOAD REPLACE is not allowed on an Archive Enabled Table Although you can use it on the Archive Table Care required if MOVE_TO_ARCHIVE = Y DB2 will also try and archive the rows when you Delete them 13

14 Retrieving Archived data Global Variable SYSIBMADM.GET_ARCHIVE Scope of the Global Variable is Session N means you only access the Active data Y includes Archived data in any SQL query against Active table SELECT * FROM SALES_ACTIVE WHERE internally becomes SELECT * FROM (SELECT * FROM SALES_ACTIVE UNION ALL SELECT * FROM SALES_ARCHIVE) WHERE. 14

15 Package Sensitivity Bind Option to Determine Sensitivity to Archive data BIND/REBIND PACKAGE & REBIND TRIGGER PACKAGE ARCHIVESENSITIVE YES/NO Trigger commands fail if an Archive-enabled table referenced in WHEN and the ARCHIVESENSITIVE option is set to YES CREATE/ALTER FUNCTION & PROCEDURE ARCHIVE SENSITIVE YES/NO If set to NO the Package ignores GET_ARCHIVE setting Only Active data will be retrieved 15

16 Performance DB2 11 for z/os Performance Redbook findings: Trigger used ~2-3x elapsed and 2x CPU of MOVE_TO_ARCHIVE Bind with ARCHIVESENSITIVE uses 10-80% more CPU Not execution time and low compared to cost of changing application Avoiding UNION ALL costs when not accessing Archive: DB2 will Bind the Statement twice First (original) section contains just the Current data Extended section includes Archive data Visible in various PLAN_TABLEs - EXPANSION_REASON = A S, B and SB already used for Temporal support Beware of ORDER BY when accessing Archive data Uses a UNION ALL so almost certain to invoke SORT 16

17 Disabling Archiving ALTER TABLE table DISABLE ARCHIVE Must be the only clause Not allowed if Views, MQT or Inline Table functions exist on an Archive enabled Table Tables get treated independently afterwards ARCHIVE table is not Deleted and no data is changed Archive Table not accessed regardless of GET_ARCHIVE or ARCHIVESENSITIVE settings Active Table data not Archived by any DELETE statement Packages are Invalidated 17

18 Miscellaneous points New Catalog columns to support feature: ARCHIVESENSITIVE in SYSPACKAGE, SYSPACKCOPY ARCHIVING_SCHEMA & ARCHIVING_TABLE in SYSTABLES TYPE = R in SYSTABLES for the Archive Table only Various IFCID changes to report on the Impact of Archiving Plan Management features check for Archiving changes Bind or Rebind will fail if you change ARCHIVESENSITIVE and use APCOMPARE or APREUSE with PLANMGMT BASIC or EXTENDED Utilities Cannot run CHECK DATA SHRLEVEL REFERENCE with DELETE YES or any of LOBERROR, AUXERROR or XMLERROR INVALIDATE LISTDEF enhanced to include ARCHIVE option 18

19 High Performance Storage Saver (HPSS) Requires an Analytics Accelerator First introduced in v3.1 and improved in 4.1 Partition rather than row based Supported by DB2 10 for z/os and higher Stores data from Partitions & tables only on the Accelerator Unlike normal partitions the data does not exist in DB2 Data in Archived partitions available using Read-Only SQL Archived data can be restored to DB2 More restrictive than DB2 11 Transparent Archiving Analytics Accelerator doesn t support all data types Does not support all SQL clauses 19

20 HPSS 2.0 enhancements Introduced with Accelerator 4.1 release Archived Partitions set to PRO status Prevents Inserts, Updates and Deletes Image copies used are protected by PRO status They will not be deleted even when MODIFY RECOVERY executed Note physical Image Copy datasets may still be vulnerable Up to 4 Image Copies created during the Archive process Improves Disaster Recovery capability Also includes more flexible naming capability Archive data on multiple Accelerators Automated Restore of data from Archive This discussion assumes you have HPSS 2.0 available 20

21 Requirements Analytics Accelerator 4.1 DB2 10 for z/os or higher Analytics Accelerator Studio 4.0 or higher Or something similar to drive the Stored Procedures used Supports UTS PBR objects 21

22 Restrictions See Manuals for full details as liable to change and complex Essentially limited by what the Accelerator will support Some are related to table structure, others on the SQL supported Some examples (far from exhaustive): Range Partitioned Only if Index controlled the Index must exist Cannot be Parent in an RI relationship DECFLOAT, BLOB, CLOB, DBCLOB, XML data types TIMESTAMP(n) where n>6 UDFs, UDTs Mixed CCSIDs in SQL (including CAST with CCSID) Correlated Table Expressions Recursive Table references 22

23 How do you Archive a Partition? Stored Procedure SYSPROC.ACCEL_ARCHIVE_TABLES Locks Partition or Table Unloads the data Creates up to 4 Image Copies (2 local and 2 remote) Used for the Recovery process Transfers data to Accelerator and Loads it Sets Base partition into UT mode Removes the data from the Base Table (LOAD REPLACE) Sets Base partition to PRO status Invalidates Statement Cache and Packages Updates SYSACCEL.SYSACCELERATEDTABLES 23

24 Running SYSPROC.ACCEL_ARCHIVE_TABLES Must generate at least one Image Copy Naming standards defined in AQTENV dataset Template support was added with PTF2 (now up to PTF4) Dataset names need to be unique use &PART or &UNIQ Older AQT_ARCHIVECOPY_HLQ deprecated remove it! Tables are processed serially, Partitions in Parallel Default is 4 Partitions at a time - defined in AQTENV If any Partition fails the operation is not rolled back Object remains in UT state fix problem and resubmit SP More serious issues may require Manual recovery Connection to client must remain active while SP runs Could take some time (hours) so care if Studio being used May be better to run the SP from a Batch job 24

25 More on the Archiving process Objects must be defined to Accelerator In either InitialLoadPending or Loaded State Incremental Update must not be active Possible to reinstate this once Archiving process complete Data in base partition pruned using LOAD REPLACE Partitions remain in DB2 using minimum Primary Allocation ALTER PRIQTY to a small value before Archiving Archiving a partition already Archived elsewhere Useful for High Availability and Disaster Recovery SP sees partition archived and unloads data from the Image Copy Archive from Image Copy mode NPI rebuilds can slow down the Archiving process 25

26 SYSACCEL.SYSACCELERATEDTABLES Column ARCHIVE Blank Table is not Archived Active requests only A Archived Handles Active & Archive requests C Archived elsewhere Active requests only B Similar to C but Archiving partially complete Active only X Table has been restored on another Accelerator disabled Column ENABLE Y Enabled and queries can be offloaded N Disabled T Transition 26

27 Accessing Archived data Managed by regular SQL When you ask for Archived data based on Range predicates Requires two settings before the request works: Acceleration must be enabled DSNZPARM QUERY_ACCELERATION sets the default Dynamic overrides uses QUERY ACCELERATION special register Static can change default using QUERYACCELERATION Bind option Access to Archived data must be permitted DSNZPARM GET_ACCEL_ARCHIVE sets the default Dynamic overrides with GET_ACCEL_ARCHIVE special register Static can change default using GETACCELARCHIVE Bind Option SQLCODE if Query needs data on Accelerator SQLCODE -901 if SQL cannot be executed on Accelerator 27

28 Restoring Archived Data Procedure SYSPROC.ACCEL_RESTORE_ARCHIVE_TABLES Restore works from Image Copies Protected by the PRO status of the Partition What does it do? Invokes RECOVER Utility Invokes REBUILD INDEX (Partitioned and NPIs) Invokes CHECK DATA Archived data moved to normal area of the Accelerator This means the table can be used for regular Accelerated queries 28

29 Recovering Archived Partitions Connection to client must remain active while SP runs Could take some time (hours) so care if Studio being used May be better to run the SP from a Batch job Tables are recovered serially, partitions in parallel Same as the Archiving process Note Indexes may be in RBPD if Recovery Fails! This could have a serious impact on regular operations Accelerator must be active If not then manual recovery can be achieved using Imagecopy Need to reset PRO status before manual recovery 29

30 Miscellaneous points on HPSS Possible to run queries while partitions are being moved But no way to tell whether Archived data is included Answer depends on when the partitions are moved and object Status How to tell what has happened and where you are? SYSPROC.ACCEL_GET_TABLES_INFO SYSPROC.ACCEL_GET_TABLES_DETAILS SYSACCEL.SYSACCELERATEDTABLES Accelerator Studio Can be good to use RTS to check that data is never updated Once it s on HPSS any updates will fail PTF3 allows Transparent Archive tables to be stored on HPSS But this will prevent you archiving more data 30

31 A note on Extended LRSN and HPSS Scenario You Archived data to HPSS using 6 byte LRSN Now want to implement Extended LRSN Not possible directly due to PRO status Option 1: Manually reset PRO status using REPAIR REORG to reset LRSN Reset PRO Recommended by Manuals but messy if you need to restore Recovered object uses Basic LRSN so will require a REORG Option 2: Restore the data, Convert and re-archive Uses more resources but neater solution 31

32 Summary Worth Considering Archiving old or redundant data Improves Performance Reduces Utility costs and improves availability DB2 11 for z/os Transparent Archiving Row based solution Can be implemented without application Changes Managed by Global Variables HPSS High Performance Storage Saver Partition based solution Based on Accelerator technology More restrictive that Transparent archiving Good for particular types of data (often Analytic) 32

33 Session Archiving DB2 comes to the rescue (twice) Steve Thomas CA Technologies 33