Modern Approaches to Archiving and Application Retirement Phil Menzies Vice President ETI-NET November 17, 2015
Agenda What is Archiving? Why do it? Conventional archiving approaches Potential problems Auditing archives Modernization of archiving Platform and application independence Retiring old applications How to go about modernization Summary References 2
Let us save what remains: not by vaults and locks which fence them from the public eye and use in consigning them to the waste of time, but by such a multiplication of copies, as shall place them beyond the reach of accident. Thomas Jefferson 3
What is archiving? Webster s dictionary: to collect and store materials (such as recordings, documents, or computer files) so that they can be found and used when they are needed Enables future querying of application data Specific data that must be retained Legal, regulatory or business requirements Examples: Stock transfer history Patient records Insurance claims Customers cleared checks Don t confuse with Backup Not all data Not for restoration of lost data Not for recovery of corrupted data Not for limited-capacity-driven data offload But, what if Usually multiple years, sometimes forever Retention must be assured Multiple copies, different media, multiple locations 4
Records handling rules & regulations Lots of them! A few examples: From FDIC Law, Regulations, Related Acts sect 8000 From Federal Financial Institutions Examination Council From Centers for Medicare & Medicaid Services And they change every so often! So content with current retention of 5yr could become 7yr Need to be able to deal with it Talk to the experts: your company s Compliance Officer 5
Records retention specific requirements An archiving methodology must: Retain for period specified for each record type Destroy after specified period specified for each record type Enforce access control rules Demonstrate chain of custody Maintain an audit trail 6
The current state of archiving 7
Conventional approach to archiving Periodically backup a specific set of data (uses same tools as for normal backups) In the past: Write to physical tape and transport to off-site storage This meant less often than regular backups perhaps once a week or a month More recently: Write to virtual tape and replicate to remote facility But long retention times can have impact on storage capacity required Poor data deduplication efficiency High costs for disk storage So back to physical tape for archives Initial copy written to virtual tape Migrated to (lower cost) physical tape via Enterprise Backup System (EBS) server Like NetBackup, Commvault, TSM, etc. 8
Evolution of Physical Tape Archives 9
Growth of Tape Capacity 10
But Each LTO tape drive technology can only read its own and previous 2 generations LTO-6 drive (current) can t read LTO-3 or earlier LTO-7 drive (available 2016) can t read LTO-4 or earlier So essentially 5-10 years of tapes are readable with current technology drives But what about tapes with content having longer retention requirement? a. Keep old technology tape drives (and servers, interfaces, O/S, applications) to read the old tapes But can they connect to current systems electrically and drivers? e.g. no tape drives older than LTO-6 on NonStop-X systems b. Or abandon tape - use disk storage But cost may be prohibitive c. Or periodically migrate content to current tape technology Some EBS products can automate this High costs for new media, tape operations 11
There may be bigger problems though Can the archived data be used? Are the originating applications still available? Can they run on current platform e.g. NonStop-X? Can they run on current O/S version? Are the usable applications backward-compatible? Can they interpret the older archived files? Do any databases have to be reverted to an older versions? 12
And even more challenges Can records retention requirements actually be met? How are content destruction requirements satisfied? An expiring tape becoming scratch does not destroy its contents But deletion of a tape encryption key does destroy the contents What about if archive tape content has mixed retention & destruction requirements? How is access control enforced? Originating application may enforce during retrieval But if files are otherwise restored or accessed? How is chain of custody demonstrable? Particularly if media goes off-site Does media have WORM capability to prevent modification? And does the archived media actually still exist? Open-loop vs. closed-loop catalog & media management Any auditing? 13
What about data protection for older archives? Protection techniques evolve with time Sensitive data saved may be encrypted in current archives e.g. Format-protecting Encryption (FPE) of PAN or Tokenized social security numbers But older tapes generally pre-date these approaches May have PAN or SSN in clear text Seldom do companies go back and encrypt/tokenize fields in the old archives This leaves data on older archives vulnerable 14
Typical BackBox Configuration with EBS for Archiving 15
BackBox + EBS for Archiving Takes advantage of virtual tape Backup via BackBox VTC Copy Virtual tapes to an EBS Server Delete from local storage & remember Archive content backed-up to one or more virtual tapes via BackBox Virtual tape images stored as files on StoreOnce or SAN, etc. EBS client runs (immediately or scheduled) to copy VT image files to EBS server EBS server stores in a tape pool or disk pool that could later migrate to tape BackBox deletes original VT image files off local storage when confirmed on EBS BackBox remembers where VT image files were on local storage Archive from NonStop to EBS-managed physical tape storage Can later ask EBS client to retrieve files (virtual tape) Validates each VT image file using timestamp & checksums 16
What could go wrong? Remember Murphy! Human error is most frequent cause of problems Retention time conflicts not set equal or longer on EBS system Mistaken configuration changes at any point in path Tape media problems Unreadable physical tape on EBS system Lost tapes, where removed from silo for off-site storage Inability to read older tape technologies for old archives EBS system problems Corruption of EBS catalog Encryption problems Loss of keys 17
So what can you do? Establish best practices Check configuration settings Particularly expiration Also storage & paths used AUDIT! Start with backup generations in DSM/TC & ensure that content can be found somewhere If on physical tape, are all tapes accounted for Does your EBS have a tape content migration capability? Some can consolidate unexpired content to fewer cartridges, moving to newer tape technologies New tool can help: ETI-NET Data Protection Auditor Automates correlation of DSM/TC content with virtual tape storage and EBS content Also generates range of reports, including storage utilization Part of an ETI-NET service offering to review customers backup and archiving processes 18
The future of archiving 19
There is a better way Back to fundamentals rather than storage mechanics Re-think archiving as a database problem Ideally would like to keep all data on original system until it expires But too big for most transaction-oriented systems & too expensive, particularly primary storage Potentially would slow down transaction processing Would make it even more difficult to transition to new applications and platforms But if you could keep the data on-line, lots of advantages: Fast access Granular access control and data retention/destruction Audit trail for records access Eliminates complexity of dealing with hierarchies of media and its storage Could eliminate the possibility of archive loss or exposure 20
Active Archiving The OAIS Reference Model Originated with NASA Consultative Committee for Space Data Systems (CCSDS) First draft in 1999 Reference model adopted as ISO 14721.2003 Doesn t require use of any particular platform, O/S, DBMS, DDL, language, user interface Principles embodied in number of vendors products Gartner Magic Quadrant for Structured Data Archiving and Application Retirement includes: InfoSphere Optim Archive (IBM) HP Structured Data Manager Informatica Data Archive EMC InfoArchive 21
OAIS Reference Model 5 Functional Entities INGEST To populate the archive with info ARCHIVAL STORAGE To receive & manage storage of the archive content DATA MANAGEMENT To maintain a searchable database of the Descriptive Information ADMINISTRATION Including access control, security, etc. ACCESS To enable querying of the Archive 22
OAIS Reference Model Information Packages 3 Types: Submission Information Package (SIP) Archival Information Package (AIP) Dissemination Information Package (DIP) Packages contain: Content information Preservation Description Info (PDI) Packaging information Descriptive information 23
Active Archiving Implementation Open architecture & standards Store data in an open, platform-independent format: XML Query using standard tools: e.g. xquery Should be storage independent move to new storage when appropriate Some synergy w. storage features: e.g. EMC InfoArchive on Isilon w. SmartLock (WORM) Ingest data via Connector Standard connectors for common products: SharePoint, MS Exchange, etc. Custom connectors for specific purposes from 3 rd parties Connector can be: Live data flows from production system in real time Batched data is accumulated for ingestion to not interfere with production Retroactive data is ingested from prior (e.g. tape) archives 24
Related Problem: Zombie Applications & Platforms Have you ceased use of a particular application? Replaced with a different application? Or gotten out of that business? Have you ceased using a particular platform? e.g. moved off OpenVMS e.g. moved off NonStop S-series to NonStop Blade But you still need to be able to retrieve years of archives Common solution: Keep running the old application & platform w/o new data input Maybe just start up the application or power on the system for historical queries But have to pay for application licenses, O/S licenses, system maintenance, etc. EXPENSIVE! 25
Active Archiving Common solution to archiving challenges & application retirement Avoids tape media hassles and expense Avoids need for media tiering (e.g. EBS offload) with virtual tape Avoids issues with application version compatibility with archives Enables moving off an application or platform Provides uniform real-time query capability into all archives Excellent ROI for application retirement Eliminates license fees for old applications Eliminates need to keep old platforms running & maintain them Operational staff savings 26
How to start modernizing your archiving Talk to your Compliance department Find out all of the records retention rules and regulations Is an Active Archiving product installed in your company? Many companies implement a central archiving system shared by departments Even if not, homework on product selection probably already done Identify any candidates for application and/or platform retirement High economic payoff - can easily justify Active Archiving implementation Work the business case Business unit responsible for application may not want to fund this But approach in conjunction with Compliance department, who will push for it Find a partner work on Active Archive implementation ETI-NET is developing Active Archiving connectors for NonStop 27
How to start modernizing your archiving (contd.) Need connector for current data from NonStop Synchronous for real-time feed of application data as SIPs to archive storage But can the feed keep up with application s data rates? Or asynchronous for batching application data as SIPs to archive storage Also need connector for prior archive data backed up by NonStop Asynchronous Read old archive tapes for backed-up data Translate to SIPs and feed to archive storage Could do this off-line from NonStop systems Need access control lists (ACLs) Need queries appropriate to data 28
Summary - Evaluation Review your approaches to: Availability Keep services available through system outages or site outages Usually most effective approach is replication (RDF, ShadowBase, etc) Restoral Deals with lost or corrupted files Or bringing system back up Most effective approach is (relatively) frequent backups to (virtual) tape Short retention requirements (days, weeks) Archiving Dictated by legal, regulatory or corporate policy Complex requirements and how to apply to prior archives Longer-term retention (years) 29
Summary - Modernization Evaluate your current archiving methodology Whatever your process it isn t foolproof! Audit your archives to make sure that they are all accounted for Understand the actual company/legal/regulatory requirements Can they be met with the current archiving process? Particularly data destruction, access control and chain-of-custody Are sensitive fields in old archives encrypted? If didn t use current levels of protection, those archives may be vulnerable to exposure Are there any zombie applications or platforms that should be retired? Active Archiving can pay for itself by savings from such retirement Active archiving puts all archives on-line, secureable, managed & queryable Find out whether your company has adopted Active Archiving for other systems Consider using connectors to pipe NonStop archives into it Other benefits include access audit trail, ability to encrypt fields, data analytics 30
References OAIS reference model: http://www.oclc.org/research/publications/library/2000/lavoie-oais.html http://en.wikipedia.org/wiki/open_archival_information_system Gartner Magic Quadrant: http://www.gartner.com/technology/reprints.do?id=1-2ibhg9g&ct=150623&st=sb InfoArchive (EMC): http://www.emc.com/content-management/infoarchive/infoarchive.htm Informatica Data Archive: http://www.informatica.com/products/data-security/data-archive.html InfoSphere Optim Archive (IBM): http://www-03.ibm.com/software/products/en/infosphere-optim-archive HP Structured Data Manager: http://www8.hp.com/us/en/software-solutions/application-database-archiving/ 31
Thank you Contact: Phil Menzies phil.menzies@etinet.com 32