Media Cloud Building Practicalities MEDIA-TECH Hamburg May 2012 Damien Bommart Director - Product Marketing www.fpdigital.com
Agenda Front Porch Digital DIVASolutions Family What is a Media Content Storage Management (CSM) system? Media Storage Technology Choice (Memory/Flash, Hard Disk, Optical, Tape)? Media Storage Format Choice? What about Cloud storage for Media? Media Cloud Services Evolution? 2
Our DIVAsolutions Family LYNX DIVAsolutions in the Cloud DIVApublish DIVArchive DIVAnet DIVAdirector DIVApublish Big Data Services SAMMAsolo SAMMArobot Online Publishing DIVAsolutions family as a hosted Service in the Cloud Content Storage Management Direct multi-format, multiplatform content publishing Leverage all CSM stored content Service currently in beta IT-centric Disaster Recovery, CSM and MAM as a Service CSM is the heart of the file based to distribute and market content Provision services on the fly Analog to Digital facility DIVArchive is the first CSM solution Simple end user access to all file based assets Leverages AXF for true asset protection 10 Legacy film and video tape asset migration/ preservation Fastest, Cost efficient Full SD and HD support Preservation and mezzanine formats generated in parallel with full AXF support Interface to ALL media devices and storage technologies 100s of Petabytes currently under DIVArchive management Global facility connectivity and distribution via DIVAnet Most efficient and scalable enterprise class workflow High SLA, limitless scalability and globally distributed Big Data centric service for a fraction of the typical costs of S3, Azure, etc. 3
Content Storage Management (CSM) CSM is an object aware, intelligent storage management solution CSM is the middleware which resides between any devices which produce or consume file-based content and commodity storage CSM is VERY different than HSM solutions (Oracle SAM-FS, Quantum StorNext Storage Manager, IBM Tivoli Storage Manager, etc.) which do not handle content as objects CSM solutions in media applications offer content aware features such as transcoding, timecode partial restore, asset/object analytics, etc. CSM offer direct integration with business systems to fundamentally link processes, resources and metadata with the content repository 4
Content Storage Management Source/Destination Devices Human/Business Systems Source/Destination Platforms Live Feed capture Ingest Servers Playout Servers MAM Systems PAM Systems News Systems Mobile OTT OVP Systems IT Servers Editing Systems CIFS, NFS, FTP, etc. Web Portal Traffic Systems CIFS, NFS, FTP, etc. Web VOD Systems CIFS, NFS, FTP, etc. Device Interface Lifecycle Mgmt Analytics Services Fixity Services Xcode Services Migration Services Partial Restore AXF Interface Web Services Storage Interface Data Movers Data Movers Data Movers CSM Asset Database 5 Nearline Storage Fast Access (SAN, NAS, DAS) Nearline Storage Slower Access (SAN, NAS, DAS) Commodity IT Storage with Varied QoS Archive Storage Limitless Scalability (Data Tape, Cloud) Offline Storage (Offsite Media)
Content Storage Management Transmission Newsroom Post Production Daily Ingest Online Publishing Migration Disaster Recovery 6
Storage Pyramid ONLINE DIRECT INTEGRATION IMMEDIACY OF ACCESS CSM NEARLINE ARCHIVE OFFLINE VALUE (STORAGE CAPACITY VS COST) 7
Storage Format Choices Hard Disk Storage Flash Storage Optical Storage Data Tape Storage Pros Fast transfer speeds Random access Random access Fast transfer speeds Good storage density Persistent storage No contact read/write Storage density Random access media Low energy costs Acquisition format Rugged media Fast mechanical times Acquisition format Emerging holographic Cost per TB Multiple R/W streams Fast access Low replication costs Extremely portable Rugged Expansion costs Low replication costs Cons Tied to host chassis Limited read/write Low storage density Sequential data access Not portable Cost per TB Low transfer speeds Head and tape wear Cost per TB Proprietary formats Questionable shelf life Slower access Cost of ownership Low transfer speeds Portability Single R/W stream Highly mechanical Low storage density Single R/W stream Expansion costs High replication costs 8
How Are My Valuable Assets Stored? We must take steps to ensure long term accessibility to valuable file-based assets stored in the CSM Key goals of the ideal storage format include: o Ensure long term accessibility o Self descriptive assets and self descriptive storage media o Preservation (Open Archival Information System - OAIS) features such as fixity, provenance, etc. o File encapsulation to wrap related metadata and files o Scalability for any number of elements of any size and type o Standardized regardless of storage media technology o Transportability and compatibility between systems What choices are there? 9
What About TAR? Tape ARchive (TAR) format has been around for many decades Despite following established standards there is no true universal TAR implementation TAR is a legacy format which cripples most of the core values of a Content Storage Management system (e.g. No Partial Restore) TAR does not maintain on-media catalog for content stored TAR does not include resiliency feature TAR does not include long term preservation features as defined in the Open Archival Information System (OAIS) model 10
What About LTFS? The Linear Tape File System (LTFS) is a simple file system for linear data tape LTFS makes data tapes appear as removable storage LTFS is very useful as a physical file-based transport mechanism but not for long term storage or preservation LTFS offers no media encapsulation and relies on simple folder hierarchies to form important asset relationships LTFS cannot scale as it has no support for spanning across storage media limiting file collection sizes LTFS can only support modern data tape technologies and is not applicable to other storage technologies 11
Archive exchange Format (AXF) AXF was developed by Front Porch Digital and provided to the community for standardization and wide-scale adoption AXF is a fully self-contained, self-describing file storage "container AXF is IT-centric and not tied to media applications only! AXF is based on an innovative file system per object approach AXF fully supports all storage technologies now and into the future! 12
What is AXF? Metadata File Payload Ancillary Files Structured Asset Components Unstructured Proprietary Open Preservation Elements Access Control Provenance Fixity Context Reference File System Universal Storage Agnostic File System AXF Object 13
AXF and Storage AXF is storage technology, operating system and file system agnostic Future Storage Technologies Spinning Disk (NTFS, FAT, etc.) Solid State Disk (NTFS, FAT, etc.) AXF Object Holographic (UDF, etc.) Blu-Ray (BDFS, UDF, etc.) Flash Media (NTFS, FAT, etc.) Data Tape (Block Based, LTFS, etc.) DVD (UDF, etc.) 14
AXF Benefits AXF Objects can scale to any size and encapsulate any number of files with full support for media spanning No need to upgrade existing storage infrastructures (e.g Tape drives) AXF guarantees long term compatibility and resiliency with selfdescribing features for both AXF Objects and AXF Media AXF overcomes all the technical, operational and functional limitations of TAR and LTFS AXF is an IT-centric implementation and is not limited to media files alone (documents, imaging data, etc.) AXF satisfies all of the previously defined "ideal storage format" goals 15
www.openaxf.org OpenAXF.org is the community portal for the AXF initiative The website includes news, documentation, videos and whitepapers providing technical details on AXF This is the focal point for the AXF community o please sign up now! AXF standardization work is well underway within SMPTE so please visit www.smpte.org to track its progress 16
What about the Cloud? Amazon, Microsoft and many others offer scalable cloud based storage services All claim reduced capital costs and high availability via globally distributed data centres Unfortunately these solutions are not tuned for big data Amazon Simple Storage Service (S3) cloud storage costs: o $0.125/GB per month to store data o No charge for archive transfers (uploads) o $0.10/GB for restore transfers (downloads) 17
Can Cloud Storage Work? Commodity IT-centric cloud solutions such as Amazon cannot work in big data environments like those typical in the media industry However CSM-based solutions in the cloud are now available and do make real economic sense These solutions offer cost effective big data storage as well as the content-centric features of CSM as a service How do different cloud services topologies fit and can the transition be handled in a staged manner? 18
Offsite Media Storage? Customer Facility Offsite Storage Facility PROS Cost effective content protection Data tapes can be shipped daily offering rapid protection FedEx offers fast transfers! Can scale to any size with little additional investment Capital costs are minimal Workflows are very well known CONS Workflows are human intensive Recall times can be excessive Tapes on the shelf are not active and must be managed over time Does not offer any redundancy to the main facility Main facility disaster can take a significant amount of time to recover from 19
The Private Cloud Customer Facility A Customer Facility B WAN PROS Private, secure and protected paths Controlled and fairly predictable workflows Business continuance possible Shipping of high density data tape media is FAST and INEXPENSIVE 20 CONS Two or more facilities necessary Infrastructure, capital and staffing costs are simple multiples Technical challenges are multiplied by the number of facilities Heavy capital and operations costs
The Hybrid Cloud 21 Customer Facilities PROS Protected and encrypted paths Controlled and predictable managed workflows Business continuance as a service possible Shipping of high density data tape media is FAST and INEXPENSIVE Services can be tuned on demand WAN CSM Cloud Service Provider CONS Content must be stored in a hosted environment Security is a key concern which must be carefully managed
CSM in the Cloud CSM Appliance Customer Facilities Cloud Service Facilities PROS CSM as a service Protected and encrypted paths Controlled and predictable managed workflows Business continuance as a service Services can be tuned on demand Global distribution and diversity Perfect for elastic applications 22 WAN CONS Content stored in a hosted environment Network costs can be prohibitive High availability WAN must be part of the solution Local caching may be necessary to ensure rapid local access
Cloud Services Evolution Disaster Recovery and Global Replication Content Distribution Services The Connected Enterprise Business Continuance as a Service Online Video Publishing as a Service CSM in the Cloud Elastic Transcoding Farm Elastic Facility Services Storage Migration Services Global Media Asset Management (MAM) Metadata Mining as a Service Hub and Spoke Distributed Facility 23
Thank You! MEDIA-TECH Hamburg May 2012 Damien Bommart Director - Product Marketing www.fpdigital.com