The Cloud. What does it mean for Media Archives? Presented by Christopher Luther, SGL

Transcription

1 The Cloud What does it mean for Media Archives? Presented by Christopher Luther, SGL

2 Overview TradiBonal digital media archives The Public storage cloud for archives The Private storage cloud for archives Archive as a service - AaaS LTFS

3 TradiBonal digital media archives Benefits

4 TradiBonal digital media archives Drawbacks

5 TradiBonal digital media archives Key technologies A solware based solubon built on a distributed service architecture Grows with demand in storage capacity and bandwidth Presents different storage media (disk, tape etc.) to the outside world as just the archive Powerful APIs to provide users and vendors with access to the archive

6 The Public cloud The clouds are coming! already here!

7 The Public storage cloud Pros Offers great freedom Dabbling/flirBng Comfort of familiarity Easy collaborabon

8 The Public storage cloud SBll evolving Security Reliability Scalability Cost

9 The Public storage cloud Other concerns Bandwidth limits will hurt adopbon Cloud vendor lock- in Customer ignorance of supplier policies Regulatory compliance and data integrity

10 The Public cloud in broadcast today Project based work of broadcast suits the cloud CollaboraBon across sites & projects Low data payloads Graphics, metadata Project rendering, proxies Can the current cloud infrastructure cope with industry demands for increased resolubons?

11 The Private storage cloud for archives Pros Familiarity Control Ease of expansion Capacity Bandwidth Resilience Green

12 The Private storage cloud Components Storage Disk Tape Host servers Database Redundant servers Data movement MigraBon Proxy generabon

13 The Private storage cloud Advantages Flexibility Intelligent scheduling PrioriBzaBon High- volume, easy DR

14 Archive as a Service - AaaS For mulbple departments Transmission News & Sport ProducBon external graphics Production NEWS&SPORT transmission Graphics For external facilibes

15 Archive as a Service ConsideraBons Secure enough? Room for expansion? Off- the- shelf IT components Ability to add new control applicabons ExisBng data migrabon? If it s well designed, your archive can become a profit center

16

17 Linear Tape File System Overview Provides interoperability Standard hardware Self- describing media Plug & play, drag & drop Drivers for mulbple OSs AgnosBc

18 Linear Tape File System Standalone Appears as directory on the desktop Simply drag and drop files to and from tape Independent of any hardware or solware Easily share content

19 IngesBng into the archive

20 ExporBng from the archive

21 Other workflow benefits Film Footage shot on disk on locabon can be backed- up to LTO via a stand- alone drive Content delivered directly to the archive is made available to a broad range of users Open standard of LTFS provides true longevity for archive content ProducBon Free up valuable field acquisibon storage devices, P2 cards etc. Post ProducBon Complete projects can now be backed- up to a single tape Smaller scale facilibes can start with a small LTFS solubon

22 The future LTO has a stable roadmap LTO6 due in early 2013 Other technologies FIMS Framework for Interoperable Media Services Gathering momentum AXF Archive exchange Format New broadcast wrapper

23 Conclusions Cloud is fast- moving Public cloud has its uses, e.g. collaborabon, approvals For media archives, private clouds more suitable (at the moment) Private cloud archives will become profit streams

24 Thank You!

25 SMPTE Meeting Presentation The Cloud What does it mean for media archives? Nik Forman, Marketing & Branding SGL, Christopher H. Luther, Director of Professional Services SGL, Rajesh A. Patel, Product Manager SGL, With thanks to Howard Twine Written for presentation at the SMPTE 2012 Abstract. Perhaps due to the close proximity IT companies now have to the broadcast industry the usage of the term 'Cloud' is becoming more and more prolific. Is this just another buzz word or are there real benefits to embracing the Cloud? This paper looks at the conventional media archive and discusses the opportunities and emerging trends for archive technology and the cloud. Some of the discussion will be about the enabling technologies like LTFS, as well as the emergence of Archiving as a Service. Additionally this paper will address the considerations to build a private 'cloud' archive in terms of users and growth expansion.. Keywords. Cloud, Archive, LTFS, LTO, DR, The authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the Society of Motion Picture and Television Engineers (SMPTE), and its printing and distribution does not constitute an endorsement of views which may be expressed. This technical presentation is subject to a formal peer-review process by the SMPTE Board of Editors, upon completion of the conference. Citation of this work should state that it is a SMPTE meeting paper. EXAMPLE: Author's Last Name, Initials Title of Presentation, Meeting name and location.: SMPTE. For information about securing permission to reprint or reproduce a technical presentation, please contact SMPTE at jwelch@smpte.org or (3 Barker Ave., White Plains, NY 10601).

26 Introduction There has been much buzz recently about how the cloud is the panacea to the storage crisis that eventually affects all Broadcasters and Media & Entertainment (M & E) companies. Much of this comes from the success of IT projects that are using distributed computing and cloud storage to lower costs and provide enhanced functionality to their users. Some of these technologies on the surface look similar, but do not address the unique requirements and demands of creative workflows. There is a place for the cloud in M & E workflows but is the media archive the most efficient use of this technology? Cloud storage comes in three flavors: public, private and hybrid. There are benefits and drawbacks to each and careful consideration of how and where they are implemented, and the complimentary technologies on which they may rely, are crucial to future success. Traditional digital media archives A traditional media archive, i.e. one that consists of standard storage technologies managed internally by an organisation, provides a number of benefits: it makes use of industry-specific technologies which are familiar and proven-to-work (e.g. PFR, formats, codec awareness); it might be standardized (file formats, standard adoption etc.); it is quantifiable (easily measured in both capacity and costs); and it is secure, using the same security technologies that are applied across the organisation. However, a traditional archive also has drawbacks: sharing of data across applications usually requires manual intervention; interfaces of control and management applications tend to be hard-coded; and there are single file and wrapper formats for each platform that the archive has to support. There are several key technologies that act as enablers for the traditional digital archive. It is generally based around software built on a distributed service architecture, which allows the archive to grow as demand for storage capacity and bandwidth increases. If implemented correctly, the archive presents its different storage media (e.g. tape and disk) to all departments simply as the archive ; in most cases, day-to-day users may not even know that they are accessing data that is stored offline. In addition, the software-based solution at the heart of the archive often provides powerful APIs that allow users and vendors direct access to the command set of the management application, and in turn direct access to the archive and restore functions and content from their own application layers. In summary, one can think of traditional broadcast archives as a closed system, with high security but a lack of flexibility in some aspects. Is there a solution that provides the same levels of security, but with the added flexibility required in the modern broadcast arena? The Public storage cloud for archives The generic term cloud storage encompasses three actual platform architectures: public, private and hybrid cloud storage. Public cloud storage involves the storage of data and content exclusively outside of the users organisation, on storage that is hosted and managed by a thirdparty provider. Content in the cloud is accessed by the organisation via high-speed wide area network connections. There are some workflows that are making successful use of the cloud both for storage and distributed processing. Some of these include Chyron (Axis World Graphics), Aspera (file transfer and delivery) and Avid (Interplay Sphere). Though not media archives they do illustrate how some M & E processes are destined to become cloud based. 2

27 For traditional archives (and indeed all technologies) the public cloud carries its own benefits (and drawbacks, of course). The consumer-driven nature of cloud computing and storage means that it offers great freedom: in many areas it is free to try (to a certain storage capacity) which makes it more appealing for those who want to trial technologies before purchase. In many cases, after a time users can then find themselves integrating these technologies into their everyday processes, almost without realizing. Indeed, certain aspects of the consumer cloud have been with us so long that there is already a high degree of comfort with the cloud storage technologies and processes that lie at the heart of their offerings (e.g. YouTube, founded Feb 2005), and many organizations have been using the cloud for their collaborative operations, such as sharing proxies and generating approvals, for many years. However, public cloud storage does have its problems. One of the major concerns highlighted in nearly all polls of prospective adopters shows that security is a major concern 1, fuelled by the high-profile data breaches that are so often reported in the media. A close second to security concerns are reliability issues: the recent outage of Amazon S3 for 8.5 hours provides a very visible, and probably very costly, example, and must raise the question Is it acceptable for broadcasters to be without their archive, without notice, for such extended periods? Scalability and transferability are further concerns will the bandwidth ever exist for transferring and reliably storing 4k content at 24 gigabytes per frame? Last, but a very long way from least, is cost: current average cloud storage costs hover around $400 per month for the same amount of data that can be stored on a single LTO6 cartridge. Further, as well as recognised existing drawbacks to cloud storage, there are a number of risks that are inherent in the existing key technologies that power it. Vendor lock-in is as apposite to cloud storage as to other broadcast operations (in the event of vendor bankruptcy, data transfer or download, if it can be achieved at all, will involve significant cost), as are other vendorspecific risks, for example the customer s lack of visibility of company operational procedures such as hiring policies. While another company may be hosting your data, regulatory compliance obligations still rest with you, the broadcaster; can you be sure that the storage company is adhering to the strict policies to which you have agreed? However, it is clear that, despite its drawbacks, the benefits provided by public cloud storage will mean that its usage in at least some elements of the digital broadcast workflow will continue to evolve. Services and products such as those offered by the broadcast-centric providers of the likes of Adobe (Adobe Anywhere) and Avid (Avid Interplay Central) will proliferate over the coming years. The Private storage cloud for archives What is meant by private storage cloud? Essentially, this takes the traditional digital media archive as outlined above, and expands it across multiple sites to create an archive that services and is serviced by autonomous yet fully integrated individual archive centers built on the traditional disk and tape storage architectures, and all managed by a single control application layer that oversees the whole. Naturally, this model expands on the benefits offered by the traditional digital archive: the familiarity of the concepts and components mean that it is instantly recognizable and implementable, and there is no need for detailed technological research, or incentivization schemes for user buy-in; as the owner and administrator of the storage platform, you the user are able to plan and implement storage policies to ensure that content of specific types is stored on the layer that is most appropriate; expansion of both capacity (extra disks and/or tapes) and bandwidth (more servers, enhanced communications) is easy and controlled in-house; private cloud storage archives also exhibit great resilience, based as they are across multiple sites and 3

28 making use of IT- and vendor-specific clustering technologies, database mirroring and inherent Disaster Recovery procedures and equipment; and finally, archives that employ a tape-based element (as opposed to public cloud storage which is almost exclusively disk-based) are greener. The average 5 year power costs (heating and cooling) of disk-based archives versus tape-based are $51,810 and $7,500 respectively 2. This is nearly 700% higher operation costs for disk than for tape based archives. Of course, private cloud archives do have drawbacks. In addition to those discussed in reference to traditional digital archives (some may remain applicable), the multi-site model does require bandwidth to allow duplication across sites, though generally the infrastructure will already be in place to allow non-archive content transfer between the centers. Also, multiple sites means that dedicated hardware maintenance at each is required. The multi-site model provides a great degree of flexibility in implementation: the same levels of resilience, scalability and cross-pollination of content can be achieved using a variety of architectures (not all individual sites within the private cloud necessarily have to be exactly the same, for instance, though they can be). Generally, however, a basic server and storage model will be used as the basic building blocks. The storage is a mixture of high-speed archive disk for rapid restore, and longer-term, deeper level tape libraries, all of which is under the control of archive management software that processes requests from the supervisory layer applications, such as asset management and automation systems, and instigates archive and restore processes using appropriate resources. The archive management resides on a network of servers, governed by a central database that manages operations both locally and at the other remote sites. To be fully resilient, the software runs redundant services, installed on all servers in the cluster, and monitored by the database; this ensures that, should a service fail, or a server be taken offline for maintenance purposes, other services on working servers can be started to ensure continuation of operations. The servers provide the I/O (input/output) functionality, instigating the data transfer processes from the source to the storage device (or vice versa in the case of data restore). These processes are generally requested by the controlling application (e.g. asset management), but may also form part of the archive s own storage management and data life cycle operations. These operations are archive-specific processes that move and copy data across different storage layers and media types, depending on user-defined rules that ensure that data is always stored at the most cost-efficient level in the archive. These operations are managed by the archive management software, completely in the background and without requirement for user intervention (beyond the initial rule configuration). The archive management software may also perform other tasks, such as the creation of browse proxy versions of content as it is archived. This helps users to correctly identify content in the archive before they restore it. Private data storage clouds give probably the greatest flexibility of the solutions examined so far. Users can control what is virtual and what is physically stored across multiple sites, and the replication and duplication that this system requires can be scheduled for periods when there will be least impact on the rest of the organization. Priorities can be set, in order that content of a certain type is immediately available, and Disaster Recovery procedures become almost inherent to the system, allowing easy duplication of content onto tapes that can be externalized and removed to a secure location. In this manner, multiple terabytes of data can be secured every day, with very few cost implications. Archive as a service As detailed above, a private cloud archive provides a large number of operational, procedural and cost benefits to its parent organization. Internally, a correctly designed and implemented 4

29 archive allows multiple departments to use a single storage plant, with no single department using a storage island to which only it has access. Archived content is presented to the entire organization as any other storage location. However, in addition to the cost benefits that are associated with this streamlining of company-internal content storage and provision, the archive can further be utilized to provide new revenue streams. Several broadcast-specific companies are beginning to offer Archive as a Service (AaaS), providing archive to other broadcasters and post-production facilities on a pay-per-gigabyte basis. AaaS allows broadcast organizations of all types to access the benefits of the private cloud archive without having to create and implement one themselves. The initial capital outlay savings are obvious, but a third party, broadcast specific archive service also means that transmission, news & sport, production and post-production houses can duplicate their content, store it on the most applicable media, free up expensive online storage and integrate the archive into their daily processes, all without having to maintain and operate their own archive. However, isn t AaaS the same as public cloud storage? After all, the service involves a thirdparty organisation storing data for their customers. In essence, this is true, and indeed AaaS is a solution that is offered by enterprise level, IT-based suppliers outside of the broadcast arena. However, the opening up of a private cloud, multi-site archive to other users differs in that it is broadcast specific, and is built upon and controlled by technologies that have been developed specifically for this operation. In addition, the broadcast AaaS is built around disk and tape storage, which is carefully managed to ensure that content is stored as efficiently as possible based upon its usage, age and type. As always, however, there are considerations that should be addressed, both by AaaS suppliers and customers. Are the archive and the underlying structure that supports it secure enough to hold other people s data? Is there sufficient capacity for expansion as customer storage requirements increase? Is it easy to add new controlling systems to the existing archive? Does the archive API allow for migration of existing data when a new system is added? Is there a mechanism (via the API) to integrate billing and other administrative systems into the archive control software? All of these considerations are important, but if you ve designed the initial private cloud archive correctly, chosen the right software vendors and used easy-to-administer, off-the-shelf IT components, then the answer to all of them should be Yes. LTFS (Linear Tape File System) The traditional digital media archive has always been viewed by many as the last port of call in the broadcast workflow. Once a clip has been ingested, edited, transmitted and stored locally for a relevant period, it can be dumped off expensive nearline disk to cheaper disk, or deeper tape storage until such a time as it might be required again, the dust is blown off and it is restored back to the area of the living media at the heart of operations. However, with the changes in content usage across multiple platforms, at a wide variety of resolutions, this linear life cycle is undergoing a radical re-examination as material is constantly re-purposed to create and exploit new revenue streams. As discussed above, the archive is now becoming a transparent storage repository, accessible from all departments and hugely expanding available capacity, whilst providing a mechanism to engage new file-based workflows that override existing conceptions of how content should be used at various stages of its broadcast life-cycle. We ve seen that a private cloud archive can provide increased flexibility, greater resilience and more cost-efficient operations over existing models, but advances in technologies associated 5

30 with the archive itself are also providing surprising new opportunities to take a fresh look at how content is managed at all stages, both inside the archive and out. One of these is the Linear Tape File System (LTFS). This new file system overlies the LTO tape format designed and created by the LTO Program (a consortium of companies including IBM, HP and Quantum), and represents something of a revolution in tape technology. LTFS is an inter-operable format, meaning that tapes written using LTFS are self-contained. In real terms, this creates true portability a tape written by one vendor s software can be read by any other software that is LTFS compliant. Additionally, LTFS tapes can be mounted on a PC using a standalone drive and nothing other than the manufacturers OS drivers, appearing as a directory structure in much the same way as a flash drive is displayed as a series of folders and files. Content can then be dragged and dropped from the PC to the tape, or vice versa, facilitating a drag-and-drop approach to backing up content, though of course this is a manual process and ignores the requirement for metadata capture and media cataloguing. LTFS drivers are available for Windows, Mac OS and Linux platforms, and the technology was recently ratified by SNIA (Storage Networking Industry Association). The portability of LTFS not only provides a simple mechanism for transferring data to tape, but also creates new, more efficient workflow possibilities, both inside and outside of the archive itself. Content can be easily captured and transferred, at any stage of the file-based workflow. LTO tapes are cheap; an LTO generation 5 cartridge is capable of storing 1.5Tb of data and costs around the $50 mark, and the recent upheavals affecting companies with production facilities operating out of Japan has seen LTO become a real competitor to other, more expensive tape formats. This cost-efficiency in content portability, coupled with the open format nature of the technology, means that LTFS is able to provide a quick, easy and extremely effective mechanism for moving content into and out of the broadcast workflow, and not simply by replacing existing workflow components. Broadcast manufacturers have been quick to see the opportunities offered by LTFS, and the last year or so has seen an influx of LTFS based products and solutions within the broadcast market. Not surprisingly, many of these have been offered by tape storage solutions providers, but there have also been LTFS-based products in the areas of video recording and on-site content capture. These new products are presenting broadcast organizations with many new avenues of revenue generation, by simplifying workflows and creating efficiencies in content life cycle. Several archive management software companies have also seen the possibilities offered by LTFS, and have implemented solutions that take advantage of LTFS s properties to facilitate content flow into and out of the archive. Even with the flexibility offered by a private cloud archive as described earlier, situations will arise where previous technology limits would introduce difficulties or would involve more convoluted workflows to reach a conclusion, for example when returning expired rights-based content resident in the archive on proprietary formatted tapes to its owner. In an archive based on LTFS formatted tapes, content can simply be consolidated onto one or more self-contained tapes and handed back to the owner, who can then access that content simply using a standalone LTFS-compliant LTO drive and the manufacturer s drivers. But there are additional benefits that may have a greater impact on traditional broadcast workflows. As stated earlier, the archive has been viewed predominantly as an end-of-the-line repository, where content ends up after undergoing the other phases of its broadcast life cycle. However, LTFS, in conjunction with intelligent archive management software, creates the opportunity for users to adopt complementary procedures. One such procedure is ingesting directly to archive: in this workflow, which is of special interest to news and sports companies, 6

31 content is captured directly to LTFS cartridge. Quick, on-location copies are made by dragging and dropping to a locally attached drive. The tapes are then added to an LTFS-capable archive system, which ingests the content, creates placeholders, and broadcasts the arrival of the new content to the over-arching asset management system via an API-based subscription service. DR copies of the raw content may also be created and exported at this time. The asset management system then has full access to the content as if it had been ingested in the traditional manner. In addition to new methods of transport into the archive, LTFS also makes content export much easier. LTFS copies can be used as shelf DR copies with no special software required to access the content, material can be delivered in bulk to remote offices, and as discussed previously, content can be returned to clients at the end of a usage period. LTFS has the potential to move the archive system from the end to the beginning of the workflow (and to various points in between). For Disaster Recovery, LTFS allows content to be duplicated automatically by an archive management system and removed from a tape library to be stored remotely. For film use, footage shot on location can be backed up quickly and onlocation (if required) from disk to LTO, via a standalone drive or dedicated recorder. Content can then be made available to a wide variety of users. In production, the cost-efficient nature of LTFS allows it to free up expensive acquisition storage, such as P2 cards, and for post, complete projects can be backed up to a single tape; finally, smaller organizations can start with a single standalone drive and expand as required. There are other emerging archive technologies that should be mentioned at this point. The Framework for Interoperable Media Services (FIMS) is still in the early stages of development, and is gathering momentum, but it is still perhaps a little early for application to broadcast archiving requirements. The Archiving Exchange Format (AXF) is close to being ratified by SMPTE, and hopes to provide a means to standardize content within an archive, albeit as yet another industry wrapper that may or may not be required. However, if LTFS, AS02, AS11 and other MXF wrapper technologies cannot deliver what they appear to promise then AXF could well be useful. Conclusion Not much will slow the pace of consumer driven usage of the Cloud as tolerances to outages and quality expectations are lower. However, in conclusion let us look at a couple of telling statistics: It would take 1,400 T-1 lines to equal the IO bandwidth of a single LTO-6 tape drive. To store a single LTO-6 tape on Amazon S3 it would cost over $400 / month plus data transfer charges. Until any drastic changes occur in broadcast workflows, or until everyone has access to unlimited bandwidth and storage for free, then perhaps the best use of public cloud storage today is for lighter data payloads such as collaboration, approvals, targeted or on demand delivery, and light backup. However, it s worth remembering that cloud storage is not just about public clouds. Given the right design, the correct implementation of hardware and software, and using the correct sub-sets of tools and technologies (e.g. LTFS), private cloud storage archives have the ability to not only provide broadcasters and other users with security, efficiency and reliability, but they can also create new revenue streams by turning some established practices on their heads. 7

32 References 1 Hoehn, John A., IBM, Cloud Computing for the Media and Entertainment Industry, < June Covey, Bob, Qualstar Corporation, Cost of Disk-Based vs. Tape-Based Archives, < March