MAY 30, 2015 TECHNICAL REPORT What to Consider When Building an Archiving Solu on for 4K UHD Media Workflows CLOUDIAN AND QSTAR TECHNOLOGIES
CONTENTS Introduc on... 3 Sec on 1: Evolu on of Content Resolu on... 3 1.1: 4K Digital Workflow Challenges... 4 1.2: Ac ve Archive... 6 Sec on 2: 4K Video Archiving Solu on... 7 2.1: Cloudian HyperStore Integra on with Qstar Archive Manager... 7 2.2: Cold Archive to S3... 8 2.3: Further 4K Workflows... 9 Conclusion... 10 2
INTRODUCTION Digital assets in broadcast and media environments can generate revenue for decades, but if they become damaged or lost, they are either impossible or very expensive to recreate. As organiza ons in the broadcast and media industries adapt and re invest for 4K or higher resolu on produc on, older workflows may no longer be as efficient as they once were. Ensuring the long term protec on and security of all high value, high resolu on assets is essen al. Cloudian and QStar Technologies regard data protec on and easy access to data files as key requirements for organiza ons wishing to maximize their use of these valuable assets. To accommodate larger volumes of data necessitated by 4K files and higher resolu on equipment, the ability to easily scale workflows, is paramount. Cloudian and QStar simplify the integra on of self healing storage into exis ng Media Asset Management (MAM) environments. The self healing ability of Cloudian HyperStore features a clusterwide data consistency and integrity check on all files upon disk failure, node failure, or so errors. The QStar Archive Manager so ware provides a NAS style archive gateway to Cloudian HyperStore s S3 based object storage pla orm. Archive Manager can be installed on physical or virtual Windows or Linux servers and connects via S3 to Cloudian HyperStore. Using either MAM so ware or standard file system search engines, assets can be easily located and retrieved to primary storage for re use. Archive Manager ghtly integrates with any MAM using FIMS based (Framework for Interoperable Media Services) web service Applica on Program Interfaces (APIs). SECTION 1: EVOLUTION OF CONTENT RESOLUTION The demand for higher resolu on video content, in movie and television broadcas ng industries, as well as for commercial and ins tu onal use cases, is leading to the next landmark in the Broadcast and Entertainment market. The game changing switch from analog to digital and the associated digital compression op ons allowed standard defini on television (SDTV) to be replaced by high defini on television (HDTV). Flat screens and projec on systems using 720p (1280 x 720) and more recently 1080p (1920 x 1080) drove the industry to create content in these preferred formats. Figure 1, depicts the evolu on hardware dimensions from the early SDTV to today s Ultra High Defini on (UHD). Many theater companies have already switched from film projec on to digital projec on, projec ng 2K (2048 x 1080) or 4K (4096 x 2160) content which provides a more life like experience for the moviegoer. This has also led to a new Ultra HD Blu ray standard for distribu ng 4K high resolu on movies with 4K Blu ray players available in the la er part of 2015. Also, Blu ray is s ll trying to catch up with 4K TVs that have been around since 2014. Some TV programs are already being broadcast in 4K. 3
Figure 1: Evolu on of video hardware Source: h p://www.crea vewaveidaho.com/blog/future coming are your marke ng videos ready 1.1: 4K DIGITAL WORKFLOW CHALLENGES The switch from high defini on (HD) to UHD, or 4K, significantly increases the data capacity and transfer performance requirements of the video edi ng system. 4K content is 7x larger than 720p and 4x larger than 1080p, hence the need for larger capacity Blu ray technology to distribute this media content. The same applies to broadband network performance where content is streamed from video/movie providers like Ne lix or Amazon Instant Video. Without today s improved broadband speeds, 4K delivery to home TV users would be impossible. To create 4K content requires a steep technology update: 4K cameras, faster video edi ng so ware, fast and expensive primary storage, bleeding edge computers with improved graphics, and a fast internal network to transmit data. NAS or SAN based workflows can be used but require Ethernet speeds of 10Gb/sec and Fibre speeds of 16Gb/sec. For example, a completed 4K movie file requires 1.7TB per hour and a transfer rate of 506MB/sec if uncompressed. (Table 1) Table 1: Data transfer rates for a 4K movie file Resolu on 4,096 x 2,160 Pixels in Frame 8,847,360 Frame Rate Bit Depth 30 frames per second 16 bits Pixels per Second 4,246,732,800 Bytes per Second 530,841,600 TeraBytes per Hour 1.7TB per hour 4
You can easily see how the total storage capacity, using 4K cameras, creates raw data content that could easily exceed the 1PB requirement, as quoted for the film Avatar in 2010. That par cular film was shot using 2K 3D. It is reported that the sequels to the movie Avatar will be shot using 4K at 48fps (50% more than the example above). TV series such as the BBC s Natural History Unit (NHU), is already filming using 4K cameras even though they have no way to transmit at this resolu on. Frame rates are also very high to capture the wildlife movements and low light se ngs (some mes as high as 150fps). Coupling this with the sheer quan ty of footage, the BBC NHU can shoot over 300 hours for a single hour used, the total storage capacity required becomes immense. New camera standards are appearing for even higher resolu ons including: 5K, 6K, 8K, and even 9K. (Figure 2) The applica on here is to shoot footage at these higher resolu ons and reduce the actual end product to 4K. This allows for a digital zoom to reframe footage or to remove ji er for cameras that were moving during filming. Consequen ally, this leads to even greater storage capaci es of raw footage. 9K footage is over 31 mes larger than that filmed at 1080p resolu ons. Figure 2: Future resolu on standards for video produc on Source: h p://www.footagebase.com/wiki/red_digital_cinema_camera_company 5
1.2: ACTIVE ARCHIVE Workflows within a video edi ng environment are rapidly changing to adapt to the change in technology. In the past, data storage workflows consisted of a fast edi ng environment and a tapebased archive environment. The edi ng environment typically consists of parallel (clustered) SANbased file systems of which there are many, with Mac, Windows, or Linux based clients accessing and edi ng content. Once the edi ng process was completed, projects were archived typically to tape for long term preserva on. This tape based workflow worked because it was an inexpensive long term storage solu on, as well as files could be retrieved when content was required for re edi ng. But in today s 4K workflow, a middle er is required where content that is not being ac vely used is relocated to lower cost storage but that can be retrieved on a file by file basis in a more efficient manner than tape. This is where archive solu ons come in to the environment that leverage Object Storage as a lower cost storage repository for data that is needed on a more short term basis. Files can then easily be moved between the performance er and the ac ve archive er before eventually being moved to the Cold archive for long term preserva on. QStar Technologies provides a file based data migra on (Network Migrator) and archive gateway so ware (Archive Manager) that integrates with MAM solu ons or uses file system stub files to relocate files to lower cost archive storage. QStar provides two so ware solu ons that work separately or together for the media workflow. The first approach uses archive aware applica ons such as MAMs and DAMs (Digital Asset Managers) to iden fy content as ready to be archived. Files are moved to QStar s Archive Manager, running on a Windows or Linux server, through a disk cache that maintains all file metadata and then to Cloudian HyperStore storage using their S3 API (Amazon Simple Storage Service Applica on Programmable Interface). Files are converted to objects of an op mal size for Cloudian which involves packing mul ple small sized files into a single object or using mul ple objects for larger file sizes. These op mal object sizes allow Cloudian HyperStore to store and retrieve objects at their maximum throughput. Figure 3. Figure 3: The combined solu on of QStar and Cloudian, solve the challenges of 4K workflows 6
Archive Manager has a set of FIMS compliant web services that MAMs can use to be er interact with the archive. Elements of the file can be retained in cache which allows for previews. Reten on management policies can be set to prevent accidental or malicious dele ons or over writes. Pre fetch policies can be included to force a set of files to be retrieved when any one file in a subdirectory is retrieved. For organiza ons that require data to be stored to two different technologies, QStar provides Archive Replicator. This upgrade to Archive Manager writes synchronous copies to two or more technologies such as Cloudian HyperStore and a remote Long Term File Storage (LTFS) tape library. The second method of using an ac ve archive is to use a data mover such as QStar s Network Migrator to monitor the primary file system and automa cally migrate assets to Archive Manager and then to Cloudian HyperStore. In many instances, a stub file is le behind that can retrieve the asset when it is accessed and needed for edi ng. Network Migrator uses policies to select the required files and specify when migra on should occur. Other policies can be created to copy data periodically; however, if the file is unchanged Network Migrator knows it already has this file in the archive. In this way all assets on primary storage can have a copy in the archive as an insurance policy copy. SECTION 2: 4K VIDEO ARCHIVING SOLUTION The QStar Archive Manager so ware is easily integrated with an organiza on s network environment and can be seamlessly combined with popular digital asset management and media asset management systems. Archive Manager also provides simple access to Cloudian HyperStore without API support. 2.1: CLOUDIAN HYPERSTORE INTEGRATION WITH QSTAR ARCHIVE MANAGER Cloudian HyperStore for archiving is desired because the data can be instantly available or ered to Amazon S3, Amazon Glacier, or another Cloudian HyperStore server. Figure 5. By using commodity hardware and choosing your own high density disk drives, the cost of the solu on can be easily managed. Cloudian and QStar provide self managing, self healing storage in exis ng MAM environments, and op onal data ering to any S3 cloud makes long term storage easy. Figure 4. Cloudian HyperStore also supports the versioning feature of the S3 API. With versioning enabled, users can access different revisions of the same file stored on the system with a S3 client. Figure 5: Cloudian and QStar for self managing, self healing storage 7
Today s capacity disk drives are so large that RAID no longer works effec vely. Rebuild mes for large capacity drives in a RAID set are so long that the chance of a second drive failure, before the rebuild is complete, becomes quite likely. RAID has tried to adapt by adding addi onal layers of redundancy but the performance of the storage is always affected during drive rebuild. Object Storage provides a be er method to support large capacity drives without RAID crashes due to mul drive failures and without a significant drop in performance. Cloudian HyperStore provides a scale up and scale out infrastructure allowing organiza ons to create almost unlimited capaci es in a single storage system with appropriate access speeds for their applica ons, but at near tape prices in terms of storage and transfer costs. Cloudian HyperStore data can be stored in erasure code or a replicated format. Both op ons remove the need for RAID technology. Replica on results in greater performance but less usable storage capacity. Erasure code provides the most cost effec ve way to store data because the data is stored in smaller chunks across the cluster. However, erasure code requires more system resources which leads to greater latency on storage requests. Choosing the right storage type will depend on your storage size and needs. If the media content stored on the cluster is meant for archiving and not accessed frequently, erasure code will be the best choice. If the media content requires the greatest performance and replica on size is not a factor, replica on is the best choice. An ac ve archive environment using Cloudian HyperStore can adapt over me, requiring no fork li upgrades. As new server and disk storage becomes available and naturally cheaper per TB over me, they can be added to the exis ng environment. Older processing and storage units can be replaced as they become inefficient and expensive to maintain. 2.2: COLD ARCHIVE TO S3 Cloudian HyperStore together with the Amazon cloud provides a hybrid cold storage archive solu on for 4K workloads. On premises HyperStore so ware integrates seamlessly with the Amazon cloud for scalable and cost effec ve hybrid cloud deployments. Figure 5. Dynamic migra on from on premises physical storage to off premises cloud storage allows near infinite capacity scaling to meet the security requirements of media and broadcas ng environments. This solu on ensures the security and control of data for business cri cal digital video workflows, as well as a low cost, offsite long term data archiving solu on Figure 5: Cold Archive to S3, Amazon 8
2.3: FURTHER 4K WORKFLOWS Many movie producers and broadcasters post produc on services are geographically dispersed and/or have many teams that need to have access to the same data files at the same me. These workgroups need to share and collaborate on content at the same. While the data is being accessed by many users, there should be ability to access and backup data. Produc on houses and broadcasters want to enhance sharing across these remote loca ons and workgroups, protect and secure the data, and with central IT administra on and control. Cloudian HyperStore storage provides a central and trusted S3 compa ble, backup and file sharing server for remote workgroups. The Cloudian HyperStore solu on combines backup and disaster recovery by crea ng two copies of the backup: one in a local data center and another in a disaster recovery loca on. The solu on also offers deployment choices that can be a fully private cloud backup where backup data stays within the firewall or a public cloud backup where all the backup data is stored in a service provider loca on. Also, Cloudian HyperStore offers a hybrid cloud backup solu on where one copy is kept locally and the second copy is stored at a service provider loca on. Figure 6. Figure 6: Hybrid Cloud 9
CONCLUSION As organiza ons in the Broadcast and Media industries adapt and re invest in 4K or higher resolu on produc on workflows, older workflow infrastructures may no longer be as efficient as they once were. To support the exponen al growth in file sizes created and the increased performance requirements for edi ng 4K content, from a storage perspec ve, the industry is being pulled in two direc ons. For edi ng, the fastest disks (o en SSD based) are required, but the cost per TB is extremely high. For archive, huge capaci es are required where cost per TB needs to be kept to a minimum. The addi onal considera on is that another storage layer is required which creates a staging area for content that will need edi ng in the near term, but not immediately. Therefore, a fast archive environment that automa cally protects content without backup, at near tape costs per TB, is desirable. An ac ve archive using Object Storage provides this er of storage while also providing op ons for addi onal ering to the cold archive using public cloud or tape libraries. QStar Archive Manager and Cloudian HyperStore combine to offer a NAS based workflow that integrates seamlessly with MAMs or DAMs and creates pools of storage to segregate content. Cloudian HyperStore constantly monitors content and automa cally takes correc ve ac on to ensure prescribed levels of protec on are constantly maintained. Leveraging the highest density disk drives and allowing organiza ons to select the number and type of processing nodes, Cloudian HyperStore provides both scale up and scale out infrastructure op ons. As digital content is the lifeblood of all organiza ons in the Broadcast and Entertainment industries, where ideally they mone ze content for decades to come, preserva on of raw 4K and higher resolu on footage without compression is paramount. For more informa on about Cloudian product offerings, visit h p://www.cloudian.com. For more informa on about QStar Technologies product offerings, visit h p://www.qstar.com. Cloudian, Inc. Metro Center Tower 950 Tower Lane, Suite 300 Foster City, CA 94404 Tel: 1.650.227.2380 Email: info@cloudian.com www.cloudian.com 2015 Cloudian, Inc. Cloudian, and the Cloudian logo are registered trademarks or trademarks of Cloudian, Inc. All other trademarks are property of their respec ve holders. 10