Private Cloud Storage for Media Applications. Bang Chang Vice President, Broadcast Servers and Storage

Transcription

1 Private Cloud Storage for Media Bang Chang Vice President, Broadcast Servers and Storage

2 Table of Contents Introduction Cloud Storage Requirements Application transparency Universal access Locally reliable, scalable, tiered, globally expandable Policy-driven self management Best of class components Cloud Storage Architecture and Components Virtual file system Global namespace Scalable, reliable local storage Usage-based, policy-driven content distribution Remote content distribution XOR Media Cloud Storage Solution Blue Whale File System Universal MediaLibrary Content Delivery Network

3 Introduction With the explosive growth of digital content at record pace, more and more content providers are deploying multiple, geographically dispersed facilities, connected via a WAN (Wide Area Network), for their content creation, production, and delivery workflows. At present each of these facilities is an isolated island, with its own media applications, local storage, and content management. Content providers rely on home grown, manually operated tools to distribute content across these multiple facilities. Not only does this ad hoc approach incur significant operating cost and WAN consumption, it also does not allow content providers to achieve their ultimate goal, which is to access any content from anywhere regardless where the content is located. It s increasingly clear that a new type of global infrastructure is required to consolidate today s isolated islands into one reliable, scalable, global content repository. This white paper discusses the requirements and architecture of such a global infrastructure, Private Cloud Storage, and why XOR Media, with the best-in-class file system and storage technology and years of experience building content delivery networks for the cable and telco operators, is in the best position to deliver such a solution. This discussion is limited to private cloud, as opposed to public cloud, so that we don t have to be concerned with multi-tenant requirements such as accounting, billing, and security. Nevertheless these additional requirements can be addressed by adding more capabilities to a private cloud solution. Cloud Storage Requirements To realize the promise of accessing any content from anywhere, a cloud storage solution needs to meet the following requirements: Application transparency Traditional media applications are only capable of accessing content on local storage via standard file system interface. A cloud storage solution needs to provide a virtualization layer that allows these media applications, without application level modifications, to access content on geographically dispersed storage systems. A cloud storage solution without such virtualization layer will experience limited adoption due to the impracticality of requiring changes to all mainstream media applications. Universal access An application should be able to access cloud storage using the most appropriate access method specific to that application, be it low cost NAS or high performance SAN, on Windows, Linux, or Mac OS, via 1G/10G IP or Fibre Channel network. In other words, it should not require significant changes to the current infrastructure when cloud storage is introduced to replace the isolated local storage islands. Locally reliable, scalable, tiered, globally expandable Cloud storage does not eliminate the need for reliable and scalable local storage systems. On the contrary, a scalable cloud storage solution is built on top of reliable and scalable local storage. A cloud storage solution should allow the content provider to deploy reliable and scalable local storage today and migrate to cloud storage infrastructure tomorrow without ripping and replacing the existing investment. A cloud storage solution should also add another level of reliability and scalability, especially in terms of protection against individual site failures. Cloud storage is also naturally suited to support tiered storage, where frequently accessed content is placed on high performance SSD based storage while relatively cold content on low cost HDD based storage. Policy-driven self management Cloud storage should be self-managed without much day to day manual intervention. Content should be cached, replicated, migrated, tiered automatically based on content access pattern, pre-defined policy, content-aware intelligence, and the ultimate goal of ensuring quality of service while reducing WAN consumption. Best of class components A cloud storage solution should allow the use of best-in-class components under one unified architecture, be it third-party local storage, state-of-theart replication technology, or centralized monitoring utility.

4 Cloud Storage Architecture and Components The following figure illustrates at a high level the component architecture of a cloud storage solution. Local Storage A Local Storage B Global Namespace Content Distribution Local Storage C Virtual file system Global namespace Media applications access the cloud storage via a virtual file system layer. Virtual file system is the key component to achieving application transparency and universal access. Traditional media applications access their persistent content through local file system interface, such as file open, read, write, and folders. By providing the same interface as local file system, virtual file system allows media applications to access geographically dispersed cloud storage without knowing the location of the content. Internally virtual file system relies on a global namespace to locate content and access the content either locally or remotely over the WAN. To achieve universal access virtual file system needs to support: A global namespace is the metadata repository where file names, their attributes, and their locations are maintained. The virtual file system relies on the global namespace to locate a file by its unique name before accessing the file content either locally or remotely. Global namespace needs to be super reliable and scalable, able to function in the case of server or site failures and scale to millions of file names. Creating redundant copies of the global namespace is a good way to protect against server or site failures. Modern hashing technology allows global namespace to locate a specific file name in a few milliseconds even amidst millions of names. Global namespace is also the optimal layer to provide access control and bandwidth management due to its centralized role in the cloud storage architecture. running on Windows, Linux, Mac OS X, and potentially Android Low cost NAS access and high performance SAN access Access via both 1G/10G IP and Fibre Channel network

5 Scalable, reliable local storage Local storage stores the actual content. In the cloud storage architecture there are multiple geographically dispersed facilities and each facility has its own local storage. Content stored on any local storage can be accessed by any media application at any site thanks to the virtual file system and the global namespace. Local storage itself needs to be reliable and scalable. RAID (Redundant Array of Identical Disks), RAIN (Redundant Array of Identical Nodes), and replication are the typical ways to provide data redundancy. High availability controller pair and server cluster are popular technologies to provide server redundancy. Cluster and grid are effective architectures to scale both storage capacity and access performance. Usage-based, policy-driven content distribution In order to ensure quality of service while minimizing WAN consumption, cloud storage monitors the usage of the content and performs content distribution based on user specified policies. For example a simple policy could be: Content frequently accessed by both site A and B will have copies at both sites Content frequently accessed by site A but not site B will be moved from B to A Content expected to be popular will be replicated to all sites Content expected to be accessed frequently will be placed on SSD based storage Remote content distribution Traditional data transfer technology such as FTP is not optimized for content distribution over the WAN. Data transfer rate by FTP over the WAN is relatively low due to TCP s limitation in the presence of long network latency. Reliable UDP based data transfer technology, which can push data close to the wire speed, is emerging as the standard for remote content distribution. Yet more data transfer technology is emerging specifically optimized for cloud storage that allows advanced features such as stream through and access while transfer, further improving quality of service and reducing WAN consumption. XOR Media Cloud Storage Solution XOR Media is in the best position to deliver the next generation cloud storage solution to the media industry. Over the last five years XOR Media has developed and delivered the following products and technology that will ultimately become the key building blocks in our private cloud storage solution. Blue Whale File System (BWFS) BWFS is a virtual file system that supports application transparency and distributed storage access. It scales to millions of file names and 144PB of managed storage capacity. BWFS has been deployed at hundreds of media facilities over the last eight years. Media centric features such as media file defragmentation, client bandwidth management, file change notification make BWFS the ideal file management system for media applications. Supporting Windows, Linux, and Mac OS, BWFS serves as the virtual file system layer in our cloud storage solution. Universal MediaLibrary (UML) UML is the result of our years of engineering effort of adapting state of the art IT technology to support media applications. It s the only storage system in the industry that supports both real time play to air and high performance production workload, the only system that supports simultaneous NAS and SAN access, and the only system that supports 10G IP and Fibre Channel connectivity. IT-based reliability features such as RAID 6 and dual HA controllers and scalable grid architecture make UML the ideal local storage platform today and the key building block for our cloud storage solution tomorrow. Content Delivery Network (CDN) XOR Media was one of the first vendors to jointly architect and develop CDN solutions with the largest cable and telco operators. Today XOR Media has deployed its CDN solution at multiple countries and in support of content consumption by millions of users. Global namespace and usage based, policy driven content distribution are at the heart of our CDN solution and will become the central intelligence system for our next generation cloud storage solution. XOR Media XOR_Media XOR Media PO Box 391, 32 Mill Street Greenville, NH USA T F _2012