Designing a Video Proxy for the Access Network

Transcription

1 Designing a Video Proxy for the Access Network Stijn Eeckhaut, Emmanuel Vanneste, Stijn De Smet, Brecht Vermeulen, Piet Demeester Ghent University - IBBT - IMEC Department of Information Technology (INTEC) Gaston Crommenlaan 8/201, B-9050 Gent, Belgium stijn.eeckhaut@intec.ugent.be, Tel: Abstract Large deployments of time-shifted TV and Video on Demand (VoD) require a distributed architecture with video proxies close to the consumers in order to be scalable. This paper illustrates how we can build such a video proxy without specialized hardware, but rather with components readily available, and with a number of transfer protocols with good open source implementations. Given the large number of video proxies needed in a TV distribution network, this implies a number of benefits while keeping performance and reliability levels at an enterprise level. 1. Introduction Figure 1 depicts a general setup of a TV distribution network. TV can be viewed live from a Broadcast Server, or on-demand from a central Video on Demand (VoD) server which has access to the full catalogue of offered TV programs. In order to be able to provide a rich catalogue, this central server will typically need several hundreds of Terabytes of storage. Current storage solutions for this amount of data typically consist of a Storage Area Network (SAN) based on the Fibre Channel (FC) protocol, and typically use expensive SCSI hard disks as storage medium. Whereas this central VoD server is ideal for not frequently requested programs, it is not for time-shifted TV or VoD of popular TV programs. In this case we will need geographically dispersed video proxies in the aggregation network, close to the customers (caching on multiple levels between the central server and the customers is also possible), in order to optimize bandwidth in the aggregation network and to relieve the load on the central system. These video proxies could also be built with the more expensive components on the market, but there are a number of reasons why it would be interesting to use common hardware and protocols for it. First, the amount of storage needed for a video proxy can be quite moderate. Suppose for example a network provider who wants to store last week s primetime programs (4 hours per day) of the 6 most popular TV channels, together with the 20 most popular films at the moment (each with a duration of 2 hours). This would mean that a capacity of 208 hours is needed. At a bit rate of 4 Mbps (~ 2 GByte/hour) approximately 500 GB of netto storage would be needed to store all this content. For bit streams of 20 Mbps this rises to nearly 2 TB of netto storage. Hard disks of 750 GB are appearing on the market at the time of writing. The throughput of 1 disk will off course not be sufficient for a video proxy, but a system (or combination of systems) with a number of disk drives combined into a RAID array could offer not only the needed capacity, but also sufficient throughput and high reliability levels. Second, a large number of video proxy servers will be needed, so a considerable cost saving could be achieved with a generic storage solution. Further, a Fibre Channel Storage Area Network would be a good idea (and current common practice) to provide enterprise reliability and storage capacity for the central VoD server or a regional (intermediate) proxy server. But Fibre Channel has also a number of disadvantages: high hardware costs, and the fact that special training is needed for the IT staff to master their Fibre Channel skills. These skills are generally not part of the common knowledge in opposition to Ethernet or IP networks. In this paper we will illustrate how a valuable alternative can be built using common (IP-based) hardware. At last, the contents of the proxy will be highly dynamic, and will only consist of a small part of the offered program catalogue. The data preservation strategy for a video proxy should merely be focused on protecting its functioning against smaller failures such as a failing disk. Data preservation against loss, disasters, should be realized on the central VoD system, so appropriate actions should be taken at the central site for that. In the following sections we will describe a number of design decisions to be made to construct a media storage solution with commodity hardware and generally known block and file transfer protocols. The proposals we present can equally be used in systems for digital file-based media production. Figure 1: TV distribution network ISBN13 : Paper Th1A5 Eeckhaut Page 1 of 5

2 2. Designing 1 storage entity: SCSI versus SATA disks Historically, low cost (S)ATA disks have been used in desktop environments, and high reliability SCSI disks have been used in high-end storage systems. SCSI disks have a lower capacity (up to 150 GB), but offer better seek times and better performance due to higher rotational speeds of the disk (up to rpm). They also offer more reliability due to their lower bit areal density (leading to higher drive design operating margins) and extended design verification testing. SCSI disks are traditionally also better armed against rotational vibrations, a phenomenon that occurs when multiple hard disk drives are packaged together and the effect of the seek acceleration is transmitted from the seeking hard disk drive to another hard disk drive that is writing. SATA disks on the other hand offer higher capacities (up to 750 GB) at a lower price, but in most cases also lower reliability and lower performance (typical rotational speed: 7200 rpm). It is however possible to obtain high storage system reliability with moderate reliability individual drives by requiring appropriate SATA drive reliability testing, RAID 1 designs that insure data reconstruction with 2 drives failed (RAID 6 for example), and storage system management of drive failure warnings. A possible procedure for this last item is to make a precautionary backup to a hot spare drive after a warning by the SMART failure prediction system of a drive [1]. Recently the storage manufacturers have also come up with Enterprise SATA disks [2], with the high capacities that can be expected from SATA disks, but with a number of improvements to give them a reliability level which could until recently only be found in SCSI disks. This will further drive the adoption of SATA disks in high-end storage systems. In our experimental comparison of a number of contemporary SCSI and SATA disks we observed that - when the caching policy of the SATA drive is set to write through - the hard disk with the better specs (SCSI) was able to give the best results. However, when the caching policy of the SATA drive is set to write back, both disk types performed equally. Although a write back policy can cause consistency problems in case of a serious failure, this is expected not to be problematic because of the nature of the stored content on a video proxy server. So SATA disks with a write back cache are a good alternative to SCSI disks in this case, both in the field of throughput as in the field of reliability (when the necessary precautions are taken). Typical sizes for a disk cache are 8 and 16 MB. 3. Designing 1 storage entity: combining multiple disks Depending on the type of workload, a single disk is able to offer 5 to 70 MB/s of throughput. In order to offer higher storage throughput and fault tolerance, disks are combined into RAID arrays. Different RAID levels exist, each with their own characteristics [3]. These RAID levels can be implemented in hardware (on a dedicated PCI-X or PCIe card) or in software. This brings us to the following 1 RAID: Redundant Array of Independent Disks. question: in what cases can we justify the extra cost of a hardware RAID controller, and in what cases is software RAID a good alternative? The answer is mainly dependent on the RAID level and the type of disks. Software RAID is most suited for RAID types not requiring high computation levels. RAID 0 for example only stripes 2 blocks over different disks without taking care of redundancy. RAID 1 only creates 2 disks which are exact copies of each other. As a consequence these RAID levels can use software RAID without loading the host CPU too much. These assertions were confirmed during our experiments. Hardware RAID shows its advantage when a computation intensive RAID level has to be implemented. RAID 5 for example not only stripes data blocks over different disks, but also calculates and stores parity information in order to be able to reconstruct the RAID array after 1 disk failure. RAID 6 is similar, but allows 2 disk failures without data loss. Our experiments showed that the CPU load of the host was considerable when using RAID 6 together with software RAID and a sequential access pattern, which proves the necessity for hardware RAID for a heavily loaded RAID 6 system. Hardware RAID will also be needed when the number of disks exceeds the number of connections on the host motherboard. Current RAID controller implementations support up to 24 SATA disks per controller [4]. Another advantage of a hardware RAID controller is its controller cache. A typical size for a RAID controller cache is 128 MB to 1 GB. Using this cache in write back mode, the controller can further optimize write traffic to the disks of the RAID array by regrouping write commands. Our experiments showed that a performance increase was possible for sequential writes using this technique for different RAID levels. In a realistic load scenario, with multiple users each writing a large sequential file to different locations on the storage, we observed that a hardware RAID 5 array (with 3 SATA disks) was able to nearly reach the same performance as a hardware RAID 5 array of SCSI disks, by using the write back controller cache. The same observations hold for a RAID 0 array. So just like using the disk cache of a SATA disk, it seems beneficial from a performance viewpoint to use the RAID controller cache. Of course, we have to make the same remarks on file system inconsistency when using the RAID controller cache. Although this can be partially countered by a journaling file system or file system check, we note again that our system is used to implement a video proxy server, so the stored data are not as valuable as the master copy on the central video server, which is the place where disaster recovery should be implemented. Given the fact that we want to store large video files, we should also consider using large stripe sizes when using a RAID 0, 5 or 6 array. 2 Striping : spreading out logically sequential blocks of a file across multiple disk drives in a round-robin fashion. ISBN13 : Paper Th1A5 Eeckhaut Page 2 of 5

3 4. Designing a system: combining storage entities into a system In one of our experiments we determined the performance of an AMD Opteron 144 system with an Areca ARC1160 PCI-X SATA RAID Controller and 12 SATA disks of 500 GB in a RAID 6 configuration with an xfs file system (5 TB of netto storage). With 100 simultaneous reads and 10 writes to the RAID array we were able to reach 50 MB/s for reading, and 20 MB/s for writing, enough to provide content from hard disk to 100 customers at video bit rates of 4 Mbps, while still writing 40 new items to the cache. If a just requested program can (partially) be kept in main memory on the proxy for a small period of time, it is even possible to serve subsequent requests for the same program directly from memory. The RAID 6 array showed a performance decrease of 30% when operating in degraded mode (1 disk failure), and 70% when operating in rebuilding mode, so scheduled rebuilds during off-peak periods (early morning) should be considered. With approximately a hundred possible users per entity, and approximately 500 DSL connections per DSLAM 3, it is clear that a number of storage entities will be needed to satisfy all cache requests of the home users during peak periods. A number of entities could be clustered together with the GFS distributed file system [5] to provide 1 single logical view of all storage (see Figure 2). In such a scenario, each entity will export its storage as a block device over an IP network (forming an IP Storage Area Network) with GNBD 4 [6] or iscsi 5 [7], and the GFS file system will abstract all storage to a single logical view. Another way to assemble multiple storage entities into a system, is to let each entity have its own file system, and to provide a kind of mapping service on top of the set of storage entities to locate the cached content (see Figure 3). As there is a tendency to expand the functionality of the DSLAM, this could be an extra function for the DSLAM. 10 Gigabit Ethernet could be used to connect storage entities to the DSLAM. iscsi or GNBD could again be used to export the block devices on the storage entities to the DSLAM. In this case, block based storage access is especially interesting because it enables home users to efficiently have interaction with the content (pause, rewind, fast forward, skip a part of a program, ). Figure 3: IP SAN with content mapping service Figure 2: IP SAN with GFS distributed file system 3 DSLAM: Digital Subscriber Line Access Multiplexer 4 GNBD: Global Network Block Device 5 iscsi: Internet Small Computer Systems Interface During our experiments we were able to fill an Ethernet Gigabit link with iscsi and NBD traffic, so on the network level these protocols promise to be a scalable solution for block based access. Redundancy on a system level can be enforced by constructing a RAID over multiple exported block devices. This can be done by a distributed file system such as GFS, or even with a manual configuration. Future possibilities also include cluster aware RAID: a number of block devices are exported to the network and seen by other nodes as 1 physical volume. Subsequently each node can define its own logical volume(s) in the physical volume and use it as its own. 5. File transfer between systems For streaming broadcast TV (on a multicast channel) from the servers to the customers, the Real Time Protocol (RTP) [8] seems a logical choice. RTP can also be used to offer time-shifted TV: when a TV program is broadcast on a multicast channel, the regional servers and proxies can cache the content with sufficiently large sliding windows [9]. This allows users to watch the content with a small time-shift, or allows them to replay the last minutes of a ISBN13 : Paper Th1A5 Eeckhaut Page 3 of 5

4 broadcast program while the content can still be offered by a video cache. For VoD, there are also other options to distribute the content to the home users. Suppose now that a home user requests a film or a previously broadcast TV program not yet available in the video proxy. The video file will have to be fetched from a regional or the central server (or a peer proxy), streamed to the client, and - depending on the expected popularity of the requested item - also stored on the video proxy system. If the decision is made to keep a copy of the item on the video proxy, a number of possibilities exist to transfer the item to the proxy. A first possibility is to make use of the RTP protocol. While the content is streamed with RTP from the central or a regional server to the home user, a copy is stored on the video proxy when the content passes the DSLAM and video proxy (see Figure 4). However, when the user starts to interact with the content (for example skipping a part of the program) only a part of the program will be cached by the video proxy. A subsequent request from another user in the same neighbourhood for the same program will be complicated by the fact that only a part of the file will be available in the video cache. Another possibility is to copy the whole file completely to the video proxy server upon reception of a request from a home user. A file transfer protocol such as FTP could be used to transfer the entire file faster than real time (see Figure 5). In the mean time, streaming (with RTP) from the proxy to the home user can already start. When the user skips a part of the content after a while, the content can be served from the proxy directly while still leaving the possibility to let another user watch the skipped content from the proxy. Another, more static solution would be to transfer new content to the proxy automatically during off-peak periods (early morning). For these faster than real time transfers, FTP - which transfers whole files with minimal overhead over a data TCP connection - is preferable to block based transfer methods like NFS or SMB/CIFS. The performance penalty that a large round trip time (RTT) between proxy and central server can cause with these block based transfer patterns can be very substantial. Special implementations of the FTP protocol exist to meet a number of specific requirements for optimized reliable data transfer in high bandwidth wide area networks (WANs). GridFTP [10] implements a number of interesting features, including multiple data channels for parallel transfers (see Figure 6), partial file transfers, third-party (direct server-to-server) transfers, reusable data channels, and command pipelining. 6. Conclusions In this paper we illustrated how commodity hardware components and a number of block and file transfer protocols with good open source implementations can help to build a VoD proxy out of generic elements, without sacrificing performance and reliability. Given the nature of the content to be stored in a VoD proxy, and the large number of video proxies needed in a TV distribution network, this forms a valuable alternative to current highend storage solutions. Figure 4: Streaming from the central server (RTP) Figure 5: File transfer to the proxy (FTP) + streaming from the proxy (RTP) Figure 6: File transfer to the proxy with parallel transfers (FTP) + streaming from the proxy (RTP) ISBN13 : Paper Th1A5 Eeckhaut Page 4 of 5

5 References 1. Hughes, G.F., Murray J.F., Reliability and Security of RAID Storage Systems and D2D Archives Using SATA Disk Drives, ACM Transactions on Storage, Vol. 1, No. 1 (Dec. 2004), pp Seagate Enterprise SATA disk: es.html 3. Description of RAID types: 4. Areca SATA Raid host adapters: 5. Red Hat Global File System (GFS): 6. Global Network Block Device (GNBD): 7. Internet Small Computer Systems Interface (iscsi): 8. Real Time Protocol : 9. Gilon, E. et al, Demonstration of an IP Aware Multi- Service Access Network, BB Europe, Bordeaux, Dec GridFTP: ISBN13 : Paper Th1A5 Eeckhaut Page 5 of 5