Moving Media Storage Technologies Applications & Workflows for Video and Media Server Platforms Karl Paulsen
CONTENTS Acknowledgments Introduction xv xvii Chapter 1 Information, Data, and Video 1 Data Structure 1 Information 2 Storing the Moving Image 3 Digital Video Recording 4 Analog Component Video 5 Colorimetry and Conversion 11 A Digital Media World: Going Forward 12 Further Readings 13 Chapter 2 Digital Disk Recording 15 Recording Moving Images 16 Dawn of the Video Disk 17 Fundamentals of Digital Disk Recording 18 Core Technologies for Digital Disk Recording 21 DDR, DVR, or VideoServer 28 Comparing DDRs to VideoServers 31 Clip Servers 32 DVR Marketplaces 35 Home Media Network Disk Recorders 36 Audio 37 Further Readings 38 Chapter 3 Video Media Servers 39 Carrying Over from the DDR 39 Universal Server Approach 41 VideoServers as Media Servers 46 The Future of VideoServers 66 Communications Protocols Used in Video Serving Platforms 66 Further Readings 71
Chapter 4 Hard Drive Anatomy 73 Magnetic Recording History 74 Drive Components 77 Disk Performance Optimization 85 Disk Scheduling 85 Access Time Components 87 Servicing and Scheduling Criteria 88 Capacity Factors, Transfer Time and Obstacles 95 Read-Write Heads 97 Advanced Format Sector Technology 102 Superparamagnetic Limit 106 Heat-Assisted Magnetic Recording. 107 Further Reading 108 Chapter 5 Buses, Ports, and Interfaces 109 Carrying and Transferring Data 110 Bus Types 110 Storage Bus Standards 112 Storage Module Device 112 Serial Storage Architecture 114 High-Performance Parallel Interface 115 IDE 116 Parallel ATA 116 SCSI 117 Further Readings 124 Chapter 6 SATA, SAS, and RAID 125 Transitioning from Parallel to Serial Technologies 126 Serial Attached SCSI (SAS) 127 SATA 133 RAID 139 Alternative Disk Sets and Configurations 155 Error Prevention Methods 159 Conclusions 163 Further Readings 163
Chapter 7 Solid State Disks 165 Solid State Storage Evolution 166 Dynamic Random Access Memory 167 Flash Memory 167 Flash Memory Functionality 168 Portability and Green Perspectives 182 Supplementing Video on Demand with SSD 183 Further Readings 185 Chapter 8 Optical Storage Media 187 Defining Removable Media 188 Optical Media Definitions 188 Blu-rayDisc 195 Holographic Storage 202 Care and Handling 210 Further Readings 211 Chapter 9 Interchange, Transcoding, and Wrappers 213 Moving Toward Non-Real Time 214 Compressed Video 216 Issues with Interchange 217 Media File Vocabularies and Terminologies 218 Content 229 Material exchange Format (MXF) 229 Task Force for MXF 230 Operational Patterns 233 Specialized Operational Pattern Atom 234 Items and Packages 236 Essence Container 238 Conversion and Transcoding 239 Converting Audio and Video Essence 240 Transcode Pipeline 244 Multiplatform Conversion and Distribution 246 Examples and Conclusions 252 Further Readings 253
Chapter 10 IP Storage 255 Internet Protocol 255 Delivering Packets by IP 256 Connected or Connectionless 256 IP Address Classes 257 Addresses Domain Naming 258 Ethernet Internet Engineering Task Force 262 IP Storage 262 iscsi iscsi RFCs 264 iscsi Protocol 264 Initiator-Target Messages 264 Transfer Directions 265 Network Address Authority (NAA). ifcp 257 259 263 265 Gateway Protocol to TCP/IP 266 Provisions 266 Fabric 266 ifcp Services 268 Transport Services 268 Other 268 Security 268 Fibre Channel Over IP (FCIP) 269 Complementary 270 Nominal Distances 270 FCIP Link 271 Clarification 272 Internet Storage Name Service (isns) 272 isns Architectural Components 273 isns Protocol (isnsp) 273 isns Client 273 isns servers 274 isns Database 274 What isns Does 275 Unified Discovery and Management 276 265
Fiber Channel Discovery 277 Zoning 277 Using SCN Functions 277 Distributed Intelligence 278 IP Storage Support 278 Quality of Service for IP Storage Networks 278 Class of Service (CoS) 278 Traffic Prioritization 279 TOS Precedence 279 Differentiated Services 279 Performance Predictability 280 Traffic Shaping 280 Theory and Research 280 Variable Delay 281 Throttling Decisions 281 Resource Reservation Protocol (RSVP) 282 Multiprotocol Label Switching (MPLS) 283 MPLS Transport Profile 284 InfiniBand 285 IP Storage Considerations 290 IP Switches and Routers 291 Determining the Need 291 Logical Separation Points 291 Security Considerations 292 Ports and Connectivity 292 Other Features 293 Management Tools 293 Acceleration and Compression 294 Security 294 Authentication 294 Fibre Channel Security Protocol (FC-SP) 295 Device Identity 296 Authorization 298 Encryption 298 Auditing 298 Integrity Checking 298 Quality of Service (QoS) 298 Further Readings 299
Chapter 11 Object-Based Storage 301 Introduction: Objects 302 Storage Categories 307 Storage Interface Progression 310 Parallel Heading 313 Applications of OSD 321 Further Readings 325 Chapter 12 Archives, Backups, and Linear Tape 327 Organizational Requirements for Data 328 Legal Requirements for Archive or Backup 329 Repacking and Repurposing 330 Added Operational Requirements 331 Backup or Archive: There are Differences 332 Backups 332 Acting like Archives 333 Data Backups Not Archives 333 Internal Application-Specific Backups 334 Archive and Transfer Management Nonlinear Editing 335 Differentiation 335 Archiving for Discovery 336 Data Archiving 337 Tape or Disk 338 Pros and Cons 339 Self-Protecting Storage Systems 340 Disk Archiving Pluses 340 Tape Archiving Pluses 341 Storage Retention Period 341 Disadvantages of Tape 341 Legacy Tape Archive Problems 342 Standards-Based Archive Approaches 342 Common Backup Formats 343 Determining the Right Archiving Method 343 Cloud Data Protection 344 Cloud Benefits 344 Scaling 345 Services Provided 345
Holographic Storage for the Archive 345 RAIT 347 Mirroring Level 1 RAIT 349 Data Striping Level 3 RAIT 349 Data Striping Level 5 RAIT 349 Not for Random Access 349 Automated Tape Libraries (ATL) 350 Stackers, Autoloaders, or Libraries 351 Server and Archive Components 351 Evolution in Digital Tape Archives 352 Tape Technology Basics 353 Tape Metrics 353 Linear Serpentine 354 Helical Scan Recording 355 Historical Development of Data Tape 355 QIC and DAT 356 Travan 356 8 mm Recording Technologies 356 Tape Error Detection 358 Tape Media Formats and Fabrication Parameters 359 Metal Particle to Advanced Metal Evaporated 359 Advanced Intelligent Tape (AIT) 360 Super and Turbo AIT 361 Digital Data Storage (DDS) 362 Data Storage Technology (DST) 362 Digital Tape Format (DTF) 363 Advanced Tape Materials 364 Digital Linear Tape (DLT) 365 Error Detection 365 - Adaptive Cache Buffering 366 Compression Algorithms 366 Duty Cycle 366 DLT1 Drive 367 Super DLT 367 RoadmapsforSDLT 368 Linear Tape Open (LTO) - 369 Genealogy 369 Ultrium First Generation 370
LTO Generation 1 370 LTO Generation 2 370 LTO Generation 3 370 LTO Generation 4 371 LTO Generation 5 371 LTFS 371 LTO Program Roadmap 372 Security WORM 373 Why Use Tape? 373 Data Archiving and Tape Storage 375 Preparing for the Archive 376 Future Standards 377 Chapter 13 Metadata and Media Asset Management 379 Media Management Systems 380 Media Assets and Content 380 Content Management System 381 Web-Based CMS 382 Digital Asset Management 382 Asset Management System 383 Media Asset Management 384 Tape-Based MAM 384 Drawbacks to Tape-Based Systems 386 MAM for VideoServer Implementations 387 Lack of Media Asset Integration 388 Collaboration Issues 388 Metadata 388 Media-Centric Metadata 389 Structural Metadata 391 Descriptive Metadata 391 Interoperability 393 Dublin Core 393 PBCore 394 Usage Metadata 396 Schemas 396 Standards, Practices, and Policies for Metadata 397 Storing Metadata 397
Metadata Services 398 Media Application Services 398 Messaging Services for MAM 400 Intelligent Integrated MAM 401 Workflow Management 402 Collaboration among Multiple Users 402 Tool Set Integration 402 Centralized Database 403 Multi-Tier Architecture 403 Scalability 404 Storage for MAM 404 Latency 405 Interfacing with Production Storage 405 Archive 407 Further Readings 408 Chapter 14 Clustering 409 Why Use Clusters? 409 Physical Components 411 Capabilities and Types of Clustering 412 Cluster Scaling Characteristics 415 Shared Structures 416 Managing Storage Clusters for Scalability 420 Grid Computing 422 Summary 428 Further Readings 428 Chapter 15 Networked Storage.. 429 Networked Storage Fundamentals 430 Networks 436 Network-Attached Storage 442 Direct Access File System 443 Storage Area Network 444 Fibre Channel SAN Components 460 Next Generation SANs 467 Further Readings 467
Chapter 16 High-Performance and Shared Storage 469 Applications for High-Performance Storage 470 Size Matters 474 Intelligent Adaptive Storage 475 Storage Accelerators 476 Storage Appliance Technologies 477 Real-World Performance 490 Shared Storage Implementations 498 Storage Scaling and Management 510 Further Readings 510 Chapter 17 Scalability, Extensibility, and Interoperability 511 Scalability 511 Flexibility 516 Reliability 520 Accessibility and Availability 526 Extensibility 533 Interoperability 539 Science and Technology 541 Further Readings 541 Chapter 18 Storage Management 545 Storage System Dependencies 546 Storage Management Concepts 549 Electronic Media Management 554 Fragmentation 554 SAN Management 560 NAS Management 564 Snapshots 571 Data Deduplication 574 Further Readings 579 Chapter 19 Resilience, Fault Tolerance, and Virilization 581 Resilience 582 Storage Tiering 583 Protecting Data and Storage 595
Visualization in File-Based Storage 605 Fault Tolerance 610 Conclusion 617 Further Readings 617 Index 619 Companion website: www.elsevierdirect.com/companions/9780240814483