Understanding AWS Storage Options Ian Massingham, Technical Evangelist @IanMmmm ianmas@amazon.com 30 April 2014 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
What s in this talk? Scalable storage Inexpensive archive storage Persistent direct attached storage Turn-key gateway solution Customer presentation: SoundCloud
We are constantly producing more data
From all types of industries
#1 Object Storage
AMAZON S3 SIMPLE STORAGE SERVICE
99.999999999% Durability
Trillions Of Unique Customer Objects
1.5 Million+ peak transactions per second
Storage Tiers: Buckets + Unlimited Objects
Reduced Redundancy Option 99.99% saves ~20%
Spotify adds over 20,000 tracks a day - RRS
Amazon S3 Website: Static Content
Amazon S3 Continuous Cost Reduction 16 price reductions since launch TCO: On-premises vs. Amazon S3 Can be challenging for some customers We can help!
RAW Storage On-Premises vs. Cloud Storage 1 PB raw storage 800 TB usable storage 600 TB allocated storage Amazon S3: Only actual usage is charged 400 TB written application storage
Use Amazon S3 When You Need Unlimited storage capacity High durability Storage for backups Single origin store with delivery via Amazon CloudFront
AMAZON GLACIER LOW-COST ARCHIVING SERVICE
1 per GB / month
$120 per TB / year
99.999999999% Durability
3-5 Hours Data retrieval
STORAGE COSTS VS RETRIEVAL COSTS
Use Amazon Glacier When You Need Inexpensive or long-term archiving Unlimited storage capacity No tape museums No tech refresh High durability
Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE
Lifecycle Rule
Archive Recovery Process with Tape + Days or Weeks
Archive Recovery Process with AWS & Amazon Glacier Amazon Glacier Amazon S3 Amazon EC2 / HPC Amazon CloudFront Generating Business Value $$ Hours
Use Amazon S3 and Amazon Glacier When You Need HSM in the cloud Archive data from Amazon S3/RRS to Amazon Glacier by policy Delete data from Amazon Glacier by policy
#2 Block Storage
AMAZON EBS ELASTIC BLOCK STORAGE
Ephemeral Storage
10 GB 1TB
Provisioned 4000 IOPS IOPS
Amazon EBS Snapshots
Use Amazon EBS When You Need Long-term persistent storage Frequent data changes Block storage for your databases Provisioned IOPS volumes Filesystem for an instance NTFS, ExtFS, RAID, LVM Access to raw, unformatted block-level storage
#3 Sync Volumes
AWS STORAGE GATEWAY
AWS Storage Gateway
What is AWS Storage Gateway? Integrates on-premise IT environments with cloud storage for departmental and remote office backup and DR Uses a virtual appliance that sits in customer datacenter Exposes compatible iscsi interface on front end Stores primary data in Amazon S3, or on-premise with data backed up to Amazon S3 as Amazon EBS snapshots
Solution Overview 1. Offsite Backup 2. DR 3. Data Mirroring 4. Dept. File Share 5. Archive/Glacier AWS Storage Gateway works with your existing backup application and moves your data into Amazon S3 as Amazon EBS snapshots Run AWS Storage Gateway in Amazon EC2 and access snapshots up to 32 TB in size Run apps in the cloud using your uploaded data HPC/Hadoop/Analytics AWS Storage Gateway Cached volumes help you create storage volumes in the cloud and keep most recent accessed data locally [Reduced SAN footprint for file shares] Virtual Tape Library version of AWS Storage Gateway helps customers move virtual tape data from on premises to Amazon S3 and then to Amazon Glacier (Virtual Tape Shelf)
IT S ALL ABOUT CHOICE PERFORMANCE-ORIENTED COST-ORIENTED
SoundCloud and AWS AWS Summit Berlin 2014 Oliver Hookins Backend Engineer, Media Publish/Delivery Team
What is SoundCloud? YouTu be
12 hours of audio every minute music / podcasts baby s Subtitle first words environmental noise you name it
petabytes of storage hundreds of millions of files hundreds of millions of Subtitle listeners fastest performance possible widest platform coverage
Horizontal Scaling Hands-Off Infrastructure HTTP Interfaces S3 to Glacier Transition Policies Subtitle Reliability Great SDKs
Thank you! https://soundcloud.com/oliver-hookins oliver@soundcloud.com We re Hiring! - https://soundcloud.com/jobs
Understanding AWS Storage Options Ian Massingham, Technical Evangelist @IanMmmm ianmas@amazon.com 30 April 2014 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.