Understanding AWS Storage Options

Transcription

1 Understanding AWS Storage Options Ian Massingham, Technical 30 April Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2 What s in this talk? Scalable storage Inexpensive archive storage Persistent direct attached storage Turn-key gateway solution Customer presentation: SoundCloud

3 We are constantly producing more data

4 From all types of industries

5 #1 Object Storage

6 AMAZON S3 SIMPLE STORAGE SERVICE

7 % Durability

8 Trillions Of Unique Customer Objects

9 1.5 Million+ peak transactions per second

10 Storage Tiers: Buckets + Unlimited Objects

11 Reduced Redundancy Option 99.99% saves ~20%

12 Spotify adds over 20,000 tracks a day - RRS

13 Amazon S3 Website: Static Content

14 Amazon S3 Continuous Cost Reduction 16 price reductions since launch TCO: On-premises vs. Amazon S3 Can be challenging for some customers We can help!

15 RAW Storage On-Premises vs. Cloud Storage 1 PB raw storage 800 TB usable storage 600 TB allocated storage Amazon S3: Only actual usage is charged 400 TB written application storage

16 Use Amazon S3 When You Need Unlimited storage capacity High durability Storage for backups Single origin store with delivery via Amazon CloudFront

17 AMAZON GLACIER LOW-COST ARCHIVING SERVICE

18 1 per GB / month

19 $120 per TB / year

20 % Durability

21 3-5 Hours Data retrieval

22 STORAGE COSTS VS RETRIEVAL COSTS

23 Use Amazon Glacier When You Need Inexpensive or long-term archiving Unlimited storage capacity No tape museums No tech refresh High durability

24 Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE

25 Lifecycle Rule

26 Archive Recovery Process with Tape + Days or Weeks

27 Archive Recovery Process with AWS & Amazon Glacier Amazon Glacier Amazon S3 Amazon EC2 / HPC Amazon CloudFront Generating Business Value $$ Hours

28

29 Use Amazon S3 and Amazon Glacier When You Need HSM in the cloud Archive data from Amazon S3/RRS to Amazon Glacier by policy Delete data from Amazon Glacier by policy

30 #2 Block Storage

31 AMAZON EBS ELASTIC BLOCK STORAGE

32 Ephemeral Storage

33 10 GB 1TB

34 Provisioned 4000 IOPS IOPS

35

36

37

38 Amazon EBS Snapshots

39

40

41 Use Amazon EBS When You Need Long-term persistent storage Frequent data changes Block storage for your databases Provisioned IOPS volumes Filesystem for an instance NTFS, ExtFS, RAID, LVM Access to raw, unformatted block-level storage

42 #3 Sync Volumes

43 AWS STORAGE GATEWAY

44 AWS Storage Gateway

45 What is AWS Storage Gateway? Integrates on-premise IT environments with cloud storage for departmental and remote office backup and DR Uses a virtual appliance that sits in customer datacenter Exposes compatible iscsi interface on front end Stores primary data in Amazon S3, or on-premise with data backed up to Amazon S3 as Amazon EBS snapshots

46 Solution Overview 1. Offsite Backup 2. DR 3. Data Mirroring 4. Dept. File Share 5. Archive/Glacier AWS Storage Gateway works with your existing backup application and moves your data into Amazon S3 as Amazon EBS snapshots Run AWS Storage Gateway in Amazon EC2 and access snapshots up to 32 TB in size Run apps in the cloud using your uploaded data HPC/Hadoop/Analytics AWS Storage Gateway Cached volumes help you create storage volumes in the cloud and keep most recent accessed data locally [Reduced SAN footprint for file shares] Virtual Tape Library version of AWS Storage Gateway helps customers move virtual tape data from on premises to Amazon S3 and then to Amazon Glacier (Virtual Tape Shelf)

47 IT S ALL ABOUT CHOICE PERFORMANCE-ORIENTED COST-ORIENTED

48 SoundCloud and AWS AWS Summit Berlin 2014 Oliver Hookins Backend Engineer, Media Publish/Delivery Team

49 What is SoundCloud? YouTu be

50 12 hours of audio every minute music / podcasts baby s Subtitle first words environmental noise you name it

51

52

53 petabytes of storage hundreds of millions of files hundreds of millions of Subtitle listeners fastest performance possible widest platform coverage

54 Horizontal Scaling Hands-Off Infrastructure HTTP Interfaces S3 to Glacier Transition Policies Subtitle Reliability Great SDKs

55 Thank you! We re Hiring! -

56 Understanding AWS Storage Options Ian Massingham, Technical 30 April Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.