Intro to AWS: Storage Services

Transcription

1 Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

2 AWS storage options Scalable object storage Inexpensive archive storage Persistent block storage Shared file system Turnkey gateway solution

3 We are constantly producing more data

4 From all types of industries

5 Object Storage

6 Amazon S3 Simple Storage Service

7 S3 usage 95% year-over-year increase in data transfer to and from S3 (Q vs Q4 2013, not including Amazon use)

8 % durability

9 Reduced redundancy option 99.99% saves ~20%

10 S3 resources: buckets and objects

11 S3 website: static content

12 S3 capacity pricing pay only for what you use! 1 PB raw storage 800 TB usable storage 600 TB allocated storage Amazon S3 400 TB application data

13 S3 continuous cost reduction Available through 11 regions globally Priced at per GB-month rates 8 price reductions since launch 51% average S3 capacity fee reduction on 4/1/2014 TCO: comparing on-premises to S3 Can be challenging for some customers We can help!

14 S3 Amazon EMR

15 Amazon S3 New Features

16 S3 event notifications Delivers notifications to Amazon SNS, Amazon SQS, or AWS Lambda when events occur in S3 S3 Notifications Notifications SNS topic SQS queue Events Notifications Lambda function Foo() { }

17 S3 versioning Preserve, retrieve, and restore every version of every object stored in your bucket S3 automatically adds new versions and preserves deleted objects with delete markers unless an explicit versioned DELETE operation is made Easily control the number of versions kept by using lifecycle expiration policies Easy to turn on in the AWS Management Console

18 S3 cross-region replication Automated, fast, and reliable asynchronous replication of data across AWS regions Use cases: Compliance store data hundreds of miles apart Lower latency distribute data to regional customers Security create remote replicas managed by separate AWS accounts Only replicates new PUTs. Once S3 is configured, all new uploads into a source bucket will be replicated Source (Ireland) Destination (Frankfurt) Entire bucket or prefix based 1:1 replication between any 2 regions Versioning required

19 S3 use cases Web-scale storage capacity and performance for web applications Single-origin store with delivery through Amazon CloudFront Staging and persistent store for Big Data applications Storage target for backup and active archive

20 Amazon Glacier Low-cost archiving service

21 $0.01 per GB-month

22 $120 per TB/year

23 % durability

24 3 5 hours data retrieval

25 Storage costs vs. retrieval costs

26 Amazon Glacier benefits Reduce cost for long-term archiving Leverage unlimited storage capacity Replace tape museums Improve durability

27 Amazon S3 Amazon Glacier integration Policy-based archiving service

28 S3 Lifecycle Policies Key prefix logs/ Transitions objects to Glacier 30 days after creation date Delete 365 days after creation date <LifecycleConfiguration> <Rule> <ID>archive-in-30-days</ID <Prefix>logs/</Prefix> <Status></Status> <Transition> <Days>30</Days> <StorageClass>GLACIER</StorageClass> </Transition> <Expiration> <Days>365</Days> </Expiration> </Rule> </LifecycleConfiguration>

29 SoundCloud leveraging Amazon Glacier for audio transcoding World s leading social sound platform Audio files must be transcoded and stored in multiple formats S3 Amazon Glacier

30 Use cases for S3 lifecycle policies Data tiering in the cloud Manage object versioning for logical data protection Deletion of data from Amazon Glacier by policy

31 Block Storage

32 Instance storage

33 Amazon EBS Elastic Block Storage

34

35

36

37 1 GB 16 Ú TB

38 EBS General Purpose (SSD) Up to 16 TB 10,000 IOPS Up to 160 Mbps Provisioned IOPS (SSD) Up to 16 TB 20,000 IOPS Up to 320 Mbps

39 EBS Price Performance Use cases Storage media Magnetic Infrequent data access Magnetic diskbacked General Purpose Provisioned IOPS Boot volumes Small to med DBs Dev and Test SSD-backed I/O intensive Relational DBs NoSQL DBs SSD-backed Max IOPS IOPS 10,000 IOPS 20,000 IOPS Latency (random read) Availability Price ms 1 2 ms 1 2 ms Designed for % $.05/GB-month $.05/million I/O Designed for % Designed for % $.10/GB-month $.125/GB-month $.065/provisioned IOPS

40 Amazon EBS snapshots

41

42 Amazon EBS use cases Persistent block storage for Amazon EC2 Transactional workloads File system for an instance NTFS, ExtFS, and so on

43 Introducing Amazon Elastic File System Cloud-based file system

44 What is Amazon EFS? Fully managed file system for EC2 instances Provides standard file system semantics Works with standard operating system APIs Sharable across thousands of instances Elastically grows to petabyte scale Delivers performance for a wide variety of workloads Highly available and durable NFS v4 based

45 Amazon EFS is designed for a broad range of use cases, such as Content repositories Development environments Home directories Web server farms Big Data

46 Amazon EFS Benefits... to IT administrators? Eliminates need to manage and maintain file system storage at scale to app owners and developers? to your business? Easy to move existing code, applications, and tools used today with existing NFS servers to the AWS cloud Simple shared file storage solution for new cloud-native applications Predictable pricing with no up-front investment Increased agility Spend less time managing file storage and more time focusing on your business

47 AWS Storage Gateway

48 AWS Storage Gateway Your on-ramp to AWS cloud storage: Back up into S3 DR on EC2 Archive into Amazon Glacier iscsi or VTL interface

49 Partner Solutions S3 &/or Amazon Glacier S3 Internet Web Services API HTTP(S)

50 Summary: the Amazon storage portfolio S3 Object storage: data presented as buckets of objects Amazon Glacier Archival storage: data presented as vaults/archives of objects EBS Block storage (analogous to SAN): data presented as disk volumes EFS File storage (analogous to NAS): data presented as a file system Storage Gateway Back up and archive data into S3 and Amazon Glacier

51 LONDON