Storage Solutions in the AWS Cloud. Miles Ward Enterprise Solutions Architect

Transcription

1 Storage Solutions in the AWS Cloud Miles Ward Enterprise Solutions Architect

2 Traditional Storage On-Premise Storage Options SAN network-attached block devices: LUNs DAS local block devices (disks) NAS network attached file systems, filers Differ in, Performance, Durability, and Cost Often used to form a Storage Hierarchy

3 Cloud Storage AWS offers multiple cloud-based storage options. Amazon S3 (Simple Storage Service) Amazon EBS (Elastic Block Storage) Amazon EC2 Local Instance Store (Ephemeral volumes) Each has a unique combination of performance, durability, cost, and interface Often used to form a Storage Hierarchy

4 Other Storage options? Amazon S3, EBS, and EC2 Instance Store volumes are the basic storage building blocks, but for more specialized uses, or managed resources, we have other options as well: Cloudfront Amazon RDS SimpleDB SQS Relational Databases on EC2

5 Storage by the Numbers Scale Durability S3 EBS Local Unlimited Storage 5TB objects 100 name spaces (buckets) % (standard) 99.99% (RRS) 1TB per volume ~44TB per server GB 1-5 volumes.1-.5% AFR Ephemeral design requires applicationlevel redundancy Latency High Low Very Low Management Zero Maintenance Snapshots, RAID, Consistency Manual Redundancy Locality Region-Centric AZ-Specific In-machine Interface REST or SOAP Web Services Block Device Block Device

6 S3 (Simple Storage Service) Pay for exactly what you use Highly Durable Limitlessly scalable Natively On-Line

7 S3 Best Practices Replicate Metadata Parallel IO for max speed Multipart Upload Sorted Writes Resource-level IAM permissions

8 S3 Use Cases and Users Media Serving Media Sharing File Storage / Backup Static Content Big Data

9 Amazon EBS Pay for what you provision POSIX compliant network attached block device Native Redundancy and Write Cache; optimized for random writes and reads.

10 EBS Best Practices Pre-warming Snapshots Striping (RAID 0) Block-device-mapping

11 EBS Use Cases and Users Application hosting Data Processing Database Large Data Envoy Media Group Case Stud Storage for multimedia content delivery on Amazon W Introd Establi that of media and po channe implem Envoy moved their IT infrastructure from a traditional co-location advantage of -as-youto share, and promptly delivered, Envoy selected Gluster in conce Challenges Envoy currently hosts their entire web presence on Amazon Web which has helped them meet the needs of their growing business base. One of their biggest challenges was ensuring the IT infrastru to respond to the highly dynamic nature of demand such as when launch and then grow exponentially. From a data storage standpo needed a solution that integrated easily with AWS and provided h Storing the data locally on EC2 instances is not suffcient since that persistent and would be lost if a server reboots. Amazon EBS volu to one instance at a time a problem if your business requires hu needed to ensure their data was easily shared.

12 EC2 Instance Store or Ephemeral Volumes Free with your EC2 Instance Zero Network Overhead; local, direct attached resource. High performance for sequential reads and writes Temporary; goes away when your instance does

13 Instance Store Best Practices App-level replication Striping (RAID 0, 1) Block-device-mapping

14 Instance Store Use Cases and Users Stateless Web Hosts Transcoding Caching HPC DNFS

15 Better Together: Cross-system patterns EBS <-> S3: Snapshots S3 <-> SimpleDB: meta-data store S3 <-> Local Storage: Caching

16 Short Detour: S3 vs EBS AMIs Two types of EC2 AMIs and Instances: Instance store ( S3-backed ) Original EC2, boot from ephemeral storage Can Start and Terminate only All data is ephemeral (unless separate EBS volume attached) EBS root ( boot from EBS ) Newer, boot from EBS volume Can Start, Stop, Create Image, and Terminate Ephemeral storage still available, but not exposed by default Use instance Block Device Mapping (command line and API only) Additional EBS volumes can be attached to either type

17 So, which should I choose? Might be all of them! S3: Static Online Content, Media Serving, Backups, Big Data, Centralized Resources EBS: Applications that require POSIX, Databases, Data Warehouses Ephemeral: Cache/Swap/Temp, Applications that require minimum latency

18 Test! It costs pennies to evaluate these technologies against your use-case, and making the right choice could save you millions.

19 How do I get it to the Cloud? AWS Import/Export Faster than Internet transfer and more cost effective than upgrading your connectivity Use cases: data migration, offsite backup, direct data interchange, disaster recovery WAN Optimization Asperatech, Riverbed, Tsunami-UDP, etc

20 Read the originals! Storage Options in the AWS Cloud: ptions.pdf Storage Use Cases in the AWS Cloud: se_cases.pdf

21 Thanks!