a unified distributed storage system Wido den Hollander Ceph trainer and consultant community member since 2009

Transcription

1 a unified distributed storage system Wido den Hollander Ceph trainer and consultant community member since 2009

2 a lot of credits for this presentation go to Sage Weil Sage is the creator and lead-developer of Ceph

3 why should you care about another storage system?

4 the old reality direct-attached storage network-attached storage expensive SAN vendor locked-in replace every 3 or 5 years scalable capacity and performance, to a point poor scaling of management and/or price

5 new user demands cloud many disk volumes, with automated provisioning larger data sets big data distributed processing unstructured data continued growth of traditional workloads

6 expanding requirements diverse use-cases object storage block devices (for VMs) (with snapshots) shared file system with POSIX, coherent caches unstructured data... files, block devices, or objects? scale terabytes, petabytes, exabytes heterogeneous hardware reliability and fault tolerance

7 cost pressures near-linear function of size or performance incremental expansion no fork-lift upgrades no vendor lock-in choice of hardware and software open

8 your time is precious ease of administration and automation no manual data migration or load balancing No more rsync, scp or netcat at 04:00 on Sunday painless scaling expansion and contraction

9 what is ceph?

10 distributed storage system large scale 10s to 10,000s of (small) machines terabytes to exabytes fault tolerant no single point of failure commodity hardware self-managing, self-healing

11

12 unified storage platform objects rich native API compatible RESTful API block file thin provisioning, snapshots, cloning strong consistency, snapshots

13 how do you design a storage system that scales?

14

15

16 (actually more like this ) () ()

17

18

19 APP APP HOST/VM CLIENT LIBRADOS A library allowing apps to to directly access RADOS, with support for for C, C, C++, Java, Python, Ruby, and PHP RADOSGW A bucket-based REST gateway, compatible with S3 S3 and Swift RBD A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for for FUSE RADOS the Ceph object store A reliable, autonomous, distributed object store comprised of of self-healing, self-managing, intelligent storage nodes

20 why start with objects? more useful than (disk) blocks names in a simple flat namespace variable size trivially parallel with rich semantics more scalable than files no hard-to-distribute hierarchy update semantics do not span objects workload is trivially parallel

21 OSD OSD OSD OSD OSD FS FS FS FS FS xfs btrfs ext4 M M M

22 Object Storage Daemons (OSDs) 10s to 10000s in a cluster one per disk, SSD, or RAID group, or... hardware agnostic serve stored objects to clients intelligently coordinate to perform replication and recovery M Monitors maintain cluster membership and state provide consensus for distributed decisionmaking small, odd number these do NOT serve stored objects to clients

23 data distribution all objects are replicated N times objects are automatically placed, balanced, migrated in a dynamic cluster must consider physical infrastructure ceph-osds on hosts in racks in rows in data centers three approaches pick a spot; remember where you put it pick a spot; write down where you put it calculate where to put it, where to find it

24 CRUSH pseudo-random placement algorithm fast calculation, no lookup repeatable, deterministic statistically uniform distribution stable mapping limited data migration on change rule-based configuration infrastructure topology aware adjustable replication allows weighting

25 hash(object name) % num pg CRUSH(pg, cluster state, policy)

26

27 CLIENT??

28

29

30 CLIENT??

31 APP APP HOST/VM CLIENT LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RADOSGW A bucket-based REST gateway, compatible with S3 S3 and Swift RBD A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file file system, with a Linux kernel client and support for for FUSE RADOS A reliable, autonomous, distributed object store comprised of of self-healing, self-managing, intelligent storage nodes

32 APP LIBRADOS native MM MM MM

33 L librados direct access to RADOS from applications C, C++, Perl, Python, PHP, Java, Erlang, Ruby direct access to storage nodes no HTTP overhead TCP/IP and RDMA (beta) Native RADOS protocol over IPv4 or IPv6

34 VM VIRTUALIZATION CONTAINER (KVM) LIBRBD LIBRADOS MM MM MM

35 CONTAINER LIBRBD LIBRADOS VM CONTAINER LIBRBD LIBRADOS MM MM MM

36 HOST KRBD (KERNEL MODULE) LIBRADOS MM MM MM

37 RADOS Block Device storage of disk images in RADOS decouple VM from host images striped across entire cluster (pool) By default 4MB objects snapshots copy-on-write clones support in mainline Linux kernel ( ) Qemu/KVM, Xen OpenStack, CloudStack, OpenNebula, Proxmox,...

38 why we do this limited options for scalable open source storage proprietary solutions expensive, don't scale (well or out) marry hardware and software users hungry for alternatives scalability, cost, features open, interoperable Inktank commercial backing for production deployments acquired by Red Hat in May 2014

39 highlights 100% open source no vendor lock-in runs on commodity hardware lots of small machines standard 10g networking scales into the petabytes storage of choice behind openstack

40 thanks! wido den