Cloud storage reloaded: Some aspects on operating distributed Linux file systems like Ceph and GlusterFS Udo Seidel
Agenda Introduction/motivation Distributed storage Ceph and GlusterFS Operational considerations Summary LinuxCon Europe 05-07/11 2012 2
Introduction LinuxCon Europe 05-07/11 2012 3
Me ;-) Teacher of mathematics & physics PhD in experimental physics Started with Linux in 1996 Linux/UNIX trainer Solution engineer in HPC and CAx environment Head of a international team with people in Erding, Miami and Sydney @Amadeus LinuxCon Europe 05-07/11 2012 4
Shared File Systems Normal 'business' for Linux Several implementation approaches Network based, e.g. NFS, CIFS 'clustered' Shared disk, e.g. OCFS2, GFS2 Parallel/distributed, e.g. Ceph, GlusterFS LinuxCon Europe 05-07/11 2012 5
Distributed File Systems 'Recent' attention on distributed storage Cloud hype Big Data Which solution to choose? Just storage or file system too? Knock-on effects Technical Non-technical LinuxCon Europe 05-07/11 2012 6
Operations Important part of the life cycle Geek vs. Enterprise Technical challenges and more LinuxCon Europe 05-07/11 2012 7
Distributed Storage LinuxCon Europe 05-07/11 2012 8
History Reviewing storage task responsibilities Block allocation Space management Extension of SCSI standard Object based storage Meta-Data handling separated from data management LinuxCon Europe 05-07/11 2012 9
Object based storage Storage objects quite general Partition, file,... Unique identifier OSD (Object based Storage Device) Hardware -> original trigger Software -> common implementation Interfaces to existing technology Part of the distributed storage solution Additional development of existing approaches LinuxCon Europe 05-07/11 2012 10
Distributed storage: Paradigm changes Block -> Object Central -> Distributed Few -> Many Big -> Small LinuxCon Europe 05-07/11 2012 11
Distributed storage Now what?!? Several implementations Different functions Support models Storage vendors initiatives Relation to Linux distributions LinuxCon Europe 05-07/11 2012 12
Ceph and GlusterFS LinuxCon Europe 05-07/11 2012 13
Ceph and GlusterFS Sufficient community presence Picked up by Enterprise Linux vendors Good amount of commonalities but still quite different LinuxCon Europe 05-07/11 2012 14
Ceph and GlusterFS High level Server part Block storage -> storage server (OSD) (POSIX) file system -> Meta data HA -> replication and distribution Client part (POSIX) file system Storage layer LinuxCon Europe 05-07/11 2012 15
Ceph LinuxCon Europe 05-07/11 2012 16
GlusterFS LinuxCon Europe 05-07/11 2012 17
Ceph and GlusterFS some details CEPH GLUSTERFS Server implementation userspace userspace Client implementation userspace/fuse and native kernel userspace/fuse File system access CEPH protocol suite GlusterFS and NFS protocol Block device access yes RFP ongoing QEMU integration yes Under development Replication traffic OSD network Client-server network Interconnect IP based IP based and/or Infiniband Encryption no not really Back-end file system XFS... BTRFS... ext4 XFS...ext4 Storage object CRUSH object file (except for striping) GEO replication not really yes LinuxCon Europe 05-07/11 2012 18
Ceph and GlusterFS more details CEPH Online growth/shrink yes yes POSIX yes alike GLUSTERFS Concurrent workload technically yes technically yes NAS replacement technically yes technically yes SAN replacment technically yes no Hadoop integration yes yes Openstack integration yes yes Community version available yes Production readiness Block layer File system layer Service provider Vendor and other Vendor and other......... yes LinuxCon Europe 05-07/11 2012 19
Operational considerations LinuxCon Europe 05-07/11 2012 20
Status quo pre distributed storage Server and Storage Clear responsibilities Known area: SCSI, NFS, FC Strict border lines Technically organizationally Storage hardware and topology Less focus on local drives Dedicated/separated storage network LinuxCon Europe 05-07/11 2012 21
Changes with distributed storage New paradigm Big overlap between server and storage Technical implication Organizational structure Strong network dependency Topology physics Support and supported setups LinuxCon Europe 05-07/11 2012 22
Challenge one: server vs. storage Need for Server admin with strong storage skills Storage admin with strong O/S skills Review of Current organizational setup Strategy for traditional storage Operation procedures HA architecture LinuxCon Europe 05-07/11 2012 23
Challenge one: possible answers 'DevOps model' for O/S and storage admins Best of both worlds Better common understanding Potential of process simplification BUT organizational changes needed! LinuxCon Europe 05-07/11 2012 24
Challenge one: possible answers 'Appliance model' for O/S and storage admins Less internal changes Potential of reusing existing processes BUT less control and visibility LinuxCon Europe 05-07/11 2012 25
Challenge two: infrastructure Dedicated vs. Shared Responsibilities server vs. storage vs. network Transport medium Local setup Spinning disks vs. SSD RAID? HA configuration of single server LinuxCon Europe 05-07/11 2012 26
Challenge two: possible answers Different flavour of performance vs. cost topic Dedicated vs. Shared Transport medium Spinning disks vs. SSD HA on server level LinuxCon Europe 05-07/11 2012 27
Challenge two: possible answers Team up with network organization Single server full blown SPOF on purpose Think big LinuxCon Europe 05-07/11 2012 28
Challenge three: support Service provider Application support LinuxCon Europe 05-07/11 2012 29
Challenge three: possible answers Combine with Linux vendor support Single point of contact Leverage vendor QA Hide the new technology XaaS Appliance approach File system level Reminder: Think big LinuxCon Europe 05-07/11 2012 30
Summary... kind of... LinuxCon Europe 05-07/11 2012 31
The message New world with several challenges Don't underestimate them Don't overestimate them Technical revolution requires non-technical changes Don't underestimate the potential Think big! ;-) Get started! LinuxCon Europe 05-07/11 2012 32
References http://www.ceph.com http://www.gluster.org your business/company LinuxCon Europe 05-07/11 2012 33
Thank you! LinuxCon Europe 05-07/11 2012 34