Seagate Kinetic Open Storage Platform. James Hughes and many others

Transcription

1 eagate Kinetic Open torage Platform James Hughes and many others

2 2

3 2

4 A 3

5 A D 3

6 A D 3

7 A D 3

8 D A 3

9 D No , LibKinetic effective Jan. 18, 2009, A 3

10 D No , LibKinetic effective Jan. 18, 2009, ProtoBuf TCP/IP/GbE A 3

11 lication Clustering Management Interconnect D No , LibKinetic effective Jan. 18, 2009, ProtoBuf TCP/IP/GbE Proprietary to ystem Vendor GPL tandard torage A Proprietary to eagate 3

12 lication Clustering Management Interconnect D No , LibKinetic effective Jan. 18, 2009, ProtoBuf TCP/IP/GbE C++, Java, Python, Erlang, DIY Proprietary to ystem Vendor GPL tandard torage A Proprietary to eagate 3

13 lication Clustering Management Interconnect D No , LibKinetic effective Jan. 18, 2009, ProtoBuf TCP/IP/GbE Proprietary to ystem Vendor GPL tandard torage A Proprietary to eagate 3

14 lication Clustering Management Interconnect LibKinetic D No , LibKinetic effective Jan. 18, LibKinetic 2009, ProtoBuf TCP/IP/GbE Proprietary to ystem Vendor GPL tandard torage A Proprietary to eagate 3

15 lication Clustering Management Interconnect LibKinetic D No , LibKinetic effective Jan. 18, LibKinetic 2009, ProtoBuf TCP/IP/GbE Proprietary to ystem Vendor GPL tandard torage A D A D A D A D A A D A D A D A D Proprietary to eagate 3

16 A versus Kinetic Open torage Object torage ystems wift Basho Riak Ceph HDF!!!!! : ubject to NDA!!!!!! 4

17 Typical HA High Density Intel server Double ocket 48GB Ram 1000w Core A tray Connected to the server A A 5

18 Low cost HA Configuration Each drive talks to both switches Each switch has 2 by 10Gb/s Ethernet Kinetic Tray talks directly to ToR No servers Eth Eth Core 6

19 ystem Hardware Typical JBOD architecture Does not require a server, just JBODs to the ToR witch 10 JBOD 60 drives 4TB = 2.4PB/Rack 7

20 ystem Hardware 8

21 Kinetic Drive Provides RPC to Key/Value database Data is pre-indexed Compression and other value is easy and transparent P2P (Drive to Drive) copy of key ranges Communicate using existing Data Center Plumbing (TCP/IP) Multiple masters - Data sharing between machines Configurable caching per command WriteThrough, WriteBack, Flush Local space management 9

22 Kinetic ystems Clustering (performance, reliability, management) Compatibility with large scale applications (3, etc.) Centralized Management Reliability, availability, durability 10

23 Existing Traffic Flow ToR ToR Data Center : Core Router ubject to NDA Data Node Client A helf A helf 11

24 Kinetic Traffic Flow ToR ToR Data Center : Core Router ubject to NDA Kinetic helf Client 12

25 Conventional HDF ystem Client Namenode Datanode Datanode Datanode Datanode... Disk Disk... Disk Disk Disk... Disk Disk Disk... Disk Disk Disk... Disk Conventional Drives 13

26 HDF on Kinetic Client Namenode DataNode Daemon Datanode Datanode... K K K K K K Disk Disk Disk Disk Disk Disk... Disk Disk... Disk Disk Disk... Disk Conventional Drives Kinetic Drives 14

27 Cumulative operations ordered by length 100% 80% operations 92% of the operations data Cumulative percentage 60% 40% 20% 32KB 0.5% of the data 0% Length (KB) 15

28 Map of Operations 0 512KB Length Time (minutes) Location (TB)

29 Performance Metrics ame normal performance expectations equential Write: 50MB/s Random Write: 50MB/s equential Read: 50MB/s Random Read: 20% slower than traditional drives 17

30 Write Performance Results 120 MB/s 6000 Puts/s MB values put rate (MB/s) 10B value put rate 18

31 Goals of API Data movement Get/put/delete/getnext/getprevious Versioned (== for success), options Range operations Multiple masters Authentication/Integrity/Authorization Cluster-able imple cluster configuration version enforcement P2P copy Management 19

32 Bootstrapping devices DHCP erver DHCP erver BMC 20

33 Bootstrapping devices Mgmt erver DHCP erver DHCP erver BMC 20

34 Bootstrapping devices Mgmt erver DHCP erver DHCP erver BMC 20

35 Bootstrapping devices M Client Object torage Client Object M torage Mgmt erver DHCP erver DHCP erver BMC 20

36 Conclusion Next Generation torage Devices Dis-intermediates cloud applications to drive Enable innovation in hardware and software ecosystem Lower TCO Integration with: wift HDF Basho Riak Ceph cality 21

37 More information