eagate Kinetic Open torage Platform James Hughes and many others
2
2
A 3
A D 3
A D 3
A D 3
D A 3
D No. 77103, LibKinetic effective Jan. 18, 2009, A 3
D No. 77103, LibKinetic effective Jan. 18, 2009, ProtoBuf TCP/IP/GbE A 3
lication Clustering Management Interconnect D No. 77103, LibKinetic effective Jan. 18, 2009, ProtoBuf TCP/IP/GbE Proprietary to ystem Vendor GPL tandard torage A Proprietary to eagate 3
lication Clustering Management Interconnect D No. 77103, LibKinetic effective Jan. 18, 2009, ProtoBuf TCP/IP/GbE C++, Java, Python, Erlang, DIY Proprietary to ystem Vendor GPL tandard torage A Proprietary to eagate 3
lication Clustering Management Interconnect D No. 77103, LibKinetic effective Jan. 18, 2009, ProtoBuf TCP/IP/GbE Proprietary to ystem Vendor GPL tandard torage A Proprietary to eagate 3
lication Clustering Management Interconnect LibKinetic D No. 77103, LibKinetic effective Jan. 18, LibKinetic 2009, ProtoBuf TCP/IP/GbE Proprietary to ystem Vendor GPL tandard torage A Proprietary to eagate 3
lication Clustering Management Interconnect LibKinetic D No. 77103, LibKinetic effective Jan. 18, LibKinetic 2009, ProtoBuf TCP/IP/GbE Proprietary to ystem Vendor GPL tandard torage A D A D A D A D A A D A D A D A D Proprietary to eagate 3
A versus Kinetic Open torage Object torage ystems wift Basho Riak Ceph HDF!!!!! : ubject to NDA!!!!!! 4
Typical HA High Density Intel server Double ocket 48GB Ram 1000w Core A tray Connected to the server A A 5
Low cost HA Configuration Each drive talks to both switches Each switch has 2 by 10Gb/s Ethernet Kinetic Tray talks directly to ToR No servers Eth Eth Core 6
ystem Hardware Typical JBOD architecture Does not require a server, just JBODs to the ToR witch 10 JBOD 60 drives 4TB = 2.4PB/Rack 7
ystem Hardware 8
Kinetic Drive Provides RPC to Key/Value database Data is pre-indexed Compression and other value is easy and transparent P2P (Drive to Drive) copy of key ranges Communicate using existing Data Center Plumbing (TCP/IP) Multiple masters - Data sharing between machines Configurable caching per command WriteThrough, WriteBack, Flush Local space management 9
Kinetic ystems Clustering (performance, reliability, management) Compatibility with large scale applications (3, etc.) Centralized Management Reliability, availability, durability 10
Existing Traffic Flow ToR ToR Data Center : Core Router ubject to NDA Data Node Client A helf A helf 11
Kinetic Traffic Flow ToR ToR Data Center : Core Router ubject to NDA Kinetic helf Client 12
Conventional HDF ystem Client Namenode Datanode Datanode Datanode Datanode... Disk Disk... Disk Disk Disk... Disk Disk Disk... Disk Disk Disk... Disk Conventional Drives 13
HDF on Kinetic Client Namenode DataNode Daemon Datanode Datanode... K K K K K K........................ Disk Disk Disk Disk Disk Disk... Disk Disk... Disk Disk Disk... Disk Conventional Drives Kinetic Drives 14
Cumulative operations ordered by length 100% 80% operations 92% of the operations data Cumulative percentage 60% 40% 20% 32KB 0.5% of the data 0% 1.00 10.00 100.00 1000.00 10000.00 100000.00 Length (KB) 15
Map of Operations 0 512KB Length 0 1 2 3 4 Time (minutes) Location (TB) 0 1 2 3 16
Performance Metrics ame normal performance expectations equential Write: 50MB/s Random Write: 50MB/s equential Read: 50MB/s Random Read: 20% slower than traditional drives 17
Write Performance Results 120 MB/s 6000 Puts/s 90 4500 60 30 3000 1500 0 0 2 4 6 8 0 0 2 4 6 8 1MB values put rate (MB/s) 10B value put rate 18
Goals of API Data movement Get/put/delete/getnext/getprevious Versioned (== for success), options Range operations Multiple masters Authentication/Integrity/Authorization Cluster-able imple cluster configuration version enforcement P2P copy Management 19
Bootstrapping devices DHCP erver DHCP erver BMC 20
Bootstrapping devices Mgmt erver DHCP erver DHCP erver BMC 20
Bootstrapping devices Mgmt erver DHCP erver DHCP erver BMC 20
Bootstrapping devices M Client Object torage Client Object M torage Mgmt erver DHCP erver DHCP erver BMC 20
Conclusion Next Generation torage Devices Dis-intermediates cloud applications to drive Enable innovation in hardware and software ecosystem Lower TCO Integration with: wift HDF Basho Riak Ceph cality 21
More information http://seagate.com/www/kinetic https://developers.seagate.com http://github.com/eagate 22