Introduction to new high performance storage engines in MongoDB

Size: px

Start display at page:

Download "Introduction to new high performance storage engines in MongoDB 2.8 3.0"

Ashlie Scott
8 years ago
Views:

1 Introduction to new high performance storage engines in MongoDB Henrik Ingo Solutions Architect, MongoDB

2 Hi, I am Henrik 2

3 Introduction to new high performance storage engines in MongoDB 2.8 Agenda: - MongoDB and NoSQL - Storage Engine API - WiredTiger configuration + performance 3.0

4 Most popular NoSQL database 4

5 5 NoSQL categories Redis, Riak Cassandra Key Value Wide Column Document Neo4j Graph Map Reduce Hadoop 5

6 MongoDB is a Document Database Rich Queries Geospatial Text Search Aggregation Map Reduce Find Paul s cars Find everybody in London with a car built between 1970 and 1980 Find all of the car owners within 5km of Trafalgar Sq. Find all the cars described as having leather seats Calculate the average value of Paul s car collection What is the ownership pattern of colors by geography over time? (is purple trending up in China?) MongoDB { first_name: Paul, surname: Miller, city: London, location: [45.123,47.232], } cars: [ } { model: Bentley, year: 1973, value: , }, { model: Rolls Royce, year: 1965, value: , } 6

$geography over time? (is purple trending up in China?) MongoDB { first_name: Paul, surname: Miller, city: London, location: [45.123,47.$

7 Operational Database Landscape 7

8 MongoDB 3.0 & storage engines

9 Current state in MongoDB 2.6 Read-heavy apps Great performance B-tree Low overhead Good scale-out perf Secondary reads Sharding Write-heavy apps Good scale-out perf Sharding Per-node efficiency wish-list: Doc level locking Write-optimized data structures (LSM) Compression Other 9 Complex transactions In-memory engine SSD optimized engine etc...

reads Sharding Write-heavy apps Good scale-out perf Sharding Per-node efficiency

10 Current state in MongoDB 2.6 Read-heavy apps Write-heavy apps Great performance B-tree Low overhead Good scale-out perf Secondary reads Sharding Good scale-out perf Sharding Per-node efficiency wish-list: Doc level locking Write-optimized data structures (LSM) Compression How to get all of the above? Other 10 Complex transactions In-memory engine SSD optimized engine etc...

Secondary reads Sharding Good scale-out perf Sharding Per-node efficiency wish-list: Doc

11 MongoDB 3.0 Storage Engine API Read-heavy app Write-heavy app Special app MMAP WiredTiger 3rd party 11

12 MongoDB 3.0 Storage Engine API One at a time: Many engines built into mongod Choose 1 at startup All data stored by the Write-heavy same engine app Incompatible on-disk data formats (obviously) Compatible client API Read-heavy app Compatible Oplog & Replication Same replica set can mix different engines MMAP WiredTiger No-downtime migration possible Special app 3rd party 12

data stored by the Write-heavy same engine app Incompatible on-disk data formats (obviously)

13 Some existing engines MMAPv1 Improved MMAP (collection-level locking) WiredTiger Discussed next RocksDB LSM style engine developed by Facebook Based on LevelDB TokuMXse Fractal Tree indexing engine from Tokutek 13

RocksDB LSM style engine developed by Facebook Based

14 Some rumored engines Heap In-memory engine Devnull Write all data to /dev/null Based on idea from famous flash animation... Oplog stored as normal SSD optimized engine (e.g. Fusion-IO) KV simple key-value engine 14

.. Oplog stored as normal SSD optimized engine (e.g. Fusion-IO) KV simple key-value engine https://github.

15 WiredTiger

16 What is WiredTiger Modern NoSQL database engine flexible schema Advanced database engine Secondary indexes, MVCC, non-locking algorithms Multi-statement transactions (not in MongoDB 3.0) Very modular, tunable Btree, LSM and columnar indexes Snappy, Zlib, 3rd-party compression Index prefix compression, etc... Built by creators of BerkeleyDB Acquired by MongoDB in 2014 source.wiredtiger.com 16

0) Very modular, tunable Btree, LSM and columnar indexes Snappy, Zlib, 3rd-party compression Index

17 Choosing WiredTiger at server startup mongod --storageengine wiredtiger 17

18 Main tunables exposed as MongoDB options mongod --storageengine wiredtiger --wiredtigercachesizegb 8 --wiredtigerdirectoryforindexes /data/indexes --wiredtigercollectionblockcompressor zlib --syncdelay

/data/indexes --wiredtigercollectionblockcompressor zlib --syncdelay 30

19 All WiredTiger options via configstring (hidden) mongod --storageengine wiredtiger --wiredtigerengineconfigstring "cache_size=8gb,eviction=(threads_min=4,threads_max=8), checkpoint(wait=30)" --wiredtigercollectionconfigstring "block_compressor=zlib" --wiredtigerindexconfigstring "type=lsm,block_compressor=zlib" --wiredtigerdirectoryforindexes /data/indexes See docs for wiredtiger_open() & WT_SESSION::create() wt.html#ga9e6adae3fc6964ef837a62795c7840ed s_e_s_s_i_o_n.html#a358ca4141d59c345f401c bbb 19

--wiredtigerindexconfigstring "type=lsm,block_compressor=zlib" --wiredtigerdirectoryforindexes /data/indexes See docs for wiredtiger_open() &

20 Also via createcollection(), createindex() db.createcollection( "users", { storageengine: { wiredtiger: { configstring: "block_compressor=none" } } )

"block_compressor=none" } } ) http://docs.mongodb.org/master/reference/method/db.

21 More... db.serverstatus() db.collection.stats() 21

22 Understanding and Optimizing WiredTiger

23 WiredTiger SE Understanding WiredTiger architecture Btree LSM Columnar Cache (default: 50%) None Snappy Zlib OS Disk Cache (Default: 50%) 23 Physical disk

24 WiredTiger SE Covering 90% of your optimization needs Btree LSM Columnar Cache (default: 50%) None Snappy Zlib Decompression time OS Disk Cache (Default: 50%) 24 Disk seek time Physical disk

25 WiredTiger SE Strategy 1: fit working set in Cache Btree LSM Columnar cache_size = 80% Cache (default: 50%) None Snappy Zlib OS Disk Cache (Default: 50%) 25 Physical disk

26 WiredTiger SE Strategy 2: fit working set in OS Disk Cache Btree LSM Columnar Cache cache_size (default: = 10% 50%) None Snappy Zlib OS OS Disk Cache (Remaining: (Default: 50%) 90%) 26 Physical disk

27 WiredTiger SE Strategy 3: SSD disk + compression to save Btree LSM Columnar Cache (default: 50%) None Snappy Zlib OS Disk Cache (Default: 50%) 27 Physical SSDdisk

28 WiredTiger SE Strategy 4: SSD disk (no compression) Btree LSM Columnar Cache (default: 50%) None Snappy Zlib OS Disk Cache (Default: 50%) 28 Physical SSDdisk

29 Performance What problem is solved by LSM indexes? Easy: Add indexes Easy: No indexes Hard: Smart schema design (hire a consultant) LSM index structures (or columnar) Fast reads Both Fast writes 29

30 2B inserts (with 3 secondary indexes) 30

HO5604 Deploying MongoDB. A Scalable, Distributed Database with SUSE Cloud. Alejandro Bonilla. Sales Engineer abonilla@suse.com

HO5604 Deploying MongoDB. A Scalable, Distributed Database with SUSE Cloud. Alejandro Bonilla. Sales Engineer abonilla@suse.com HO5604 Deploying MongoDB A Scalable, Distributed Database with SUSE Cloud Alejandro Bonilla Sales Engineer abonilla@suse.com Agenda SUSE Cloud Overview What is MongoDB? 2 Getting familiar with the Cloud