Building a RESTful service with MongoDB and Spring MVC

Transcription

1 Building a RESTful service with MongoDB and Spring MVC p e n t a l o g. f r

2 O v i d i u S t a n c i u o s t a n c i p e n t a l o g. f r M a r c h p e n t a l o g. f r 2

3 Action plan What is NoSQL? What is MongoDB? Mongo operations using the JavaScript client Scaling MongoDB Spring Data MongoDB features Cloud Catalog demo Afterthoughts M a r c h p e n t a l o g. f r 3

4 What is NoSQL? 1998 Carlo Strozzi creates an open source database without a SQL interface, which he names NoSQL 2009 last.fm conference NoSQL used to describe all distributed databases which don't adhere to ACID principles NoSQL characteristics : Schema-free : normal RDBMS constraints do not apply No SQL No full ACID guarantees eventual consistency Limited transactions Distributed (horizontally scalable) and fault-tolerant M a r c h p e n t a l o g. f r 4

5 What is NoSQL? (continued) Types of NoSQL databases : Document oriented information encapsulated in text encoded documents (XML, YAML, JSON / BSON etc) or binary (PDF, MS Office formats etc) MongoDB, CouchDB, Jackrabbit, BaseX, SimpleDB, Oracle NoSQL Database, Couchbase Graph database nodes (contain the information) and edges (relationship between nodes) Neo4j, IBM DB2 (>=10), InfiniteGraph, DEX, FlockDB Key-Value store Cassandra, Riak, Redis, memcached, Google Appengine Datastore, Amazon DynamoDB M a r c h p e n t a l o g. f r 5

6 What is MongoDB? Document oriented, schema-less storage solution Developed by 10gen, first release in 2009 Open Source, AGPL and commercial licenses Written in C++ Windows, Linux, OS X, Solaris BSON (Binary JSON) { } "_id": ObjectId("4efa8d2b7d284dad101e4bc7"), "lastname": "PELLERIN", "firstname": "Franck", "dateofbirth": " ", "phonenumbers": [{ "type": "home", "number": " " },{ "type": "fax", "number": " ", "verified": false }], "address": {"street": "1 chemin des Loges", "city": "VERSAILLES"}, M a r c h p e n t a l o g. f r 6

7 What is MongoDB? (continued) Storage data files mapped in RAM (swapping managed by the operating system) Features : High Availability Journaling Replication Sharding (partitioning) Indexing Geoindexing (how close am I to?) Aggregation MapReduce Capped collections GridFS Drivers for : C, C++, C#, Erlang, Java, Javascript, Perl, PHP, Python, Ruby (official) Many other languages community supported M a r c h p e n t a l o g. f r 7

8 What is MongoDB? (continued) HTTP interfaces REST DrowsyDromedary (Ruby) MongoDB Rest (Node.js) alpha Mongodb Java REST server (Jetty) HTTP Sleepy Mongoose (Python) HTTP console HTTP interface which offers information / statistics about the server instance (installed by default and accessible on the server port ) M a r c h p e n t a l o g. f r 8

9 What is MongoDB? (continued) Tools : mongo interactive shell JavaScript mongostat server statistics mongotop similar to the Linux/UNIX top tool statistical information about actions performed on document collections mongosniff package sniffer for incoming / outgoing server data, similar to Wireshark (which also supports the MongoDB protocol) Unix-like OS only! mongoimport, mongoexport import / export tools (JSON, CSV, TSV) mongodump, mongorestore importing / exporting binary dumps M a r c h p e n t a l o g. f r 9

10 What is MongoDB? (continued) Glossary of terms : SQL / MySQL / Oracle term DB server / mysqld / oracle DB client / mysql / sqlplus database table row column table joins primary key group by MongoDB term mongod mongo database collection BSON document field embedded documents or linking _id field aggregation M a r c h p e n t a l o g. f r 10

11 What is MongoDB? (continued) Popularity Compared with other NoSQL solutions M a r c h p e n t a l o g. f r 11

12 What is MongoDB? (continued) Used with server-side languages : M a r c h p e n t a l o g. f r 12

13 MongoDB installation Windows : Linux : OS X : 32bit, 64bit, optimized 2008R2 or Windows 7 or higher Latest version (2.2) dropped support for XP No official installer 32bit, 64bit Debian and Red Hat packages available Homebrew or MacPorts packages 10gen binaries compiled for OS X Can be installed in under 5 minutes (without taking into account package download time) M a r c h p e n t a l o g. f r 13

14 The JavaScript shell (mongo) Synopsis : mongo [OPTIONS] [DB_ADDRESS] [FILE+] Very similar to mysql (JavaScript vs SQL) Commands : Command / function Description help Help me! db.help() db.<collection>.help() show dbs use <db> show collections show users show profile Lists all db object functions (singleton representing the current database) Collection functions (<collection> may not even exist) List all existing databases on this server Switches to the <db> database. From now on, the db object will represent the new database Lists the existing collections on the current database Lists the existing users on the current database Lists information about the last 5 operations which took more than 1 millisecond to execute M a r c h p e n t a l o g. f r 14

15 CRUD Insert / Update Insert : db.<collection>.insert(<document>) db.languages.insert({ name: 'JavaScript', release_year: 1994, designer: 'Brendan Eich', developers : ['Netscape', 'Mozilla Foundation'] }); Document identification Unique in the collection May have any type Automatically indexed the _id field Auto-generated if omitted at document insertion Mandatory for top level documents (optional for embedded) { "_id" : ObjectId("511f8e771f67a8d45558afa3"), "name" : "JavaScript", "release_year" : 1994, "designer" : "Brendan Eich", "developers" : [ "Netscape", "Mozilla Foundation" ] } M a r c h p e n t a l o g. f r 15

16 CRUD Insert / Update (continued) db.<collection>.save(<document>) Insert unspecified or missing _id Update, otherwise db.<collection>.update(<query>, <update> [, upsert[, multi]]) <query> equivalent to the SQL <update> Simple document field : value Document with $set field upsert default false WHERE clause multi default false only one document matched by <query> will be updated M a r c h p e n t a l o g. f r 16

17 CRUD Delete and referencing db.collection.remove([<query>] [, <justone>]) With no <query> it will empty the collections <justone> accepts both boolean or 0 / 1 Default false Referencing documents in other collections No joins Recommended construct : my_reference: {$ref: '<collection_name>', $id: <document_id>} db.companies.insert({_id: 'P5', name: 'Pentalog'}); db.employees.insert({_id: 1, name: 'Ovidiu Stanciu', company: {$ref: 'companies', $id: 'P5'}}); var ovidiu = db.employees.find({_id: 1}); db[ovidiu.company.$ref].find({_id: ovidiu.company.$id}); M a r c h p e n t a l o g. f r 17

18 CRUD Querying db.collection.find([<query>] [, <fields>]) <fields> mappings of the fields to be returned db.collection.findone([<query>]) <query> optional? it will return the first document in the collection (by default the oldest) Conditionals expressed as JSON objects Selectors : Comparison : $all, $gt, $gte, $in, $lt, $lte, $ne, $nin Logical : $and, $or, $not, $nor Element : $exists, $mod, $type JavaScript : $regex, $where Geospatial : $near, $within etc Array : $elemmatch, $size M a r c h p e n t a l o g. f r 18

19 CRUD Querying (continued) Controlling the result set - the cursor object batchsize() count() explain() foreach() hint() limit() map() max(), min() size() skip() sort() M a r c h p e n t a l o g. f r 19

20 Indexing At the collection level (maximum of 64 per collection) On one or more document fields (including embedded documents) B-tree Each query (including update) uses at most one index Exception queries with $or ensureindex(keys [, options]) Limitations in index usage : Negations : $ne, $not Arithmetic operations : $mod Some regular expressions : Yes : /^Brasov/, /^Bra.*/, /^bra.*/i No : /Brasov/ $where M a r c h p e n t a l o g. f r 20

21 Aggregation distinct(<field> [, <query>]) group({key, reduce, initial [, keyf] [, cond] [, finalize]}) key fields to group reduce(document, partialresult) aggregate function initial initializer code for the aggregate function keyf optional, function for creating a custom key object for grouping cond optional, filtering condition for the documents to be grouped finalize optional, function that will be executed for each item in the result set before returning the final value extra processing of the result set M a r c h p e n t a l o g. f r 21

22 Aggregation (continued) SQL term / function MongoDB aggregation operator WHERE GROUP BY HAVING SELECT ORDER BY LIMIT SUM() COUNT() $match $group $match $project $sort $limit $sum $sum M a r c h p e n t a l o g. f r 22

23 MapReduce Term inspired from the names of two functions commonly used in functional programming (their purpose in MapReduce is slightly different than the original functions) Map : transforming the dataset in key-value pairs Dividing the big problem into smaller, equivalent problems Reduce (aka fold, accumulate, compress, inject) : takes the mapping result and assembles it to construct the final Collecting and aggregating partial results obtained from solving sub-problems Example files containing pairs City recorded temperature M a r c h p e n t a l o g. f r 23

24 MapReduce (continued) db.collection.mapreduce(<map>, <reduce>, { out: <collection>, query: <document>, sort: <document>, limit: <number>, finalize: <function>, scope: <document>, jsmode: <boolean>, verbose: <boolean> }) map = function() {... emit(key, value); } reduce = function(key, values) {... return result; } M a r c h p e n t a l o g. f r 24

25 Scaling MongoDB High Availability : system design approach which aims to provide operational performance Target : 99.99% (four nines) That means 1 hour of downtime per year Perspective : 90% = 36 days of downtime per year Before scaling MongoDB : Optimize queries (that includes indexing) Know your working set data Tune your filesystem (ext3 vs ext4) Use performance disks (seek time matters), RAID matters, SSD is awesome M a r c h p e n t a l o g. f r 25

26 Scaling MongoDB Replication Performed via Replica Sets Purpose : read scalability, data backup and automatic failover Transparent to the client application Master Slave replication Writes performed on Master, then replicated between slaves To avoid confusion with older versions of MongoDB Master = Primary Slave = Secondary Each member of the replica set is a mongod instance Recommended replica set configuration : 3 instances = Primary + 2 x Secondaries Minimum : 3 instances = Primary + Secondary + Arbiter Maximum : 12 instances, but only 7 voting Replica set should have an odd number of data instances M a r c h p e n t a l o g. f r 26

27 Scaling MongoDB Replication (continued) M a r c h p e n t a l o g. f r 27

28 Scaling MongoDB Replication (continued) Automatic replica set failover : master election Rigging the election the priority property of a mongod Between 0 and 100, default is 1 Types of replica set members : Secondary-only cannot become primary (priority = 0) Hidden priority = 0 and hidden = true Hidden from client applications Delayed slavedelay > 0. Changes on primary replicate after the specified delay. Should also be hidden Arbiters hold no data. Exist only to participate in elections (break ties) Non-Voting votes = 0 (default is 1). Data instances in replica sets with more than 7 nodes M a r c h p e n t a l o g. f r 28

29 Scaling MongoDB Replication (continued) Write Concern and replica sets Write Concern = level of guarantee that the server will provide in response to a write operation It's the client's responsibility and configurable on every operation Issue the getlasterror command after a write operation Values : Errors ignored = connection and server exceptions (e.g. duplicate key) are ignored Unacknowledged = client receives only connection errors Receipt acknowledged = client receives connection + server exceptions Journaled = server confirms write operation once it has been committed to the journal (if enabled) Default write concern : < November 2012 (Java < v2.10.0) Unaknowledged ( Fire and Forget ) >= November 2012 (Java >= v2.10.0) Receipt acknowledged M a r c h p e n t a l o g. f r 29

30 Scaling MongoDB Replication (continued) Write Concern and replica sets Specify replica set default write concern using getlasterrordefaults getlasterror parameters : j (boolean) : true to specify journaled write concern. Default false w : number of servers to replicate the write before returning Default 1 (only the primary) majority = replicate the write to half + 1 servers Can specify a number greater than the number of existing servers the write will never return Use together with wtimeout wtimeout : milliseconds to wait for write propagation If the write doesn't complete in the specified timeframe, getlasterror will return an error status fsync (boolean) : wait for the server to write data to disk before returning Default false Prefer j to ensure durability (performance hit) M a r c h p e n t a l o g. f r 30

31 Scaling MongoDB Partitioning Autosharding Kristina Chodorow Scaling MongoDB (feb. 2011, O'Reilly) At the collection level big collections split over several mongod instances MongoDB sharding goals : Make the cluster invisible to the client Cluster always available for reads and writes Cluster should be easy to expand Actors : Router mongos service Configuration service mongod --configsrv instance Data node mongod instance Client connects to the router Most important aspect choosing an appropriate shard key Future versions of MongoDB may provide support to generate shard keys M a r c h p e n t a l o g. f r 31

32 Scaling MongoDB Partitioning (continued) M a r c h p e n t a l o g. f r 32

33 Scaling MongoDB Partitioning (continued) Ultimate sharding goal have similar load on all partitions If you have most of the load on one partition, then you're doing it wrong mongos router Entry point for interacting with the cluster Dispatches queries to partitions based on the shard key No shard key used in the query it's dispatched to all partitions Doesn't hold any data Config server Lightweight instance that stores data used by mongos Stores the relationship between chunks and where they reside One instance for testing purposes Cluster of 3 config servers for production If one config server goes down sharding cluster becomes read-only If all config servers go down cluster read-only and until restarted mongos perform M a r c h p e n t a l o g. f r 33

34 Scaling MongoDB (continued) Large production deployment schema M a r c h p e n t a l o g. f r 34

35 Scaling MongoDB Afterthoughts Scaling involves losing the ability to examine the data as a single snapshot Consistency and durability versus performance Reflect on operation write concerns Flexible deployment mongod instances performing different roles Audit instances Reporting instances Snapshot instances Easy setup and administration (compared to other solutions) Partitioning should be planned ahead, don't wait until it's too late Adding shards involves initial performance loss as data is re-partitioned and moved between shards MongoDB autosharding : Good it's automatic Bad it's automatic Scaling is where MongoDB really shines M a r c h p e n t a l o g. f r 35

36 Spring Data - MongoDB Vendor : VMWare / SpringSource Licence : Apache Licence, Version 2.0 Current version : GA M a r c h p e n t a l o g. f r 36

37 Spring Data MongoDB (continued) Features : Templating Resource abstraction Configure connections to mongod / mongos instances Collection lifecycle (create / drop) MapReduce / Aggregation Object Mapping @Indexed etc. Classes mapped to collections, objects to documents Repository support Queries derived from method signatures Geospatial queries M a r c h p e n t a l o g. f r 37

38 Spring Data MongoDB (continued) Configuration : Usage : M a r c h p e n t a l o g. f r 38

39 Spring Data MongoDB (continued) Object Document Mapping (ODM) : M a r c h p e n t a l o g. f r 39

40 Spring Data MongoDB (continued) Repositories : Declare an interface extending MongoRepository<T, ID> Built-in methods for CRUD : save, findone, findall, delete Built-in pagination support Derive queries at runtime using the method signature M a r c h p e n t a l o g. f r 40

41 Cloud Catalog Demo Demo application using : MongoDB server Maven 3 Eclipse + m2e + m2e-wtp Spring (Core, Context, WebMVC and AOP) REST via Spring MVC controllers JSON views via Jackson Mapper Spring Data MongoDB Mongo Java Driver Deployed on Tomcat 7 E n j o y! M a r c h p e n t a l o g. f r 41

42 Downsides of MongoDB No absolute guarantees about data integrity Default consistency model eventual consistency Write concern < 2.2 unacknowledged >= 2.2 receipt acknowledged Don't store valuable data in MongoDB! No full transactional support atomic operations on a document level No joins could lead to a lot of document duplication Pay attention to spelling (using the driver in a strongly typed language reduces the risk) Data size on 32 bit machines approx. 2GB of data MapReduce is quite slow Slower reads / writes compared to RDBMS solutions and even other NoSQL storages Depends on the actual context, schema design M a r c h p e n t a l o g. f r 42

43 Strengths DevOps : Schema migrations become much easier Applications are inconsistency aware Self-healing data layer Batches which verify and fix data inconsistency Background index generation Dumping / restoring collections or databases is trivial Empowers the developers Free Replication and sharding very easy to set up and administer Horizontal scaling Hardware maintenance RAM mapped data files OS in charge of swapping Can be combined with other solutions (Solr, Hadoop etc) M a r c h p e n t a l o g. f r 43

44 Q u e s t i o n s? M a r c h p e n t a l o g. f r 44

45 References / Additional resources MongoDB documentation : MongoDB in Action : MongoDB downloads : Hadoop and MongoDB : Source code for this presentation : Repo BitBucket (Mercurial) Direct download: M a r c h p e n t a l o g. f r 45

46 T h ank you! p e n t a l o g. f r