Transactions and ACID in MongoDB Kevin Swingler Contents Recap of ACID transactions in RDBMSs Transactions and ACID in MongoDB 1
Concurrency Databases are almost always accessed by multiple users concurrently A user may be a person, or a process or program Different users can interact in a way that causes the database to become inconsistent or simply introduce errors Example Relational Integrity Imagine a database containing a table of managers and the staff they manage Imagine process 1needs to remove manager A from the table as he is leaving Check manager has no team members Delete manager And process 2needs to assign a new worker to a manager Identify manager with fewest team members Assign new worker to manager 2
Possible Problems 1. Process 1verifies that manager Ahas no team members 2. Process 2 looks for the manager with fewest members - finds manager A(with none) 3. Process2assigns new team member to manager A 4. Process 1deletes manager A Now the database has lost integrity as the new team member references a manager who is not in the database Example Lost Update Bank process is adding interest While person is removing cash from machine Adding Interest Read balance Calculate interest Add to balance figure Write new balance Removing cash Read balance Subtract amount withdrawn Write new balance Removed cash is overwritten by new interest calculation! So called lost update 3
Transactions The notion of a transaction is designed to remove the risk of examples like those above This is covered in detail in another course, but involves: The definition of a transaction as a series of database operations Locking of fields to prevent other processes writing until a transaction is complete Queries and Transactions A query is a single database operation Read, write, delete, etc. A transaction is a series of queries, often interspersed with other calculations Read, Add, Write Transactions may be spread over time if user interaction is required Read, wait for user input, write... 4
ACID Transactions ACID transactions are core to relational databases Atomic Cannot be broken into smaller components All or Nothing Consistent Always leave the database in a consistent state Independent Do not interfere with other transactions Durable Once complete, cannot be undone (as in the bank example) Transactions in NoSQL Different NoSQLdatabases have different levels of ACID support. For some applications, the notion of a transaction is unnecessary For others it is essential There are a number of ways of handling it 5
Concurrent Queries Queries can be run in serial or parallel Both cases can cause inconsistency, but the parallel case has some extra problems Shardeddatabases can run concurrent queries across multiple shards The database server chooses the order in which queries are run (usually in temporal order as they arrive) Concurrency in MongoDB docs.mongodb.org/master/faq/concurrency/ describes concurrency in 3.0 Locking used to be at the database level As of version 3.0, locking is at the collection level 6
Transactions in MongoDB MongoDBwrite operations are Atomic at the document level (including documents within a document) Transactions across multiple documents can be made atomic using two phase commits http://blog.mongodirector.com/atomicity-isolation-concurrency-in-mongodb/ Two Phase Commit An attempt at bringing transactions to MongoDB Considered a bit of a hack by many Okay if you really need NoSQLand transactions are not the main requirement Otherwise, will a RDBMS be better? http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/ http://cookbook.mongodb.org/patterns/perform-two-phase-commits/ 7
Two Phase Commit Set up a collection called transactions { Target document, source document, value, state } Add a pendingtransactions=[] field to documents Create a new transaction with state = initial When transaction starts, set state = pending Store transaction id in pendingtransactions[] Apply transactions to both documents Set state = committed Use find() to see if documents are correct If so, set state = done Example Add pendingtransactions field to accounts documents Add record to transactions collection 8
Example Get the transaction from the collection Update the balances - put transaction id in pendingtransactions array Example Set the transaction state to done and remove the pending transactions array from the account document 9
CAP Theorem States that you can have at most two of: Consistency Accessibility Partition Tolerance Consistency In a distributed database, maintaining consistency means ensuring that every read gets the most recent data and every write is durable Write inconsistency can occur if two versions of the database (each on a different machine) are updated at the same time Read inconsistency occurs if a read is made from one machine after another is updated 10
Eventual Consistency Replication consistency means that every read, no matter which replication it is made from, gives the same answer Requires writes to propagate fully to every node before a read can take place: not always necessary Eventual consistency allows some nodes to be a little behind others, but to catch up eventually (really, quite quickly) Examples Facebook not a problem if a friend in the UK can see a new photo of your cat while a friend in America has to wait a few more seconds before it appears Paypal needs to be sure the balance it reads is correct, and that another node hasn t spent the remaining money 11
Read Your Writes Consistency Imagine a blog database, distributed across several nodes If I write to one node and you read from another, you won t see my post until it propagates to your node eventual consistency But, if I write to one node and then, due to load balancing, read from another my post has vanished! Sticky Sessions To ensure read your writesconsistency, a session between the user and the node can be maintained so that the entire interaction is consistent Can reduce the efficiency of load balancing 12
Availability One way to maintain consistency is to make sure updates are fully propagated or writes are forced through a master node That means that a node might be reachable on the network, but still unavailable because it either hasn t been updated or can t contact the master node So available really means able to respond Read / Write Available In the case where writes need to go through a master node, but reads don t, availability depends on the request Read available Write unavailable 13
Hotel booking system Example Read from a slave (might be out of date) Write through master If no rooms available, report room was lost If master not available, either report error or write to slave and deal with conflict later Keeps reads (most frequent query) fast using slaves Keeps writes consistent using master Partition Tolerance A network becomes partitioned when one or more links fail causing some machines to become isolated from some others If a master node is in one partition, then the slaves in the other can t reach it So those slaves become unavailable until the partition is repaired and they are updated 14
Without Partition Tolerance A database can be partition tolerant if it is happy to lose either consistency or availability as soon as it is partitioned It can keep consistent by making some nodes unavailable (CP) Or stay available but accept that it will become inconsistent (AP) While everything is working (no partitions) a database can be consistent and available Consistency Latency It takes some time (however small) to update all nodes in a network after a write That latency is like temporary partition So in a sense, you always have brief partitions So you can only really choose between consistency and availability 15
Really a Continuum In reality, the CAP qualities are not all or nothing options, but a continuum. You need to think about: How much do I need consistency? How long are users prepared to wait for it? Can I get away with write consistency only? How can conflicts be solved later, and at what cost? Read / Write Quora Replication is generally only an additional two nodes, so three copies in total Latency not much of a problem as updates propagated fast Can speed things up more by using a read or write quorum Write is acknowledged once two of the three nodes have it, then a read accesses two of the three and picks the most recent 16
Trade-off of Read/Write Quorum Write to 3, read from 1 Write to 2, read from 2 Write to 1, read from 3 The Write to part means write that many and then acknowledge write as complete Durability Memory is MUCH faster than disk, even SSD Running a DB in memory is desirable where speed is crucial Disk writes can be at intervals or, for temporary stores, never Node crashes cause permanent data loss Worth it for things like web session data 17