Storing billions of images in a hybrid relational and NoSQL database using Oracle Active Data Guard and Oracle NoSQL.

Size: px

Start display at page:

Download "Storing billions of images in a hybrid relational and NoSQL database using Oracle Active Data Guard and Oracle NoSQL."

Moses Carpenter
7 years ago
Views:

1 Storing billions of images in a hybrid relational and NoSQL database using Oracle Active Data Guard and Oracle NoSQL Aris Prassinos

2 Who I am Chief Engineer MorphoTrak, SAFRAN Group Oracle ACE 2007, Oracle ACE Director 2009 Oracle Magazine Content Management Architect of the Year (2005) ODTUG KSCOPE13 Slide 2

3 Agenda Introduce a real world use case Quick recap of relevant Oracle NoSQL concepts Present a solution that straddles relational and Oracle NoSQL databases Discuss things to consider and lessons learned ODTUG KSCOPE13 Slide 3

4 The use case Application storing large amounts of binary data images, documents, short videos High value data data consistency and minimal data loss Write Once Read Many / occasional updates, deletes Random reads throughout database regardless of age of record ODTUG KSCOPE13 Slide 4

5 Assumptions / Constraints High transaction volume with low latency Reads dominate writes High Availability using Active Standby datacenter Low latency between Primary and Standby datacenters Clients can arbitrarily connect to any datacenter System can tolerate a temporary loss of availability to guarantee consistency ODTUG KSCOPE13 Slide 5

6 Traditional RDBMS approach Oracle Active Data Guard / Synchronous replication to Standby(s) Reader farm of Standby's to scale Reads Real-time intelligent routing of calls between Primary and Standby(s) based on transaction requirements (e.g. read own writes) and Standby lag Oracle Global Data Services to load balance Reads on Standbys Oracle 12c feature Minimal loss of availability during failover Third Observer site needed when two datacenters are used ODTUG KSCOPE13 Slide 6

7 Why a hybrid solution High licensing cost for traditional RDBMS solution Other lower cost RDBMS not well suited for large scale image handling Concerns with storing images as regular files on the filesystem: durability / recovery consistency / integrity performance / scalability high availability / replication security Store the binary data and minimal metadata in Oracle NoSQL and everything else in Oracle database ODTUG KSCOPE13 Slide 7

8 Consistency between RDBMS and Oracle NoSQL Lack of transactions in Oracle NoSQL possibility of orphaned records left in Oracle NoSQL RDBMS is the source of truth Only query Oracle NoSQL using unique id obtained from RDBMS avoid queries on metadata in Oracle NoSQL Make NoSQL inserts idempotent (upserts) Oracle NoSQL may be ahead of an Active Data Guard Standby create a new version for each NoSQL update and store version# in RDBMS don't immediately delete from NoSQL but flag as deleted instead ODTUG KSCOPE13 Slide 8

9 Oracle NoSQL key-value store Major key : id, version Minor keys (columns): binary_data, binary_data.lob, mime_type, size, signature, delete_flag Internal representation id/version/binary_data => data id/version/binary_data.lob => data id/version/signature => data etc Note that Oracle NoSQL offers native streaming of large LOBs but different columns (binary_data, binary_data.lob) are needed to store both non-streamed and streamed LOBs under the same major key ACID semantics only for data within same Major key ODTUG KSCOPE13 Slide 9

10 Oracle NoSQL Scalability Reader farm approach to scale Reads single Master / multiple Replicas architecture run-time load balancing of Reads based on real-time latency stats Oracle NoSQL latency aware driver Sharding to scale NoSQL Writes Multiple Oracle NoSQL Replication Groups Each shard (Replication Group) has its own Master and Replicas Transparent to the client A physical node may host the Master of one shard and the Replicas of other shards replicas may later be moved to dedicated physical nodes for scaling out ODTUG KSCOPE13 Slide 10

11 Oracle NoSQL Master election Master is elected using PAXOS algorithm Master election held when a group of nodes forming a majority does not have a Master when less than half of the nodes remain alive the number that constitutes a majority can be manually overridden using je.rep.electablegroupsizeoverride possible to end up with two Masters if former Master is in group that consists of a minority of nodes after a network disconnect multiple Master problem solved by always requiring quorum writes ODTUG KSCOPE13 Slide 11

12 Two datacenter configuration Nodes split equally between the two sites A third lightweight arbiter site needed to allow automatic master election otherwise Standby site nodes cannot form a majority when Primary site lost Arbiter is an upcoming feature Arbiter placed at same site as Data Guard Observer otherwise manually set je.rep.electablegroupsizeoverride for manual failover By default NoSQL Master may not be on the same site as Data Guard Primary overridden using je.rep.node.priority for the preferred Master site nodes ODTUG KSCOPE13 Slide 12

13 NoSQL Writes 1. Attempt a quorum write (Durability.ReplicaAckPolicy.SIMPLE_MAJORITY) In terms of CAP favor Latency over Consistency. Possible inconsistencies can be handled when reading 2. If quorum write fails retry a few times to allow for potential new master election 3. If quorum write still fails decision depends on number of nodes (RF) In terms of CAP favor Consistency over Availability. If RF=2 attempt to write only to Master (Durability.ReplicaAckPolicy.NONE). If RF>2 an administrator needs to intervene since there is a possibility of having two masters if network partitioned and writing without quorum could cause divergent transactions. ODTUG KSCOPE13 Slide 13

14 NoSQL Reads 1. Read from any replica using unique id and version. Due to quorum writes some replicas may not yet have the requested data 2. If data not found then attempt to read from Master. We expect the data to exist since we query based on information in the RDBMS which is the source of truth. 3. If Master not reachable return error. This approach achieves strong consistency, minimal latency, high availability most Reads to proceed when Master is temporarily unavailable only Reads resulting in no data found return error ODTUG KSCOPE13 Slide 14

15 NoSQL log structured storage and backups Data always appended to log files up to a configurable size inserts, updates, deletes go to the latest log file Durability.SyncPolicy controls how often data written / flushed to disk old log files are immutable and may contain deleted or overwritten data Compaction process may move data from old log files to new ones when user defined percentage of valid records remains in a log file this is one of the primary candidates for tuning in a production system Snapshots and RSYNC can be used to perform incremental NoSQL backups ODTUG KSCOPE13 Slide 15

16 NoSQL backups using replication NoSQL data since time of last snapshot lost after a restore when only a single node used i.e. ReplicationFactor = 1 Imperative to use NoSQL replication for loss-less recovery Possible to setup a node as a dedicated backup server for one or more nodes define each backup Rep Node as non electable master (i.e. always be Replica) plan change-parameters... configproperties="je.rep.node.priority=0" ODTUG KSCOPE13 Slide 16

17 Conclusion Hybrid approach superior to just keeping pointers to binary data stored on the filesystem replication, sharding, adjustable consistency / durability Strong data consistency achieved with careful application coding keeping in mind the consistency / availability / latency tradeoffs Very good architectural fit with Active Data Guard master / replica architecture, observer / arbiter third site Better suited for a replicated NoSQL configuration rather than a single node minimum of two server nodes and if necessary a third arbiter ODTUG KSCOPE13 Slide 17

18 Questions - Answers ODTUG KSCOPE13 Slide 18

Practical Cassandra. Vitalii Tymchyshyn tivv00@gmail.com @tivv00

Practical Cassandra. Vitalii Tymchyshyn tivv00@gmail.com @tivv00 Practical Cassandra NoSQL key-value vs RDBMS why and when Cassandra architecture Cassandra data model Life without joins or HDD space is cheap today Hardware requirements & deployment hints Vitalii Tymchyshyn