CENG 553 DATABASE. BIG DATA:NOSQL SYSTEMS PRESENTATION Okan Yaman/Eray Ölgün/Emin İnal

Transcription

1 CENG 553 DATABASE BIG DATA:NOSQL SYSTEMS PRESENTATION Okan Yaman/Eray Ölgün/Emin İnal

2 CONTENTS INTRODUCTION TYPES OF DATABASES (ARCHITECTURES) 1. GRAPH DB 2. KEY VALUE DB 3. COLUMN BASED DB

3 INTRODUCTION WHAT IS BIG DATA? WHAT MAKES DATA BIG?

4 INTRODUCTION No single standard definition Big Data is data whose scale, diversity, and complexity require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it

5 INTRODUCTION Credit: The data deluge, Economist; Understanding Big Data, Eaton et al. The world is creating ever more data (and it s a mainstream problem) Mankind created data 150 exabytes in 2005 (exabyte is a billion gigabytes) 1200 exabytes in exabytes in 2020 (expected by IBM) 5

6 INTRODUCTION Examples: U.S. drone aircraft sent back 24 years worth of video footage in 2009 Large Hadron Collider generates 40 terabytes/second Bin Laden s death: 5106 tweets/second Around 30 billion RFID tags produced/year Oil drilling platforms have 20k to 40k sensors Our world has 1 billion transistors/human Ref.

7 INTRODUCTION (A Quick Primer on Data Sizes)

8 INTRODUCTION

9 INTRODUCTION

10 INTRODUCTION

11 INTRODUCTION

12 INTRODUCTION

13 INTRODUCTION

14 INTRODUCTION

15 INTRODUCTION

16 INTRODUCTION Characteristics of Big Data: 1-Scale (Volume) Data Volume 44x increase from From 0.8 zettabytes to 35zb Data volume is increasing exponentially Exponential increase in collected/generated data 16

17 INTRODUCTION Characteristics of Big Data: 2-Complexity (Varity) Various formats, types, and structures Text, numerical, images, audio, video, sequences, time series, social media data, multi-dim arrays, etc Static data vs. streaming data A single application can be generating/collecting many types of data To extract knowledge all these types of data need to linked together 17

18 INTRODUCTION Characteristics of Big Data: 3-Speed (Velocity) Data is begin generated fast and need to be processed fast Online Data Analytics Late decisions missing opportunities Examples E-Promotions: Based on your current location, your purchase history, what you like send promotions right now for store next to you Healthcare monitoring: sensors monitoring your activities and body any abnormal measurements require immediate reaction 18

19 INTRODUCTION Big Data: 3V s 19

20 INTRODUCTION Some Make it 4V s 20

21 INTRODUCTION Harnessing Big Data OLTP: Online Transaction Processing (DBMSs) OLAP: Online Analytical Processing (Data Warehousing) RTAP: Real-Time Analytics Processing (Big Data Architecture & technology) 21

22 INTRODUCTION Who s Generating Big Data Mobile devices (tracking all objects all the time) Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Sensor technology and networks (measuring all kinds of data) The progress and innovation is no longer hindered by the ability to collect data But, by the ability to manage, analyze, summarize, visualize, and discover knowledge from the collected data in a timely manner and in a scalable fashion 22

23 INTRODUCTION The Model Has Changed The Model of Generating/Consuming Data has Changed Old Model: Few companies are generating data, all others are consuming data New Model: all of us are generating data, and all of us are consuming data 23

24 ARCHITECTURES GRAPH DB WHAT IS A GRAPH DB?(Neo4j; InfoGrid; Sones GraphDB; AllegroGraph; InfiniteGraph)

25 ARCHITECTURES GRAPH DB Graph databases replace relational tables with structured relational graphs of interconnected key-value pairings

26 ARCHITECTURES GRAPH DB They are similar to object-oriented databases as the graphs are represented as an objectoriented network of nodes (conceptual objects), node relationships ( edges ) and properties (object attributes expressed as keyvalue pairs).

27 ARCHITECTURES GRAPH DB They are the only of the four NoSQL types discussed here that concern themselves with relations, and their focus on visual representation of information makes them more human-friendly than other NoSQL DMS.

28 ARCHITECTURES GRAPH DB Graph NoSQL Database (Source: )

29 ARCHITECTURES GRAPH DB SUITABLE USE CASES 1. CONNECTED DATA (SOCIAL NETWORKS) 2. ROUTING,DISPATCH AND LOCATION-BASED SERVICES 3. RECOMMENDATION ENGINES

30 ARCHITECTURES GRAPH DB WHEN NOT TO USE?

31 KEY-VALUE DB Key Value is one of the non-relational database model like graph and document oriented database. Ex: Dynamo (Amazon); Voldemort (LinkedIn); Redis; BerkeleyDB; Riak Key value stores allow the application developer to store schema-less(schema-free) data.

32 KEY-VALUE DB This data is usually consisting of a string which represents the key and the actual data which is considered to be the value in the «key - value» relationship. This schema-free data store is highly flexible and scalable.

33 KEY-VALUE DB Data searches in key-value data stores can usually only be performed against keys, not values, and are limited to exact matches

34 KEY-VALUE DB WINDOWS AZURE TABLE STORAGE (WATS) Windows Azure Table Storage is a faulttolerant, ISO certified NoSQL key-value store. Windows Azure Table Storage can be useful for applications that must store large amounts of non-relational data, and need additional structure for that data.

35 KEY-VALUE DB WINDOWS AZURE TABLE STORAGE Tables offer key-based access to schema-free data at a low cost for applications with simplified data-access patterns. While Windows Azure Table Storage stores structured data without schemas, it does not provide any way to represent relationships between the data.

36 KEY-VALUE DB WINDOWS AZURE TABLE STORAGE A Table is a set of entities (ROWS). An Entity is a set of properties (COLUMNS).

37 KEY-VALUE DB WINDOWS AZURE TABLE STORAGE Each Entity must have (fixed in schema) A partition key A row key A time stamp Each entity can have up to 252 properties. Partition key and Row key properties are the unique ID of the entity

38 KEY-VALUE DB WINDOWS AZURE TABLE STORAGE Partition Key enables scalability. The entities with the same partition key are served by a single server, that is, you cannot split entities to different storage nodes with same partition key. Row Key uniquely identifies the entity in the partition.

39 KEY-VALUE DB WINDOWS AZURE TABLE STORAGE Property Types Partition Key and Row Key String (up to 64kb) Other Properties String (up to 64kb) Binary (up to 64kb) Bool DateTime Guid Int Int64 Double

40 KEY-VALUE DB WINDOWS AZURE TABLE STORAGE Microsoft PDC 2008 Presentation

41 WINDOWS AZURE TABLE STORAGE Consider using Windows Azure Table Storage when: Your application must store significantly large data volumes (expressed in multiple terabytes) while keeping costs down. Your application stores and retrieves large data sets and does not have complex relationships that require server-side joins, secondary indexes, or complex server-side logic.

42 WINDOWS AZURE TABLE STORAGE Your application requires flexible data schema to store non-uniform objects, the structure of which may not be known at design time. You need to achieve a high level of scaling without having to manually shard your dataset.

43 KEY-VALUE DB In most cases, Key-Value data stores are used to store 1. SESSION INFO 2. USER PROFILES,PREFERENCES 3. SHOPPING CART DATA

44 SESSION INFO KEY-VALUE DB Use Cases Generally, every web session is assigned to a unique SessionID on disk or in the RDMS will greatly benefit from moving to key value data store, since everything in session can be stored by a single PUT request or retrieved using GET.

45 KEY-VALUE DB Use Cases USER PROFILES, PREFERENCES Almost every user has a unique UserID, UserName or some other attributes, as well as preferences such as language, color, time zone, which products the user has access to, and so on. This can all be put into an object, so getting preferences of a user takes a single GET operation.

46 SHOPPING CART DATA KEY-VALUE DB Use Cases E-commerce web sites have shopping carts tied to the user. As we want the shopping carts to be available all the time, across browsers, machines and sessions, all the shopping cart data can be put into the value where the key is the UserID.

47 KEY-VALUE DB WHEN NOT TO USE? If you have 1. RELATIONSHIPS AMONG DATA 2. MULTIOPERATION TRANSACTIONS 3. QUERIES BY DATA 4. OPERATIONS BY SETS

48 KEY-VALUE DB In conclusion, if you are going to make queries against key values and you need schema-free data stores, you should consider using Key-Value data stores. For more information and pricing details about Windows Azure Table Storage, visit

49 COLUMN-BASED DB

50 COLUMN-BASED DB Contents Raw Oriented Storage Column-Store Introduction Column-Store Data Model Column Family Who uses Column Oriented DB Conclusion

51 COLUMN-BASED DB Row-Oriented Storage In row-oriented databases, row value data is usually stored contiguously:

52 COLUMN-BASED DB What is Columnar Database? A column-oriented DBMS is a database management system that stores its content by column rather than by row.

53 COLUMN-BASED DB Column-Oriented Storage Column-oriented databases primarily work on columns All columns are treated individually Values of a single column are stored contiguously This allows array-processing the values of a column

54 COLUMN-BASED DB Are These Two Fundamentally Different? The only fundamental difference is the storage layout. However: we need to look at the big picture.

55 COLUMN-BASED DB RDBMS vs. Columnar Oriented DBMS (Physical Level)

56 COLUMN-BASED DB Column vs Row So column stores are suitable for read-mostly, read-intensive, large data repositories

57 COLUMN-BASED DB Column-Oriented Example Use Case - 1 Column stores can greatly improve the performance of queries that only touch a small amount of columns Simple math: table t has a total of 10 GB data, with Column a: 4GB Column b: 2GB Column c: 3GB Column d: 1GB If a query only uses column d, at most 1GB of data will be processed by a column store In a row store, the full 10 GB will be processed

58 COLUMN-BASED DB Column-Oriented Example Use Case - 2 The physical structure of a column-family database enables you to partition your data vertically. This strategy helps to minimize the amount of data that the database software actually needs to read from disk (or memory) to satisfy a query, or that it needs to save to disk (or memory) to update information.

59 COLUMN-BASED DB Column-Oriented Example Use Case - 2

60 COLUMN-BASED DB What is Column Family? Column family stores are modeled on Google s BigTable. The data model is based on a sparsely populated table whose rows can contain arbitrary columns, the keys for which provide natural indexing.

61 COLUMN-BASED DB What is Column Family? The simplest unit of storage is the column itself, consisting of a name-value pair. Any number of columns can be combined into a supercolumn, which gives a name to a sorted set of columns. Columns are stored in rows, and when a row contains columns only, it is known as a column family. When a row contains super columns, it is known as a super columnfamily.

62 COLUMN-BASED DB The four building blocks of column family storage

63 COLUMN-BASED DB Super Column Family It provides the nested hashmap structure into which we decompose our data. we show how we might map a recording artist and his albums into a super column family structure logically, it s really nothing more than maps of maps

64 COLUMN-BASED DB Why Column Oriented Database? Most data warehousing applications make more number of reads and lesser number of writes. Can be significantly faster than row stores for some applications Fetch only required columns for a query Better cache effects Better compression (similar attribute values within a column) Row oriented databases have the overhead of seeking through all columns.

65 COLUMN-BASED DB Who uses Column-oriented DB?

66 Conclusion In summary let me say this: I believe, generally speaking, that each of these three data storage technologies offer specific features and therefore should be used in specific ways. ROW based databases should prevail when you want a complex, but not too-huge data set that requires efficient storage and retrieval for OLTP and even some OLTP usage; COLUMN based database are clearly aimed at analytics; optimized for aggregations coupled with huge data compression and should be adopted for most business intelligence usage; NoSQL based data solutions step in when you need to ingest BIG DATA, fast, Fast, FAST and when you only really need to make simple correlations across the data quickly;

67 Thank You! Thank you for listening to us..

68 REFERENCES

69 REFERENCES Judith Hurwitz, Alan Nugent, Dr. Fern Halper and Marcia Kaufman (2013). Big Data For Dummies.John Wiley & Sons, Inc. ISBN Pramod Sadalage and Martin Fowler (2012). NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence. Addison-Wesley. ISBN Moniruzzaman AB, Hossain SA (2013). "NoSQL Database: New Era of Databases for Big data Analytics - Classification, Characteristics and Comparison". 12presentations/fcsm_june2012_cooper_mell.pdf

70 QUESTIONS / COMMENTS?