Experimentation on Cloud Databases to Handle Genomic Big Data

Transcription

1 Experimentation on Cloud Databases to Handle Genomic Big Data Presented by: Abraham Gómez, M.Sc., B.Sc. Academic Advisor: Alain April. Ph.D,M.Sc.A, B.A.

2 Agenda Concepts & Overview Technology description Case study Methodology Obtained Results 6 Conclusion Page 2

3 Concepts & Overview At present, industries & academics have become focused on delivering information. Past local applications are moving to applications in the Cloud. After all, this migration is easy? Page 3

4 Concepts & Overview Page 4

5 Concepts & Overview This new field of research is called big data. Often time-sensitive Velocity Variety Extends beyond structured data: text, audio, video, etc. Volume Comes in one size: large Page 5

6 Concepts & Overview Why genetics/genomic are related to Big Data? -The time and cost of DNA sequencing has been reduced by a factor of 1 million. -Near future, mapping a personal genome could cost a few hundred dollars. Page 6

7 Concepts & Overview Why genetics/genomic are related to Big Data? -Now, a personal genome = 100 GB. -If everybody wants its own genome = hundreds of petabytes of data. (Volume). -If everybody wants get it fast (Velocity). -If nobody wants only text (Variety). Page 7

8 Agenda Concepts & Overview Technology description Case study Methodology Obtained Results 6 Conclusion Page 8

9 Concepts & Overview Companies presently use relational database (RDB) technology. Big data applications also use RDB technology Page 9

10 Concepts & Overview However, case study shows that accessing petabytes of data efficiently (in the cloud) is challenging. -Solutions: -Sharding but creates technological problems. -Buy specialized hardware but creates budget problems. Page 10

11 Concepts & Overview Emerging Cloud Computing (CC) database technology (No-SQL) is a new research domain that addresses these challenges. Page 11

12 Hadoop Project Page 12

13 Hadoop Project Several subprojects: HBase is the Hadoop database. So, is a distributed, scalable, big data store. Page 13

14 Agenda Concepts & Overview Technology description Case study Methodology Obtained Results 6 Conclusion Page 14

15 Case study Data from Centre d Excellence en Neuromique de l'université de Montréal (CENUM) has been used. The research group uses a fivesteps process to collect the information saved in the database: Page 15

16 Case study Page 16

17 Case study The current RDB start to show weakness by the impressive amount of data. Some tables only with a sample of 25% have: By next 2 year will have 200% more data. Coverage 20GB Page 17 Variant 2,1GB

18 Case study SQL 1 = Exclude some variants present in 1 or more individuals (list of samples). SELECT DISTINCT variant_id FROM sample_variant WHERE sample_id in (146,162,167,189,193) GROUP BY variant_id ORDER BY variant_id Page 18

19 Case study Some SQL requests become very slow or sometimes even impossible. Page 19

20 Case study The problem has three dimensions: Data Analysis Data Storage Data Retrieval Page 20

21 Agenda Concepts & Overview Technology description Case Study Methodology Obtained Results 6 Conclusion Page 21

22 Methodology Several case studies show that HBase is a feasible solution. HBase Pros: -Avoid the sharding problems. -Avoid to buy specialized Hardware. HBase Cons: -Redesign database schema. Page 22

23 Methodology Case study analysis. -Exactly what they need? -Thinks a long term data store. -Improve data retrieval. Redesign the DB schema based on: -SQL request ( Read / Write ). Page 23

24 Methodology Redesign the DB schema based on: -Take reference tables and mixed in one. -Faster data retrieval. -Avoid some relationships. Using Sqoop for data migration. Page 24

25 Methodology Page 25

26 Agenda Concepts & Overview Technology description Case Study Methodology Obtained Results 6 Conclusion Page 26

27 Obtained Results Create an standard in migration process from RDB to No-SQL DB. -Proc #1: Create column families according access / write pattern (SQL request). -Proc #2: Merge reference tables. -Proc #3: Spreading relationships manually. Use of Sqoop. Page 27

28 Obtained Results The SQL request has a better performance: HBase RDB Page 28

29 Agenda Concepts & Overview Technology description Case Study Methodology Obtained Results Conclusion Page 29 Rules for Converting a Relational Database to a No-SQL Database for Cloud Computing

30 Conclusion Migration from RDB to HBase can be standardized. More data from case study will be needed. Until now the results support that HBase is faster than RDB. (Controlled environment). Page 30

31 Conclusion HBase is a good solution for: 1. Data Storage because is possible to save petabytes of data. 2. Data Retrieval because you can obtain your data fast. 3. Data Analysis if you can uses other Hadoop subprojects (MapReduce, Hive) from HBase. Page 31

32 Thanks!!! Questions? Page