Big Data, Big Challenges

Transcription

1 Big Data, Big Challenges Big Data, Big Challenges DeIC Conference 2013 Michael Sullivan, M.D.

2 Big Data Variety Volume Visualiza0on Velocity

3 Variety Roger Ebert Roger was diagnosed with cancer of a salivary gland in 2002 and died in What I believe is that all clear- minded people should remain two things throughout their life:mes: curious and teachable.

4 TesGng for Cancer Today Diagnosis: Sublingual Adenocarcinoma (rare) Standard treatment: None Tissue biopsies showed a rare cancer. Serial CT scans of the lung showed disease progression.

5 Cancer Diagnosis and Treatment in the Future Central Dogma of Molecular Biology

6 DNA Sequencing Results This Circos plot shows gene expression before (T1) and arer (T2) treatment.

7 Signaling Network 3D RET protein structure Disrupted signaling pathways drive tumor proliferagon.

8 Heterogenous Data

9 NIH/NCI Cancer Genomics Cloud Ini0a0ve u The Cloud - move the compute, not the data. u 3 pilot centers maybe CGHub, BioNimbus, and Broad. u Preloaded with tools and data (TCGA will have 2.5 PB). u Assumes exisgng infrastructure base. u EsGmated $5 million per site per year.

10 Trans- NIH: BD2K and Infrastructure Plus u Big Data to Knowledge (BD2K) has 4 parts: v FacilitaGng Broad Use of Data (catalog, metadata) v Analysis Methods and SoRware (access to HPC) v Enhancing Training (data science) v Centers of Excellence (6-8 centers) u Infrastructure Plus Intramural (campus) upgrade

11 Volume May 5, 2013 mega = 10 6 giga = 10 9 tera = peta = exa = ze`a = yo`a = googol = googolplex = 10 googol

12 Growth of Storage at NCBI ( )

13 NCBI Outbound Data (TB/Month) Trillions Total Feb Apr Jun Aug Oct Dec Feb Apr Jun Aug Oct Dec Feb Apr Jun Aug Oct Dec Feb Apr Jun Aug Oct Dec Feb Apr Jun Aug

14 InformaGon Content: u = log 2 (n) in silico: n = 2, u = 1 1 byte = 8 bits in vivo: n = 4, u = 2 1 byte = 4 base pairs

15 Storage in silico vs. in vivo NCBI or EBI 20 Petabytes Human 160 ZeHabytes* Ra0o: 1 : 8,000,000 * CalculaGon: 6.4 Gbp 1 Byte 10 microbe bp cells 160 ZB X X X = cell 4 bp 1 human bp person person GB = Gigabyte (10 9 ) PB = Petabyte (10 15 ) ZB = Ze`abyte (10 21 ) bp = base pair Gbp = Gigabase pair (10 9 ) Assume all cells are diploid.

16 Visualiza0on Physics LHC Lead Ion Collision Source: CERN (ALICE detector)" 16 10/1/13, 2012 Internet2 Life Sciences MRI Monkey Brain Source: Van Wedeen, M.D., Martinos Center and Dept. of Radiology, Massachusetts General Hospital and Harvard University Medical School"

17 Circos Mapping of Dog and Human Genes Human 23 Chromosomes 3.1 Gb Dog 39 Chromosomes 2.4 Gb

18 Human Dog Synteny by Dog Chromosome

19 Protein- Protein InteracGons

20 The Human Connectome

21 Velocity

22 White House Champions of Open Science David Lipman, NCBI Atul Bu`e Stanford University David Altshuler Broad InsGtute Stephen Friend Sage Bionetworks

23 Science DMZ Science DMZ h`p://fasterdata.es.net/science- dmz/science- dmz- security/

24 Growth of perfsonar ~30 Countries ~200 Domains ~850 Instances

25 NCBI EBI Performance Problem: 10 Mb stream (expected 500Mb) SoluGon: perfsonar at the endpoints localized the problem

26 Dr. Lin Fang US China 10 Gbps Link Fed Ex: Internet + FTP: China- US 10G Link: 2 days 26 hours 30 seconds Dr. Dawei Lin Sample.fa (24GB) 26 10/1/13, 2012 Internet2

27 Scalability 300 million light years ~ meter Galaxy Neuron

28 NIST Big Data Reference Architecture NIST RA M0226v5 INFORMATION VALUE CHAIN System Orchestrator Data Provider DATA SW Applica0on Provider Collec0on Cura0on Analy0cs Visualiza0on Access DATA SW IT Provider Big Data Processing Frameworks (analy0c tools, etc.) Horizontally Scalable Ver0cally Scalable Big Data Pla`orms (databases, etc.) Horizontally Scalable Ver0cally Scalable Infrastructures Horizontally Scalable (VM clusters) Ver0cally Scalable DATA SW Data Consumer Security & Privacy Management IT VALUE CHAIN Physical and Virtual Resources (Networking, Compu0ng, etc.) 09/04/2013 NIST Big Data WG / Ref Arch Subgroup 28

29 Internet2 Network Infrastructure Topology

30 Innovation Platform 100 GigE Layer 2 Connec0on Science DMZ Soaware Defined Networking Internet2 innova0on backbone delivered as 100G L1 High- Performance Layer 2/3 Switch/Router SDN Control Server Performance Node Switches, data stores for data- intensive science R&E IP TR- CPS IP Network Layer 3 Your Research Sta0c Layer 2 Dynamic Layer 2 GENI Experiments GENI? Tradi0onal regional and commodity providers Tradi0onal Campus Border Router Tradi0onal L3 Campus Border Security Campus Enterprise Network For more informa0on, see fasterdata.es.net Tradi0onal Services Tradi0onal Switch Substrate Op0cal System Dark Fiber Innova0on Services Soaware Defined Networking Substrate 30 10/1/13, 2012 Internet2

31 Condo of Condos

32 DemocraGzaGon of Sequencing 2,386 Genome Sequencers Worldwide 30 May 2013 Source: Map of High-throughput Sequencers" 32 10/1/13, 2012 Internet2

33 NaGonal Cyberinfrastructure XSEDE NSF- funded Supercomputers HPC resources Internet2 250 universiges XSEDEnet NCGAS Indiana University TACC SDSC PSC Source: h`ps:// /1/13, 2012 Internet2

34 NCGAS Virtual Instrument Indiana University 6 PB Storage TACC NSF- Funded or XSEDE AllocaGon Federally Funded NCGAS Galaxy Portal POD Galaxy Portal Mason POD 5 PB D.C. 100 Gig Internet2 5.5 PB Storage 4 PB Storage SDSC PSC Sequencing Center 10 Gig NLR NCBI Source: Barne`, W.K., and R.D. LeDuc, Next Genera:on Cyberinfrastructures for Next Genera:on Sequencing and Genome Science, presented at 2013 AAMC GIR Conference, Vancouver, BC

35 A note here. Apache Hadoop Ecosystem

36 AWS Cluster Pricing

37 HPC and Cloud Convergence?

38 OpenStack