A Physics Approach to Big Data. Adam Kocoloski, PhD CTO Cloudant

Size: px

Start display at page:

Download "A Physics Approach to Big Data. Adam Kocoloski, PhD CTO Cloudant"

Pierce Waters
8 years ago
Views:

1 A Physics Approach to Big Data Adam Kocoloski, PhD CTO Cloudant

3 Solenoidal Tracker at RHIC (STAR)

4 The life of LHC data Detected by experiment Online filtering (hardware and software) Transferred to CERN main campus, archived & reconstructed Transferred to T1 sites, archived, reconstructed & skimmed Transferred to T2 sites, reconstructed, skimmed, filtered & analysed Written into locally analyzable files, put on laptops Turned into a plot in a paper

archived, reconstructed & skimmed Transferred to T2 sites, reconstructed, skimmed,

5 Dig big tu nne ls

6 C h a i n u p s e r i e s o f a t o m smashers

7 Put sensitive cameras in awkward places

8 Re co rd e ve nts

9 The life of LHC data Detected by experiment Online filtering (hardware and software) Transferred to CERN main campus, archived & reconstructed Transferred to T1 sites, archived, reconstructed & skimmed Transferred to T2 sites, reconstructed, skimmed, filtered & analysed Written into locally analyzable files, put on laptops Turned into a plot in a paper

10 CMS Data Flow We have a big digital camera

11 It takes photos of this

12 Which come out looking like this

13 CMS Data Flow We have a big digital camera

14 CMS Data Flow We have a big digital camera Which goes into lots of computers (HLT)

15 CMS Data Flow We have a big digital camera Which goes into lots of computers (HLT) And then into lots of disk (Storage Manager)

16 CMS Data Flow We have a big digital camera ~200 GB/s Which goes into lots of computers (HLT) ~2 GB/s And then into lots of disk (Storage Manager)

17 Throw away data ASAP

18 The life of LHC data Detected by experiment Online filtering (hardware and software) Transferred to CERN main campus, archived & reconstructed Transferred to T1 sites, archived, reconstructed & skimmed Transferred to T2 sites, reconstructed, skimmed, filtered & analysed Written into locally analyzable files, put on laptops Turned into a plot in a paper

19 10 PB / year

20 The life of LHC data Detected by experiment Online filtering (hardware and software) Transferred to CERN main campus, archived & reconstructed Transferred to T1 sites, archived, reconstructed & skimmed Transferred to T2 sites, reconstructed, skimmed, filtered & analysed Written into locally analyzable files, put on laptops Turned into a plot in a paper

21 Tiers 1PB/week Maybe a bit different Many (>100) sites with 100 s TB storage, s worker nodes Why so many? Politics, power budget, cost

22 The life of LHC data Detected by experiment Online filtering (hardware and software) Transferred to CERN main campus, archived & reconstructed Transferred to T1 sites, archived, reconstructed & skimmed Transferred to T2 sites, reconstructed, skimmed, filtered & analysed Written into locally analyzable files, put on laptops Turned into a plot in a paper

23 Analysis Workflow Each analysis is ~unique Query language is C++ Runs on distributed and local resources Myriad cut selections to identify interesting events Data in final plot substantially reduced from the original dataset

26 Multivariate analyses extract signals that are impossible to find via hand-drawn cuts

27 Humans find signals even when none exist

29 Workflow Ladder Number of users Large datasets (>100 TB) } Complex computation Large datasets (>100 TB) Simple computation Shared datasets (>500 GB) Complex computation Use Grid compute and storage exclusively Shared datasets ( GB) Complex computation Shared datasets ( GB) Simple computation Shared datasets ( GB) Simple computation Private datasets ( GB) Simple computation } } Work on departmental resources, store resulting datasets to Grid storage Work on laptop/desktop machine, store resulting datasets to Grid storage

30 Collaborative skimming remains valuable

31 Let s Review

32 Let s Review Particle physics is special

33 Let s Review Particle physics is special But not as special as it used to be vis a vis data operations

34 Let s Review Particle physics is special But not as special as it used to be vis a vis data operations And that s a good thing

35 Let s Review Particle physics is special But not as special as it used to be vis a vis data operations And that s a good thing Keep only the data you can analyze

36 Let s Review Particle physics is special But not as special as it used to be vis a vis data operations And that s a good thing Keep only the data you can analyze Remember that we re biased to find patterns, and lots of data means lots of patterns

37 Let s Review Particle physics is special But not as special as it used to be vis a vis data operations And that s a good thing Keep only the data you can analyze Remember that we re biased to find patterns, and lots of data means lots of patterns Multivariate analyses win in the end

38 Let s Review Particle physics is special But not as special as it used to be vis a vis data operations And that s a good thing Keep only the data you can analyze Remember that we re biased to find patterns, and lots of data means lots of patterns Multivariate analyses win in the end Skim datasets for fun and profit

CMS: Challenges in Advanced Computing Techniques (Big Data, Data reduction, Data Analytics)

CMS: Challenges in Advanced Computing Techniques (Big Data, Data reduction, Data Analytics) With input from: Daniele Bonacorsi, Ian Fisk, Valentin Kuznetsov, David Lange Oliver Gutsche CERN openlab technical