Commentary on Techniques for Massive- Data Machine Learning in Astronomy

Transcription

1 1 of 24 Commentary on Techniques for Massive- Data Machine Learning in Astronomy Nick Ball Herzberg Institute of Astrophysics Victoria, Canada

2 The Problem 2 of 24 Astronomy faces enormous datasets Their size, dimensionality, and complexity require intelligent, automated investigation Exponential increase in data size: algorithms cannot scale worse than O(N log N) Most data mining algorithms naïvely scale as N 2 or worse

3 The Solution 3 of 24 Make data mining algorithms that scale as N log N! (or better) May have to compromise accuracy slightly Deploy them so that astronomers are willing and able to use them They must work on real astronomical data

4 Collaboration is Vital 4 of 24 Successful use of astrostatistics and data mining requires expertise in computer science, statistics, and astronomy Collaboration enables novelty that would not arise from a single group So, computer scientists supplying algorithms in this way is excellent

5 But 5 of expertise in computer science, statistics, and astronomy Successful collaborations have involved astronomers who are experts in computing/statistics, or who are working closely and over time with these experts

6 And 6 of 24 Astronomy data are messy: - Large, complex, increasingly high-dimensional, timedomain - Missing data: non-observation or non-detection - Heteroscedastic, non-gaussian, underestimated errors - Outliers, artifacts, false detections, systematic effects - Correlated inputs - Etc.

7 An Example 7 of 24 How do you apply astrostatistics and fast algorithms to this?

8

9 The Next Generation Virgo Cluster Survey 9 of 24 10σ point source limiting magnitude g = 25.7 (faint!) Photometric (few spectra), ~100 deg 2, 5 bands (ugriz, like Sloan) galaxies, 2.6 terabytes data 40 people at at 23 institutions in Canada, France, etc. (PI Laura HIA)

10 Virgo is an actual cluster of galaxies, the nearest large one to us

11 NGVS Statistical Challenges 11 of 24 Object detection and classification Photometric redshifts (photo-z) Virgo cluster membership / background Missing data Field-to-field variation Multi-wavelength data Completeness(mag, SB, etc. etc.)

12 Object detection: low surface brightness galaxies

13 13 of 24 Cluster membership: photometric redshift using k nearest neighbours

14 14 of 24 Missing data: NGVS fields (not final) don t all contain all 5 bands ugriz

15 Multi-wavelength data

16 Canadian Astronomy Data Centre CADC is one of the world s largest astronomy data centres ~500 terabytes of data (will grow to petabytes) Uses Virtual Observatory standards Staffed by astronomers and computer specialists, but not statisticians 16 of 24

17 CANFAR 17 of 24 Canadian Advanced Network for Astronomical Research, at CADC Combines cluster job scheduling with cloud computing resources Users manage their own virtual machines

18 So 18 of 24 Put fast data mining tools on the CANFAR infrastructure... but early days, not much to say yet

19 Guide to Data Mining in Astronomy 19 of 24 Virtual Observatory KDD-IG guide: IvoaKDDguide Emphasizes data mining, which is part of astroinformatics But this overlaps with astrostatistics -> potential outreach channel to wider community

20 knn Quasar Photometric Redshifts 20 of 24 Use kd-tree for fast knn assignment of photo-zs to Sloan Digital Sky Survey quasars Single neighbour, perturb input features to make a PDF in redshift Removing multi-peaked PDFs removes almost all catastrophic outliers

21 knn Quasar Photometric Redshifts 21 of z mean = z spec 20

22 knn Quasar Photometric Redshifts 22 of z one peak = z spec 20

23 Questions 23 of 24 Can we overcome the problems of real data? Will there be data of high intrinsic dimension? Will astronomers be able to deploy the algorithms? Where do GPUs fit? (GPU+brute force may be just as fast?)

24 Conclusions 24 of 24 Provided the data can be suitably prepared, and the science-driven usage of the algorithm intelligently motivated, the fast algorithms presented here have excellent potential for advancing astronomical research