Making astronomical discoveries on the web

Transcription

1 Making astronomical discoveries on the web David W. Hogg Center for Cosmology and Particle Physics, New York University Max-Planck-Institut für Astronomie, Heidelberg 2011 July 12

2 Conclusions It is possible to make astronomical discoveries in images posted (for other reasons) on the Web. Astrometry.net makes a lot of useless data useful. We make our discoveries by modeling the full data stream. Everything (in my view) is a model. Scientific interpretability is related to probabilistic justification. Lang & Hogg, Lang et al., Lang & Hogg, Barron et al.,

3 principal collaborators Jon Barron (Berkeley) Mike Blanton (NYU) Jo Bovy (NYU IAS) Dustin Lang (Princeton) Sam Roweis (deceased) Christopher Stumm (Microsoft Etsy)

4 search Comet Holmes on Yahoo! (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) (o) (p) (q) (r) (s) (v) (t) (u) (w)

5 Comet Holmes

6 Comet Holmes

7 Comet Holmes

8 Comet Holmes

9 Comet Holmes number of images date

10 Comet Holmes 100 CANON 80 number of images NIKON SBIG OLYMPUS SONY FUJIFILM CELESTRON KAF-6303 Other manufacturer

11 Comet Holmes: the model p(α i Ω i, ω, θ) = p(α i t i, Ω i, ω, θ) p(t i Ω i, θ) dt i p(α i t i, Ω i, ω, θ) = p good p fg (α i t i, Ω i, ω, θ) + [1 p good ] p bg (α i ) { [η Ωi ] p fg (α i t i, Ω i, ω, θ) = 1 comet in η sub-image 0 comet not in η sub-image p bg (α i ) = [4π] 1 ; (1) p(t i Ω i, θ) = { pemp (t i ) if no t EXIF p EXIF p(t i t EXIF ) + [1 p EXIF ] p emp (t i ) if t EXIF in Ω

12 Comet Holmes: the model It is a model of the way people point their cameras. We don t trust the meta-data. Meta-data reconstruction often requires a model of the meta-data provider. See also GalaxyZoo. It requires informative priors. That doesn t mean we have to make strong assumptions. This is Citizen Science with unwitting participants. Lang & Hogg, 2011,

13 Comet Holmes: results

14 Comet Holmes: results 1000 EXIF time - Comet in image time (days) image number (sorted by comet traversal duration)

15 Comet Hyakutake

16 Comet Hyakutake

17 Comet Hyakutake

18 Comet Hyakutake

19 Conclusions It is possible to make astronomical discoveries in images posted (for other reasons) on the Web. Astrometry.net makes a lot of useless data useful. We make our discoveries by modeling the full data stream. Everything (in my view) is a model. Scientific interpretability is related to probabilistic justification. Lang & Hogg, Lang et al., Lang & Hogg, Barron et al.,

20 My day job Infer the detailed properties of the dark matter. Understand the detailed structure of the Milky Way and other galaxies. Measure the growth of structure in the Universe. All these problems are modeling problems. They all involve inferences about unseen material. The fundamental model is simple, but the auxilliary models are not.

21 Astrometry.net Non-text search: Here is an image, what is this an image of? In the process of answering this, we also vet and calibrate it. Calibration: Produce standards-compliant world-coordinate systems for images of unknown provenance. Repair damaged or wrong image headers. Provide astrotagging services.

22 Astrometry.net web demo

23 Astrometry.net In flickr: 14,000 submissions. On the web: tens of thousands; in projects: millions. Source detection, geometric hashing, Bayesian decision theory. A probabilistic model of how detected stars are distributed within images! mixture model or foreground background model Make decisions that opimize our long-term discounted free cash flow. requires utility specification requires customer model Lang et al., 2010,

24 Astrometry.net

25 Astrometry.net

26 Astrometry.net

27 Orion astrometry

28 Orion astrometry

29 Orion astrometry

30 Orion astrometry There is more information in the collection of images uploaded to flickr than in any individual professional astronomical catalog. It just needs to be extracted and combined. Stumm, Lang, Hogg, forthcoming

31 Faint-source proper motions ( ): brown dwarf

32 Faint-source proper motions ( ): z 6 quasar

33 Faint-source proper motions ( ): faint galaxy

34 Faint-source proper motions ( ): defect

35 Faint-source proper motions ( ): results Co-addition of the data (averaging) can detect the faintest sources but not measure their time-dependent properties. To measure their time-dependent properties you must model the uncombined pixels. This works despite the fact that the sources are not clearly detectable in those pixels. (We discovered a dozen brown dwarfs and re-discovered a handful of z > 6 quasars.) Lang et al., 2008,

36 What do I mean by model? I mean p(d x). x contains both parameters of the Universe and your instrument. If you only care about the Universe you have to marginalize. We should work as close to the telescope readouts as possible (D should be as raw as possible).

37 What s wrong with current imaging projects? reducing data with point estimates building catalogs with point estimates catalog matching working on the individual images and a co-add

38 What s wrong with current imaging projects? reducing data with point estimates building catalogs with point estimates catalog matching working on the individual images and a co-add All of these throw away information. Does it matter?

39 What s wrong with current imaging projects? reducing data with point estimates building catalogs with point estimates catalog matching working on the individual images and a co-add All of these throw away information. Does it matter? Lang and I are betting it does: thetractor.org

40 The Tractor

41 The Tractor

42 The Tractor Region above threshold Symmetric template Flux attributed to template Model Sum of models

43 The Tractor (1) (2) (3) (4) (5)

44 Pipeline model Tuned model Data Pipeline χ Tuned χ

45 The Tractor σ Data Pipeline model Tuned model

46 The Tractor Data Data Pipeline Galfit 3 Galfit 1 Galfit 4 Galfit 2 Galfit 5

47 The Tractor Measurements in data should be made by modeling. This starts with a likelihood function. The likelihood function includes a noise model. Modeling makes results better: Classification of sources can be probabilistic. Permits marginalization over descriptions of different complexity. Measured properties have noise propagated correctly. Properly down-weights bad data. Properly combines heterogeneous data.

48 Blind Date ( )

49 Blind Date ( )

50 Blind Date ( )

51 Blind Date ( )

52 Blind Date ( ) We can use the (tiny) motions of stars to age-date images. Precisions measured in years. Possibly far more information available in periodic variables?

53 Conclusions It is possible to make astronomical discoveries in images posted (for other reasons) on the Web. Astrometry.net makes a lot of useless data useful. We make our discoveries by modeling the full data stream. Everything (in my view) is a model. Scientific interpretability is related to probabilistic justification. Lang & Hogg, Lang et al., Lang & Hogg, Barron et al.,