Exploring Space in Cyberspace: Cyber-Enabled Research and

Transcription

1 011 Exploring Space in Cyberspace: Cyber-Enabled Research and Discovery in Astronomy S. George Djorgovski (Caltech) CDI Workshop, Seattle, Nov Overview Astronomy in the era of information abundance Exponential growth in data volume and complexity The Virtual Observatory concept and status A domain-specific, community-wide, distributed, open framework for science with massive and complex data sets Technology-enabled, but science-driven Examples of VO science drivers Exploration of parameter spaces Exploration of the time domain: distributed analysis and mining of massive data streams Some general comments and musings on Cyber-science What is really new here? What are the important trends? The enhanced science and technology synergies

2 Galactic Center Region (a tiny portion) 2MASS NIR Image Digital Sky Surveys The dominant source of data in astronomy today, typically several tens of TB each, ~ sources detected, ~ attributes per source; all wavelengths, radio to!-ray Examples: SDSS, Palomar surveys, 2MASS, A single survey feeds a broad range of science, from statistical studies of major constituents of the universe, to discovery of rare types of objects Federated in the Virtual Observatory framework Current surveys are mainly single-snapshot; the next generation will be synoptic (multi-pass), opening the time-domain astronomy (cosmic cinematography); Peta-scale data sets Examples: PanSTARRS, LSST; SKA; etc.

3 Radio Far-Infrared Panchromatic Views of the Universe Visible Crab Star forming complex Data Fusion! A More Complete, Less Biased Picture Visible + X-ray Radio + IR Understanding of complex phenomena requires complex data Theoretical Simulations Are Also Becoming More Complex and Generate Many TB s of Data Structure formation in the Universe Supernova explosions Numerical simulations are not just a weak substitute for the analytical theory - they are an inevitable methodology to study theoretically many complex phenomena, e.g., star or galaxy formation, etc.

4 An Example of Where We Are Heading Large digital sky surveys are becoming the dominant source of data in astronomy: ~ TB/survey (soon PB), ~ sources/survey, many wavelengths Data sets many orders of magnitude larger, more complex, and more homogeneous than in the past Data! Knowledge? doubling t! 1.5 yrs The exponential growth of data volume (and also complexity, quality) driven by the exponential growth in detector and computing technology but our understanding of the universe increases much more slowly! (from CCDs A. Szalay) Glass

5 The Virtual Observatory Concept Astronomy community s response to the scientific and technological challenges posed by the exponential growth of data sets and data complexity Technology-enabled, but science-driven: harness the IT advances in service of astronomy A complete, dynamical, distributed, web-based, open research environment for astronomy with massive and complex data sets Provide content (data, metadata) services, standards, and analysis/compute services Federate the existing + forthcoming digital sky surveys and archives, facilitate data inclusion and distribution Develop and provide data exploration and discovery tools From Traditional to Survey to VO-Based Science Traditional: Survey-Based: Another Survey/Archive? Telescope Data Analysis Survey Telescope Archive Target Selection Data Mining Follow-Up Telescope Results Results Highly successful and increasingly prominent, but inherently limited by the information content of individual surveys What comes next, beyond survey science is the VO science

6 Nat l Virtual Observatory: A Systemic View Primary Data Providers Surveys Observatories Missions Digital libraries Survey and Mission Archives Numerical Sim s User Community NVO Data Services: Data discovery Warehousing Federation Standards Compute Services: Data Mining and Analysis, Statistics, Visualization Networking Secondary Data Providers Follow-Up Telescopes and Missions International VO s Virtual Observatory Is Real! ivoa.net

7 VO as a New Research Environment The VO is not yet another data center, archive, mission, or a traditional project It does not fit into any of the usual structures today It is inherently distributed, and web-centric It is based on a rapidly developing technology (IT/CS) It transcends the traditional boundaries between different wavelength regimes, agency domains It has an unusually broad range of constituents and interfaces It is inherently multidisciplinary It is inherently trans-national in its reach The VO represents a novel type of a scientific organization for the era of information abundance Many other fields are building VOs of their own They are always discipline-based, not institution-based Virtual Observatory Enabled Science Statistical astronomy done right Precision cosmology, Galactic structure, stellar astrophysics Discovery of significant patterns and multivariate correlations Poissonian errors unimportant Systematic exploration of the observable parameter spaces Searches for rare or unknown types of objects and phenomena Low surface brightness universe, the time domain Multi-wavelength data fusion to disentangle complex processes and superpositions e.g., interpretation of the precision CMBR measurements Confronting massive numerical simulations with massive data sets + things we have not thought of yet

8 Integrated SZ Grav. Lensing Integ. Sachs-Wolfe Understanding the CMBR Foregrounds CMB Signal Gal. Nonthermal Galactic Thermal Radio Sources Galaxies (SF) Exploration of Parameter Spaces How many different types of objects are there? Which ones are identifiable with known, physically distinct types (e.g., stars, galaxies, quasars, etc.)? Are there rare and/or previously unknown classes, seen as outliers? Are there intermediate or transition types? Are there negative clusters? Anomalies possibly indicative of problems with the data? Are there new multivariate correlations?

9 An Example: Discoveries of High-Redshift Quasars and Type-2 Quasars in DPOSS Known, astrophysically interesting but rare types of objects, with a known or predictable parameter space signature Color-color parameter space used for selection Spectroscopic identification Normal Stars High-z QSOs Type-2 QSOs Rare Types of Objects Discovered as Outliers in a Color Parameter Space Peculiar types of quasars: Peculiar types of stars: DQ White Dwarf Highly unusual CV (Fan et al., SDSS; Djorgovski et al., DPOSS)

10 Dwarf Planets, Flying Rocks, and Snowballs Dwarf planets and KBOs M. Brown et al. Quauar Sedna, Xena,? NEAT, Catalina, etc. Killer Asteroids Tunguska A Rich Variety of Time-Domain Phenomena Flaring stars Novae, Cataclysmic Variables Supernovae Gamma-Ray Bursts Gravitational Microlensing Accretion to SMBHs

11 Blazars: Accelerators in the Sky They are quasars where we are looking straight down the relativistic jet (" ~ ). Instabilities and shocks produce strong variability Known sources of!-rays (up to a few TeV), and probable sources of ultrahigh energy cosmic rays (up to ~ ev ~ 10 8! LHC!) The future of particle (astro)physics? Probes of relativistic physics, AGN, and cosmic star formation history TeV!-ray Detections Donald Rumsfeld s Epistemology There are known knowns, There are known unknowns, and There are unknown unknowns

12 And Some Mysteries Megaflares in normal stars! An example from DPOSS: A normal, mainsequence star which underwent an outburst by a factor of > 300 (orders of magnitude more than the Solar flares). The cause, duration, and frequency of these bursts is currently unknown Archival optical transients! Seen in many surveys (DPOSS, DLS, PQ, SN surveys, ). Their physical nature is unknown The Palomar-Quest Event Factory Detect ~ 1-2!10 6 sources per half-night scan Compare with the baseline sky R tonight baseline Find ~ 10 3 apparent transients (in the data) Remove instrum. artifacts Identify ~ 2-4!10 2 real transients (on the sky) Remove asteroids Identify ~ 1-10 possible Astrophysical transients I Classification and follow-up

13 The VOEventNet Project A telescope sensor network with a feedback Scientific measurements spawning other measurements and data analysis in the real time (time scales ~ minutes/hours/days) Immediate web-based dissemination and publishing Please see P48 External archives PQ Event Factory Compute resources VOEN Engine PI: R. Williams Spons. NSF/DDDAS P60 Robotic telescope network Raptor Web Event Archive Paritel Follow-up obs. Broader and Societal Benefits of a VO Professional Empowerment: Scientists and students anywhere with an internet connection would be able to do a first-rate science A broadening of the talent pool in astronomy, democratization of the field Interdisciplinary Exchanges: The challenges facing the VO are common to most sciences and other fields of the modern human endeavor Intellectual cross-fertilization, feedback to IT/CS Education and Public Outreach: Unprecedented opportunities in terms of the content, broad geographical and societal range, at all levels Astronomy as a magnet for the CS/IT education Weapons of Mass Instruction

14 VO Education and Public Outreach Google Sky: uses DSS, SDSS, HST data, etc., for easy sky browsing Soon also: Microsoft s World Wide Telescope Transformation and Synergy We are entering the second phase of the IT revolution: the rise of the information/data driven computing The impact is like that of the industrial revolution and the invention of the printing press, combined All science in the 21st century is becoming cyber-science (aka e-science) - and with this change comes the need for a new scientific methodology The challenges we are tackling: Management of large, complex, distributed data sets Effective exploration of such data! new knowledge These challenges are universal There is a great emerging synergy of the computationally enabled science, and the science-driven IT

15 Information Technology! New Science The information volume grows exponentially Most data will never be seen by humans! The need for data storage, network, database-related technologies, standards, etc. Information complexity is also increasing greatly Most data (and data constructs) cannot be comprehended by humans directly! The need for data mining, KDD, data understanding technologies, hyperdimensional visualization, AI/Machineassisted discovery We need to create a new scientific methodology on the basis of applied CS and IT VO is the framework to effect this for astronomy A Modern Scientific Discovery Process Data Gathering (e.g., from sensor networks, telescopes ) Data Farming: Storage/Archiving Key Technical Challenges Indexing, Searchability Data Fusion, Interoperability +feedback } Database Technologies Data Mining (or Knowledge Discovery in Databases): Pattern or correlation search Clustering analysis, automated classification Outlier / anomaly searches Hyperdimensional visualization Data Understanding New Knowledge Key Methodological Challenges

16 The key role of data analysis is to replace the raw complexity seen in the data with a reduced set of patterns, regularities, and correlations, leading to their theoretical understanding However, the complexity of data sets and interesting, meaningful constructs in them is starting to exceed the cognitive capacity of the human brain Universal Challenges: Towards The New Scientific Methodology Data farming and harvesting Semantic webs, computational and data grids, universal or transdisciplinary standards and ontologies Digital scholarly publishing and curation (libraries) data, metadata, virtual data, hierarchical data products; legacy vs. dynamical; open vs. proprietary; data, knowledge, and codes; persistency; peer review; web samizdat vs. officially blessed and supported; mandates; etc., etc. Data mining and understanding, knowledge extraction Scalable DM algorithms Hyperdimensional visualization Empirical validation of numerical models Computer science as the new mathematics The art and science of scientific software systems Architecture, design, implementation, validation

17 Some Distinguishing Characteristics of Data/Comp. Enabled (e-)science Data-intensive: massive (TB-scale and beyond) data sets Poissonian errors not important, systematics dominate Data complexity: multi-wavelength and/or multi-scale and/or multi-epoch data sets, 100 s or 1000 s parameters per source, combining imaging, spectroscopy, etc. Heterogeneity and visualization are key issues Computationally intensive Traditional solutions do not scale to the scope of new problems Need new tools and scalable algorithms Data and computing resources (an experise) are generally geographically distributed Inherently cross-cutting in many ways (CS/Astro, multi-# ) Some Thoughts on CyberScience Enables a broad spectrum of users and contributors From large teams, to small teams, to individuals Data volume ~ team size, but scientific returns! f (team size) Human talent is distributed very broadly geographically Open, distributed, web-based nature of new science is a key feature Transition from data-poor to data-rich science Chaotic! Organized regulation vs. creative freedom Can we learn to ask a new kind of questions? Information is cheap, but expertise is expensive Just like the hardware/software situation Computer science as the new mathematics It plays the role in relation to other sciences which mathematics did in ~ 17 th - 20 th century

18 Summary Comments Cyber-enabled (computationally and data-enabled) science is a practical necessity Complex problems! simulations, complex and massive data sets Distributed resources (data, facilities, people )! virtual scientific organizations (VO is an example) It is IT-enabled, and has a potential to drive transformative scientific and practical advances The key challenge is to think differently (computationally) Remember the origins of WWW; now grid, semantic web, knowledge extraction tools, MP/HPC design and apps There is a great deal of methodological commonality between different fields and this commonality can lubricate some genuine multi- or interdisciplinary research, with a great discovery potential Let s avoid wasteful replication of efforts, share the tools, methods