Visualization and Exploration of huge data volumes

Transcription

1 Visualization and Exploration of huge data volumes Claudio Gheller (CINECA),

2 CINECA ( CINECA is a non profit Consortium, made up of 36 Italian universities, The National Institute of Oceanography and Experimental Geophysics - OGS, the CNR (National Research Council), and the Ministry of University and Research. Today it is the largest Academic HPC centre in Italy It is partners of the major HPC/GRID computing EU supported projects: DEISA, PRACE, HPCEUROPA)

3 Research topics and collaborations HPC computing applications in astrophysics Management and Visualization of scientific data Istitute of Radioastronomy, Bologna, G. Brunetti, F. Vazza Astronomy Dept. of Trieste, S. Borgani, L. Tornatore, G. Murante Visualization CINECA, S. Imboden, L. Calori Astronomical Observatory of Catania and Portsmouth University (see later) Enzo simulations Gadget simulations MAF development; outreacheducational applications VisIVO toolkit development MPA-Munich, K.Dolag Splotch for astro-simulations and public outreach S.Diego Supercomputing Center, R.Wagner Developing a protocol for simulation data access (SimDAP) European Projects DEISA 2 Responsible of JRA (The DEISA Development Environment Using Ecplipse PTP for DEISA) HPC-EUROPA2 JRA1: Innovative Parallel Programming Paradigms Evaluation and Parallel I/O Tools JRA3: Tools for scientific data services

4 Introduction Experiments, observations, numerical applications produce huge amount of data: the data volume approx. doubles each year. This data must be properly stored and managed. This is commonly indicated as the Data problem. Furthermore, the data must be accessible and proper tools for its exploration, processing analysis should be available. My focus is: numerical simulations in Astrophysics However, problems and requirements are common to most of scientific areas disciplines (CFD, planetary sciences, geophysics ).

5 Chapter 1: THE (HUGE) DATA

6 Astrophysical simulations: a (very) short overview Astrophysical simulations are challenging since they have to deal with a large number of different physical processes, on a huge dynamical range. They requires sophisticated algorithms and high resolution. They are computationally expensive and produce tons of data usually RAW data

7 Detailed observations Courtesy Alexis Finoguenov, Ulrich Briel, Peter Schuecker, (MPE) electron density gas temperature gas pressure

8 Require detailed modeling (experiments not possible) Courtesy G.Lemson and V. Springel

9 Temperature map of a cluster of galaxies (evolution from redshift z=6)

10 Historical simulations Toomre & Toomre, 1972 Volker Springel Di Matteo, Springel and Hernquist, 2005

11 Simulations data: Size Let s see what happens for a cosmological simulation Pure gravity (N-body): Size: particles elements For each particles we save (at least) 3D position and velocity: 6 variables Each variable is a float number: 4 bytes We want to save a number of time steps: e.g. 10 (but for millennium, e.g., it is 64!!!) 6x10 10 elements 2.4x10 11 bytes = 240 GBs 2.4 TBs In a hydrodynamics simulation, you must save further variables, at least density, energy and momentum of the gas, so the data size is more or less doubled!!!

12 Simulations data: other issues Apart from its size, data produced by numerical simulations is: Monolithic (few files contains plenty of data) Uncompressible Non standard (propretary formats are the rule) Non portable (depend from simulation machine) No (or few) annotations metadata Heterogeneous in units (often code units)

13 Chapter 2: Data Exploration, Visualization and Processing

14 Overview Various different tools offer powerful instrument for automatically analyzing large volumes of data, for classification, association, clustering, etc. (e.g. data mining or statistical tools) In general, data analysis is characterized by accurate and sophisticated (i.e. complex and computationally expensive) algorithms which often scale as N 2 or even N 3 (non-linear behavior) and that are not or cannot be optimized/parallelized (not suitable for HPC system) But an extremely accurate approach is not always necessary

15 Visualization e.g. some problems require a overall data exploration approach, as that provided by visualization Visualization offers an intuitive and immediate insight into data and has the capability to show simultaneously different species and different properties of data, to navigate inside data to find out and select interesting regions and features at a glance. What takes hours for a CPU can take a glance for the human eye!!! Therefore, the visualization process can play a fundamental role in understanding the data

16 Visualization Tools In fact, the astronomical community has always dedicated special attention to graphical and visualization tools Traditionally, the most popular software for astronomers can be subdivided into two main categories: tools for image display and processing (IRAF, NOAO, MIDAS ) and tools for plotting data (IDL, SMongo, Gnuplot, Tipsy ) However, most of these tools are highly specialized and cannot cooperate with each other

17 Visualization Tools: The New Generation A new generation of graphic software tools is maturing. These tools are designed to overcome the limits and the barriers of traditional software by exploiting the latest technological opportunities. Main objectives: High performance and large data support Interoperabilty Access to remote and distributed resources Tools like VisIVO, ParaViev, Visit, Aladin and Topcat have been recently developed to achieve these goals.

18 High performance and large data support High performance allows to exploit at best the resources of your computing system: Multi threading implementation (for multi core processors) Parallel implementation (for multi processors architectures) Exploitation of graphics/processing accelerators (GPUs, FPGA ) Usage of specialized programming paradigms (CUDA, shaders, OpenCL ) Large data support provides solutions for large data volumes: 64 bit applications (exploit large - > 2 GB memories) Visualization tricks to render plenty of data Levels of details Parallel systems

19 Interoperability Different tools (software) are good in specific tasks It s impossible to have a single tool that can do everything in the best way Interoperability means take different tools and make them work in a cooperative way, exploiting their best functionalities Various possible solutions. E.g. (for astronomy): SAMP: Simple Application Messaging Protocol, and its precursor PLASTIC: PLatform for AStronomical Tool InterConnection Plastic HUB Messages management

20 A couple of examples: 1. The VisIVO Toolkit All of this can be still implemented as a traditional desktop application: o Data is on the workstation o Computing and rendering is done by the same workstation And this was also our* historical approach which led to the VisIVO software. Gabriella Caniglia Software Developer Astrophysical Observatory Catania Italy Marco Comparato Software Manager/ Developer Astrophysical Observatory Catania Italy * Our stands for: Alessandro Costa Web Services & TVO Astrophysical Observatory Catania Italy CINECA (myself and S.Imboden) The Italian Institute for Astrophysics-INAF-Catania Observatory (U.Becciani, M.Comparato, G.Caniglia, A.Costa, P.Massimino) And, more recently, the School of Creative Technologies, University of Portsmouth (M. Krokos, Z. Jin) Zef Han J. Software developer University of Portsmouth UK Mel Krokos Co-Principal Investigator University of Portsmouth UK

21 VisIVO (

22 A couple of examples: 2. Splotch ( cosmo/splotch/) Splotch is a light and fast, public available, ray-tracer software tool which supports the effective visualization of cosmological simulations data. The algorithm it relies on is designed in order to deal with point-like data. Used for both scientific and outreach purposes. Hybrid parallelization: o MPI based each node compute an image (under development) o OpenMP processor in a node calculates part of a image (public) Example: Turin Planetarium movie: Approx 360 sec. Framerate=30 images per second (approx: frames) Stereographic=doubled the number of frames (approx 20000) Resolution 1400 x 1050, Input dataset, approx points Each frame takes approx 3-4 minutes to be generated I.E. each a full animation takes approx 20 CPU hours to be completed. A large number of tests required Parallelization absolutely necessary

23 Useful links and references VisIVO : Aladin: Topcat: Plastic: VTK: MAF: IVOA: Visualization, Exploration and Data Analysis of Complex Astrophysical Data M. Comparato, U. Becciani, A. Costa, B. Garilli, C. Gheller, B. Larsson, J. Taylor, 2007, The Publications of the Astronomical Society of the Pacific, Volume 119, Issue 858, pp

24 Chapter 3: Remote and Distribute Data

25 Data Management & Access o Data produced by supercomputers are stored at the HPC center o They will (probably) stay there (cannot be moved over a WAN) o Network speed can increase, but data size increases faster Tools MUST be deployed in order to manage and access effectively remote stored data. STEP 1: MOVING from the HPC center to the HPC-DATA center

26 Simulations data: limits Size; Monolithic; Uncompressible. Non standard; Non portable. No metadata; Heterogeneous in units Difficult to move (download); Difficult to process (CPU/memory/disk demanding); Plenty of usless data manipulated. Difficult to share and distribute; Difficult to reuse; Difficult to access. Difficult to search; Difficult to reuse and share; Difficult to preserve; Difficult to access by humans and machines; Difficult to publish.

27 Toward the data center Present situation Local UNIX filesystem. Hierachical. FTP Propretary No description Directory browsing XML None Repository Organization Perspective irods, DIGS Transport protocol Toward high performance: GridFTP File format Common exchange format, HDF5 Data desciption Theroetical Data Model (see Archive exploration and search DB queries, Google-like search Information encoding language XML Available services Quality of service/data Data search Data visualization and processing Data selection, decimation, compression

28 Extending the perspective: distributed data resources Are simulations datasets interesting for more than one? Can data produced by a simulation be published becoming available for the scientific community? o It can be used for testing new codes or algorithms o It can be analyzed in many different aspects (possibly not interesting/ expected/considered by authors o Data publication can increase its popularity and that of its authors o It Bridges the gap in specializations: not everyone has required expertise to create simulations, though they can analyse them. STEP 2: Moving from Local HPC-Data Centers to Distributed HPC-Data Centers Various approaches can be followed..

29 The Virtual Observatory In Astronomy Most of these problems characterized also observational data. That s why the international communitiy started a joint effort for overtaking them. The result of this effort is the International Virtual Observatory A virtual observatory is a collection of interoperating data archives and software tools which utilize the internet to form a scientific research environment in which astronomical research programs can be conducted. (Wikipedia)

30 The International Virtual Observatory Alliance The basic requirement of the VO is to define STANDARDS to make everything INTEROPERABLE The development of such standards, hence of the VO is coordinated by the International Virtual Observatory Alliance (IVOA, see The IVOA goal is "To facilitate the international coordination and collaboration necessary for the development and deployment of the tools, systems and organisational structures necessary to enable the international utilisation of astronomical archives as an integrated and inter-operating virtual observatory" (IVOA mission, from

31 The theoretical Virtual Observatory (TVO) Up to now, most of the IVOA effort has been devoted to observational data, but simulated data are getting a growing attention. Broadly speaking, the goal of the TVO development work is to create a distributed archive of simulated data accessible from anywhere by web based tools in a easy and transparent way. Using the same metaphor of the VO (and of the web) its just like the researcher has all the simulation data in his pc. The TVO is defining standards for Data description (repositories) Data access (services)

32 Back to visualization In this approach, visualization is exploited for DATA EXPLORATION and SELECTION: o See the data o Find out interesting features o Select interesting objects o Extract them o Process them o Download the results o Do your standard desktop work on them

33 Exploring data Search for simulations with Lambda>0.7 Data is too large!!! I like this one It s too large!!! Let s select a sub-region!!! Extract a sub region it is still large Metadata VOTable Binary data file Perform the analysis on-site Finally I have a jpeg cannot be too large!!!

34 VisIVO Server: Basic Architecture TVO XML Document New VBT Local or Remote (http ref) User Data VBT: VisIVO Binary Table

35 VisIVO Server main features VisIVO Server is just a collection of C++ applications (most of them derived by the Desktop version). New modules can be added just supporting the I/O protocols. Application are o Linux/Unix compliant o 64 bits o Data size independent o Parallel (MPI/OpenMP work in progress) At present visualization is static, i.e. only sequences of images can be produced

36 Path Forward In order to have interactive visualization different approaches are followed: - Development of interactive web browser embedded interfaces is in progress (OpenSceneGraph + javascript based). Graphics is at client side. Critical are: o Multi-browser Multi-platform support o partitioning schema (tree-based? For performances) - Visualization node: Images are produced at server side and sent to a client. Critical are o Light protocol o Effective compression Experimenting IBM-DCV+VNC solution (RVN node on the supercomputing BCX cluster Linux cluster)

37 A few examples ITVO Catania (Italy) Thanks to Ugo Becciani & Alessandro Costa ITVO Trieste (Italy) Thanks to Patrizia Manzato & Fabio Pasian (experimental installation) ITVO Portsmouth (UK) Thanks to Mel Krokos Simulation data San Diego (USA) Thanks to R. Wagner Millennium Run MPI Munich (Germany) Thanks to G. Lemson

38 TVO usage: 1 starting to search (standard query) Search query Find all SPH simulations of the Concordance model with data between z=0 and z=0.1 Registry User The registry is a web resource Finds all the available suitable data services

39 TVO usage: 2 data discovery Search query Find all SPH simulations of the Concordance model with data between z=0 and z=0.1 Registry User Query submission Only Munich and Trieste Have SPH simulations Archive with SimDB Archive with SimDB Archive with SimDB Search results List of the hits

40 TVO usage: 3 previewing data User Archive with SimDB Archive with SimDB Archive with SimDB Search results List of the hits Reduced data Preview I like this cluster Inside this snaphot!!! The data is in Munich Ask for previews

41 TVO usage: 4 get the data User Archive with SimDB Archive with SimDB Archive with SimDB Select the Region of Interest and Get the data Cutout service get the data (large bandwith for large data) DATA Cutout FINAL RESULT VOTable (small bandwidth, Small data)

42 TVO concept Search query Find all SPH simulations of the Concordance model with data between z=0 and z=0.1 Registry User Query submission Only Munich and Trieste Have SPH simulations Archive with SimDB Archive with SimDB Archive with SimDB Search results List of the hits Reduced data Preview I like this cluster Inside this snaphot!!! The data is in Munich Ask for previews Cutout FINAL RESULT