Data analy(cs workflows for climate

Transcription

1 Data analy(cs workflows for climate Dr. Sandro Fiore Leader, Scientific data management research group Scientific Computing CMCC Prof. Giovanni Aloisio Director, Scientific Computing CMCC ISENES2 Workshop on Workflow Solu(ons in Earth System Modelling Hamburg, June 03-05, 2014

2 Data analytics requirements! A set of requirements have been discussed with our scientists to identify the key data analytics needs. Preliminary requirements and needs focus on: Time series analysis Data reduc6on (e.g. by aggrega6on) Data subse<ng Model intercomparison Mul6model means Data transforma6on (through array- based primi6ves) Param. Sweep experiments (same task applied on a set of data) Comparison between historical data and future scenarios Maps genera6on Ensemble analysis Data analy(cs worflow support But also Performance, re- usability, extensibility

3 Defining some workflows: some real examples Some workflows have been designed with CMCC climate scientists to collect their needs and requirements about more complex processing chains: Extreme Precipitation Tendencies analysis Climate change signal Massive data reduction. Ophidia Workflow Modeling Language IMPORT (comments) SUBSETTING [ dim1, dim2,, dimn ] (ranges) GENERIC WORKFLOW INTERCUBE Operation (comments) DETAILED WORKFLOW Data Analytics Workflow Modelling Language 0-D TASK 2-D TASK 1-D TASK 3-D TASK REDUCTION APPLY EXPORT Task transition [ dim1,, dimn ] operation (aggregation set) [ dim ] operation (comments) (comments) 3-D+ TASK n I/O operation Model/Simulation output Feedback Positive comments from users regarding the data analytics workflow modelling language to describe their use cases Sharing & re-usability of workflows across CMCC divisions GENERIC TASK SUPERVISED TASK MODEL REPOSITORY Exported Files File CMCC-CM CMCC-CM Model RCP8.5 RCP8.5 Scenario Atmos Atmos Variable TAS TAS (Frequency)

4 The Ophidia Project Ophidia is a research effort carried out at the Euro Mediterranean Centre on Climate Change (CMCC) to address big data challenges, issues and requirements for climate change data analytics S. Fiore, A. D Anca, C. Palazzo, I. Foster, D. N. Williams, G. Aloisio, Ophidia: toward bigdata analytics for escience, ICCS2013 Conference, Procedia Elsevier, Barcelona, June 5-7, 2013

5 Architecture 2.0: Ophidia Server & Workflow support Workflow tasks Front-end and security REST WS- I OGC GSI/VOMS Request parser Each task is associated to an Ophidia operator OphidiaDB manager OphidiaDB Workflow queue Request validator Task selector Task starter Mul(ple cube management Single cube submission No6fica6on manager Notification from the framework Workflow engine Notification manager Towards the framework Resource manager

6 The analytics framework: datacube operators (about 50)! OPERATOR NAME OPERATOR DESCRIPTION Operators Data processing Domain-agnostic OPH_APPLY(datacube_in, datacube_out, array_based_primitive) OPH_DUPLICATE(datacube_ in, datacube_out) OPH_SUBSET(datacube_in, subset_string, datacube_out) OPH_MERGE(datacube_in, merge_param, datacube_out) OPH_SPLIT(datacube_in, split_param, datacube_out) OPH_INTERCOMPARISON (datacube_in1, datacube_in2, datacube_out) OPH_DELETE(datacube_in) Creates the datacube_out by applying the array-based primitive to the datacube_in Creates a copy of the datacube_in in the datacube_out Creates the datacube_out by doing a sub-setting of the datacube_in by applying the subset_string Creates the datacube_out by merging groups of merge_param fragments from datacube_in Creates the datacube_out by splitting into groups of split_param fragments each fragment of the datacube_in Creates the datacube_out which is the element-wise difference between datacube_in1 and datacube_in2 Removes the datacube_in Data Access (sequential and parallel operators) Metadata management (sequential and parallel operators) Data processing (parallel operators, MPI & OpenMP based) Import/Export (parallel operators) OPERATOR NAME OPERATOR DESCRIPTION Operators Data processing Domain-oriented OPH_EXPORT_NC Exports the datacube_in data into the (datacube_in, file_out) file_out NetCDF file. OPH_IMPORT_NC Imports the data stored into the file_in (file_in, datacube_out) NetCDF file into the new datacube_in datacube Operators Data access OPH_INSPECT_FRAG Inspects the data stored in the (datacube_in, fragment_in) fragment_in from the datacube_in OPH_PUBLISH(datacube_in) Publishes the datacube_in fragments into HTML pages Operators Metadata OPH_CUBE_ELEMENTS Provides the total number of the (datacube_in) elements in the datacube_in OPH_CUBE_SIZE Provides the disk space occupied by the (datacube_in) datacube_in OPH_LIST(void) Provides the list of available datacubes. OPH_CUBEIO(datacube_in) Provides the provenance information related to the datacube_in OPH_FIND(search_param) Provides the list of datacubes matching the search_param criteria

7 Architecture 2.0: Ophidia Server & Workflow support Workflow tasks Front-end and security REST WS- I OGC GSI/VOMS Request parser OphidiaDB manager OphidiaDB Workflow queue Request validator Task selector Task starter Mul(ple cube management Single cube submission No6fica6on manager Notification from the framework Workflow engine Notification manager Towards the framework Resource manager

8 Single Cube Task Execution of a data/metadata operator through a single declarative statement Single Cube Task OperatorA Data OperatorB Metadata Metadata info oph_cubeio cube=doi;branch=children oph_list level=3 oph_explorecube cube=doi.

9 Single Cube Task (oph_cubeio & provenance) Execution of a data/metadata operator through a single declarative statement

10 Multiple cubes tasks Execution of the same data operator over a group of data cubes through a single declarative statement Multiple cubes tasks can be used: to import a large number of datasets into the Ophidia platform to process a large set of data cubes with similar properties Multiple Cubes Task OperatorA Fork OperatorA OperatorA Join OperatorA Filter [ file=*.nc; ] Data cubes to be processed by a multiple cubes operator can be selected by applying a set of filters

11 Array based primitives Ophidia provides a wide set of array-based primitives (about 100) to perform data reduction (by aggregation), sub-setting, predicates evaluation, statistical analysis, compression, and so forth. They are provided both for byte-oriented and bit-oriented arrays Primitives can be nested to get more complex functionalities Compression is a primitive too! New primitives can be easily integrated as additional plugins by the end users Primitives come as plugins and are applied on a single datacube chunk (fragment)

12 Array based primitives: OPH_MATH ( SIGN ) oph_math(measure, OPH_SIGN, "OPH_DOUBLE ) Single chunk or fragment (input) Single chunk or fragment (output)

13 Array based primitives: OPH_BOXPLOT oph_boxplot(measure, "OPH_DOUBLE ) Single chunk or fragment (input) Single chunk or fragment (output)

14 Array based primitives: nesting feature oph_boxplot(oph_subarray(oph_uncompress(measure), 1,18), "OPH_DOUBLE ) Single chunk or fragment (input) Single chunk or fragment (output) subarray(measure, 1,18)

15 Ophidia primitives (about 100) Mathematical primitives: oph_math, oph_mul_scalar, oph_sum_array, oph_sum_scalar, oph_sum_array_r, oph_gsl_sd, oph_gsl_complex_get_abs, oph_gsl_complex_get_arg, oph_gsl_complex_get_imag, oph_gsl_complex_get_real, oph_gsl_complex_to_polar, oph_gsl_complex_to_rect, oph_gsl_dwt, oph_gsl_fft, oph_gsl_idwt, oph_gsl_ifft, oph_gsl_quantile, oph_gsl_sort, oph_gsl_stats, oph_gsl_histogram, oph_gsl_boxplot, oph_petsc_vec_norm, oph_petsc_vec, oph_petsc_vec_r, oph_sum_scalar2, oph_mul_scalar2, oph_compare, Data transformations: oph_to_bin, oph_to_bit, oph_permute, oph_reverse, oph_dump, oph_convert_d, oph_shift, oph_rotate, oph_concat, oph_predicate, oph_roll_up, oph_get_subarray, oph_get_subarray2, oph_get_subarray3, oph_mask, oph_cast, oph_drill_down (stored procedure), oph_reduce, oph_reduce2, oph_reduce3, oph_aggregate_operator, oph_operator, oph_aggregate_stats, oph_extract, Compression: oph_compress, oph_uncompress, oph_compress2, oph_uncompress2, oph_id_to_index, oph_size_array, oph_count_array, oph_find, oph_find2, Bit measures processing: oph_bit_aggregate, oph_bit_import, oph_bit_export, oph_bit_size, oph_bit_count, oph_bit_dump, oph_bit_operator, oph_bit_not, oph_bit_shift, oph_bit_rotate, oph_bit_reverse, oph_bit_subarray, oph_bit_subarray2, oph_bit_concat, oph_bit_find, oph_bit_reduce, Integration of numerical libraries: math, GNU GSL, PetsC. Integration of some CDO operators

16 Integrating some CDO operators as new primitives CDO Module CDO Operator Ophidia primi(ve fldmax FLDSTAT fldmin fldsum oph_cdo_reduce(meas, OPH_MAX ) oph_cdo_reduce(meas, OPH_MIN ) oph_cdo_reduce(meas, OPH_SUM ) TIMSTAT CONDC COMPC 6mmax 6mmin 6msum i]henc,c ifno_henc,c ltc,c gtc,c oph_cdo_aggregate_operator(meas, OPH_MAX ) oph_cdo_aggregate_operator(meas, OPH_MIN ) oph_cdo_aggregate_operator(meas, OPH_SUM') oph_cdo_condc(meas,'oph_ifthenc',const) oph_cdo_condc(meas,'oph_ifnotthenc',const) oph_cdo_compc(meas,'oph_ltc',const) oph_cdo_compc(meas,'oph_gtc',const) SMOOTH smooth9 oph_cdo_smooth9(meas, OPH_SMOOTH9,nlon) REGRESSION regres detrend trend subtrend (trend ofiles) oph_cdo_regres(meas, OPH_REGRES ) oph_cdo_regres(meas, OPH_DETREND ) oph_cdo_regres(meas, OPH_TREND ) oph_cdo_subtrend (measure, oph_cdo_regres (measure,'oph_trend'),'oph_subtrend'))

17 Ophidia Architecture 2.0: features summary New server front-end and workflow management support Data analytics workflow modelling language Ophidia workflow JSON schema New native I/OServer and query engine Support for different storage devices including memory All of these aspects are strongly connected each other to fully address (near) real time parallel data analytics workflows for Climate P.S.: Some of the slides related to the Ophidia 2.0 architecture have been removed due to copyright issues (papers under publication). Please send an to the contact points in the last slide in case you are interested in getting more information on this research effort

18 References and contacts! [1] G. Aloisio, S. Fiore, I. Foster, D. N. Williams, Scientific big data analytics challenges at large scale, Big Data and Extreme-scale Computing (BDEC), April 30 to May 01, 2013, Charleston, USA (position paper). [2] S. Fiore, A. D'Anca, C. Palazzo, I. Foster, Dean N. Williams, Giovanni Aloisio, Ophidia: Toward Big Data Analytics for escience, ICCS 2013, June 5-7, 2013 Barcelona, Spain, Procedia Computer Science, Elsevier, pp [3] S. Fiore, C. Palazzo, A. D Anca, I. Foster, D. N. Williams, G. Aloisio, A big data analytics framework for scientific data management, Workshop on Big Data and Science: Infrastructure and Services, IEEE International Conference on BigData 2013, October 6-9, 2013, Santa Clara, USA, pp [4] S.Fiore, A. D Anca, D. Elia, C. Palazzo, I. Foster, D. N. Williams, G. Aloisio, Ophidia: a full software stack for scientific data analytics, Workshop on Big Data Principles, Architectures & Applications, HPCS2014, Bologna, USA, July 21-25, Contacts: Sandro Fiore, Giovanni Aloisio CMCC and University of Salento sandro.fiore@cmcc.it, giovanni.aloisio@cmcc.it, Team: S. Fiore, A. D Anca, D. Elia, C. Palazzo, A. Mariello, P. Nassisi, G. Aloisio Acknowledgments: Dean Williams Lawrence Livermore National Lab Ian Foster Argonne National Laboratory & University of Chicago

19 Questions?!