XML enabled databases. Non relational databases. Guido Rotondi

Transcription

1 Programme of the ESTP training course on BIG DATA EFFECTIVE PROCESSING AND ANALYSIS OF VERY LARGE AND UNSTRUCTURED DATA FOR OFFICIAL STATISTICS Rome, 5 9 May 2014 Istat Piazza Indipendenza 4, Room Vanoni A laboratory approach in managing very large datasets, which are emerging as primary sources feeding most up to date statistical processes. Students will be introduced to the appropriate use of technology for managing the ETL processes resulting from collecting and feeding data from large structured and unstructured data sources. The course also provides a collection of methods and techniques to integrate the sources, to compare the archives against reference metadata sets and to discover and eventually resolve source anomalies. The attendee will be introduced in the theoretical fundamentals, which underlie any presented methodology and will finally be brought to a real implementation by using innovative techniques and algorithms. Day 1, 5 May 2014 Old and new data manipulation paradigms Opening Too big to ignore: a matter of balance. Evolution in data management; scenario The need for alternative computing paradigms. Antonino Virgillito Classification of data sources The Internet of Things Case study: synthesising a Big Data driven framework. Diego Zardetto Sharing experiences, expectations and critical aspects. Giulio Barcaroli International activities on Big Data in Official Statistics Carlo Vaccari XML as integration paradigm. Service Oriented Architecture.

2 XML enabled databases. Non relational databases Handling XML sources. Non structured XML Tables Dealing with XSD schemas. Structured XML Tables Merging XML data in the business process: the Resource Description Framework Conclusions

3 Day 2, 6 May 2014 A roadmap toward Big Data Opening The Map Reduce programming model. Antonino Virgillito The World of Hadoop. Antonino Virgillito NoSQL databases Robust concurrent computing architectures and the Byzantine agreement problem. Single Point Of Control. Single Point Of Failure Using Big Data technologies (part one): massive computing. Antonino Virgillito Using Big Data technologies (part two): dealing with unstructured data examples and applications ' ' Implementing the Map Reduce programming model on a parallel enabled database: aggregating functions. Profiling the Map Reduce model on a real enterprise infrastructure. Implementing and evaluating simple Map Reduce algorithms Conclusions

4 Day 3, 7 May 2014 Big Data in Official Statistics Opening Introduction to Big Data in Official Statistic. The concept of Big Data; overview of Big Data sources. Methodological issues in using Big Data for Official Statistics. Antonino Virgillito Giulio Barcaroli IT Issues in using Big Data for Official Statistics Using mobile phones for analyzing mobility of city users. Antonino Virgillito Improving Labor Force Survey estimates by the effective usage of Google Trends Internet as a data source: web scraping and text mining for estimating ICT usage by enterprises and public Institutions. Privacy, Security and Safety: Recipes for securing data, recipes for disclosure control, trusted computing Conclusions

5 Day 4, 8 May 2014 Improving data availability and processing efficiency Opening Data location and partitioning. Indexing. Problem splitting. Actor systems. Storage virtualisation. Examples of improving data location and partitioning. Effective usage of indexes Improving database (serial) operations. Code profiling. Bulk operations. Pipelined functions. Sustained data streaming. Partition swapping. External tables in performing fast bulk operations. Application of a pipelined function to an ETL process. Managing changes of a big micro data set Quasi real time analytics. Diego Zardetto Fundamentals of parallel computing. Definitions, metrics, workload, critical aspects. Distributed vs Symmetric Multi Processing Parallel database operations. Scheduled concurrent tasks. Parallel enabled pipelined functions. Parallel queries. Embedded relational objects, aggregating functions. Self-made parallelism vs controlled tasks.benefits of parallel data streaming. Multipath data querying. Embedded relational objects. Design of central aggregating functions Conclusions

6 Day 5, 9 May 2014 The analysis of massive datasets Opening Geometric interpretation of data structures and the introduction of regular languages and expressions Getting involved with regular expressions Mapping techniques for studying anomalies in structured data: Probabilistic ranking of event patterns Stochastic characterisation of unstructured data sets Characteristics of a Big Data Analysis Framework: a distributed approach Inference techniques used for Official Statistics (Part-1) Diego Zardetto Inference techniques used for Official Statistics. (Part-2) Diego Zardetto Where can we go from here? Golden rules Final remarks Giulio Barcaroli Antonino Virgillito