Big data analy+cs for global change monitoring and research in forestry and agriculture. Lubia Vinhas

Size: px
Start display at page:

Download "Big data analy+cs for global change monitoring and research in forestry and agriculture. Lubia Vinhas"

Transcription

1 Big data analy+cs for global change monitoring and research in forestry and agriculture Lubia Vinhas

2 Earth observa+on satellites and geosensor webs provide key informa+on about global change but that informa+on needs to be modelled and extracted

3 Mo+va+on: EO data is now free and big LANDSAT: 600,000 images since 1972 (150 M km 2 ) Image source: NASA

4 More data is coming... How best to use it? ResourceSat- 2 Landsat- 8 CBERS- 4 Sen$nel- 2B Sen$nel- 2A ResourceSat- 3

5 ST arrays allows new ques$ons: Are biofuels replacing food produc$on in Brazil? 5 source: B. Rudorff, INPE

6 Project topic: global land observatory Design and develop novel ST analysis methods and use innova+ve array databases to build a system for global monitoring of forests and agriculture, based on free Earth observa+on images. Researchers will be capable of doing analysis that have not been possible before.

7 Where we want to get to Remote visualiza$on and analysis Big data EO management and analysis 40 years of Earth Observa+on data of land change accessible in a single data set for analysis and modelling. Original slide: Leilani Battle (MIT)

8 Analysis on mul+dimensional ST arrays t y X g = f (<x,y,t> [a 1,.a n ])

9 Array databases: all data from a sensor put together in a single array t result = func+on (points in space- +me ) y X

10 Global Land Observatory: describing change in a connected world R: Powerful data analysis methods SciDB: array database for big scien+fic data Soaware goes where the data is! Free satellite images

11 Global Land Observatory: describing change in a connected world Methods for land change for foresty and agriculture uses 40 years of LANDSAT + 12 years of MODIS + SENTINELs + CBERS Unique repository of knowledge and data about global land change Free satellite images

12 What we know 30 years of EO experience Powerful analysis engine (R) EO database tech (Terralib) Time series EO analysis SciDB: innova$ve DMAS for big arrays Original slide: Leilani Battle (MIT)

13 R: Leadership, SciDB- R

14 Accessing TerraLib databases using R- sp Database

15 Image Algorithms (100+) Mosaic Clipping through iterators Segmenta+on Classifica+on

16 What we know we don t know 1: data How to put all EO data together? How to work with different ST resolu$ons? Different satellites have different calibra+ons Geometric and radiometric problems Original slide: Leilani Battle (MIT)

17 What we know we don t know 2: databases How to organize scien$fic data in array databases? How to match data seman+cs to arrays? What s the equivalent of transac+on? What about concurrency control? How to support worldwide users? Original slide: Leilani Battle (MIT)

18 What we know we don t know 3: methods What are good tools for space- $me modelling of EO data? How to combine +me series with spa+al sta+s+cs? How to do space- +me object and event detec+on? How to develop a library of methods for SciDB- R env? Original slide: Leilani Battle (MIT)

19 What we know we don t know 4: applica+ons How best to use ST EO data for global forest studies? How best to use ST EO data for global food studies? Original slide: Leilani Battle (MIT)

20 Our research with array databases for handling big data EO Lubia Vinhas Gilberto Ribeiro de Queiroz Karine Reis Ferreira Ricardo Cartaxo M. de Souza Raphael Willian da Costa Roger Victor

21 Mo+va+on to work with array databases Conceive, build and deploy a knowledge plalorm for organiza+on, access, processing and analysis of big Earth Observa+on data Show that this knowledge plalorm allows scien+sts to produce informa+on on land use and land cover change in a completely innova+ve way Use the SciDB array database manager for storing and processing large sets of remote sensing images

22 e- Sensing projeto de 3 anos financiamento pela FAPESP SciDB EO And now? How do start? Temporal series of remote sensing data for change detection. MODIS data for land use land cover and agriculture How to create SciDB databases?

23 Tools to upload data to SciDB modis2scidb: a C++ tool for conver+ng a MODIS HDF file to the 1D binary format accepted by SciDB: Each selected subdataset from the HDF file is converted to an aoribute of the array. Add one extra aoribute with the encoded +me, row and column of the aoribute. Ex: $ modis2scidb - - f <modis- hdf- file> - - o <output- file> - - b "0,1,2" - - t <time- point> Available at: hops://github.com/gqueiroz/modis2scidb

24 Tools to upload data to SciDB modis2scidb- loader: a Python script that orchestrates the load of a set of MODIS HDF files to a 3D array: Save a log into regular tables of a PostgreSQL database Use modis2scidb to generate the 1D binary file to be loaded into SciDB Ex: $ modis2scidb- loader.py - - config mod13q1.json Available at: hops://github.com/gqueiroz/modis2scidb- loader

25 Databases MODIS data: 33 Tiles covering South America ~15 years( ) MOD09Q1, files HDF- 4, 3 variables MOD13Q1, ~ files HDF- 4, 12 variables MCD43A4,7 variables

26 WTSS Client coverage=mod09q1,attributes=red,nir& longitude=- 54,latitude=- 12&start= &end= JSON Document {"result": { "attributes":[ { "name": "red", "values": [ 1004, 1160, 241 ] }, { "name": "quality", "values": [ 4842, 3102, 2116 ] } ], "timeline": [ " ", " ", " " ], WTSS (TerraLib + SciDB C++ API) SciDB (Arrays) "center_coordinates": { "latitude": , "longitude": } }, "query": { "coverage": "MOD09Q1", "attributes":[ "red", "quality" ], "latitude": - 12, "longitude": - 54, "start": " ", "end": " " } } PostgreSQL (Metadados) WTSS Web Time Series Service: a lightweight service for handling remote sensing imagery as +me series

27 Time series visualiza+on: web

28 Plugin SITS Viewer Plugin for Quantum GIS Source: https://github.com/vwmaus/sits_viewer

29 R client for Web Time Series Service (WTSS) obj = wtss("http://chronos.dpi.inpe.br") coverages = list_coverages(obj) cv_desc = describe_coverage(obj, "MOD09Q1") ts1 = get_time_series(obj, coverage="mod09q1", attributes=c("nir", "quality", "red", "evi2"), latitude=-12, longitude=-45, start=" ", end=" ") Source: https://github.com/albhasan/rwtss

30 JavaScript API for Web Time Series Service (WTSS) <html> <head> <script type="text/javascript" src="https://www.google.com/jsapi"></script> <script type="text/javascript" src="../../src/js/wtss.js"></script> <script type="text/javascript" src="../../src/js/tschart.js"></script> <script type="text/javascript"> var wtss_server = wtss("http://chronos.dpi.inpe.br"); wtss_server.time_series( { "coverage": "mod09q1", "attributes": ["red", "nir"], "longitude": -54.0, "latitude": -12.0, "start": " ", "end": " "}, fill_time_series ); Source: https://github.com/gqueiroz/wtss

31 What comes next? Some experiments and tests SciWCS: the prototype is under development in Python using Django + SciDB Python API WCS Evalua+on: WCPS Evalua+on: GeoServer Plugin (SciWCS): Use of WMS to allow dynamic visualiza+on Performance evalua+on according to SciDB configura+on: #instances, chunk size, etc.

32 BigData SDI

33 Did Amazon forests green up during the 2005 drought? (An exercise on reproducible science using SciDB) Gilberto Câmara 1,2, Alber Sanchez 2 1 Na+onal Ins+tute for Space Research (INPE), Brazil 2 Ins+tute for Geoinforma+cs, University of Münster, Germany

34 Compu+ng environment CPU: 1 Intel Xeon cores Memory: 132 GB Disk: 10 Tb Soaware: Ubuntu 12.04, SciDB 14.3

35

36 Did Amazon forests green up during 2005 drought? (An exercise on reproducible science) July - September 2005 standardized anomalies (A) precipita+on (TRMM ) (B) forest canopy greenness (MODIS EVI ) Published by AAAS S R Saleska et al., Science 2007;318:612

37 MOD09Q1 product image: NASA 250 mts spa+al resolu+on, 8 days temporal resolu+on 4800 x 4800 pixels, 3 bands (red, nir, qc) 13 years of data (since 2000)

38 Reproducing Saleska s paper in Science with SciDB Array Func+onal Language 1. Recover all +les of MOD09Q1 product for the whole Brazil and load them into SciDB 2. Extract the subarray covering Amazonia 3. Compute EVI for each cell in all +me steps 4. Compute EVI mean and stdev for JAS for each cell 5. Compute EVI mean for JAS 2005 for each cell 6. Compare EVI mean (JAS 2005) to the JAS mean

39 Extrac+ng a subarray for MODIS MOD09Q1 for JAS in Amazonia (with quality filter) store(between(filter(mod09q1_br_2000_2013, time_id % 46 >= 23 and time_id % 46 <= 34 and quality = 4096), 48000, 38400, 0, 67199, 52799, 275), MODIS_AMZ_BQ_JAS); dimensions: col_id, row_id, time_id attributes: red, nir, quality

40 Calculate EVI2 for all cells in all +me steps store(apply(modis_amz_bq_jas, evi2, 2.5*((nir - red)/(nir + 2.4*red + 1))), MODIS_AMZ_BQ_JAS_EVI2); attributes: red, nir, quality, evi2 image: NASA

41 Calculate EVI2 mean and stdev (JAS ) image: U. Arizona store (aggregate (MODIS_AMZ_BQ_JAS_EVI2, avg(evi2) as evi2_avg_2000_2006, stdev(evi2) as evi2_stdev_2000_2006, col_id, row_id), MODIS_AMZ_BQ_JAS_EVI2_AVG_2000_2006); Attributes: evi2_avg_2000_2006, evi2_stdev_2000_2006

42 Extract data for JAS 2005 and get EVI2 mean image: U. Arizona - - Filters data for 2005's 3rd quarter store(between (MODIS_AMZ_BQ_JAS_EVI2, 48000, 38400, 253, 67199, 52799, 264), MODIS_AMZ_BQ_JAS_EVI2_2005); - - Average for 2005's 3rd quarter store(aggregate(modis_amz_bq_jas_evi2_2005, avg(evi2) as evi2_avg_jas_2005, col_id, row_id), MODIS_AMZ_BQ_JAS_EVI2_2005_AVG);

43 Joining two arrays (JAS 2005 and JAS ) image: U. Arizona store(join (MODIS_AMZ_EVI2_BQ_JAS_AVG_2000_2006, MODIS_AMZ_EVI2_BQ_JAS_2005_AVG), MODIS_AMZ_EVI2_COMP); Attributes: evi2_avg_2000_2006, evi2_stdev_2000_2006 evi2_avg_jas_2005

44 AFL: EVI anomalies (JAS 2005) store(apply (MODIS_AMZ_EVI2_COMP, evi_anomaly, (evi2_avg_jas_ evi2_avg_2000_2006) /evi2_stdev_2000_2006), MODIS_AMZ_EVI2_ANOM);

45 7 lines of SciDB commands 4,000 MODIS +les (92 billion cells) 4.6 hours processing on a single medium- sized server 6 months learning curve

Fields as a Generic Data Type for Big SpaLal Data

Fields as a Generic Data Type for Big SpaLal Data Fields as a Generic Data Type for Big SpaLal Data Gilberto Camara, Max J. Egenhofer, Karine Ferreira, Pedro Andrade, Gilberto Queiroz, Alber Sanchez, Jim Jones, and Lubia Vinhas image: INPE mobile devices

More information

INPE Spatial Data Infrastructure for Big Spatiotemporal Data Sets. Karine Reis Ferreira (INPE-Brazil)

INPE Spatial Data Infrastructure for Big Spatiotemporal Data Sets. Karine Reis Ferreira (INPE-Brazil) INPE Spatial Data Infrastructure for Big Spatiotemporal Data Sets Karine Reis Ferreira (INPE-Brazil) INPE (Brazil) and Big Data I work at INPE (Brazilian National Institute for Space Research). Main gate

More information

The Big Data Lab for Interdisciplinary SpatiallyEnabled Science (BLISS)

The Big Data Lab for Interdisciplinary SpatiallyEnabled Science (BLISS) The Big Data Lab for Interdisciplinary SpatiallyEnabled Science (BLISS) Goal Setting up a large database of time series of changes in land use for most of the agricultural areas of the planet Agenda The

More information

TerraAmazon - The Amazon Deforestation Monitoring System - Karine Reis Ferreira

TerraAmazon - The Amazon Deforestation Monitoring System - Karine Reis Ferreira TerraAmazon - The Amazon Deforestation Monitoring System - Karine Reis Ferreira GEOSS Users & Architecture Workshop XXIV: Water Security & Governance - Accra Ghana / October 2008 INPE National Institute

More information

A Hybrid Architecture for Mobile Geographical Data Acquisition and Validation Systems

A Hybrid Architecture for Mobile Geographical Data Acquisition and Validation Systems A Hybrid Architecture for Mobile Geographical Data Acquisition and Validation Systems Claudio Henrique Bogossian 1, Karine Reis Ferreira 1, Antônio Miguel Vieira Monteiro 1, Lúbia Vinhas 1 1 DPI Instituto

More information

TOLOMEO. ORFEO Toolbox. Jordi Inglada - CNES. TOoLs for Open Mul/- risk assessment using Earth Observa/on data TOLOMEO

TOLOMEO. ORFEO Toolbox. Jordi Inglada - CNES. TOoLs for Open Mul/- risk assessment using Earth Observa/on data TOLOMEO ORFEO Toolbox Jordi Inglada - CNES TOoLs for Open Mul/- risk assessment using Earth Observa/on data Outline ORFEO Toolbox : general characteris>cs Example of OTB features OTB Applica>ons & Processing Chains

More information

EUMETSAT EO Portal. End User Image Access using OGC WMS/WCS services. EUM/OPS/VWG/10/0095 Issue <1> <14/01/2010> Slide: 1

EUMETSAT EO Portal. End User Image Access using OGC WMS/WCS services. EUM/OPS/VWG/10/0095 Issue <1> <14/01/2010> Slide: 1 EUMETSAT EO Portal End User Image Access using OGC WMS/WCS services Slide: 1 Overview Introduction: status of data access and visualization EUMETSAT datasets Architecture Web Map Service implementation

More information

.nl ENTRADA. CENTR-tech 33. November 2015 Marco Davids, SIDN Labs. Klik om de s+jl te bewerken

.nl ENTRADA. CENTR-tech 33. November 2015 Marco Davids, SIDN Labs. Klik om de s+jl te bewerken Klik om de s+jl te bewerken Klik om de models+jlen te bewerken Tweede niveau Derde niveau Vierde niveau.nl ENTRADA Vijfde niveau CENTR-tech 33 November 2015 Marco Davids, SIDN Labs Wie zijn wij? Mijlpalen

More information

DNS Big Data Analy@cs

DNS Big Data Analy@cs Klik om de s+jl te bewerken Klik om de models+jlen te bewerken! Tweede niveau! Derde niveau! Vierde niveau DNS Big Data Analy@cs Vijfde niveau DNS- OARC Fall 2015 Workshop October 4th 2015 Maarten Wullink,

More information

TerraLib as an Open Source Platform for Public Health Applications. Karine Reis Ferreira

TerraLib as an Open Source Platform for Public Health Applications. Karine Reis Ferreira TerraLib as an Open Source Platform for Public Health Applications Karine Reis Ferreira September 2008 INPE National Institute for Space Research Brazilian research institute Main campus is located in

More information

NetCDF and HDF Data in ArcGIS

NetCDF and HDF Data in ArcGIS 2013 Esri International User Conference July 8 12, 2013 San Diego, California Technical Workshop NetCDF and HDF Data in ArcGIS Nawajish Noman Kevin Butler Esri UC2013. Technical Workshop. Outline NetCDF

More information

e-sensing: Big Earth observation data analytics for land use and land cover change information

e-sensing: Big Earth observation data analytics for land use and land cover change information e-sensing: Big Earth observation data analytics for land use and land cover change information Thematic Project (48 months) November 2014 October 2018 Project leader: Prof. Dr. Gilberto Câmara (INPE Instituto

More information

Interactive Visualization of Big Data Leveraging Databases for Scalable Computation. Leilani Marie Battle

Interactive Visualization of Big Data Leveraging Databases for Scalable Computation. Leilani Marie Battle Interactive Visualization of Big Data Leveraging Databases for Scalable Computation by Leilani Marie Battle B.S., University of Washington (2011) Submitted to the Department of Electrical Engineering and

More information

Obtaining and Processing MODIS Data

Obtaining and Processing MODIS Data Obtaining and Processing MODIS Data MODIS is an extensive program using sensors on two satellites that each provide complete daily coverage of the earth. The data have a variety of resolutions; spectral,

More information

Licensing++ for Clouds. Mark Perry

Licensing++ for Clouds. Mark Perry Licensing++ for Clouds Mark Perry Plan* 1. Cloud? 2. Survey 3. Some ques@ons 4. Some ideas 5. Some sugges@ons (that would be you) * Plan 9 future events such as these will affect you in the future Clouds

More information

BLACKBRIDGE SATELLITE IMAGERY THROUGH CLOUD COMPUTING

BLACKBRIDGE SATELLITE IMAGERY THROUGH CLOUD COMPUTING BLACKBRIDGE SATELLITE IMAGERY THROUGH CLOUD COMPUTING Jason Setzer Cloud Product Manager Slide 1 THE RAPID EYE CONSTELLATION 5 Identical Satellites in same obit Up to 5 million km² collected daily 1 billion

More information

Data-centric Renovation of Scientific Workflow in the Age of Big Data

Data-centric Renovation of Scientific Workflow in the Age of Big Data Data-centric Renovation of Scientific Workflow in the Age of Big Data Ryong Lee, Ph. D. ryonglee@kisti.re.kr Dept. of Scientific Big Data Research Korea Institute of Science and Technology Information

More information

NERSC Data Efforts Update Prabhat Data and Analytics Group Lead February 23, 2015

NERSC Data Efforts Update Prabhat Data and Analytics Group Lead February 23, 2015 NERSC Data Efforts Update Prabhat Data and Analytics Group Lead February 23, 2015-1 - A little bit about myself Computer Scien.st Brown, IIT Delhi Real- 3me Graphics, Virtual Reality, HCI Computa3onal

More information

Big Data and Analytics: A Conceptual Overview. Mike Park Erik Hoel

Big Data and Analytics: A Conceptual Overview. Mike Park Erik Hoel Big Data and Analytics: A Conceptual Overview Mike Park Erik Hoel In this technical workshop This presentation is for anyone that uses ArcGIS and is interested in analyzing large amounts of data We will

More information

Darrel L. Williams 1, Samuel N. Goward 2, and Ta9ana Loboda 2

Darrel L. Williams 1, Samuel N. Goward 2, and Ta9ana Loboda 2 Darrel L. Williams 1, Samuel N. Goward 2, and Ta9ana Loboda 2 1 Global Science & Technology, Inc., 7855 Walker Drive, Suite 200, Greenbelt, MD 20770 2 U. of Maryland College Park, Department of Geographical

More information

Matsu Workflow: Web Tiles and Analytics over MapReduce for Multispectral and Hyperspectral Images

Matsu Workflow: Web Tiles and Analytics over MapReduce for Multispectral and Hyperspectral Images Matsu Workflow: Web Tiles and Analytics over MapReduce for Multispectral and Hyperspectral Images Open Cloud Consortium Open Data Group September 7, 2012 1 Materials for Matsu Matsu is a public project.

More information

HPSA Agent Characterization

HPSA Agent Characterization HPSA Agent Characterization Product HP Server Automation (SA) Functional Area Managed Server Agent Release 9.0 Page 1 HPSA Agent Characterization Quick Links High-Level Agent Characterization Summary...

More information

Satellite derived Groundwater Storage Es4mates and Opportuni4es for Expanding Research in Arid Environments

Satellite derived Groundwater Storage Es4mates and Opportuni4es for Expanding Research in Arid Environments Satellite derived Groundwater Storage Es4mates and Opportuni4es for Expanding Research in Arid Environments NASA Ames DEVELOP Program Summer 2011 Amber Kuss, San Francisco State University William Brandt,

More information

Advanced Image Management using the Mosaic Dataset

Advanced Image Management using the Mosaic Dataset Esri International User Conference San Diego, California Technical Workshops July 25, 2012 Advanced Image Management using the Mosaic Dataset Vinay Viswambharan, Mike Muller Agenda ArcGIS Image Management

More information

Bringing Big Data Modelling into the Hands of Domain Experts

Bringing Big Data Modelling into the Hands of Domain Experts Bringing Big Data Modelling into the Hands of Domain Experts David Willingham Senior Application Engineer MathWorks david.willingham@mathworks.com.au 2015 The MathWorks, Inc. 1 Data is the sword of the

More information

Agile Retrieval of Big Data with. EarthServer. ECMWF Visualization Week, Reading, 2015-sep-29

Agile Retrieval of Big Data with. EarthServer. ECMWF Visualization Week, Reading, 2015-sep-29 Agile Retrieval of Big Data with EarthServer ECMWF Visualization Week, Reading, 2015-sep-29 Peter Baumann Jacobs University rasdaman GmbH baumann@rasdaman.com [co-funded by EU through EarthServer, PublicaMundi]

More information

The USGS Landsat Big Data Challenge

The USGS Landsat Big Data Challenge The USGS Landsat Big Data Challenge Brian Sauer Engineering and Development USGS EROS bsauer@usgs.gov U.S. Department of the Interior U.S. Geological Survey USGS EROS and Landsat 2 Data Utility and Exploitation

More information

A RDF Vocabulary for Spatiotemporal Observation Data Sources

A RDF Vocabulary for Spatiotemporal Observation Data Sources A RDF Vocabulary for Spatiotemporal Observation Data Sources Karine Reis Ferreira 1, Diego Benincasa F. C. Almeida 1, Antônio Miguel Vieira Monteiro 1 1 DPI Instituto Nacional de Pesquisas Espaciais (INPE)

More information

Sisense. Product Highlights. www.sisense.com

Sisense. Product Highlights. www.sisense.com Sisense Product Highlights Introduction Sisense is a business intelligence solution that simplifies analytics for complex data by offering an end-to-end platform that lets users easily prepare and analyze

More information

A Benchmark to Evaluate Mobile Video Upload to Cloud Infrastructures

A Benchmark to Evaluate Mobile Video Upload to Cloud Infrastructures A Benchmark to Evaluate Mobile Video Upload to Cloud Infrastructures Afsin Akdogan, Hien To, Seon Ho Kim and Cyrus Shahabi Integrated Media Systems Center University of Southern California, Los Angeles,

More information

AAFC Medium-Resolution EO Data Activities for Agricultural Risk Assessment

AAFC Medium-Resolution EO Data Activities for Agricultural Risk Assessment AAFC Medium-Resolution EO Data Activities for Agricultural Risk Assessment North American Drought Monitor (NADM) Ottawa, Ontario, Canada. October 15-17 2008. A. Davidson 1, A. Howard 1,2, K. Sun 1, M.

More information

TLD Data Analysis. ICANN Tech Day, Dublin. October 19th 2015 Maarten Wullink, SIDN. Klik om de s+jl te bewerken

TLD Data Analysis. ICANN Tech Day, Dublin. October 19th 2015 Maarten Wullink, SIDN. Klik om de s+jl te bewerken Klik om de s+jl te bewerken Klik om de models+jlen te bewerken Tweede niveau Derde niveau Vierde niveau TLD Data Analysis Vijfde niveau ICANN Tech Day, Dublin October 19th 2015 Maarten Wullink, SIDN Wie

More information

!"#$%&'()*#"+,&-(.#,"*'/'.%-*

!#$%&'()*#+,&-(.#,*'/'.%-* !"#$%&'()*#"+,&-(.#,"*'/'.%-*!01234567* #0894:6;90* '!#'?* 15* =@3* 03A* B30346;90* 98* 10=3B46=3C* 59DA643* 894* %0=34E4153* &359F4G3* -606B3:30=* >%&-?* =@6=* E4921C35* =@3* 836=F435* 60C* 8F0G;90671;35*

More information

Review for Introduction to Remote Sensing: Science Concepts and Technology

Review for Introduction to Remote Sensing: Science Concepts and Technology Review for Introduction to Remote Sensing: Science Concepts and Technology Ann Johnson Associate Director ann@baremt.com Funded by National Science Foundation Advanced Technological Education program [DUE

More information

Geoinformatics and Environmental Modelling for Supporting Global Land Use Change Research

Geoinformatics and Environmental Modelling for Supporting Global Land Use Change Research CAPES CHAIR UNIVERSITY OF MÜNSTER June 2013 May 2015 Geoinformatics and Environmental Modelling for Supporting Global Land Use Change Research Final Report Gilberto Câmara (INPE - Instituto Nacional de

More information

Handling Heterogeneous EO Datasets via the Web Coverage Processing Service

Handling Heterogeneous EO Datasets via the Web Coverage Processing Service Handling Heterogeneous EO Datasets via the Web Coverage Processing Service Piero Campalani a*, Simone Mantovani b, Alan Beccati a, a a a Jacobs University Bremen (DE) / b MEEO Srl (IT) * {p.campalani@jacobs-university.de

More information

DEVELOPMENT OF AN INTEGRATED IMAGE PROCESSING AND GIS SOFTWARE FOR THE REMOTE SENSING COMMUNITY. P.O. Box 515, São José dos Campos, Brazil.

DEVELOPMENT OF AN INTEGRATED IMAGE PROCESSING AND GIS SOFTWARE FOR THE REMOTE SENSING COMMUNITY. P.O. Box 515, São José dos Campos, Brazil. DEVELOPMENT OF AN INTEGRATED IMAGE PROCESSING AND GIS SOFTWARE FOR THE REMOTE SENSING COMMUNITY UBIRAJARA MOURA DE FREITAS 1, ANTÔNIO MIGUEL MONTEIRO 1, GILBERTO CÂMARA 1, RICARDO CARTAXO MODESTO SOUZA

More information

Detection of Selective Logging for Estimating and Monitoring Forest Degradation: methodologies and experiences in Brazil

Detection of Selective Logging for Estimating and Monitoring Forest Degradation: methodologies and experiences in Brazil Detection of Selective Logging for Estimating and Monitoring Forest Degradation: methodologies and experiences in Brazil Thelma Krug National Institute for Space Research INPE Ministry of Science and Technology

More information

The CBERS satellite, data policy and social benefits

The CBERS satellite, data policy and social benefits GEO Capacity Building Workshop, May 2006 The CBERS satellite, data policy and social benefits Gilberto Câmara Director, National Institute for Space Research From EO to societal benefits source: NASA Knowledge

More information

Data Warehousing. Yeow Wei Choong Anne Laurent

Data Warehousing. Yeow Wei Choong Anne Laurent Data Warehousing Yeow Wei Choong Anne Laurent Databases Databases are developed on the IDEA that DATA is one of the cri>cal materials of the Informa>on Age Informa>on, which is created by data, becomes

More information

Oklahoma s Open Source Spatial Data Clearinghouse: OKMaps

Oklahoma s Open Source Spatial Data Clearinghouse: OKMaps Oklahoma s Open Source Spatial Data Clearinghouse: OKMaps Presented by: Mike Sharp State Geographic Information Coordinator Oklahoma Office of Geographic Information MAGIC 2014 Symposium April 28-May1,

More information

Visualization of Semantic Windows with SciDB Integration

Visualization of Semantic Windows with SciDB Integration Visualization of Semantic Windows with SciDB Integration Hasan Tuna Icingir Department of Computer Science Brown University Providence, RI 02912 hti@cs.brown.edu February 6, 2013 Abstract Interactive Data

More information

INTEROPERABLE IMAGE DATA ACCESS THROUGH ARCGIS SERVER

INTEROPERABLE IMAGE DATA ACCESS THROUGH ARCGIS SERVER INTEROPERABLE IMAGE DATA ACCESS THROUGH ARCGIS SERVER Qian Liu Environmental Systems Research Institute 380 New York Street Redlands, CA92373, U.S.A - qliu@esri.com KEY WORDS: OGC, Standard, Interoperability,

More information

Landsat Monitoring our Earth s Condition for over 40 years

Landsat Monitoring our Earth s Condition for over 40 years Landsat Monitoring our Earth s Condition for over 40 years Thomas Cecere Land Remote Sensing Program USGS ISPRS:Earth Observing Data and Tools for Health Studies Arlington, VA August 28, 2013 U.S. Department

More information

An Open Dynamic Big Data Driven Applica3on System Toolkit

An Open Dynamic Big Data Driven Applica3on System Toolkit An Open Dynamic Big Data Driven Applica3on System Toolkit Craig C. Douglas University of Wyoming and KAUST This research is supported in part by the Na3onal Science Founda3on and King Abdullah University

More information

«Shanoir : une solu/on pour la ges/on de données distribuées en imagerie in- vivo» Jus/ne Guillaumont Isabelle Corouge

«Shanoir : une solu/on pour la ges/on de données distribuées en imagerie in- vivo» Jus/ne Guillaumont Isabelle Corouge «Shanoir : une solu/on pour la ges/on de données distribuées en imagerie in- vivo» Jus/ne Guillaumont Isabelle Corouge Shanoir: a solu-on for neuro- imaging data management Jus/ne Guillaumont, Isabelle

More information

Big Data Volume & velocity data management with ERDAS APOLLO. Alain Kabamba Hexagon Geospatial

Big Data Volume & velocity data management with ERDAS APOLLO. Alain Kabamba Hexagon Geospatial Big Data Volume & velocity data management with ERDAS APOLLO Alain Kabamba Hexagon Geospatial Intergraph is Part of the Hexagon Family Hexagon is dedicated to delivering actionable information through

More information

The NASA NEESPI Data Portal to Support Studies of Climate and Environmental Changes in Non-boreal Europe

The NASA NEESPI Data Portal to Support Studies of Climate and Environmental Changes in Non-boreal Europe The NASA NEESPI Data Portal to Support Studies of Climate and Environmental Changes in Non-boreal Europe Suhung Shen NASA Goddard Space Flight Center/George Mason University Gregory Leptoukh, Tatiana Loboda,

More information

Best Practices for Sharing Imagery using Amazon Web Services. Peter Becker

Best Practices for Sharing Imagery using Amazon Web Services. Peter Becker Best Practices for Sharing Imagery using Amazon Web Services Peter Becker Objectives Making Imagery Accessible Store massive volumes of imagery on inexpensive cloud storage Use elastic compute for image

More information

GRNET Cloud Compu7ng Services An Overview

GRNET Cloud Compu7ng Services An Overview GRNET Cloud Compu7ng Services An Overview Panos Louridas louridas@grnet.gr GN3 Innova7on Workshop Copenhagen, October 10 11 2011 Greek Research and Technology Network 2 Outline u okeanos IaaS u pithos

More information

Silviu Panica, Marian Neagul, Daniela Zaharie and Dana Petcu (Romania)

Silviu Panica, Marian Neagul, Daniela Zaharie and Dana Petcu (Romania) Silviu Panica, Marian Neagul, Daniela Zaharie and Dana Petcu (Romania) Outline Introduction EO challenges; EO and classical/cloud computing; EO Services The computing platform Cluster -> Grid -> Cloud

More information

Remote Sensing Image Server for GMS (Greater Mekong Sub-Region) Countries

Remote Sensing Image Server for GMS (Greater Mekong Sub-Region) Countries Remote Sensing Image Server for GMS (Greater Mekong Sub-Region) Countries Kiyoshi HONDA, Sarawut Ninsawat, Teerayut Horanont, Ryuzo Yokoyama, Amor V.M. Ines Space Technology Applications and Research (STAR)

More information

PART 1. Representations of atmospheric phenomena

PART 1. Representations of atmospheric phenomena PART 1 Representations of atmospheric phenomena Atmospheric data meet all of the criteria for big data : they are large (high volume), generated or captured frequently (high velocity), and represent a

More information

Precipitation Remote Sensing

Precipitation Remote Sensing Precipitation Remote Sensing Huade Guan Prepared for Remote Sensing class Earth & Environmental Science University of Texas at San Antonio November 14, 2005 Outline Background Remote sensing technique

More information

Welcome to NASA Applied Remote Sensing Training (ARSET) Webinar Series

Welcome to NASA Applied Remote Sensing Training (ARSET) Webinar Series Welcome to NASA Applied Remote Sensing Training (ARSET) Webinar Series Introduction to Remote Sensing Data for Water Resources Management Course Dates: October 17, 24, 31 November 7, 14 Time: 8-9 a.m.

More information

Evaluation of Open Source Data Cleaning Tools: Open Refine and Data Wrangler

Evaluation of Open Source Data Cleaning Tools: Open Refine and Data Wrangler Evaluation of Open Source Data Cleaning Tools: Open Refine and Data Wrangler Per Larsson plarsson@cs.washington.edu June 7, 2013 Abstract This project aims to compare several tools for cleaning and importing

More information

Jozef Matula. Visualisation Team Leader IBL Software Engineering. 13 th ECMWF MetOps Workshop, 31 th Oct - 4 th Nov 2011, Reading, United Kingdom

Jozef Matula. Visualisation Team Leader IBL Software Engineering. 13 th ECMWF MetOps Workshop, 31 th Oct - 4 th Nov 2011, Reading, United Kingdom Visual Weather web services Jozef Matula Visualisation Team Leader IBL Software Engineering Outline Visual Weather in a nutshell. Path from Visual Weather (as meteorological workstation) to Web Server

More information

George Mason University (GMU)

George Mason University (GMU) George Mason University (GMU) Center for Spatial Information Science and Systems Organization (CSISS) 4400 University Drive, MSN 6E1 George Mason University Fairfax, VA 22030, USA Telephone: +1 703 993

More information

An Integrated GeoAgro Webtool for Spatial Data Visualization and Dissemination

An Integrated GeoAgro Webtool for Spatial Data Visualization and Dissemination OPEN ACCESS Conference Proceedings Paper Remote Sensing An Integrated GeoAgro Webtool for Spatial Data Visualization and Dissemination Jalal Eddin Omari and Chandrashekhar M. Biradar International Center

More information

Partek Flow Installation Guide

Partek Flow Installation Guide Partek Flow Installation Guide Partek Flow is a web based application for genomic data analysis and visualization, which can be installed on a desktop computer, compute cluster or cloud. Users can access

More information

Enabling Science in the Cloud: A Remote Sensing Data Processing Service for Environmental Science Analysis

Enabling Science in the Cloud: A Remote Sensing Data Processing Service for Environmental Science Analysis Enabling Science in the Cloud: A Remote Sensing Data Processing Service for Environmental Science Analysis Catharine van Ingen 1, Jie Li 2, Youngryel Ryu 3, Marty Humphrey 2, Deb Agarwal 4, Keith Jackson

More information

Reprojecting MODIS Images

Reprojecting MODIS Images Reprojecting MODIS Images Why Reprojection? Reasons why reprojection is desirable: 1. Removes Bowtie Artifacts 2. Allows geographic overlays (e.g. coastline, city locations) 3. Makes pretty pictures for

More information

VITO Centre of Image Processing

VITO Centre of Image Processing 07/11/2013 VITO Centre of Image Processing Towards an Improved Scientific Exploitation of EO Data Sources in Support of Vegetation Monitoring Erwin Goor, VITO nv Outline» Introduction» The role & and objectives

More information

Resolutions of Remote Sensing

Resolutions of Remote Sensing Resolutions of Remote Sensing 1. Spatial (what area and how detailed) 2. Spectral (what colors bands) 3. Temporal (time of day/season/year) 4. Radiometric (color depth) Spatial Resolution describes how

More information

On the Efficient Evaluation of Array Joins

On the Efficient Evaluation of Array Joins [co-funded by EU through H2020 EarthServer -2] On the Efficient Evaluation of Array Joins Big Data in the Geosciences Workshop IEEE Big Data, Santa Clara, US, 2015-oct-29 Peter Baumann, Vlad Merticariu

More information

Big Data and Big Analytics

Big Data and Big Analytics Big Data and Big Analytics Introducing SciDB Open source, massively parallel DBMS and analytic platform Array data model (rather than SQL, Unstructured, XML, or triple-store) Extensible micro-kernel architecture

More information

Processing of Mix- Sensi0vity Video Surveillance Streams on Hybrid Clouds

Processing of Mix- Sensi0vity Video Surveillance Streams on Hybrid Clouds Processing of Mix- Sensi0vity Video Surveillance Streams on Hybrid Clouds Chunwang Zhang, Ee- Chien Chang School of Compu2ng, Na2onal University of Singapore 28 th June, 2014 Outline 1. Mo0va0on 2. Hybrid

More information

CollaboraLve Development of a Standards- Based XML System for IGS Site Log Metadata Management and DisseminaLon

CollaboraLve Development of a Standards- Based XML System for IGS Site Log Metadata Management and DisseminaLon CollaboraLve Development of a Standards- Based XML System for IGS Site Log Metadata Management and DisseminaLon Fran Boler, Nicholas Brown, Roger Fraser, Carey Noll, Melinda Squibb, Markus Bradke, Carine

More information

An Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database

An Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database An Oracle White Paper June 2012 High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database Executive Overview... 1 Introduction... 1 Oracle Loader for Hadoop... 2 Oracle Direct

More information

Towards Analytical Data Management for Numerical Simulations

Towards Analytical Data Management for Numerical Simulations Towards Analytical Data Management for Numerical Simulations Ramon G. Costa, Fábio Porto, Bruno Schulze {ramongc, fporto, schulze}@lncc.br National Laboratory for Scientific Computing - RJ, Brazil Abstract.

More information

I.Jonckheere, E. Lindquist & A. Pekkarinen FAO Forestry Department

I.Jonckheere, E. Lindquist & A. Pekkarinen FAO Forestry Department Space Data Management System (SDMS) Acquire, Query, Process and Deliver Earth Observation Data and Forest Information Products to Developing Countries Pilot & Project I.Jonckheere, E. Lindquist & A. Pekkarinen

More information

AQA GCSE in Computer Science Computer Science Microsoft IT Academy Mapping

AQA GCSE in Computer Science Computer Science Microsoft IT Academy Mapping AQA GCSE in Computer Science Computer Science Microsoft IT Academy Mapping 3.1.1 Constants, variables and data types Understand what is mean by terms data and information Be able to describe the difference

More information

County of Los Angeles. Chief Information Office Preferred Technologies for Geographic Information Systems (GIS) September 2014

County of Los Angeles. Chief Information Office Preferred Technologies for Geographic Information Systems (GIS) September 2014 County of Los Angeles Chief Information Office Preferred Technologies for Geographic Information Systems (GIS) September 2014 CIO Preferred Technologies for GIS This document lists the preferred Geographic

More information

A SciDB-based Framework for Efficient Satellite Data Storage and Query based on Dynamic Atmospheric Event Trajectory

A SciDB-based Framework for Efficient Satellite Data Storage and Query based on Dynamic Atmospheric Event Trajectory A SciDB-based Framework for Efficient Satellite Data Storage and Query based on Dynamic Atmospheric Event Trajectory ABSTRACT Luboš Krčál Nanyang Technological University, Singapore Czech Technical University

More information

Distributed Computing and Big Data: Hadoop and MapReduce

Distributed Computing and Big Data: Hadoop and MapReduce Distributed Computing and Big Data: Hadoop and MapReduce Bill Keenan, Director Terry Heinze, Architect Thomson Reuters Research & Development Agenda R&D Overview Hadoop and MapReduce Overview Use Case:

More information

Cloud Based Tes,ng & Capacity Planning (CloudPerf)

Cloud Based Tes,ng & Capacity Planning (CloudPerf) Cloud Based Tes,ng & Capacity Planning (CloudPerf) Joan A. Smith Emory University Libraries joan.smith@emory.edu Frank Owen Owenworks Inc. frank@owenworks.biz Full presenta,on materials and CloudPerf screencast

More information

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan Data Management in the Cloud: Limitations and Opportunities Annies Ductan Discussion Outline: Introduc)on Overview Vision of Cloud Compu8ng Managing Data in The Cloud Cloud Characteris8cs Data Management

More information

ECBDL 14: Evolu/onary Computa/on for Big Data and Big Learning Workshop July 13 th, 2014 Big Data Compe//on

ECBDL 14: Evolu/onary Computa/on for Big Data and Big Learning Workshop July 13 th, 2014 Big Data Compe//on ECBDL 14: Evolu/onary Computa/on for Big Data and Big Learning Workshop July 13 th, 2014 Big Data Compe//on Jaume Bacardit jaume.bacardit@ncl.ac.uk The Interdisciplinary Compu/ng and Complex BioSystems

More information

A remote sensing instrument collects information about an object or phenomenon within the

A remote sensing instrument collects information about an object or phenomenon within the Satellite Remote Sensing GE 4150- Natural Hazards Some slides taken from Ann Maclean: Introduction to Digital Image Processing Remote Sensing the art, science, and technology of obtaining reliable information

More information

Background on Elastic Compute Cloud (EC2) AMI s to choose from including servers hosted on different Linux distros

Background on Elastic Compute Cloud (EC2) AMI s to choose from including servers hosted on different Linux distros David Moses January 2014 Paper on Cloud Computing I Background on Tools and Technologies in Amazon Web Services (AWS) In this paper I will highlight the technologies from the AWS cloud which enable you

More information

Using RDBMS, NoSQL or Hadoop?

Using RDBMS, NoSQL or Hadoop? Using RDBMS, NoSQL or Hadoop? DOAG Conference 2015 Jean- Pierre Dijcks Big Data Product Management Server Technologies Copyright 2014 Oracle and/or its affiliates. All rights reserved. Data Ingest 2 Ingest

More information

2009-04-15. The Representation and Storage of Combinatorial Block Designs Outline. Combinatorial Block Designs. Project Intro. External Representation

2009-04-15. The Representation and Storage of Combinatorial Block Designs Outline. Combinatorial Block Designs. Project Intro. External Representation Combinatorial Block Designs 2009-04-15 Outline Project Intro External Representation Design Database System Deployment System Overview Conclusions 1. Since the project is a specific application in Combinatorial

More information

APPLICATION OF MULTITEMPORAL LANDSAT DATA TO MAP AND MONITOR LAND COVER AND LAND USE CHANGE IN THE CHESAPEAKE BAY WATERSHED

APPLICATION OF MULTITEMPORAL LANDSAT DATA TO MAP AND MONITOR LAND COVER AND LAND USE CHANGE IN THE CHESAPEAKE BAY WATERSHED APPLICATION OF MULTITEMPORAL LANDSAT DATA TO MAP AND MONITOR LAND COVER AND LAND USE CHANGE IN THE CHESAPEAKE BAY WATERSHED S. J. GOETZ Woods Hole Research Center Woods Hole, Massachusetts 054-096 USA

More information

BIRT Application and BIRT Report Deployment Functional Specification

BIRT Application and BIRT Report Deployment Functional Specification Functional Specification Version 1: October 6, 2005 Abstract This document describes how the user will deploy a BIRT Application and BIRT reports to the Application Server. Document Revisions Version Date

More information

Tim Blevins Execu;ve Director Labor and Revenue Solu;ons. FTA Technology Conference August 4th, 2015

Tim Blevins Execu;ve Director Labor and Revenue Solu;ons. FTA Technology Conference August 4th, 2015 Tim Blevins Execu;ve Director Labor and Revenue Solu;ons FTA Technology Conference August 4th, 2015 Governance and Organiza;onal Strategy PaIerns of Fraud and Abuse in Government What tools can we use

More information

A Process and Environment for Embedding The R Software

A Process and Environment for Embedding The R Software A Process and Environment for Embedding The R Software into TerraLib Pedro Ribeiro de Andrade Neto 1, Paulo Justiniano Ribeiro Junior 1 1 Laboratório de Estatística e Geoinformação (LEG) Statistics Department

More information

LSST Database Design Jacek Becla

LSST Database Design Jacek Becla LSST Database Design Jacek Becla Database and Data Access Lead October 21-25, 2013 FINAL DESIGN REVIEW October 21-25, 2013 Name of Mee)ng Loca)on Date - Change in Slide Master 1 Outline Driving requirements

More information

WCS as a Download Service for Big (and Small) Data

WCS as a Download Service for Big (and Small) Data WCS as a Download Service for Big (and Small) Data INSPIRE 2013 Florence, Italy, 2013-jun-25 Peter Baumann 1, Stephan Meissl 2, Alan Beccati 1 1 Jacobs University rasdaman GmbH, Bremen, Germany 2 EOX GmbH,

More information

LoadRunner and Performance Center v11.52 Technical Awareness Webinar Training

LoadRunner and Performance Center v11.52 Technical Awareness Webinar Training LoadRunner and Performance Center v11.52 Technical Awareness Webinar Training Tony Wong 1 Copyright Copyright 2012 2012 Hewlett-Packard Development Development Company, Company, L.P. The L.P. information

More information

smespire - Exercises for the Hands-on Training on INSPIRE Network Services April 2014 Jacxsens Paul SADL KU Leuven

smespire - Exercises for the Hands-on Training on INSPIRE Network Services April 2014 Jacxsens Paul SADL KU Leuven smespire - Exercises for the Hands-on Training on INSPIRE Network Services April 2014 Jacxsens Paul SADL KU Leuven These exercises aim at people who already have some basic knowledge of INSPIRE Network

More information

March 10 th 2011, OSG All Hands Mee6ng, Network Performance Jason Zurawski Internet2 NDT

March 10 th 2011, OSG All Hands Mee6ng, Network Performance Jason Zurawski Internet2 NDT March 10 th 2011, OSG All Hands Mee6ng, Network Performance Jason Zurawski Internet2 NDT Agenda Tutorial Agenda: Network Performance Primer Why Should We Care? (15 Mins) GeNng the Tools (10 Mins) Use of

More information

Using Remote Sensing to Monitor Soil Carbon Sequestration

Using Remote Sensing to Monitor Soil Carbon Sequestration Using Remote Sensing to Monitor Soil Carbon Sequestration E. Raymond Hunt, Jr. USDA-ARS Hydrology and Remote Sensing Beltsville Agricultural Research Center Beltsville, Maryland Introduction and Overview

More information

Data Management in the Cloud

Data Management in the Cloud With thanks to Michael Grossniklaus! Data Management in the Cloud Lecture 8 Data Models Document: MongoDB I ve failed over and over and over again in my life. And that is why I succeed. Michael Jordan

More information

SharePoint Capacity Planning Balancing Organiza,onal Requirements with Performance and Cost

SharePoint Capacity Planning Balancing Organiza,onal Requirements with Performance and Cost SharePoint Capacity Planning Balancing Organiza,onal Requirements with Performance and Cost Kirk Devore / J.D. Wade SharePoint Consultants Horizons Consul;ng Agenda Expecta;ons Defining SharePoint Capacity

More information

Implementation of information system to respond to a nuclear emergency affecting agriculture and food products - Case of Morocco

Implementation of information system to respond to a nuclear emergency affecting agriculture and food products - Case of Morocco Centre National de l énergie des sciences et techniques nucléaires (CNESTEN-Morocco) Implementation of information system to respond to a nuclear emergency affecting agriculture and food products - Case

More information

Digital image processing

Digital image processing 746A27 Remote Sensing and GIS Lecture 4 Digital image processing Chandan Roy Guest Lecturer Department of Computer and Information Science Linköping University Digital Image Processing Most of the common

More information

Fields as a Generic Data Type for Big Spatial Data

Fields as a Generic Data Type for Big Spatial Data Fields as a Generic Data Type for Big Spatial Data Gilberto Camara 1,2, Max J. Egenhofer 3, Karine Ferreira 1, Pedro Andrade 1, Gilberto Queiroz 1, Alber Sanchez 2, Jim Jones 2, Lúbia Vinhas 1 1 Image

More information

SQream Technologies Ltd - Confiden7al

SQream Technologies Ltd - Confiden7al SQream Technologies Ltd - Confiden7al 1 Ge#ng Big Data Done On a GPU- Based Database Ori Netzer VP Product 26- Mar- 14 Analy7cs Performance - 3 TB, 18 Billion records SQream Database 400x More Cost Efficient!

More information

Hadoop- Based Data Explora1on for the Healthcare Safety- Net Technical & Sociocultural Challenges to Big Data Usability

Hadoop- Based Data Explora1on for the Healthcare Safety- Net Technical & Sociocultural Challenges to Big Data Usability Hadoop- Based Data Explora1on for the Healthcare Safety- Net Technical & Sociocultural Challenges to Big Data Usability David Hartzband, D.Sc. Research Affiliate, SSRC, MIT & Director, Technology Research

More information

INCREMENTAL, APPROXIMATE DATABASE QUERIES AND UNCERTAINTY FOR EXPLORATORY VISUALIZATION. Danyel Fisher Microso0 Research

INCREMENTAL, APPROXIMATE DATABASE QUERIES AND UNCERTAINTY FOR EXPLORATORY VISUALIZATION. Danyel Fisher Microso0 Research INCREMENTAL, APPROXIMATE DATABASE QUERIES AND UNCERTAINTY FOR EXPLORATORY VISUALIZATION Danyel Fisher Microso0 Research Exploratory Visualiza9on Ini9al Query Process query Get a response Change parameters

More information