WP22: GEONETCast tools and user support. Tools for data processing

Transcription

1 Page : 1 of 37 WP22: GEONETCast tools and user support Tools for data processing Date Topic WP22 Prepared by Dr. B.H.P. Maathuis (UTwente ITC) and T. Jacobs (VITO) Diffusion All AGRICAB partners, other public Public Page 1 of 37

2 Page : 2 of 37 Version Prepared by Date 1.0 B. Maathuis (Utwente ITC) and T. Jacobs (VITO) Public Page 2 of 37

3 Page : 3 of 37 CONTENTS 1 Background and introduction Tools and training material portals The AGRICAB project website and the Africa Platform Software tool portals Training material portals Overview of currently used tools Tools for Data Management Overview of Capability of the main tools within the Suite ILWIS Generic RS and GIS package ILWIS toolbox plug ins and scripts for data pre processing and retrieval SPIRITS and other free time series processing tools INPE s suite of processing tools Data exchange utilities Further developments Additional and changed satellite images and products Building synergies with national use case development activities Conclusions Public Page 3 of 37

4 Page : 4 of 37 Public Page 4 of 37

5 Page : 5 of 37 1 BACKGROUND AND INTRODUCTION One of the guiding principles in the AGRICAB project is to further develop and promote the use of open source and freeware software tools, in connection with the sustained data access, both online and via low cost GEONETCast satellite broadcast. While dedicated agriculture and forestry management applications took the time to set up properly, ITC, VITO and INPE spent efforts to link their free software packages. The underlying idea was not to modify the tools and merge them (as they are independently developed and used by different communities), but rather to build a stronger tool family with simplified data exchanges between the tools (see figure 1). The current implementation thus comprised mainly of a set of import/export routines to facilitate the data exchanges between: ITC/52N s generic GIS/Remote Sensing software ILWIS and its toolbox extensions (notably GEONETCast and ISOD toolboxes) VITO s new time series analysis tool, SPIRITS, and the product conversion utility VGTExtract INPE s Terra family (TerraView, TerraAmazon, underlying TerraLib) and GeoDMA plugin extension In addition, time was spent on updating existing toolboxes that allow to import and pre process remote sensing data, adding for instance additional useful data sources (e.g. DEIMOS) or updating them to changes in provided products (e.g. switch to MSG10). Figure 1: AGRICAB Tools for Environmental Data Access and (pre )Processing Public Page 5 of 37

6 Page : 6 of 37 These additional and updated data exchange routines were particularly put to the test in two regional training workshops on GEONETCast & image processing tools, using real time GEONETCast data reception. In these workshops, special attention was given to time series data and data transformation as this is a cumbersome activity, taking a lot of time. Experiences obtained from the workshop participants and ad hoc user support requests have been used as well, for instance to upgrade the GEONETCast Data Manager to ensure compatibility with other ground reception configurations, like those used for the meteorological or environmental communities in Africa. This further development of free and open source tools, and data exchanges between them, is just a start. Further developments will be needed to include the upcoming data sources (PROBA V, CBERS 3, Landsat 8), changes to existing data sources (e.g. VEGETATION product changes in context of the Copernicus Global Land service that started in Jan 2013) and new ways to access those data sources (e.g. WP42 on the use of interoperable web services for data access and online processing). More importantly, these free software tools need to be aligned with the other capacity enhancement efforts, in particular the development and flow of national use case applications. Of course, relevant training material may also require updates, and both the software and training materials should be structured along the same end to end application flow (see Training and Workshop plan). As an example (see Figure 1), related to the national level use case development on crop yield forecasting and modelling, Alterra aims to develop a link between SPIRITS and the Crop Growth Model (CGMS) Statistics Tool (CST) they have been using in their training efforts. This development will need to be followed up in WP2, to stimulate synergies. The tools development should thus be seen as a continuous (on going) effort. This document lists the currently implemented and used tools and provides the locations where the software utilities and related documentation and training material can be retrieved. The concept and roadmap for further development of tools within AGRICAB remains to be described in a separate document. Public Page 6 of 37

7 Page : 7 of 37 2 TOOLS AND TRAINING MATERIAL PORTALS Building on past and related activities, AGRICAB aims to build an online Community of Practice with a portal to share data, software tools, training material and other information and to engage users in active discussion and exchange of ideas. While it should thus be noted that this online community and portal development is an on going process and the tools used are from various sources (for some, development started over 25 years ago), all tools used as part of the GEONETCast data access and related tool development efforts in AGRICAB are under active development, freely available and can be retrieved online. This section highlights the main online portals where these tools and the related training materials can be found today. 2.1 The AGRICAB project website and the Africa Platform Project website: Africa Platform: After setting up a short summary web site for the project s visibility, AGRICAB project leader VITO started the development of a new online platform. As a first step towards this, VITO developed its Africa Platform, bringing together over 10 years of experiences of projects, related to the use of Earth Observation techniques for food security (early warning) and management of agriculture, forests and natural resources, all in an effort to strengthen Africa's sustainable development. As tools like VGTExtract and SPIRITS were created as part of these activities, they can now be obtained via the Africa Platform. The portal further provides a listing of workshops conducted and workshop reports, as well as showing examples of integrated applications and on going PhD research. In addition, ITC has started to develop, using the free Moodle environment, a central portal to access training materials. 2.2 Software tool portals Here is an overview of the main portals that provide tools used within AGRICAB so far. Other (underlying or related) tools used, together with the online links to retrieve them and short descriptions of their main functionality, are provided in chapter 3. The Earth Observation and ILWIS Communities: (available at: and The Earth Observation Community promotes the use of free near real time environmental and Earth Observation data (in situ, airborne and space based) and derived products to a worldwide user community. Using inexpensive, off the shelf equipment, the data can be directly received from communication satellites. This capability, in conjunction with data from freely accessible archives, provides the possibility to obtain a multitude of environmental and Earth Observation related data. The Earth Observation community builds on the existing geospatial processing utilities provided by the ILWIS community. This community provides a free and open GIS/RS software framework for research and software development to implement up to date training components, scientific Public Page 7 of 37

8 Page : 8 of 37 toolboxes and (web) services. Recent versions of ILWIS Open integrate image, vector and thematic data in a desktop package. It delivers a wide range of features including import/export, digitizing, editing, analysis and display of data, as well as production of maps. For further modelling and statistical analysis of spatio temporal data, web services, etc. there are links with other communities. The INPE suite of tools: (available at: and mediawiki/geodma/index.php?title=main_page) INPE has developed TerraView, which is a GIS application using the TerraLib GIS library. TerraView s main goal is to make available to the GIS Community an easy geographic data viewer with resources that include database queries and data analysis, using the TerraLib library. TerraView handles vector data (polygons, lines and points) and raster data (grids and images), both stored in a relational or geo relational database, including Microsoft ACCESS, PostgreSQL, MySQL and Oracle. TerraAmazon is a large environmental database developed in TerraLib and can be regarded as the world's first rainforest surveillance system. Recently the Geographic Data Mining Analyst (GEODMA) has been added. GEODMA is a toolbox for integrating remote sensing imagery analysis methods with data mining techniques producing a usercentered, extensible, rich computational environment for information extraction and knowledge discovery over large geographic databases. 2.3 Training material portals The ITC GEONETCast pages: (available at: GEONETCast) These pages provide a lot of information on how to establish a GEONETCast Ground Reception infrastructure and provides access to the so called GEONETCast DevCoCast Application Manual, containing a suite of applications using data delivered through GEONETCast, in conjunction with local in situ observations making use of the tools described below. Application fields covered range from vegetation, biomass and different types of agricultural assessments, natural habitat conservation, insect and pest monitoring, rainfall and evapotranspiration estimation, hot spot and fire detection, drought monitoring and even some marine applications are included. Study areas selected are from South America, Eastern and Southern Africa. The ITC ILWIS pages: (available at: An extensive set of exercises demonstrating various applications can be obtained from the online application guide ( Although initially written for an older ILWIS version most of the menu touch and feel has not changed much until ILWIS version 3.8. Also the older (but stable) ILWIS 3.3 (and update to 3.3.1) can still be obtained online from here. Public Page 8 of 37

9 Page : 9 of 37 The Earth Observation Capacity Building Portal: (available at: The Earth Observation Capacity Building Portal supports the use of free near real time environmental and Earth Observation data (in situ, airborne and space based) and derived products for a worldwide user community. This capability, in conjunction with data from freely accessible archives, provides the possibility to obtain a multitude of environmental and Earth Observation related data. A start has been made making the information presented during the regional GEONETCast courses available to a wider audience. A moodle based (freeware) digital learning environment was established to allow users to access the materials. It is anticipated that this portal is further developed and updated. INPE TERRAVIEW pages: (available at: The TERRAVIEW Tutotial is an online tutorial, consisting of 15 chapters, providing the basic concepts of TERRAVIEW and how to use it. It also provides a link to the different data models which can be applied (and provides also a link to online resources describing the TerraLib Data Model). Public Page 9 of 37

10 Page : 10 of 37 Public Page 10 of 37

11 Page : 11 of 37 3 OVERVIEW OF CURRENTLY USED TOOLS Table 1 below provides an alphabetical list of the tools which are currently used within the integrated AGRICAB suite of tools for increased Earth Observation capacity for better agricultural and forestry management in Africa. Their internet URL is provided for retrieval or for further reference and online documentation. Table 1: Primary Tools integrated in various routines Tool name AMESD SADC Toolbox BUFRDisplay BUFRTool Data Manager GDAL GeoDMA GEONETCast Toolbox HANTS ILWIS IrfanView ISOD Toolbox METAR Weather PANOPLY SPIRITS TERRAVIEW TIMESAT TimeSeriesConversion Tool VGTExtract WFS E Toolbox WGET WGRIB and WGRIB2 Link observation/aboutamesd sadc lighthouse.com/cipher/bufrtool.html observation/aboutreception stations e=main_page observation/aboutgeonetcast observation/aboutisod toolbox observation/about tsconvert observation/about wfse toolbox The functionality of a number of tools will be described later into more detail. For those not described, here a short description on their main functionality is provided. BUFRDisplay and BUFRTool are used to decode the Binary Universal Formatted data (according to WMO conventions). Various data types within GEONETCast are provided in this format. Bufrdisplay is used for visualization and Bufrtool is used for import of the relevant data layers contained within a single Bufr file into ILWIS format. In these routines data produced by the Meteorological Product Extraction Facility at EUMETSAT and disseminated real time through GEONETCast can be imported and processed. The Geospatial Data Abstraction Library (GDAL) is a translator library for raster geospatial data. As a library, it presents a single data model for transformation into most of the common raster data Public Page 11 of 37

12 Page : 12 of 37 formats and, via the included OGR library, also vector data formats. It also comes with a variety of useful commandline utilities for data translation and processing. Harmonic Analysis of Time Series (HANTS) is a time series analysis tool which can be used for removal of clouds. The tool accomplishes two tasks, first it is screening and removing cloud contaminated observations and secondly it performs a temporal interpolation of the remaining observations to reconstruct gapless images at a prescribed time interval. To achieve this the tool calculates a Fourier series to model a time series of pixelwise observations, while at the same time it identifies outliers relative to the model of the time series. The algorithm susequently eliminates these outliers and replaces them with the value given by the computed Fourier series. Additonal software has been developed to allow efficient data exchange of time series data between HANTS and ILWIS. IrfanView is a visualization packages which is capable to deal with the most common graphical picture formats. For images not converted into a GIS or remote sensing format this tool is used for visualization of these pictures. It has various command line options to automate the visualization process and is also used to create animated sequences. Metar Weather decodes weather reports disseminated through the GEONETCast Meteorological Data Dissemination (MDD) Service (the TAF and METAR reports) and displays them in a simple weather report table or as overlay on e.g. Google Earth. Panoply is an application which plots geo gridded arrays from netcdf, HDF and GRIB datasets. Here the tool is used for visualization of data retrieved from the European Centre for Medium Range Weather Forecasting (ECMWF), the so called Era Interim daily fields. Various reprocessed climatological parameters (from 1983 onwards), relevant for agriculture and crop growth modeling, can be retrieved and visualized. TIMESAT is used for analysing time series of satellite sensor data. TimeSat is developed to be able to investigate the seasonality of satellite time series data and their relationship with dynamic properties of vegetation, such as phenology and temporal development. Furthermore the software allows for temporal filtering of time series data. Additonal software has been developed to allow efficient data exchange of time series data between TimeSat and ILWIS. GNU Wget is an open source, free network utility to retrieve files from the World Wide Web using HTTP and FTP, the two most widely used Internet protocols. It can be easily automated, can work non interactively (in the background) and allows both recursive retrievals (e.g. making mirrors of entire web or FTP sites) and is able to continue interrupted downloads. WGRIB and WGRIB2 are command line utilities to transform the Gridded Binary formatted data (versions 1 and 2) into ILWIS. Within the GEONETCast data stream some data is delivered in GRIB format, like the MSG Multi Sensor Precipitation Estimate (MPE). Beyond the tools already integrated into the AGRICAB tool suite, a number of other tools have been shortlisted that are potentially relevant for for pre or post processing of the data. These standalone tools are listed in table 2 and their main functionality is described further below. Table 2: Linkage to other free tools used for pre and post processing or satellite tracking Tool name BEAM VISAT BILKO BRAT Link consult.de/cms/web/beam/ Public Page 12 of 37

13 Page : 13 of 37 OSGeo4W R WXTRACK project.org/ BEAM is an open source package for viewing, analysing and processing of remote sensing raster data. Originally developed to facilitate the utilisation of image data from ENVISAT it also supports a number of other raster data formats such as GeoTIFF and NetCDF as well as data formats of other EO sensors such as MODIS, AVHRR, AVNIR, PRISM and CHRIS/Proba. Various data and algorithms are supported by dedicated extension plug ins. An example is presented below in figure 2 demonstrating the use of Beam as a pre processor to retrieve the ESA s Soil Moisture and Salinity (SMOS) data and using the results in ILWIS. Figure 2: SMOS orbit over the USA and retrieved soil moisture profile BILKO is used as a capacity development tool for remote sensing image analysis, in first instance applied to the use of remote sensing for oceanography and coastal management. BILKO routines currently include a wide range of standard image processing functions. Routines are developed to export the data processed within ILWIS to BILKO to make use of functionality offered by this tool, especially relevant for the marine products disseminated within the GEONETCast data stream. The Basic Radar Altimitry Toolbox (BRAT) facilitates the use of radar altimetry data. It can read most of the distributed radar altimetry data, from ERS 1 & 2, Topex/Poseidon, Geosat Follow on, Jason, Envisat to the future Cryosat missions, and can perform processing and data editing, extraction of statistics, and visualisation of results. Within the GEONETCast toolbox an import routine has been made to import the data which is pre processed using BRAT. The OSGeo4W provides a broad set of open source geospatial software, ready for installation under Windows 32 bit environments (Windows XP, Vista, etc), including the underlying GDAL library and utilities, FWTools viewer OpenEV, QuantumGIS and many more. Public Page 13 of 37

14 Page : 14 of 37 R is a language and environment for statistical computing and graphics and is available as free software. R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time series analysis, classification, clustering, etc.) and graphical techniques, and is highly extensible. Export routines are developed to ensure that these added statistical capabilities can be used for further (geo spatial) statistical analysis. WXTrack: to deal with selection of recordings of satellites the actual positions of these (polar orbiting) satellites at a certain time needs to be known. For this purpose the WXtrack package can be used. With updated Keplers (parameters which describe the satellite orbit, the so called 2 line elements ) the position of the satellite can be retrieved at a user defined (UTC) time to know exactly when a satellite has made (or will make) a recording over a certain area of interest. An automated routine has been added to download the updated Keplers from the internet. Figure 3 is showing a number of satellites recording the Earth. The data from these satellites and associated products are disseminated through GEONETCast. Figure 3: Satellites and their positions at 08:30:52 UTC on 11 March 2013 Public Page 14 of 37

15 Page : 15 of 37 4 TOOLS FOR DATA MANAGEMENT Backbone of the capability for enhanced agricultural and forestry management, amongst others, is the provision of (near) real time data disseminated through the GEONETCast global data dissemination system. This system delivers a multitude of environmentally relevant images and (derived) products to a global user community using low cost and off the shelf equipment using a parabolic antenna in conjunction with a Digital Video Broadcasting (DVB) board. Figure 4 shows the low cost ground reception facility established at the ITC UTwente, the Netherlands. Currently two broadcasts are received, one over Europe and the other over Africa. Two licenses are used, one for science and education and also a manufacturer license was obtained, to ensure that all data can be accessed and if regarded relevant appropriate import routines can be developed. Some systems are used to process the data received, e.g. to create real time animations of METEOSAT 2 nd Generation visible channels, to process rainfall data, like the 15 minutes Multi Sensor Precipitation Estimate, the GOES Rainfall Estimate (over Latin America) or the TAMSAT Rainfall Estimate over Ethiopia and make these available to the user community via FTP. This facility has recently been upgraded with new computers and is also used for other purposes, like hosting a web processing server and is used for training system adminstrators or staff from e.g. African Universities who want to establish a ground reception station themselves. In the recent past also the ESA Data Dissemination System (both the Europe and Africa service) was operated but due to the loss of ENVISAT these activities have ceased. Figure 4: Ground reception facility at the Faculty of Geo Information and Earth Sciences With increasing usage and availability of free remote sensing data, and applications requiring a sensible combination of data, the need for appropriate management and archiving of the retrieved data is evident and even increasing. To help address this need, a Data Manager tool was developed. Its primary purpose is to organize the large amount of data files received on a GEONETCast receiving station a computer with a DVB device connected to a satellite dish, configured to receive the GEONETCast data. The data received on the GEONETCast receiving station must also be transferred to a computer storage that is independent of the receiving station's disk storage before users can access (and process) them, in order to avoid interruption in the data reception. This automated data management system can be easily configured using a simple ascii text file from which it builds its own Graphical User Interface (see figure 5). Once the Data Manager (developed in Java and thus workable on Windows and Linux systems) is activated, an ASCII input configuration file Public Page 15 of 37

16 Page : 16 of 37 needs to be specified. A sample configuration file that is used at ITC for the Ku Band GEONETCast broadcasting is resulting in the Graphical User Interface appearance as given in Figure 5. Having the capability to easily start, adapt functionality (copy or move) and modify configure the menu, changes to the GEONETCast broadcast stream (e.g. new or changed products) can be incorporated easily without the need to wait for or install a software update. Using simple copy paste or other text editing, the configuration file can be tuned accoring to the needs of the user. The system manageradministrator thus does not need to have any programming experience to do these types of manipulations. Figure 5: Data Manager Graphical User Interface As an example of the ever changing data stream, also in the future, the figure below provides the long term position change of the meteorological satellites operated by EUMETSAT. With the satellites changing position, other (new) satellites take over the nominal operation, in turn resulting in different file names for the received data. The same holds true for other data providors, like NOAA or CMA. For construction of a long term archive, this requires adaptation of the data manager configuration file to cope with these changes. Public Page 16 of 37

17 Page : 17 of 37 Figure 6: Spacecraft relocation over the next couple of years (source: EUMETSAT) Following feedback from the participants in AGRICAB s regional GEONETCast training workshops, an updated version of the Data Manager was developed to include an automatic start option after system re start and a copy option, leaving the data in the source folder of the ground receiving station to be available to other applications. This allows the construction of time series in a structured archive even if the other software routines running on the ground reception station are deleting the data from the source folder from time to time. Having access to an (external) archive with the raw data allows the users to utilize also the tools described here, next to those operated within the organization already. Public Page 17 of 37

18 Page : 18 of 37 Public Page 18 of 37

19 Page : 19 of 37 5 OVERVIEW OF CAPABILITY OF THE MAIN TOOLS WITHIN THE SUITE 5.1 ILWIS Generic RS and GIS package The recent ILWIS version has greatly changed and improved visualization functionality, especially to cope with the multitude of (high temporal) data delivered through GEONETCast. A comprehensive update of the graphics system has resulted in an extensive number of options for managing visualization, (multiple synchronized) animations and tuning it for visual analysis. Some of the major changes are: Updated graphics system. Animations and 3D have similar functionality and ease of use to that of the regular 2D visualizations; New Layer tree. The former display options form has been replaced by a Layer tree in the MapWindow. New MapWindow command line. The MapWindow now has its own command line for calculating maps and directly adding them to the MapWindow; New Base maps. A set of base maps can be used as extra layers in a MapWindow. Each base map acts as a system object. Is including OpenStreetMap, for which internet connectivity is required; PixelInfo. The former PixelInfo is now a regular part of the MapWindow; New applications. Due to the increased importance of animation, the need for certain operations for whole map lists (apart from what can be done with MaplistApplication) resulted in the development of a set of applications for map lists, like drawing of profiles sections, Hovmüller diagram, interactive representations, etc; Revamped translation system. The translation system has been revised. A new Spanish translation is now available, next to French and English; ILWIS as a WPS Server. A web server system has been added so that ILWIS can now function as a Web Processing Service (WPS) server and client; Improved help system editability. The help system has been moved to a set of html files that can be freely adapted. Previously, the internal help files were already html files, but because they were bundled in the microsoft help format they were dificult to edit; New printing process. The old layout system has been removed. Maps are now printed via WYSIWYG copy/paste into an appropriate print program or into a document; Correct time definition. A "Time" domain (basically a date time) has been added to enable correct time definition; Extended operation list. The catalog operation list has been extended with a finder option to quickly filter through the long list of ILWIS applications; Updated Surface Energy Balance System (SEBS) module. A document highlighting this new functionality into more detail can be obtained from the ILWIS Community. Other changes are described in the online change log. A number of starter exercises, including sample data, can be retrieved from ftp://ftp.itc.nl/pub/52n/intro_exercises/. An example of the new capabilities is presented in figure 7, using sample data from DEIMOS (one of the medium high resolution satellite data providers in AGRICAB), showing the NE of the Netherlands and NW part of Germany. The bottom layer is from OpenStreetMap, and on top are displayed the Public Page 19 of 37

20 Page : 20 of 37 atmospherically corrected reflectances, the SAVI and NDVI respectively. Visualization is shown as a 3 D stack with an offset in vertical direction to see the various layers from an oblique point of view. Figure 7: DEIMOS sample data import and visualization 5.2 ILWIS toolbox plug ins and scripts for data pre processing and retrieval To retrieve and process the multitude of free satellite images and environmental data disseminated through GEONETCast and available online, in a mixture of data formats, attention was given to the development of import and pre processing libraries. Given that a large number of routines have to be used for data import and pre processing, these have been integrated into more user friendly toolboxes, developed as plug ins (extensions) in ILWIS. This was done with the following rationale: The toolboxes need to be open and allow modification and full control by the users; Being able to update and modify the toolbox independently of ILWIS software updates; Using the extensive GIS/RS functionality already offered by ILWIS and, as it is an inhouse developed package, additional GIS/RS functionality can be easily added; To built links with other free tools relevant for environmental data pre and post processing (like the VITO and INPE tools in the frame of AGRICAB). The figure below shows the main toolbox(es) development concept. The relevant data, disseminated through GEONETCast, are stored on a storage device that can be accessed within an organization using the automated data management system. After this, a whole suite of tools is at disposal of the user, which can be selected through a Graphical User Interface, to bring the various data sources Public Page 20 of 37

21 Page : 21 of 37 (that require further processing and analysis) into a common GIS/RS environment. The user can further integrate online archived data, using the In situ and Online Data (ISOD) Toolbox. Figure 8: Representation of the main development concept behind the toolbox plug ins Through various collaborations in different projects and with partners from e.g. Africa, as well as the conducted training workshops, it was noted that the generic GEONETCast Toolbox, while ideal to explore the potential of GEONETCast, required further refinement to meet the needs of the different user communities. This triggered the development of more dedicated toolboxes which allow retrieval and pre processing tailored to specific geographical regions and/or thematic applications (omitting irrelevant datasets). Examples of such toolboxes are the Water and Food Security Ethiopia toolbox and the SADC Thema toolbox, described below. While additional toolboxes have been defined and will be further developed in future, for instance for ECOWAS and CEMAC regions, or for the Asian GEONETCast (CMACast) data stream, four toolbox plug ins are currently at disposal to the user community. Main features of the GEONETCast Toolbox are: friendly user interface, automated data management, import routines for various geostationary satellites like METEOSAT 1 st and 2 nd Generation satellites, GOES, Fengyun and MTSAT, products from the Meteorological Product Extraction Facility (MPEF), images and products delivered through the Rapid Scanning Service (RSS), products from the Satellite Application Facilities (like the LSA SAF) for Europe, Africa and Latin America, products from the Chinese Meteorological Administration (CMA) based on observations Public Page 21 of 37

22 Page : 22 of 37 from Fengyun 2D and 2E for SE Asia. Routines to handle data from so called 3 rd party data providers like TAMSAT (Univ. Reading, UK), MODIS, SPOT VEGETATION basic and derived products from GMES/Copernicus Global Land, EAMNET marine data, integration of METOP AVHRR and JASON 2 data, METOP A/B soil surface moisture and ocean vector wind products, generic geotif import routine, export routines to BILKO and R, calculation of solar and MSG zenith and azimuth angles, bufr data visualization option, automatic real time visualization of METEOSAT Second Generation visible channels (for various pre defined windows) and GOES MSG Composite thermal channel animated visualization. The time series data transformation utility developed, described into more detail below, is also integrated within this toolbox. Main features of the ISOD Toolbox are a friendly user interface, automatic retrieval and import of data from various free online archives related to in situ climatological observations and Meteorological Area Reports (METAR), gauge and various satellite derived rainfall estimates and climatologies, weather and pressure forecasts, potential evapotranspiration, normalized difference vegetation indices from various instruments for Africa and at global level and global elevation information at different spatial resolutions, global ocean data on sea surface temperature and chlorophyll, Global Land Data Assimilation System 3 hourly climatological data from the Noah model, global daily day and night time surface soil products, AMESD SADC online Thema products on Agriculture, Drought, Fire and long range temperature and rainfall forecasts and web mapping services, like visualization of data from the European Centre for Medium Range Weather Forecasting and Sea Surface Temperature and anomalies from the NOAA Climate Prediction Centre. Main features of the AMESD SADC Toolbox are: friendly user interface, import and pre processing of all products disseminated through GEONETCast by the AMESD SADC program for southern Africa related to agriculture, drought, fire and long range forecasting. A total of 31 dekadal agricultural products, 17 dekadal drought related products, 6 different types of fire product, including fire danger forecasting indices and daily MODIS colour composites for each of the SADC countries and long range rainfall and temperature (minimum and maximum) forecasts. In fact, this toolbox is developed by staff from the Botswana Department of Meteorological Services, with some guidance. Main features of the Water and Food Security Ethiopia Toolbox (WFS E) are: friendly user interface, most products are retrieved for the Ethiopian region only and are resampled to a standard map projection, automated data management, processing of selected GEONETCast data relevant for water and food security monitoring and analysis for Ethiopia, like import routines for METEOSAT 1 st and 2 nd Generation satellites, automatic real time visualization of MSG HRIT SEVIRI visible and thermal channels, for the Ethiopia window data can be imported from MPEF, Land Surface Analysis SAF, actual evapotranspiration aggregation routine, TAMSAT rainfall and anomaly products, various products from SPOT VEGETATION, Meteorological Data Dissemination Service routines to import MDD 1 (METAR and TAF) and MDD 3 (like Mean Sea Level Pressure, Geopotential Height at 500 hpa and Winds at various pressure levels), surface soil moisture derived from the ASCAT instrument on METOP A/B, MODIS and MSG based fire products. Export routines to Livelihood Early Warning & Protection software (LEAP used and provided by the World Food Program Ethiopia country office and national authorities) are gradually being added. For all toolboxes, updated user and installation manuals are available at the Earth Observation Community portal, describing their functionality and use in more detail. Sample data (for those interested, but not operating ground receiving stations) have been made available on FTP (ftp://ftp.itc.nl/pub/52n/) that can be used in conjunction with exercises, comprising the last part of each of the user manuals. Public Page 22 of 37

23 Page : 23 of 37 Below a few examples are shown of various toolbox routines, in conjunction with the generic ILWIS processing capabilities. Figure 9 shows a number of products derived from the Meteorological Product Extraction Facility operated by EUMETSAT after their import with the GEONETCast toolbox. The top left image shows a Cloud Analysis Images (CLAI) product. The top right shows the one of the retrieved Global Instability Index (GII) products, here the K Index data is shown. Other datasets processed from the GII product are the KO Index, Lifted Index, Maximum Buoyancy and Precipitable Water. Below are the daily (left) and dekadal (right) NDVI products. The examples show the mean NDVI, but also the maximum and minimum NDVI values, per data set, are processed. Figure 9: Products derived from MPEF EUMETSAT Figure 10 below shows the products derived from the ASCAT instrument onboard of METOP B. The left image shows the surface soil moisture product (plotted on top of OpenStreetMap) and the right one shows the Ocean Vector Winds product. Both examples are taken from / along the Mediterranean Sea, zooming in to one of the ASCAT swath strips recorded. Public Page 23 of 37

24 Page : 24 of 37 Figure 10: Surface soil moisture aand ocean vector winds from METOP B Figure 11 show the extracted drainage network from the ASTER GDEM Digital Terrain Model for a region east of the Rift in Ethiopia, retrieved using the ISOD toolbox. The output tables provide the topological information facilitating the required input for dedicated external rainfall runoff models. The satellite precipitation estimates, contained within the GEONETCast data stream (via GEONETCast toolbox), or from time series archives (via the ISOD toolbox), can be used for model input as well. Figure 11: ASTER GDEM and extracted drainage network A further example is presented in figure 12. Here, various derived morphometric parameters are used to develop a prioritization scheme for catchments requiring conservation measures, like (agro ) Public Page 24 of 37

25 Page : 25 of 37 forestry measures. Linear parameters are used like bifurcation ratio, stream frequency, drainage texture and density but also shape parameters like elegation and circularity ratio s as well as form factor. All have different relations with respect to (potential) erodibility and should guide the efforts related to water resources management and conservation work. The example shows the prioritization of the Upper Manyame catchment, an important hydrologic area in Zimbabwe. Figure 12: Catchment prioritization of the Upper Manyame catchment in Zimbabwe Next example shows the real time visualization capability. During day time the visible channels of MSG can be used, eventually in combination and merged with the high resolution visible channel. During night time, the thermal channel is used and eventually the cloud top temperatures can be colour coded in a few classes (slices with specific temperature thresholds) to ease visual detection and interpretation. These operations are performed automatically every 15 minutes and an animated sequence of these images shows clearly the temporal developments within the atmosphere. This visualization, as implemented in the Water and Food Security Ethiopia toolbox, not only shows Ethiopia but also the Eastern African (IGAD) region. The GEONETCast toolbox includes another real time visualization, again with animation capability, for the thermal composite of MSG and GOES, as produced by INPE every 15 minutes. Public Page 25 of 37

26 Page : 26 of 37 Figure 13a: Real time visualization of MSG over Ethiopia and Eastern Africa using the visible and HRV channel Figure 13b: Real time visualization of MSG over Ethiopia using the Thermal IR channel Finally, figure 14 shows the onset of rains map for the SADC region. It is determined using a threshold amount and distribution of rainfall received in three consecutive dekades. The start of season is established when there is at least 25 mm of rainfall in one dekade followed by a total of at least 20 mm of rainfall in the next two consecutive dekades. The Dry Matter Productivity products (in kgdm/ha/day) are 10 day maps, oriented towards crop monitoring and yield estimation. For dry matter productivity assessment over the growing season, an analysis needs to be made of a multitemporal set of those images so that the progress of the growing season and changing growth rates can be assessed (e.g. by looking at the differences of consecutive images in the same season, or by comparing to previous years or seasons or long term averages). Alternatively, the Dry Matter Productivity images can be cumulated over time from the start of the season in order to estimate the final dry matter which has been produced by the vegetation over time. The DMP product can be integrated into yield prediction models. These products are disseminated via GEONETCast and can be obtained online as well. All products can be imported using the AMESD SADC or ISOD toolboxes. Public Page 26 of 37

27 Page : 27 of 37 Figure 14: Onset of rainfall and Cumulative Dry Matter Productivity The examples above only demonstrate a few capabilities. In total over 550 routines are at the disposal of the user when operating all toolboxes. Detailed information on the various routines is provided in the respective online Installation and User Manuals. Some online data sources, like VITO s SPOT VGT, METOP AVHRR S10, MERIS Reduced Resolution S10 or ESA s Culture Meris, can only be downloaded with specific username/password restriction. To import these data, an online ILWIS script library (ftp://ftp.itc.nl/pub/52n/ilwis_scripts/) is provided in addition to the regular toolbox plug ins. Once the scripts are downloaded and the zip file is extracted into the ILWIS sub directory \Scripts, they are available to the user in the ILWIS software. The zip file also contains a document describing how to operate the script and specifies relevant processing details. An example from the MERIS RR S10 script is given in the figure below, showing the NDVI and FAPAR for the African continent. Figure 15: MERIS RR S10 NDVI and FAPAR import through ILWIS scripts Generic vector and polygon layers which are used on a continuous basis, like country boundaries, main rivers and cities, are made available online as well as ILWIS BaseMaps (ftp://ftp.itc.nl/pub/52n/ilwis_basemaps/). Within the ILWIS38 or higher version these basemaps are Public Page 27 of 37

28 Page : 28 of 37 integrated into the system together with online access to OpenStreetMap, which can be used as a topographical reference. 5.3 SPIRITS and other free time series processing tools The Software for the Processing and Interpretation of Remotely sensed Image Time Series (SPIRITS) is a Windows based software aiming at the analysis of remotely sensed earth observation data. Although it includes a wide range of general purpose functionalities, the focus lies on the processing of time series of images, derived from low resolution sensors such as SPOT VEGETATION, NOAA AVHRR, METOP AVHRR, TERRA MODIS, ENVISAT MERIS and MSG SEVIRI. SPIRITS has been developed by VITO s remote sensing unit on behalf of (and sponsored by) the European Commission s Joint Research Centre (EC JRC) in Ispra, Italy. The JRC MARS group (Monitoring Agricultural ResourceS) continuously supplies the EC directorates with agro statistical information on crop areas and yields for Europe and the major production areas of the world. This information is partly based on remote sensing images and it is used by the EC to adjust its agricultural interventions and food security policies. SPIRITS forms the follow up of another software tool developed by VITO, called GLIMPSE (GLobal IMage Processing SoftwarE), which is a set of command line driven image processing routines, developed since GLIMPSE programs can be easily concatenated by means of scripting languages to establish dedicated processing chains. At VITO s they are systematically used in this way for the automated processing of incoming satellite data. The SPIRITS development started in 2009 and first aimed at setting up a Graphical User Interface (GUI) for more convenient access to the GLIMPSE modules. Gradually, a number of new tools were incorporated in SPIRITS that go well beyond GLIMPSE. An exhaustive list of all SPIRITS functionalities is provided in the online User Manual. Here a short summary is presented in three groups: a. Image processing routines in the following three domains:: Spatial: resampling, thinning, filtering; Thematic: rescaling, band combinations, masking, extraction of biophysical indicators, unsupervised classification; Temporal: profile smoothing, compositing, detection of phenologies and anomalies, similarity analysis. b. Downstream analysis tools: Generation of quicklook maps; Extraction and management of databases with regional unmixed means ; Graphical analysis of the database and creation of charts. The generated quicklook maps and charts can be ingested in agro meteorological bulletins which provide assessments of the current crop conditions and yield forecasts. And, using customizable templates, similar maps and graphs can be automatically created for the full time series (a considerable save in time). c. Other facilities: Import/export of external image formats; Rastering of vector files; Public Page 28 of 37

29 Page : 29 of 37 Option to setup new processing chains via so called user tools ; Project management; Help functionalities & tutorial. The online User Manual describes the image formats, first in general terms, later focussing on the modified ENVI format adopted by GLIMPSE and SPIRITS. Subsequently it covers a full description of all the individual SPIRITS modules. The annexes provide an overview of the software installation, the included open source components and the used acronyms. Related to this User Manual the software also comes with a Tutorial which demonstrates the most important functionalities of SPIRITS by means of some practical exercises and sample data. The example below shows some of the output generated over an area of interest in Sudan, by one of the participants attending the Regional GEONETCast AGRICAB course in Nairobi in The figures below are showing the relationship between satellite derived precipitation estimates and the subsequent NDVI response as well as the crop specific NDVI profiles over various years for a selected irrigated versus a rainfed area. The flexible configuration of user tools allows users to run their own programs on individual products or even the full time series. In fact, these user tools can be used to repeatedly call the ILWIS toolbox batch routines, effectively automating the ILWIS import for a full time series. Figure 16: NDVI response to rainfall for a region in Sudan Public Page 29 of 37

30 Page : 30 of 37 Figure 17: Crop specific NDVI response for irrigated versus a rainfed area Large rainfed Grassland 5.4 INPE s suite of processing tools TerraView is a user friendly geographic data viewer with resources that include database queries and data analysis. TerraView handles vector data (polygons, lines and points) and raster data (grids and images) ( In addition to visualization tools, TerraView provides a set of analysis tools. TerraView creates and handles geographical databases in a Database Management System. The data is stored according to a data model defined in the underlying TerraLib library. A TerraLib/TerraView database can store both vector and raster data. Spatial data analysis methods, like the spatial proximity or neighbourhood matrix, are included. Next to this, various plug ins are available which add to the overall functionality of TerraView, like a plug in for pattern identification and analysis. There is an online tutorial to help the users to learn the basic concepts of TerraView. The Geographical Data Mining Analyst (GeoDMA) runs as an add on to the TerraView software. GeoDMA offers data mining, object oriented classification, a tool that is highly relevant for construction of (high resolution) crop masks. GeoDMA is able to extract several features, from spatial to statistical and spectral attributes, performing the complete data mining process, including attribute(s) selection, training, classification, visualization and validation. By exploring spatial and spectral features together, more accurate classification results can be obtained. For this purpose GeoDMA incorporates features for best attribute set selection, by applying the pattern metrics such as Perimeter, Area, Perimeter Area ratio, Shape index, Fractal dimension index, Related circumscribing circle, Contiguity index and Radius of gyration. Using the TerraView interface, GeoDMA becomes capable of storing every data mining operation in the local database for further access, providing an easier way of working. The system can deal with all types of spatial applications where the input is described by images and regions. Figure 18 shows a MSG image, retrieved using the MSG Data Retriever available from the GEONETCast toolbox and thus transformed from a geostationary projection into Lat Long and then exported to GeoTiff format for use in TerraView. The country boundaries, initially vector data in ILWIS, were transformed into Shape format, using the ILWIS38 vector export option. The MSG raster and boundary vector can then be seamlessly overlaid in TERRAVIEW. Public Page 30 of 37

31 Page : 31 of 37 Figure 18: Compatibility between ILWIS and INPE s suite of tools 5.5 Data exchange utilities Users often spend a lot of time in the preparation of their input data and the data exchanges (e.g. format conversion) between the different software tools used in their analysis. These operations are particularly tedious and time consuming when they need to deal with time series data. As more and more software are added to the mix, in order to make full use of the various capabilities, dedicated utilities become necessary to facilitate the exchanges. One such utility, developed using the GDAL translation library to facilitate the exchange of time series data between, on one hand, the generic ILWIS and, on the other, the time series analsysis software SPIRITS, TIMESAT and HANTS. Figure 19 provides a screenshot of this tool, that facilitates the exchange by renaming of a map list constructed in ILWIS, by adding temporal information to the name of the layers within a time series. After this, the data can be exported from ILWIS to the time series tools, including updated header files (for use in SPIRITS) or text files with references to the newly created output files (for HANTS and TIMESAT). The output result of the analysis in these packages can then be transformed back into an ILWIS format for ingestion into other processing or visualization routines. Figures 20 and 21 show examples of the SPOT VEGETATION NDVI time series from Burkina Faso, used during the second regional AGRICAB GEONETCast course, after export to TIMESAT. After the export to TIMESAT, TIMESAT s filtering procedure is used to remove outliers. The filtered (smoothed) data can then be exported back to ILWIS for subsequent visualization or further processing. Note that in figure 20 the initial time series is from 2010 till October 2012, for the TimeSat processing only 2 years of data was used (2010 and 2011). While not shown here, similar exchanges can also be performed with SPIRITS and HANTS to utilize their time series processing capabilities. Public Page 31 of 37

32 Page : 32 of 37 Figure 19: Time Series Data Conversion tool Figure 20: Generation of file list and Time Series filtering using TimeSat A second utility is VITO s VGTExtract, that facilitates the integration of VEGETATION data into a variety of GIS and Remote Sensing software (raster format export). It includes translation to ILWIS or the modified ENVI format used in SPIRITS, for the processing, analysis and visualization. VGTExtract covers almost all formats of basic and derived VEGETATION data, not only those provided via GEONETCast. At present, it allows only the selection of inputs on folder, product or data layer basis, but will include time series inputs soon. Public Page 32 of 37

33 Page : 33 of 37 Figure 21: Comparison of original and filtered time series data Public Page 33 of 37

34 Page : 34 of 37 Public Page 34 of 37

35 Page : 35 of 37 6 FURTHER DEVELOPMENTS 6.1 Additional and changed satellite images and products The launches of the PROBA V and CBERS 3 satellites, both providing relevant data for AGRICAB, were unfortunately postponed in PROBA V s launch is now scheduled in mid April and CBERS 3 towards end of first or early second semester 2013 (e.g. May or July). This means the first data will only be available, after satellite commissioning, towards late Q3 or Q4 2013, effectively pushing back their routine use to the 2014 season (for the crop applications). Following an earlier demonstration in the regional GEONETCast workshop in Nairobi, AGRICAB will try to get Landsat 8 Data Continuity Mission (DCM) data for the use case study areas. In addition, the startup of the Copernicus (former GMES) Initial Operations, Global Land service in Jan 2013 will lead to a number of changes in the VEGETATION derived product creation and distribution (e.g. new online distribution platform, products derived from PROBA V input data, operational review cycles for product changes). This will furthermore add Soil Water Index (TUWien) products from METOP ASCAT on GEONETCast. These important changes will of course require updates to or the development of additional routines to facilitate the retrieval and use of those new images and products. 6.2 Building synergies with national use case development activities In parallel to the tools development and GEONETCast broadcast efforts in AGRICAB described in this document, which were broad scale and mainly at regional level in Africa, the national level use case applications have been prepared in the focus countries. This included steps such as study area identification, first field surveying to assess the currently used methods and first meetings with national stakeholders to well capture their requirements and needs. To build synergies between both efforts, a comprehensive framework for enhancement of capacities is proposed. This encompasses user communication (e.g. online portals), training modules and software and is structured along the end to end flow of the various applications: from data management, preprocessing to qualitative analysis for monitoring and quantitative analysis & modelling. The next step is thus to fit the wider data access and tools developments into the framework. This may require fine tuning of the software solutions provided in order to further facilitate the application flows (or at least those flows identified as best practice or ideal), fill in gaps. In 2013, specific attention will be given to the Tunesia regional irrigated agriculture use case, as there is already interest in using GEONETCast and thus room to build such a synergie. Within the regional AGRICAB workshop, planned for the 3 rd quarter of 2013, a couple of days will be devoted to the use of data disseminated through GEONETCast for the purpose of irrigation assessment. Public Page 35 of 37

36 Page : 36 of 37 In the training plan, additional training activities on the data management and (pre processing) routines are currently envisaged at regional level, as supporting topics to the main thematic application topics of the upcoming regional workshops. Pursuing other potential synergies or inclusion into the national training efforts will however require more careful preparation. Therefore, the full framework needs to be further clarified first and tried out, e.g. on EU side, prior to being added to the operational activities in the focus countries. Public Page 36 of 37

37 Page : 37 of 37 7 CONCLUSIONS Within this report an overview is given of the suite of tools which can be applied to handle the near real time environmental data delivered to the global user community through GEONETCast and time series data which is available through online data archives. In short, with all tools in place, users are now capable to: efficiently create a structured archive themselves, based on continuous operation of a GEONETCast ground receiving station, complemented with relevant information from all kinds of web archives; use the import routines to ingest the received into a common GIS/RS and exchange it with other software for e.g. time series analysis or object oriented classification; use the functionality and flexibility of these GIS/RS tools for setting up their own processing and analysis. The various elements of the suite of tools have been demonstrated and tested at the regional AGRICAB GEONETCast courses. Continuous development is taking place to ensure that the tools will be able to handle the changes resulting from modifications within the GEONETCast data stream, but also add new functionality to cope with new products which are relevant for the AGRICAB objectives, to ensure that the information is used for better Agriculture and Forestry management. Public Page 37 of 37