27/01/2014. Workflows for Remote Sensing Data Processing: Software Development Strategy & Designing Operational Scenarios

Transcription

1 27/01/2014 Workflows for Remote Sensing Data Processing: Software Development Strategy & Designing Operational Scenarios

2 SW Development strategy Life cycle model Project-driven evolutionary life-cycle model, i.e. build a larger prototype/operational system by pooling single projects to serve a common objective» Objective 1 (Internal): Continuously upgrading and consolidating (= operational code) the department-internal technical and scientific capacity of operational photogrammetric & hyperspectral image/data processing (terrestrial, airborne, satellite).» Objective 2 (External): Project dependent 2

3 SW Development strategy Life cycle model CVB: Processing Power + Archive (raw data + metadata & data products & database) Customer1 Operational platform V1 Customer2 Operational platform V1 V2 SLA (Service Level Agreement) 1 SLA Valorization cycles Research, assembly and integration test platform Innovation by means of classical project work (FP7, BELSPO, IWT, ): AGIV, VITO, universities, companies, Innovation cycles Operational platform According clear and measurable Service Level Agreements (SLA) From Level1 onwards, the Customer1 and Customer2 workflows are 100% the same 3

4 Development strategy Life cycle model verification ESA software engineering standards: life cycle verification and validation approach. BSCC (ESA Board for Software Standardisation and Control) ECSS-E-40 is the ECSS standard for software engineering and ECSS-Q-80 is the ECSS standard for software Product Assurance. These standards cover a wide range of applications, some of them may not be applicable for small, low-cost projects. The selection process of the appropriate requirements for a particular project is called a tailoring. To be accurate, a tailoring must follow a certain number of rules and generally has to be done with the help of a standards expert. Unfortunately, the "Guide to applying the ESA standards to small software projects, BSSC(96)2 Issue 1, May 1996" has no equivalence in ECSS, as the ECSS standards are intended to be tailored to the need of each project, be it "small". However, for small projects, since (a) the BSSC(96)2 provides direct hands-on guidelines which can be easily adopted by developers lacking the ECSS standardization knowledge and (b) the project resources are way to limited to include a standardization expert, it is advisable to follow the guidelines as defined in BSSC(96)2. 4

5 Development strategy Life cycle model verification 1. In ESA terminology, unit testing refers to the process of testing modules against the detailed design. The inputs to unit testing are the successfully compiled modules from the coding process. These are assembled during unit testing to make the largest units, i.e. the components of architectural design. The successfully tested architectural design components are the outputs of unit testing. At VITO a Functional Test Framework (FTF) is continuously maintained (grouping of data, auxiliary data, configuration files, executables, logging and results) to test (a) module behavior according design and (b) to validate the ATBD. An up to date FTF is essential in the Rapid Application Development (RAD) process structure. 2. This FTF is also being used for integration testing. For example, atmospheric correction involves a number of subsequent modules (creating MODTRAN lookup tables, image-based AOD retrieval, image-based water-vapor estimation, BRDF correction, land/water/cloud identification, the actual atmospheric correction). Once each module is unit tested, the series of modules can be integrated in a processing sequence that can be implemented in the FTF by means of a simple batch file. As such, the FTF can be used to validate the behavior and results of this module sequence, which thus functions as an integration test. 3. After passing the unit testing and integration testing and once integrated in the workflow software system, system tests validate the resulting system against the SRD Bèta Hardware System 4. At VITO, it is preferred that the final acceptance tests are executed by the project partner which represent the user segment. And this is formalized by a project acceptance review meeting. When an external partner is not available due to the given project consortium layout, an internal acceptance review meeting is organized. 5

6 Development strategy Functional Test Framework A Functional Test Framework can also be used as your desktop Processing & Archiving Facility. 6

7 Functional Test Framework and Operational Scenarios A Functional Test Framework can be used to design your hardware needs. High Performance Computing Cluster for Research, questions to solve: System dimensioning via testing: 1. CPU needs 2. Storage needs 3. Reliability needs 4. Availability needs 5. Scalability needs 6. Connectivity needs 7

8 Bèta (Development) Hardware System 8

9 Software System Quality Assurance & Control (Operators, SW/HW Developers, Scientists, Support Software) Coordinated Data Acquisition: airborne multispectral, airborne hyperspectral Level 0 Raw image data, raw metadata. SW system components: 1. Middleware Workflow system Database Workflow monitoring Hardware system monitoring Network interfaces 2. Algoritmes VITO C++, Java, Python code Bestaande open source libraries All based on proven records (scientific papers) Product workflows (fast temporary processing storage + processing cluster) Archiving Workflow L0 L1 Level 1 Geometric, radiometric and spectral calibrated image data. Workflow L1 L2 Level 2 Atmospheric corrected orthophoto products. GBG/GBA Workflow L2/3 L4 Level 3 Mosaics covering a region of interest. Workflow L2 L3 Archive (storage + database + life cycle management) Level 4 Change detection products. Network services: WWW interface archive & processing cluster, FTP, external hard-disks, OGC Web Services Research & Development: innovation (FP7, BELSPO, IWT, projects) User Community: Internal / external operators, external users 9

10 Software System Functional Flow Subdivided in 3 workflows grouped in one single WWW user interface: 1. Level1 (raw) to Level2 (geometric and atmospheric corrected block of images) 2. Level2 to Level3 (mosaic of Level2) 3. Level2/3 to Level4 (e.g. change detection products, soft classifications, ) This allows for: 1. Different processing entry points. 2. The option to forward Level2, 3 or 4 products in the archive system 10

11 Software System: Middleware Middleware is computer software that connects software components or applications. The middleware software consists of a set of enabling services that allow multiple processes running on one or more machines to interact across a network. Airborne missions generate thousands of images need for distributed computing need to chose patterns for parallelism: Master/Worker: Master application constructs a job-list and maintains the jobdependency. Worker applications ask the master for a job and execute this job. Task/Data Decomposition: algorithmic module is executed on smaller subsets of data. Master applications implements the task and data decomposition. Task decomposition = functional decomposition (orthorectification, atmospheric correction, building MODTRAN4 lookup tables, ) Parallelism is implemented in the middleware, NOT in the algorithmic applications (this keeps the C++/C/Fortran/Java/IDL/Python code of the applications as simple as possible) 11

12 Software System Processing Cluster: Middleware Middleware for cluster/distributed computing is developed by VITO: Message passing (over reliable TCP/IP sockets) and job-pulling Master-Worker pattern developed in Java (VITO, Dept. Remote Sensing) Multiple Masters can run next to each other. Multiple masters can run on one machine. Masters can be configured to take only processing jobs from specific registered users/operators. One worker per machine. Workers keep multiple threads alive which invoke the running of applications. The number of threads can be altered on-the-fly. Workflow Monitoring Software: Java GUI application for monitoring and configuring the processing cluster 12

13 Software System: Middleware vs Hardware Scalability» Multiple masters allow for flexibility to deploy multiple workflows in parallel next to each other» Dynamically adding and removing workers allows for adding extra horsepower as the need arises Image/Data Archive: Enterprise-level storage. Short Term File Servers (STFS) are optimized for I/O speed.» Short Term Storage servers are written to in a round robin fashion to load-balance I/O output The master decides to which STFS a worker job will write output to.» Further details see presentation hardware. 13

14 Software System Processing Cluster: other middleware MySQL 5.X or PostgreSQL: product/user/order database CACTI ( monitoring of memory, disk space, system load, network traffic NAGIOS ( service monitoring 14

15 Example Level1 Level2 workflow (orthorectification and atmospheric correction) APEX flight calibration data: 23/06/2009 level1 APEX level 2 data: Water leaving reflectance 15

16 Example Level2 Level3 workflow (seamless mosaic generation) Automatic seam-line via: 1. cost grid based on combination of gradient smoothness (similarity between neighboring pixels) and similarity between the overlapping pixels. 2. Iterative cost grid masking. 3. Ford-Fulkerson Graph-Cut on masked cost grid pixels. 4. and optional multi-resolution spline blending. Iteratively masked cost grid: searching for the best possible solution (just before connectivity fails). 16

17 Example Level2/3 Level4 workflow (automated change detection in large scale national/regional vector databases of civil infrastructure) Example: GRB mutation and anomaly detection. GRB (Grootschalig Referentiebestand) change detection: blue polygons are already mapped buildings, green polygons are the result of an automatic building detection process (combined K-means, Quadratic Discriminant Analysis with Mahalanobis distance metrics and post-classification logic) 17