NEP-02: Service Oriented Scientific Computing for SAFORAH Non-linear Denoising Parallelization Demo

Transcription

1 NEP-02: Oriented Scientific Computing for SAFORAH Non-linear Denoising Parallelization Demo Ashok Agarwal, Patrick Armstrong, Andre Charbonneau, Hao Chen, Ronald J. Desmarais, Ian Gable, David G. Goodenough, Piper Gordon, Aimin Guan, Roger Impey, Kelsey Lang, Belaid Moa, Susan Perkins, Wayne Podaima, Randall Sobie February 15, 2011 canadien Council of de recherches

2 Road Map Introduction Introduction to non-linear denoising Problem algorithm is expensive Solution parallelize Non-linear denoising application How we parallelize using the grid? Application work done during extension Non-linear Denoising application SAFORAH: CUDOS web interface Future work canadien Council of de recherches

3 Introduction Hyperspectral noise has been assumed to be linear Linear Stochastic Processes were used to model the noise Recently, Han and Goodenough (2009) have proved that this is not the case Non-linear analysis is used to reduce the noise Results show significant signal to noise ratio SNR boost: canadien Council of de recherches

4 Introduction Non-linear Denoising Results: Original Image Denoised Image Difference Image Aviris Image (600x1000x179) RGB: 1503, 750, 645 nm Linear 2% trimming canadien Council of de recherches

5 Introduction Typical hyperspectral image cube 600 samples, 1000 lines, 179 bands Over 200 MB Problem: The algorithm is computationally very expensive. This 200MB image will take approximately four days to process on a single cpu. Solution: Give the algorithm more cpu s! canadien Council of de recherches

6 Non-linear Denoising Solution: Give the algorithm more cpu s: Parallelize using the computational grid. Embarrassingly parallel problem: No spatial data interdependence Data can be divided spatially for processing canadien Council of de recherches

7 Non-linear Denoising The image is split into N pieces using the gdal library: Number of pieces based on image size. For example, this sample image is broken into 250 pieces. canadien Council of de recherches

8 Non-linear Denoising Each piece is sent to the grid with the application source code and a script to compile and run it: Denoise.cpp Makefile Denoise.sh Job # n Computational grid canadien Council of de recherches

9 Non-linear Denoising The pieces are sent to the metascheduler The metascheduler allocates resources for the jobs: Non-linear Denoise Computational grid Job # 2 Job # 3 Job # Job # n Metascheduler Resource 1 Resource 2 Resource m canadien Council of de recherches

10 Non-linear Denoising The nodes copy the completed images back to the local machine The Non-linear Denoising application stitches the pieces together: Non-linear Denoising Computational grid canadien Council of de recherches

11 Advantages: Two orders of magnitude faster! Completed in as little as 1 hour on a grid with 74 nodes, instead of 4 days, or about 100 hours. The time to process dependent on: Non-linear Denoising Size of image file Number of nodes available on the grid How many other jobs are running on the grid canadien Council of de recherches

12 The activities in this quarter dealt with making the parallel Non-linear Denoising (NLD) application available in the CUDOS GUI, and preparing the application and software infrastructure for longevity. Developed the parallel NLD application Improved stability Improved response time Extension Made NLD available from CUDOS GUI Ran end-user testing for NLD The infrastructure activities included: Improvements to source code version management canadien Council of de recherches

13 Extension Changes to the Architecture canadien Council of de recherches 13

14 Extension System Flow Chart Product Request Splitter Scatterer Job Scheduler Pool of Computational Resources Gatherer Merger Gridway Metascheduler Portable Batch System (PBS)( Results canadien Council of de recherches

15 Demo of Automated Parallelization of Denoising SAFORAH CUDOS canadien Council of de recherches

16 Future Work Update the other existing EO applications according to the user requirements on parallelization (Radarsat 2 decomposition, Aboveground carbon mapping) Update Grid Web Integration to use supported APIs Globus 4.0 no longer supported Other possibilities: Condor Globus 5.x Cloud computing Add more nodes to grid to improve performance. Make more applications available on OGC and on CUDOS canadien Council of de recherches

17 Conclusion Accomplished what we set out to do for extension: Parallelized Non-linear Denoising. Using grid-computing for speed and efficiency. Obtained significant performance improvements from parallelization: ~100 x faster!! Made the application available on SAFORAH Improved the system infrastructure Thank you CANARIE! canadien Council of de recherches

18 CUDOS Portal -Create New Product canadien Council of de recherches 18

19 CUDOS Portal -Product Parameters canadien Council of de recherches 19

20 CUDOS Portal -Product Order canadien Council of de recherches 20

21 Results on noise reduction (1) The effectiveness of noise reduction by the LGP was evaluated by comparing the noise intensity and signal-to-noise ratio (SNR) between the original and the de-noised images. A covariance method proposed by Roger and Arnold (1996) was employed here for noise estimation. Without relying on any additional information, this method is capable of producing reasonable noise estimates based on the covariance matrix of the hyperspectral image alone. The average noise intensity of the original AVIRIS image was estimated at 1.2, which varies between 0.3 (minimum) and 11.3 (maximum) across all bands. The LGP removed an average of 34.4% of the noise in the AVIRIS image, which left the noise intensity in the post-denoised image varying between 0.2 and 9.0 with an average of 0.9. canadien Council of de recherches