Automatic 3D Mapping for Infrared Image Analysis

Transcription

1 Automatic 3D Mapping for Infrared Image Analysis i r f m c a d a r a c h e V. Martin, V. Gervaise, V. Moncada, M.H. Aumeunier, M. irdaouss, J.M. Travere (CEA) S. Devaux (IPP), G. Arnoux (CCE) and JET-EDA contributors Workshop on usion V. Martin Data et al. Processing 1 (19) Validation WDPVA, and Analysis, ENEA rascati ENEA rascati, /03/12 March 2012

2 3D IR Scene Calibration JET #81313 KL7 (images in DL) W coated CC Bulk Be Bulk Be Be coated linconel W coated CC Bulk Be Bulk W V. Martin et al. 2 (19) WDPVA, ENEA rascati 28/03/12

3 3D IR Scene Calibration JET #81313 KL7 (images in DL) Issue: a complex thermal scene 1. Wide angle views with high geometrical effects: depth of field and curvature 2. Many metallic materials (Be, W) with different and changing optical (reflectance) and thermal (emissivity) properties W coated CC Bulk Be Bulk Be Be coated linconel W coated CC Bulk Be Bulk W V. Martin et al. 3 (19) WDPVA, ENEA rascati 28/03/12

4 3D IR Scene Calibration JET #81313 KL7 (images in DL) Issue: a complex thermal scene 1. Wide angle views with high geometrical effects: depth of field and curvature 2. Many metallic materials (Be, W) with different and changing optical (reflectance) and thermal (emissivity) properties Bulk Be W coated CC Bulk Be Objective: Match each pixel with the 3D scene model of in-vessel components for: 1. getting the real geometry of the viewed objects 2. reliable linking between viewed objects and their related properties Be coated linconel W coated CC Bulk Be Bulk W V. Martin et al. 4 (19) WDPVA, ENEA rascati 28/03/12

5 3D IR Scene Calibration JET #81313 KL7 (images in DL) Issue: a complex thermal scene 1. Wide angle views with high geometrical effects: depth of field and curvature 2. Many metallic materials (Be, W) with different and changing optical (reflectance) and thermal (emissivity) properties Bulk Be W coated CC Bulk Be Objective: Match each pixel with the 3D scene model of in-vessel components for: 1. getting the real geometry of the viewed objects 2. reliable linking between viewed objects and their related properties Be coated linconel W coated CC Bulk W Bulk Be Applications 1. Image processing (event characterization) 2. IR data calibration: T surf = f(material emissivity) V. Martin et al. 5 (19) WDPVA, ENEA rascati 28/03/12

6 Methodology V. Martin et al. 6 (19) WDPVA, ENEA rascati 28/03/12 Calibration chain NUC Dead pixel Map Reference image 2D/3D scene models Knowledge base of the thermal scene Camera Image Correction Image Stabilization 2D/3D Scene Model Mapping Image Processing Registered & Calibrated Image

7 Illustration of Motion in Images V. Martin et al. 7 (19) WDPVA, ENEA rascati 28/03/12 Camera vibrations lead to misalignments of ROIs (PC RT protection) = false alarms or worth missed alarms Image stabilization is a mandatory step for heat flux deposit analysis based on T surf (t)-t surf (t-1) estimations

8 Important factors for method selection Image Stabilization Deformation type: planar (homothety), non-planar Target application: real-time processing, off-line analysis Data quality and variability: noise level, pixel intensity changes, image entropy Required precision level: pixel, sub-pixel Applications in tokamaks (non-exhaustive list) Motion amplitude Target application Precision required Difficulty JET KL7 wide-angle 5-10 pixels (camera vibrations) Hot spot detection PC protection pixel low image entropy JET KL7 windowed up to 15 pixels (disruptions) Physics analysis (e.g. heat load during disruptions ) pixel pixel intensity changes JET KL9 divertor tiles <1 pixel (sensor affected by magnetic fields) Physics analysis (power deposit influx) sub-pixel low resolution, slow motion, aliasing V. Martin et al. 8 (19) WDPVA, ENEA rascati 28/03/12

9 Image Stabilization V. Martin et al. 9 (19) WDPVA, ENEA rascati 28/03/12 See Zitova s survey, Image and Vision Computing, vol. 21(2003), pp

10 Image Stabilization V. Martin et al. 10 (19) WDPVA, ENEA rascati 28/03/12 Classical Methodology See Zitova s survey, Image and Vision Computing, vol. 21(2003), pp

11 Image Stabilization V. Martin et al. 11 (19) WDPVA, ENEA rascati 28/03/12 Classical Methodology 1. eature Detection Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu talk), SIT, SUR, AST Global descriptors: Tsallis entropy (see Murari talk), edge detectors ourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping See Zitova s survey, Image and Vision Computing, vol. 21(2003), pp

12 Image Stabilization V. Martin et al. 12 (19) WDPVA, ENEA rascati 28/03/12 Classical Methodology 1. eature Detection Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu talk), SIT, SUR, AST Global descriptors: Tsallis entropy (see Murari talk), edge detectors ourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping 2. eature Matching Spatial cross-correlation techniques: normalized cross-correlation, Hausdorff distance ourier domain: normalized cross-spectrum and its extensions See Zitova s survey, Image and Vision Computing, vol. 21(2003), pp

13 Image Stabilization V. Martin et al. 13 (19) WDPVA, ENEA rascati 28/03/12 Classical Methodology 1. eature Detection Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu talk), SIT, SUR, AST Global descriptors: Tsallis entropy (see Murari talk), edge detectors ourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping 2. eature Matching Spatial cross-correlation techniques: normalized cross-correlation, Hausdorff distance ourier domain: normalized cross-spectrum and its extensions 3. Transform Model Estimation Shape preserving mapping (rotation, translation and scaling only) Elastic mapping: warping techniques See Zitova s survey, Image and Vision Computing, vol. 21(2003), pp

14 Image Stabilization V. Martin et al. 14 (19) WDPVA, ENEA rascati 28/03/12 Classical Methodology 1. eature Detection Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu talk), SIT, SUR, AST Global descriptors: Tsallis entropy (see Murari talk), edge detectors ourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping 2. eature Matching Spatial cross-correlation techniques: normalized cross-correlation, Hausdorff distance ourier domain: normalized cross-spectrum and its extensions 3. Transform Model Estimation Shape preserving mapping (rotation, translation and scaling only) Elastic mapping: warping techniques 4. Image transformation 2D Interpolation: nearest neighboor, bilinear, bicubic See Zitova s survey, Image and Vision Computing, vol. 21(2003), pp

15 Proposed Algorithm V. Martin et al. 15 (19) WDPVA, ENEA rascati 28/03/12 1. Masked T-based image registration [1] Deterministic computing time Accelerating hardware compatible algorithm (e.g. T on GPU) real time applications Local analysis with dynamic intensity-based pixel masking (e.g. mask the divertor bright region) 2. with sub-pixel precision [2] Slow drift compensation 3. and dynamic update of the reference image Robust to image intensity changes (context awareness) Evaluation of registration quality over time [1] D. Padfield, IEEE CVPR 10, pp , 2010 [2] M. Guizar-Sicairos et al., Opt. Lett., vol. 33, no. 2, pp , 2008

16 Principle of ourier-based Correlation V. Martin et al. 16 (19) WDPVA, ENEA rascati 28/03/12

17 Principle of ourier-based Correlation V. Martin et al. 17 (19) WDPVA, ENEA rascati 28/03/12 Let I ref a reference image, I t an image at time t and DT the Discrete 2D ourier transform such as I t ( x, y ) = I ref ( x-x 0, y-y 0 ) 1 2 DT ( I DT ( I (.,.) ref DT t ) ) (.,.) 1 (.,.) 1-1 ( ) 2 2 (.,.) (.,.) I ref I t

18 Principle of ourier-based Correlation V. Martin et al. 18 (19) WDPVA, ENEA rascati 28/03/12 Let I ref a reference image, I t an image at time t and DT the Discrete 2D ourier transform such as I t ( x, y ) = I ref ( x-x 0, y-y 0 ) 1 2 DT ( I DT ( I (.,.) ref DT t ) ) (.,.) 1 (.,.) 1-1 ( ) 2 2 (.,.) (.,.) I ref I t is the Normalized Cross Correlation figure (image) and the position of the peak gives the coordinates of the translation ( x 0, y 0 ) x0, y0 arg max x,y (I ref, I t ) max ((I ref, I t ))

19 Sub-pixel Precision V. Martin et al. 19 (19) WDPVA, ENEA rascati 28/03/12 Up-sample k times the DT of (trigonometric interpolation): ( u, v) ( 0 u k, v k ) if. k isan integer otherwise DT ( ) (low frequencies) (high frequencies) 0 (anti - aliasing) DT -1 ( )

20 Sub-pixel Precision V. Martin et al. 20 (19) WDPVA, ENEA rascati 28/03/12 Up-sample k times the DT of (trigonometric interpolation): ( u, v) ( 0 u k, v k ) if. k isan integer otherwise DT ( ) (low frequencies) (high frequencies) 0 (anti - aliasing) DT -1 ( ) The peak coordinates ( x 0, y 0 ) give the translation with 1/k pixel of precision: 1 x0, y0 arg max k x, y

21 Reference Image Updating V. Martin et al. 21 (19) WDPVA, ENEA rascati 28/03/12 Goal: maintaining a good reliability of the motion estimator ( peak value) while image appearance changes during the pulse.

22 Reference Image Updating V. Martin et al. 22 (19) WDPVA, ENEA rascati 28/03/12 Solution: use the peak value to trigger the update of I ref such as: if then Tmin max( ( t)) I ref I t T max update I ref update I ref update I ref update I ref peak too low, no I ref update

23 Results V. Martin et al. 23 (19) WDPVA, ENEA rascati 28/03/12 JET #81313 (MARE, disruption), KL7, 480x512 pixels, 50 Hz, 251 frames k=1/4 pixel

24 Results V. Martin et al. 24 (19) WDPVA, ENEA rascati 28/03/12 JET #80827 (disruption), KL7, 128x256 pixels, 540 Hz, frames k=1/2 pixel

25 96 pixels Results V. Martin et al. 25 (19) WDPVA, ENEA rascati 28/03/12 JET #82278, KL9B (slow drift), 32x96 pixels, 6 khz, 4828 frames T stab T unstab 25, pixels

26 Computational Performance V. Martin et al. 26 (19) WDPVA, ENEA rascati 28/03/12 High frame rate performance using GPU 256x256, k=1/4 700 fps

27 rom 2D to 3D Challenge transform pixel coordinates into machine coordinates: (x, y) (r, θ, φ) Method Ray-tracing method from 3D/simplified CAD files V. Martin et al. 27 (19) WDPVA, ENEA rascati 28/03/12

28 V. Martin et al. 28 (19) WDPVA, ENEA rascati 28/03/12 3D Scene Model for Image Processing S. Palazzo, A. Murari et al., RSI 81, , Z Map (depth) mm 2 Blobs 1 & 2 must not be merged! 2m 1 7m V. Martin et al.

29 Integrated ramework V. Martin et al. 29 (19) WDPVA, ENEA rascati 28/03/12 An integrated software for IR data stabilization & analysis NUC Dead pixel Map Reference image 2D/3D scene models Knowledge base of the thermal scene Camera Image Correction Image Stabilization 2D/3D Scene Model Mapping Image Processing Registered & Calibrated Image

30 Integrated ramework V. Martin et al. 30 (19) WDPVA, ENEA rascati 28/03/12 An integrated software for IR data stabilization & analysis NUC Dead pixel Map Reference image 2D/3D scene models Knowledge base of the thermal scene Camera Image Correction Image Stabilization 2D/3D Scene Model Mapping Image Processing Registered & Calibrated Image Plasma ImagiNg data Understanding Platform (PIN)

31 Integrated ramework V. Martin et al. 31 (19) WDPVA, ENEA rascati 28/03/12 An integrated software for IR data stabilization & analysis Load/save translations Set mask Set sub-pixel precision factor

32 Integrated ramework V. Martin et al. 32 (19) WDPVA, ENEA rascati 28/03/12 An integrated software for IR data stabilization & analysis Used for event triggering Used for temperature evaluation Used for PC protection

33 Conclusion Summary Complex IR scenes require a new approach for reliable data analysis including image stabilization and 3D mapping. A robust and fast image stabilization algorithm with sub-pixel precision has been proposed. A first demonstration of 3D model for IR data analysis has been successfully carried out at JET on the wide-angle ITER-like viewing system (KL7). An integrated software (PIN) implementing these features is available for users upon request. Outlook Test of the stabilization algorithm on visible imaging data (JET KL8) with rotation compensation ull integration of 3D scene models into PIN Improvement of image processing algorithms (e.g. hot spot detection) with 3D information V. Martin et al. 33 (19) WDPVA, ENEA rascati 28/03/12