Image analysis for photogrammetric data capture Christian Heipke IPI, Leibniz Universität Hannover

Transcription

1 Image analysis for photogrammetric data capture Christian Heipke IPI, Leibniz Universität Hannover

2 Contents Introduction A flavour of the involved algorithms snakes: parametric active contour models normalised cuts segmentation MRF and CRF SVM classification Examples Conclusions and outlook

3 Some trends in photogrammetric data acquisition merging of airborne and terrestrial data acquisition unmanned aerial vehicles (UAV), oblique imaging sensor fusion: images, point clouds etc. data acq. in geosensor networks, linked to the web increasingly, update more important than new acq. image sequences rather than single images real-time monitoring (roads, train tracks, traffic, safety and security applications, ) terrabytes (and soon petabaytes) of data per day dump data (pixels)

4 OGC: We desire the ability to discover and integrate observations from any sensor that meets our needs adapted from: Botts, 2004 OGC Helping the world to communicate geographically

5 Glacier movement Landsat scenes of Jakobshavn Isbrae, Greenland Maas et al., PFG 2006

6 Automation only way to deal with the huge amounts of incoming data not necessarily full automation: operator can supervise process, check and (if necessary) edit automatically obtained results need for automation goes far beyond photogrammetry & remote sensing, especially in close range prerequisite for real-time applications surprisingly difficult!!

7 Typical questions Were are the roads? Which building is this? How many people live here? Which type of animal is this? Image analysis I would like to have an image concerning this topic Photogrammetry I need more images concerning this building Identification, localisation, typification, recognition of detail, search (e.g. on the web)

8 Challenges: representation

9 Challenges: context

10 Image analysis - definition automatic generation of an explicit meaningful description of physical objects in the real world from images (Rosenfeld, 1982) model of the real world, projection model real world images description

11 A flavour of the involved algorithms

12 Active contour models Kass, M., Witkin, A. and Terzopoulus, D., Snakes: Active Contour Models. International Journal of Computer Vision, Vol. 1, pp

13 Active contour models subjective contour

14 Parametric active contours Contours are represented as a parametric curve ( s) ( x( s),y( s) ) v = where s is the arc length and x and y are the image coordinates The contour is evaluated with an energy functional, which has to be minimized: E * snake Min E 1 1 * snake = 0 0 E ( ()) [ snake v s ds = Eimg ( v( s) ) + Eint ( v( s) ) + Econ( v( s) )]ds

15 Image energy Image energy E img describes the features of an image obtaining an optimal delineation of the object boundaries For example image intensities ( lines): ( v( s) ) I( v( s) ) E img = negative square of the magnitude of the gradient image ( edges): E img ( v() s ) = I ( v( s) ) 2

16 Internal energy Internal Energy E int describes the ideal shape of the object boundaries (model): E int ( ()) 2 2 v s = 1 α() s v () s + β () s v () s 2 s ss First term α(s) controls the elasticity: large values let the contour become very straight Second term β(s) controls the rigidity: large values let the contour become smooth, small values allow the generation of corners

17 Minimization of the total energy E * snake 1 = 0 ( v() s ) + E ( v( s) ) [ E ] ds Min img int Calculus of variations: minimization of a functional 1 0 E ( s, v, v, v ) ds Min s ss Minimum of the total energy can be derived by solving the Euler equation: E img v ( v( s) ) () s α v ss () s + β v () s = 0 ssss

18 Approximation of the derivatives Converted to vector notation with v i =(x i, y i ) and with E img (v(s))/ v(s)=f v (v) the Euler equations read: α i ( vi vi 1 ) α i+ 1( vi+ 1 vi ) + βi 1( vi 2 2 vi 1 + vi ) 2βi ( vi 1 2vi + vi+ 1 ) + βi + 1( vi 2 vi vi + 2 ) + f v ( v) = 0 Rewritten in matrix form: Iterative solution Av + f v v ( ) = 0

19 Extraction of field boundaries

20 Extraction of field boundaries

21 Normalized cuts segmentation segmentation: grouping of pixels into segments based on some homogeneity criterion prerequisite for object extraction Region growing complex scenes: simple segmentation schemes are not applicable

22 Normalized cuts segmentation graph based method: pixels as nodes edge weights = pixel similarities cut into segments with Normalized Cut criterion: link( A, B) link( A, B) Ncut( A, B) = + = link( A, V ) link( B, V ) min link( A, B) = w pq advantages: w pq p A, q B : weight between pixels p and q combination of features for similarity measure incorporation of local and global characteristics

23 Normalized cuts segmentation graph based method: pixels as nodes edge weights = pixel similarities cut into segments with Normalized Cut criterion: link( A, B) link( A, B) Ncut( A, B) = + = link( A, V ) link( B, V ) min link( A, B) = w pq advantages: w pq p A, q B : weight between pixels p and q combination of features for similarity measure incorporation of local and global characteristics

24 Normalized cuts segmentation graph based method: pixels as nodes edge weights = pixel similarities cut into segments with Normalized Cut criterion: link( A, B) link( A, B) Ncut( A, B) = + = link( A, V ) link( B, V ) min link( A, B) = w pq advantages: w pq p A, q B : weight between pixels p and q combination of features for similarity measure incorporation of local and global characteristics for actual minimisation see Shi, Malik, T-PAMI (2000)

25 NC segmentation for aerial images Edge intensity between two pixels Colour space distance of two pixels Hue difference between two pixels NDVI region difference between two pixels Oversegmented result (as desired) Grouping and network generation required as further steps (not discussed here)

26 MRF Markov Random Fields y i y i Graphical model node: site (pixel, segment) edge: spatial dependencies y i x i y i x i Markov Random Fields consideration of context knowledge Bishop, 2006 a label is assigned to each site based on its features labels interact bi-directional estimation of the most problable configuration of all labels x i x i

27 CRF - Conditional Random Fields 1 P exp A x, I x,x, Z i S i S j Ni ( x y ) = i( i y ) + ij( i j y ) x i : label ; y: data A i : association potential I ij : interaction potential N i : neighbourhood of i Z: partition function association potential is a function of all data, not only of those of the site interaction potential is not only a function of labels but also of the data

28 Inference Loopy Belief Propagation For graphs with cycles No guarantee for ideal result Iteration steps: 1. Update of edge information 2. Update of node probabilities 3. Convergence test

29 Classification by Support Vector Machine non-parametric classification (no need to assume normal, or in fact any, distribution) classification into 2 classes (can be extended to many)

30 SVM: a two-class problem given: training data set: ( x y ), K,( x, y ) x 1, 1 m m mit n i R und yi + { 1, 1} := +1 := -1

31 SVM: a two-class problem w x + b>0 w w x + b=0 given: training data set: ( x y ), K,( x, y ) x 1, 1 m m mit n i R und yi + needed: { 1, 1} w R n, b R w x + b<0 := +1 := -1

32 Different hyperplanes... an infinite number of them

33 Which hyperplane to choose?... select the one with the largest margin

34 Maximise the margin M Goal 1: Classify all training vectors correctly: w w x i x i + + b b 1 1 for y i = +1 for y i = -1 y( w x + b) 1 i i for all i m Goal 2: simultaneously maximise margin M: M = 2 w equivalent to: minimise 1 w w = w 2 Result: quadratic optimisation problem with constraint (to be solved using Lagrange multipliers): w min y( w x + b) 1 i i i

35 SVM in higher dimensions Often classes can be separated better in higher dimensions: for details see literature

36 SVM in higher dimensions feature 2 SVM-hyperplane urban background road types feature 1

37 A flavour of the algorithms - summary Local and global (snakes, NC cuts) Discrete rather than continuous mathematics (NC cuts) Graphical models and probability (MRF, CRF) Optimisation in many different forms (SVM, all) Semantics is and remains important adequate description of the model

38 Examples...

39 Road extraction in rural areas Mayer 1998

40 Result Rural areas: completeness and correctness up to 90 % Baumgartner et al.1999

41

42

43 Road extraction based on NC segmentation

44 Building and tree extraction from image and lidar data Grangemouth Size of the testsite 60,000 m² No of Trees 235 Completeness 95% Correctness 89% Error in Position Error in Height Error in Radius 0.9 m 0.2 m 0.7 m

45 Building and tree extraction from image and lidar data

46 CRF classification RapidEye 2009 reference ML (64,2%) CRF mono (74,5%) CRF multi (79,1%)

47 Automatic quality control of GIS data geospatial data reality (orthophoto) automated comparison

48 Example: orthophoto

49 Example (ctd.): orthophoto and ATKIS

50 Incorrect labels (based on classification)

51 Quantitative results Different test areas in Germany black/white and colour images a few 100 km 2 System Human operator green red more than ATKIS objects percentages refer to no. of objects green 69 % efficiency 22 % interactive check 5 % 4 % red undetected errors interactive check

52 Results... operator time per orthophoto, 2km 2km time for completely manual processing 4 h time for semi-automatic approach 1 h 20 min Productivity increase by factor of 3 verified for many satellite images and other object classes system in use at BKG since a number of years

53 Conclusions and outlook degree of automation in image analysis increases (slowly) successful for schematic tasks bottleneck: self evaluation semi-automatic approaches are starting to be used in practical applications proper integration of human and machine (work flow, GUI,...) some products available: (Inpho InJect, Cyber City CC Modeler) Image Easytrace research will continue to tackle boundaries of automation integration of 3D (e.g. lidar) machine learning for building models

54 Some observations a visual world 3D models are in Google Earth, Microsoft Bing, increasing amounts of images and sensors 24/7, I want it NOW the WWW is more and more becoming the dominating driver automation is key to process vast amounts of image data quickly, for geo-referencing and exploitation sensor fusion, sensor networks, collaboration many useful algorithms in neighbouring disciplines (CV, PR, ) semi-automatic approaches are feasible in practice change detection, monitoring and process modelling, real-time

55 The future? Thank you for your attention!

56 ISPRS Hannover Workshop 2011 High-Resolution Earth Imaging for Geospatial Information Hannover, June 14-17, 2011 hosted by sponsored by