RIPL. An Image Processing DSL. Rob Stewart & Deepayan Bhowmik. 1st May, Heriot Watt University

Transcription

1 RIPL An Image Processing DSL Rob Stewart & Deepayan Bhowmik Heriot Watt University 1st May, 2014

2 Image processing language Rathlin = + FPGA

3 Motivation

4 Application scenario

5 FPGAs good fit for remote image processing reconfigurable energy efficient FPGA constraints Memory Task scheduling Language design tradeoffs Solution: Small DSL closely coupled to FPGA instruction set

6 Requirements

7 Design

8 High level imperative language Language choices image algebra Existing languages/libraries FPGA instruction set abstraction Platform independent reference interpreter GPUs & CPUs

9 RIPL Language Features Functions and procedues Assignment let rgb img a = foo(..) {.. } ; action(..) {.. } ; let rgb image a =.. ; var grey image b; b :=.. ; Iteration & conditional branching for i in 0.. n {.. } ; if (.. ) {.. } else {.. } ; Image algebra implementation Overloading b := (a (+) s)ˆ2 ; c := max( sum(b), d) ; let rgb image a =.. ; let rgb img b =.. ; let rgb image c = a - b ; let int x = 3 ; let int y = 4 ; let int z = x - y ;

10

11 Constructing Images in RIPL image : pointset valueset /* RGB img */ let rgb image a = [1:3,1:2] {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)};

12 Constructing Images in RIPL image : pointset valueset /* RGB img */ let rgb image a = [1:3,1:2] {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)}; /* Same RGB img to image algebra notation b F Y */ let ptset Y = [1:3,1:2] ; let valset F = {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)}; let rgb image b = FˆY ;

13 Constructing Images in RIPL image : pointset valueset /* RGB img */ let rgb image a = [1:3,1:2] {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)}; /* Same RGB img to image algebra notation b F Y */ let ptset Y = [1:3,1:2] ; let valset F = {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)}; let rgb image b = FˆY ; /* mutable variables */ var grey image c; c := [1:2,1:3] {221,244,230,165,102,124};

14 Overloaded Operations /* add two integers */ let int i = 3; let int j = 4; print(i+j); /* add two value sets */ let valset v1 = {3,4,5}; let valset v2 = {1,2,6}; print(v1+v2); /* add two point sets */ let ptset pt1 = {(1,2),(4,3)}; let ptset pt2 = {(3,1),(5,2)}; print(pt1+pt2); /* add a dog and a cat */ var rgb image dog, cat, friends; cat = readfile("cat.bmp"); dog = readfile("dog.bmp"); friends = cat + dog; writefile(friends,"out.bmp");

15

16 Thresholding I I Segment into regions of interest Semi thresholding: pixels within threshold are retained var grey image a; var grey image b; a := readfile("pumpkin.jpg") ; b := X[ ](a) ; writefile(b,"segmented.jpg") ;

17 Edge detection Contour with abrupt brightness change Important for segmentation & scene analysis Convolve two kernels over original image to calculate approximations of the X & Y derivatives G = s 2 + t 2

18 let grey image a = readfile( pumpkin.bmp ) ; let int template s = [3,3] {-1, 0, 1, -2, 0, 2, -1, 0, 1 } ; let int template t = [3,3] {-1,-2,-1, 0, 0, 0, 1, 2, 1 } ; let grey image newimg = (((a (+) s)ˆ2) + (((a (+) t)ˆ2)))ˆ(1/2) ; writefile(b,"out.bmp");

19 RIPL let rgb image a = readfile("images/bike.bmp"); /* Sobel template definitions */ let grey image newimg = (((a(+)s)ˆ2) + (((a(+)t)ˆ2)))ˆ(1/2); writefile(newimg,"pumpkin-edges.bmp"); OpenCV Mat src, dst, grad_x, grad_y, abs_grad_x, abs_grad_y;; src = imread( pumpkin.bmp ); Sobel(src, grad_x, ddepth, 1, 0, 3, 1, 0, BORDER_DEFAULT); convertscaleabs( grad_x, abs_grad_x ); Sobel(src, grad_y, ddepth, 0, 1, 3, 1, 0, BORDER_DEFAULT); addweighted( grad_x, 0.5, grad_y, 0.5, 0, dst ); convertscaleabs( grad_y, abs_grad_y ); imwrite( pumpkin-edges.bmp,dst);

20 Tackling the pyramid with image algebra Image enhancement Edge detection Thresholding Connected components Morphological transformations Shape detection Image features

21 Image Algebra Operations to express all image-to-image transformations Small number of concise & simple operations 25 point operations 15 point set operations 9 value set operations 30 image operations 37 template operations 4 neighbourhood operations Amenable optimisation techniques that are machine: independent formal mathematical systems dependent FPGAs, CPUs & GPUs

22 Image Algebra Value sets Numeric data for points of types Z, R, or Z 2 k Point sets Spatial relationship between points Image pixels Tuple of point & value function (x, a(x)) Image Function from points to values F -valued image on X is a : X F, or a F X. Rectangular point set X = Z + m Z + n where Z + m Z+ n = {(x 1, x 2 ) Z 2 : 1 x 1 m, 1 x 2 n}

23 Thresholding For source image a R X and threshold range [h, k], semithreshold image b R X is given by: { a(x) if h a(x) k b(x) = 0 otherwise Semithresholded image b R X over [100, 255] is b := a χ [100,255] (a) var grey image a; var grey image b; a := readfile("pumpkin.jpg") ; b := X[ ](a) ; writefile(b,"segmented.jpg") ;

24 Edge detection ( [ Edge enhanced image b R Y is b := (a s) 2 + (a t) 2] ) 1/2 2 x = (i 1, j) 1 x = (i 1, j 1), (i 1, j + 1) s (i,j) (x) = 1 x = (i + 1, j 1), (i + 1, j + 1) 2 x = (i + 1, j) 0 otherwise 2 x = (i, j + 1) 1 x = (i 1, j + 1), (i + 1, j + 1) t (i,j) (x) = 1 x = (i 1, j 1), (i + 1, j 1) 2 x = (i, j 1) 0 otherwise

25 let grey image a = readfile( pumpkin.bmp ) ; let int template s = [3,3] {-1, 0, 1, -2, 0, 2, -1, 0, 1 } ; let int template t = [3,3] {-1,-2,-1, 0, 0, 0, 1, 2, 1 } ; let grey image newimg = (((a (+) s)ˆ2) + (((a (+) t)ˆ2)))ˆ1/2 ( [ (a s) 2 + (a t) 2] 1/2 ) writefile(b,"pumpkin-edges.bmp");

26 Tool Support

27 Syntax highlighting & code completion

28 Rendering RIPL programs as image algebra Video demonstration

29 Implementation

30 RIPL syntax described in labelled BNF notation Prog. Program ::= [Decl] Body ; CmdIf. Command ::= SelectionStm ; EENorm. Exp ::= "[[" Exp "]]2" ; BNot. Exp ::= " " Exp ; ESumIA. Exp ::= "\\sum" Exp ;... Compiled to lexer, parser & AST RIPL Interpreter traverses user program using AST

31 BNF Converter ELisp backend

32 Symbolic RIPL Markup Operation RIPL RIPL-IA IA symbol Negation -x -x x Ceiling ceil(x) \ceil*{x} x Floor floor(x) \floor*{x} x Rounding [x ] [x ] [x] Projection p(i,x) p i(x) p i (x) Sum sum(x) \sum x x Product product(x) \Pi x Π x Maximum max(x) \vee x x Minimum min(x) \wedge x x Euclidean norm [[x ]]2 x 2 x 2 Characteristic X x (z) \chi x(z) χ X (z)

33 Symbolic RIPL Markup

34 RIPL Interpreter

35 Evaluation

36 Parallel image processing Current status: Profiling RIPL on Heriot-Watt Beowulf cluster CPU 2Ghz Intel Xeon, 12Gb memory GPU 1.6Ghz GeForce GT 610, 1Gb memory Feeding repa & accelerate benchmarks to community Goal: to match OpenCV performance (Optimising data parallel code is hard)

37 Future Work Full coverage of image algebra operations Implement algorithm libraries in RIPL RIPL dataflow compiler Hardware support for low-level image algebra operations

38 References Handbook of Computer Vision Algorithms in Image Algebra 2nd Ed., G. Ritter & J Wilson, Supporting image algebra in the Matlab programming language for compression research, M. Schmalz et al. SPIE, Efficient parallel stencil convolution in Haskell, B Lippmeier & G Keller, Haskell, Accelerating Haskell array codes with multicore GPUs, M Chakravarty et al. DAMP, 2011.

39 Thanks