RIPL An Image Processing DSL Rob Stewart & Deepayan Bhowmik Heriot Watt University 1st May, 2014
Image processing language Rathlin = + FPGA
Motivation
Application scenario
FPGAs good fit for remote image processing reconfigurable energy efficient FPGA constraints Memory Task scheduling Language design tradeoffs Solution: Small DSL closely coupled to FPGA instruction set
Requirements
Design
High level imperative language Language choices image algebra Existing languages/libraries FPGA instruction set abstraction Platform independent reference interpreter GPUs & CPUs
RIPL Language Features Functions and procedues Assignment let rgb img a = foo(..) {.. } ; action(..) {.. } ; let rgb image a =.. ; var grey image b; b :=.. ; Iteration & conditional branching for i in 0.. n {.. } ; if (.. ) {.. } else {.. } ; Image algebra implementation Overloading b := (a (+) s)ˆ2 ; c := max( sum(b), d) ; let rgb image a =.. ; let rgb img b =.. ; let rgb image c = a - b ; let int x = 3 ; let int y = 4 ; let int z = x - y ;
Constructing Images in RIPL image : pointset valueset /* RGB img */ let rgb image a = [1:3,1:2] {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)};
Constructing Images in RIPL image : pointset valueset /* RGB img */ let rgb image a = [1:3,1:2] {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)}; /* Same RGB img to image algebra notation b F Y */ let ptset Y = [1:3,1:2] ; let valset F = {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)}; let rgb image b = FˆY ;
Constructing Images in RIPL image : pointset valueset /* RGB img */ let rgb image a = [1:3,1:2] {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)}; /* Same RGB img to image algebra notation b F Y */ let ptset Y = [1:3,1:2] ; let valset F = {(1,56,35),(94,22,42),(155,134,99), (56,7,21),(245,1,32),(42,211,111)}; let rgb image b = FˆY ; /* mutable variables */ var grey image c; c := [1:2,1:3] {221,244,230,165,102,124};
Overloaded Operations /* add two integers */ let int i = 3; let int j = 4; print(i+j); /* add two value sets */ let valset v1 = {3,4,5}; let valset v2 = {1,2,6}; print(v1+v2); /* add two point sets */ let ptset pt1 = {(1,2),(4,3)}; let ptset pt2 = {(3,1),(5,2)}; print(pt1+pt2); /* add a dog and a cat */ var rgb image dog, cat, friends; cat = readfile("cat.bmp"); dog = readfile("dog.bmp"); friends = cat + dog; writefile(friends,"out.bmp");
Thresholding I I Segment into regions of interest Semi thresholding: pixels within threshold are retained var grey image a; var grey image b; a := readfile("pumpkin.jpg") ; b := X[100..255](a) ; writefile(b,"segmented.jpg") ;
Edge detection Contour with abrupt brightness change Important for segmentation & scene analysis Convolve two kernels over original image to calculate approximations of the X & Y derivatives G = s 2 + t 2
let grey image a = readfile( pumpkin.bmp ) ; let int template s = [3,3] {-1, 0, 1, -2, 0, 2, -1, 0, 1 } ; let int template t = [3,3] {-1,-2,-1, 0, 0, 0, 1, 2, 1 } ; let grey image newimg = (((a (+) s)ˆ2) + (((a (+) t)ˆ2)))ˆ(1/2) ; writefile(b,"out.bmp");
RIPL let rgb image a = readfile("images/bike.bmp"); /* Sobel template definitions */ let grey image newimg = (((a(+)s)ˆ2) + (((a(+)t)ˆ2)))ˆ(1/2); writefile(newimg,"pumpkin-edges.bmp"); OpenCV Mat src, dst, grad_x, grad_y, abs_grad_x, abs_grad_y;; src = imread( pumpkin.bmp ); Sobel(src, grad_x, ddepth, 1, 0, 3, 1, 0, BORDER_DEFAULT); convertscaleabs( grad_x, abs_grad_x ); Sobel(src, grad_y, ddepth, 0, 1, 3, 1, 0, BORDER_DEFAULT); addweighted( grad_x, 0.5, grad_y, 0.5, 0, dst ); convertscaleabs( grad_y, abs_grad_y ); imwrite( pumpkin-edges.bmp,dst);
Tackling the pyramid with image algebra Image enhancement Edge detection Thresholding Connected components Morphological transformations Shape detection Image features
Image Algebra Operations to express all image-to-image transformations Small number of concise & simple operations 25 point operations 15 point set operations 9 value set operations 30 image operations 37 template operations 4 neighbourhood operations Amenable optimisation techniques that are machine: independent formal mathematical systems dependent FPGAs, CPUs & GPUs
Image Algebra Value sets Numeric data for points of types Z, R, or Z 2 k Point sets Spatial relationship between points Image pixels Tuple of point & value function (x, a(x)) Image Function from points to values F -valued image on X is a : X F, or a F X. Rectangular point set X = Z + m Z + n where Z + m Z+ n = {(x 1, x 2 ) Z 2 : 1 x 1 m, 1 x 2 n}
Thresholding For source image a R X and threshold range [h, k], semithreshold image b R X is given by: { a(x) if h a(x) k b(x) = 0 otherwise Semithresholded image b R X over [100, 255] is b := a χ [100,255] (a) var grey image a; var grey image b; a := readfile("pumpkin.jpg") ; b := X[100..255](a) ; writefile(b,"segmented.jpg") ;
Edge detection ( [ Edge enhanced image b R Y is b := (a s) 2 + (a t) 2] ) 1/2 2 x = (i 1, j) 1 x = (i 1, j 1), (i 1, j + 1) s (i,j) (x) = 1 x = (i + 1, j 1), (i + 1, j + 1) 2 x = (i + 1, j) 0 otherwise 2 x = (i, j + 1) 1 x = (i 1, j + 1), (i + 1, j + 1) t (i,j) (x) = 1 x = (i 1, j 1), (i + 1, j 1) 2 x = (i, j 1) 0 otherwise
let grey image a = readfile( pumpkin.bmp ) ; let int template s = [3,3] {-1, 0, 1, -2, 0, 2, -1, 0, 1 } ; let int template t = [3,3] {-1,-2,-1, 0, 0, 0, 1, 2, 1 } ; let grey image newimg = (((a (+) s)ˆ2) + (((a (+) t)ˆ2)))ˆ1/2 ( [ (a s) 2 + (a t) 2] 1/2 ) writefile(b,"pumpkin-edges.bmp");
Tool Support
Syntax highlighting & code completion
Rendering RIPL programs as image algebra Video demonstration
Implementation
RIPL syntax described in labelled BNF notation Prog. Program ::= [Decl] Body ; CmdIf. Command ::= SelectionStm ; EENorm. Exp ::= "[[" Exp "]]2" ; BNot. Exp ::= " " Exp ; ESumIA. Exp ::= "\\sum" Exp ;... Compiled to lexer, parser & AST RIPL Interpreter traverses user program using AST
BNF Converter ELisp backend https://github.com/robstewart57/bnfc/tree/elisp-backend
Symbolic RIPL Markup Operation RIPL RIPL-IA IA symbol Negation -x -x x Ceiling ceil(x) \ceil*{x} x Floor floor(x) \floor*{x} x Rounding [x ] [x ] [x] Projection p(i,x) p i(x) p i (x) Sum sum(x) \sum x x Product product(x) \Pi x Π x Maximum max(x) \vee x x Minimum min(x) \wedge x x Euclidean norm [[x ]]2 x 2 x 2 Characteristic X x (z) \chi x(z) χ X (z)
Symbolic RIPL Markup
RIPL Interpreter
Evaluation
Parallel image processing Current status: Profiling RIPL on Heriot-Watt Beowulf cluster CPU 2Ghz Intel Xeon, 12Gb memory GPU 1.6Ghz GeForce GT 610, 1Gb memory Feeding repa & accelerate benchmarks to community Goal: to match OpenCV performance (Optimising data parallel code is hard)
Future Work Full coverage of image algebra operations Implement algorithm libraries in RIPL RIPL dataflow compiler Hardware support for low-level image algebra operations
References Handbook of Computer Vision Algorithms in Image Algebra 2nd Ed., G. Ritter & J Wilson, 2000. Supporting image algebra in the Matlab programming language for compression research, M. Schmalz et al. SPIE, 2009. Efficient parallel stencil convolution in Haskell, B Lippmeier & G Keller, Haskell, 2011. Accelerating Haskell array codes with multicore GPUs, M Chakravarty et al. DAMP, 2011.
Thanks