Methods for two dimensional stroke based painterly rendering. Eects and applications

Size: px
Start display at page:

Download "Methods for two dimensional stroke based painterly rendering. Eects and applications"

Transcription

1 University of Pannonia Veszprém, Hungary Information Science and Technology PhD School Methods for two dimensional stroke based painterly rendering. Eects and applications Levente Kovács Department of Image Processing and Neurocomputing Supervisor: Prof. Dr. Tamás Szirányi A thesis submitted for the degree of Doctor of Philosophy September 3, 2006

2 I would like to dedicate this work to my parents who were brave enough to search for new horizons, repeatedly, opening the door towards new vistas for their children. And to my sister, for a proof that in a dark room with no exits a jackhammer comes handy. Also, I'd like to include one of my favourite qoutes here, without further ado: "The universe is driven by the complex interaction between three ingredients: matter, energy, and enlightened self-interest."

3 Acknowledgements I'd like to give all my thanks to my parents and also my sister for their constant support during all those years I can remember, and more. I'd like to thank my supervisor, professor Tamás Szirányi, for his support, help, guidance, advices and ideas during the years that led to the writing of this work. Then I'd like to thank my collegues at the image processing group of the Department of Image Processing and Neurocomputing, University of Veszprém, with whom I spent the most part of the workdays during the last three years for being supportive, professional and cool, and providing an environment in which our work could be pursued mostly enjoyably. I also thank all my long time good old friends, spread across countries and continents, for their support, insightful advices, and I also give thanks for the common beer parties which kept some of us (?) sound of mind and which ought to have been somewhat more frequent.

4 Methods for two dimensional stroke based painterly rendering. Eects and applications Abstract This work presents the author's results in 2D non-photorealistic stroke based rendering research. The focus is on automatically generated painterly eects on images using articial stroke template based rendering approaches, but also on other areas like optimizing template based painting, representations and encoding of rendered images, and applications. The main results of the work are in developing new stroke based painting methods, in optimizations and coding of generated paintings, in connecting the painting methods with scalespace feature extraction and also in creating focus based level of detail control during the painting process. Both the theoretical and application results are presented.

5 Methode für künstlich-erzeugte Gemälde-eekte auf Bildern in zwei Dimensionen. Eekte und Anwendungen Extract Diese Arbeit stellt die Resultate der Forschung des Autors im 2D nicht photorealistisches pinselgegründeten Rendering dar. Der Fokus ist auf automatisch erzeugten Gemälde-eekten an Bildern durch Verwendung von künstlichen Pinselschablonen, und gleichzeitig auf anderen Bereichen wie Optimierung von Schablonenbegründeten nicht photorealistiches Darstellungen, Representierung und Kodierung der hergestellten Bildern und andere unterschiedliche Anwendungen. Die Hauptresultate der Arbeit sind die Entwicklung neuer Methode von Pinselschablonen gegründete künstliche Gemälde-eekte, Optimierung der Rendering Prozessen, Kodierung der generierten künstlische Gemälde, Forschung nach Verbindungmöglichkeiten der Render-methoden und sogenannten "scale-space" Bildeigenschaften und auch die Entwicklung einer Fokus-begründete, Position abhängige Steuerung des Niveaus des Details während des künstlischen Gemälde-generierung Prozesses. Sowohl theoretische Resultate als auch unterschiedliche Anwendungen werden vorgelegt und presentiert.

6 Kétdimenziós mesterséges festési eljárások.hatások és alkalmazások Kivonat A jelen disszertáció f témája nem-fotorealisztikus, ecsetvonás-alapú, mesterségesen el állított festményszer képi hatások, eektusok létrehozása, illetve az így létrehozott modellképreprezentációk leírása, kódolása, gyakorlati hasznosítása. A bemutatott módszerek alapját olyan kétdimenziós mesterséges festési eljárások adják, amelyek ecsetvonás-mintázatokat és bels képi struktúra-információkat hasznosítanak festményszer hatások automatikus el állításához. A diszszertációt eredményez munka során a szerz célja az említett típusú nem-fotorealisztikus képalkotási eljárások kidolgozása, vizsgálata, alternatív megoldások kidolgozása, különféle képfeldolgozási és képi struktúra-információ kinyerési módszerek alkalmazása az ecsetvonás-alapú képalkotásokban, újabb festés-alapú képalkotási módszerek kidolgozására volt, és párhuzamosan az ezekkel a módszerekkel létrehozott képek hatékony leírási, tárolási, kódolási lehet ségeinek vizsgálata. A kutatói munka során elért eredmények kerülnek bemutatásra a disszertációban, amelyek kulcsszavakkal a következ k: különféle többszintü ecsetmintázatok használata a festésszer képalkotási folyamatban, festési optimalizálások, a létrehozott képek ecset-sorozat alapú leírásának hatékonysága, scale-space módszerek alkalmazása a ecsetmintázat alapú festésben, alkalmazás képsorozatokra, módszer relatív fókusztérképek automatikus kinyerésére illetve ennek alkalmazása a festési eljárásban a részletgazdagság vezérlésére, létrehozott festett képek vektorgrakus reprezentációja, illetve néhány alkalmazási terület amelyekre a disszertációban b vebben kitér a szerz. Az eredmények tételes felsorolását és rövid leírását a disszertáció végén található téziscsoportok tartalmazzák.

7 Contents 1 Introduction Context, goals, motivation Context Goals Motivation NPR/SBR Overview New scientic results in this work Outline Stochastic painterly rendering Stochastic painting Notes and starter ideas Painting with templates Extensions: layer lling, edges Optimizing layers and stroke scales Scalability, coding, compression, storage Convergence Painting with scale-space Rendering with multiscale image features Automation by feature control Convergence Focus maps Automatic focus map extraction by blind deconvolution Localized deconvolution overview Extraction of Focus Maps Painting with focus extraction Color segmentation for quick background generation The painting process I

8 5 Applications Creation and coding of stroke-based animations Transformation details Stroke-rendered paintings Realtime painting viewer Rendering strokes into scalable vector graphics Focus-based painting to vector graphics Using focus maps for feature extraction Conclusions 92 Bibliography 94 Thesis Groups 100 II

9 List of Figures 2.1 Stochastic painting algorithm steps Sample of painted image (red ower, in Figure 2.3) with 10 layer coarse-to-ne fully stochastic painting using rectangular strokes. Top: rst step (after rst layer), bottom: last step (nal painted image) Samples of images used during tests (top-to-bottom: buttery, parrot, car, theatre, airplane, lena, red ower) Some examples of user-dened stroke templates Sample painted image using stroke style as shown in the upper left corner. Left image: after rst step, right: last step Coarse painting with rectangular (left), uy (middle) and large circular (right) stroke Sample painted by starting with the rst of the two templates shown, and nished with the second one Area covered partially (left), then morphed (right) Sample of painted image by using interpolative layer lling (only 70% of the canvas is covered) and edge following stroke orientation. a: model image, b: used edge map, c: rst layer, d: rst layer morphed, e: middle step, f : nal painted image Sample for image painted with combination of edge following stroke orientation and layer lling PSNR similarities (see equation 2.1) of painted image plotted against placed stroke numbers (and a magnication of relevant areas): rendering with dierent stroke sets (decreasing from 1 to 10), while morph70 is 3-layer painting combined with interpolative layer lling (with 70% coverage) Transformation times and PSNR at the 5000 placed strokes point from Figure Using 2-3 layers give better transformation times at the same quality (compared to using more layers) Some more examples for painted image quality (vertical) plotted against stroke numbers (horizontal) Stochastic 10-layer painting compared to enhanced painting with edge following and layer lling. It is shown (for consecutive frames of the Mother and Daughter standard test video) that for achieving similar quality the new method is much faster. 24 III

10 2.15 Rendering of 6 dierent images, all with 5 dierent runs with dierent stroke scales (labeled by roman numbers), plotting compression ratios (CR) against PSNR (see equation 2.1). Points show the positions where the rendering processes converged to. Sample images in Figure Detailed runs for the buttery image from Figure 2.15, plotting render times and compression ratios (CR) against PSNR (see equation 2.1). With a selected goal e.g. given quality in the shortest time one can select the appropriate rendering mode and vice versa Sample showing scalability. The images are all displayed from the same painted stroke-stream Sample for a painterly rendered image displayed from SVG format Comparing lossless compression rates of two example rendered image bitmaps by dierent codecs and by stroke-stream compressions. Subgure a) also contains stored in SVG format (last 2 columns) Lossless coding ratios of output stroke-series relative to lossless coded bitmaps of painted images JPEG coded oil painting segment (left) and the JPEG recoded with Paintbrush rendering (right) - showing the ability to mimic stroke styles and noise ltering eects Extending stochastic painting to be controlled by internal image structure improves eciency: same PSNR quality (in this case 27.7dB for the Lena image and 28dB for the ower image in Figure 2.3) can be achieved in about half the time as fully stochastic painting requires Sample of painted image with the multiscale edge/ridge extraction based one layer sequential painterly rendering model. Top: edge map and ridge map used, bottom: rst step, and after closing the holes Example of image painted following edge orientations (random stroke positions, sequential 10 layer stroke scales) "normal" is the fully stochastic painting method with 10 layers, "espt" is the new method with random stroke positions but following edge orientations when placing the strokes. It shows (for consecutive frames of the Mother and Son qcif video) that for achieving similar PSNR the new method needs much less time Example of image painted following edge orientation and scaling by ridge scales (with random stroke positions) Main steps of the 3rd painting approach Example of image painted following edge orientations, scales determined from ridge scales, stroke positions are sequential Examples of image painted following edge orientations, scales determined from ridge scales, stroke positions are sequential. On a) the main edge contours are also overlaid Example for an extracted relative focus map (left: input, middle: a coarse 2D map, right: a corresponding class map) IV

11 4.2 Steps of the iteration Error curves for 8 neighboring blocks (each curve stands for one block) on a blurred texture sample (top) for the same blur with ADE (left), and MSE (right). Ideally, curves of the same measure should remain close to each other Comparison of a) input image's b) MSE distance based extraction), c) PSF-based classication and d) the new angle deviation error metric Example for focus map extraction with contrast weighting. a) input, b) focus map without contrast weighting, c) focus map with contrast weighting and d) the extracted relative focus surface Example for extraction on the a) input image, with maps on the b) pixel level, c) with overlapping blocks, and d) non-overlapping blocks Visual comparison for a) input image, for b) deconvolution, c) autocorrelation and d) edge content based extractions Example images showing blurred texture discrimination properties of our (top right), autocorrelation (lower left) and edge content based (lower right) methods. a) Top left is the input, numbers show increasingly blurred sub-areas. b) Top left is the input containing blurred areas shown with black borders Columns from left to right for each row: input image with a blurred area, deconvolution based map, autocorrelation based map, edge content based map. Higher intensity means higher detected focusness Evaluation (lower is better) of the performance of deconvolution/ autocorrelation/ edge content extraction methods. For deconvolution both the new error measure (deconv) and the mean square error approach (deconv-mse) are investigated. Horizontal axis shows increasing blur, vertical axis shows extraction error relative to the real size of the blurred area. Blurred areas are shown with black borders. Errors larger than 100% mean that areas outside the ground truth area were also falsely detected as being blurred, thus increasing the error Examples for focus extraction on images with various textures (top row: input, bottom row: respective focus maps) Two examples of focus extraction on video frames: top row shows the video frames, bottom row shows their respective focus maps (higher intensity means higher focus) Following the move of focus across a scene (focus shifted from the left of the image to the stairs in the background). Upper row are the input frames, bottom row are their respective focus maps with higher intensity meaning higher focus Map extraction results on some high depth of eld images Map extraction results on images with a focused area on blurred background. Columns in each row from left to right: input, deconvolution based map, autocorrelation based map, edge content based map Samples for relative focus map extraction. Top: inputs and relative focus maps, bottom: relative focus surfaces with four classes V

12 4.17 Sample image generation by lling the background layer with larger strokes and rening the foreground with smaller strokes Samples for color segmentation and cluster colorization. Left column: input, middle: segmentation into 32 classes, right: colorized clusters Sample generated image (from a to e): model, segmentation map for background generation, colorized segmentation map, higher detail area, nal image Painted outputs. From left to right: input, and focus map; painted object on painted background; painted on original; original on painted Rendering samples. Top: input images and their focus maps. Middle row: painted objects on low-detail painted background. Bottom row: painted objects on segmentationcolorized background Cut detection used in the video painting process: cuts are detected by thresholding a function which is a weighted product of edge and pixel dierence maps of consecutive frames Painting of videos: painted frame (a), motion eld extracted (b), generated frame from previous frame and motion eld (c) Encoding the painted frames as a stroke stream losslessly is more ecient than encoding them as raster images with typical image coders in lossless mode.(espt stands for enhanced stochastic painting with edge following stroke orientations and layer lling, see Chapter 2). Numbers show average frame sizes for video samples a) Variation of output size relative to uncompressed input avi size (in %) with changing keyframe density (every nth frame is keyframe) for videos painted with dierent no. of stroke-sets. b) Relation between coding time and keyframe density for a sample video Using the enhanced painting process (espt) with layer lling and edge following stroke orientations produces more stable frame sizes at the same quality which is better from the encoding point of view (normal stands for stochastic painting, espt stands for enhanced painting with edge following stroke orientation and layer lling) Sample painted video frames Sample painted video frames Rendered oil painting (top:original) (Prole of a Woman / Zorka by József Rippl- Rónai, 1916) a) Rendered painting with uy template. b) Examples of painterly rendered images (from top to bottom, top: model image, bottom: nal and comparison) Examples of painterly rendered images (from top to bottom, top: model image, bottom: nal) of real oil paintings MPVPlay, a test OpenGL painted image viewer Painterly rendered output as SVG, displayed in dierent sizes Storing the painted images as stroke series is more ecient than coding them with image coders In many cases the SVG representation is shorter than the binary stroke sequence.. 81 VI

13 5.15 Some images used in the evaluation Viewing image rendered into SVG in a web browser Editing image rendered into SVG in Inkscape Left: SVG background formed from the buttery image. Right: The background is formed from distinct color areas; some of them are here picked and placed to the side Left: the rened focus area with small strokes. Right: the fully painted image, the rened area added to the background From left to right: downscaled model image; low detail background in SVG; rened focus area with small strokes; nal rendered image Some examples of rendered images Query-response example. Left: the query and its map below, others: response images and their maps Query-response example. Top-left: the query and its map; others: response images Precision-Recall graph for 10 dierent queries, in the case of a) the returned closest 10 matches and b) the use of a thresholded error measure for deconvolution- (deconv), autocorrelation- (AC) and edge content- (EC) based queries. The curves on the b) show values for a single query image for the 3 methods, given varying retrieval thresholds VII

14 List of Tables 1.1 Short feature list of referred methods Possible similar emulations of mentioned stroke representations by a common painting model (multiscale, template-based, stochastic) for using similar compression schemes. Last column shows approximations of bytes needed to store a stroke if simulated by the model Contributions by theses VIII

15 1. Introduction 1 Chapter 1 Introduction This rst chapter will be dedicated to an introduction of the eld of two dimensional non-photorealistic stroke-based rendering (NPR/SBR), basic techniques, methods and tools. A short historical background, the motivation behind this work, the goals and the steps towards achieving these goals will be presented. This work will be placed among the vast amount of achievements in the eld of NPR/SBR during approximately the last decade. 1.1 Context, goals, motivation Context Non-photorealistic rendering (NPR) techniques started emerging about a decade ago. Non-photorealistic is just what the name suggests - images created not with the goal to reproduce original imagery as close as possible, but to produce dierent representations of a model, something that might even be considered artistic. The rst goals of these techniques were to somehow imitate real life painting eects articially. Since the rst trials many dierent methods appeared, both in 2 and 3D, for many dierent purposes. Yet the rst goal has always remained: produce visually pleasant representations of model images or create such images articially with dedicated tools. Non-photorealistic rendering has been used since then for many dierent purposes: to create painting-like eects on images, to produce cartoon-like renderings, to create dierent types of sketches, drawings, or even to aid the creation of illustrations from ordinary images (i.e. in medical imaging). Stroke based rendering (SBR) methods form a special subset of non-photorealistic rendering

16 Context 2 techniques. In SBR the painterly eects are produced by simulating real life brush stroke to an extent, and the painting process itself, to produce images by a so called articial painting process. Simulated brushes can be of a wide variety from simple geometrical shapes to multilevel height maps or complex curves. As we will show later, there are many ways of generating painting-like images from ordinary pictures. The rough outline of these algorithms are as follows. We take an image that we wish to create a painterly rendered abstraction from, a model, and a blank image that we call canvas, on which we will create the painterly rendered image. The rendered image will consist of a series of brush strokes that will be placed on the canvas. The strokes can have dierent size, shape, orientation and color. These properties are the so-called stroke parameters which are the data needed to be known if we would wish to reconstruct the painted image later from the stroke-series it consists of. Strokes can be very versatile, and almost every researcher came up with their own version of the idea of a stroke. There are methods which use simple shapes (e.g. rectangles, lines, etc.) as strokes, others that use long curves with constant or varying width, then there are ones who use dierent templates or texture patterns as stroke models. The methods dier at most in the way they place the strokes on the canvas in order to obtain a plausible artistical representation of the model image. There are manual, semi-automatic and fully automatic painting techniques. A manual painting method is a painting application, fully controlled by the user. Stroke sizes, positions, color, i.e. everything is determined by the user. In manual painting approaches sometimes external haptic devices are also used to aid the artist in creating articial paintings. In semi-automatic painting techniques some parameters (for example stroke curve shapes or thickness) are determined by the user, but color/texture, lling and placing is done automatically. In automatic painting techniques the whole painting process is automatically controlled by the algorithm (sometimes parameters need to be set before starting the rendering process). SBR algorithms can again be classied by a dierent property, into two main groups: greedy and optimization algorithms. Greedy algorithms usually consist of a method which tries to place strokes which locally or globally seem to improve the image on the canvas, with no or minimal consideration of stroke neighborhoods, stroke density or other parameters. Optimization algorithms usually contain some kind of built-in intelligence in the stroke placement and/or acceptance steps, which help the painting process converge faster or to converge more to some global minimum rather than to a local dead-end. Usually the output of the rendering process is a series of stroke parameters, which can be

17 Goals 3 stored and later be used to reconstruct the painted images (and possibly also for other purposes as we will show later on) Goals Our work in the eld of NPR/SBR has been concentrated around 2D, model based SBR techniques. The goals we pursued were to achieve automatic painterly eects on model images and videos, and at the same time to investigate dierent aspects of SBR like optimizations, coding, compression, scalability, etc. Thus in our work and research in 2D SBR the source is a usual 2D image which is used as a model for the painting process. All data needed for the painterly transformation is obtained from this model image using dierent image analysis techniques. Our additions to the world of 2D SBR has been various, the most important probably having been a combined point of view not just from computer graphics but also from image processing and computer vision. This gave the opportunity to see dierent aspects of SBR besides the generation of painterly eects. Also, we have to state that our goal has not been to produce painting or painter-aiding applications in which some real painter can create paintings by using simulated digital counterparts of real painters' tools, as in [5]. Our methods and techniques provide means to create painterly eects automatically, by using model images and stroke templates, without no user interaction. Basically we have continuously targeted areas which have not been fully covered by graphics research in the eld of 2D SBR, mostly by combining techniques from image processing, pattern recognition and image coding. Thus, while we have been producing painterly eect generation methods, we always have also been dealing with storing, compressing and managing such produced images. This thesis work presents most of our previous research in the eld and also some new achievements that are being published or being prepared for publication Motivation The author's motivation in this work comes from the challenging, and widely unresolved tasks the 2D NPR/SBR techniques contain. As we will show in the next section, there are many NPR, and among them many 2D SBR methods. Some of them target specic application areas, like pencil or charcoal drawings, others try to simulate watercolor painting, there are ones that mimic oil paintings, of course there are many drawing-aiding applications which create an interactive

18 1.2. NPR/SBR Overview 4 painting environment for the users, and so on. Among these methods there are some, which try to produce (semi-)automatic painterly eect generators, which is also our target area. These methods produce painting-like images in a wide variety of styles and ways, each having its pros and cons. Our motivation to work in this eld was to create fully automatic, and totally image structure and feature extraction based SBR methods. Of course other methods have also incorporated some image information to some extent during the rendering process, like edge or texture information, still, there were no methods which would base the control of the entire process on image features and remain fully automatic. This was one area which we targeted. Another area which we tried to contribute to, was something which most of SBR techniques did not address: what should happen with the generated images, in what format and in what way should they be stored and handled? Generally the produced images are stored as raster images, or video frames, or 3D meshes in the case of 3D NPR. This can cause a lot of problems, e.g. how should these images be scaled (scaling the raster images with interpolation methods, regenerating the painting into a dierent scale?), how should they be compressed (since line, or stroke based renderings easily gain compression artifacts), how should the use for animations be handled, and so on. We tried to address these areas and investigated these issues. 1.2 NPR/SBR Overview In [25] Haeberli introduced a simple interactive painterly rendering program. The technique was to follow the cursor, point sample the color and paint a brush stroke with that color. A stochastic process was responsible for putting the strokes on user-dened positions. Haeberli has also described a relaxation method for painting based on a kind of Voronoi algorithm. While the later algorithm is more automatic, it gives a quite synthetic result. In Salisbury et al.'s [91] an interactive technique was introduced which produced high quality pen and ink type of contour-based sketches of model images. A similar silhouette painting algorithm is presented in Northrup et al.'s [79]. In [11] Curtis et al. created a watercolor paint system and demonstrated how computer-generated watercolor can be used as part of an interactive watercolor paint system, as a method for automatic image watercolorization, and as a mechanism for nonphotorealistic rendering of three-dimensional scenes. In Litwinowitcz's work [71] strokes with a given center, length, radius and orientation are generated, adding random perturbations and variation, gradient-based orientation. The author also uses stroke clipping for retaining edges and anti-aliasing. Hertzmann presented an algorithm [31]

19 1.2. NPR/SBR Overview 5 which painted an image with a series of B-spline modeled strokes on a grid over the canvas. The paintings were built in a series of layers, starting with a rough sketch then rened with smaller brushes where necessary. Later he implemented a back-end for rendering brush strokes [35] produced by painterly rendering systems like this or like Meier's one described later. Hertzmann also extended his method to video processing [32]. In Szirányi et al.'s [104,105] the so-called Paintbrush Image Transformation was introduced, which was a simple random painting method using rectangular stroke templates, up to ten different stroke scales in a coarse-to-ne way also for segmentation and classication purposes. A multiscale image decomposition method was presented, the resulting images of which looked like good quality paintings, with well dened contours. The image was described by the parameters of the consecutive paintbrush strokes. In Szirányi et al.'s [106] developments were presented, which included the usage of dynamic Monte Carlo Markov Chain optimization at the stroke acceptance step. Faster convergence was supported by a dynamic Metropolis Hastings rule. In [45] Kaplan et al. presented an algorithm for rendering subdivision surface models of complex 3D scenes using an interactively editable particle system. Geograftals are used (special types of triangle strips), following the works of Meier [76] and Kowalski [63]. In [76] Meier used NPR techniques to create hand-paint like animations. 3D models and particle systems are used, a technique not really suited for 2D painterly processing. An automatic painting system was introduced by Shiraishi et al. [96] The method generates rectangular brush strokes. Properties of strokes are obtained from moments of the color dierence images between the local source images and the stroke colors. The method controls the density of strokes as well as their painting order based on their sizes. The density is controlled by a dithering method with space-lling curves. Hertzmann's other work [33] is a relaxation technique to produce painted imagery from images and video. He uses an energy function similar to the one used by the Paintbrush algorithm [104] by Szirányi et al. and minimize it in a quite sophisticated way (Hertzmann's implementation deletes, adds and relaxes the strokes, where the relaxation means the modication of some of the stroke parameters). Modifying this energy function, he is able to follow several painting styles. In Szirányi et al.'s [105] the relation between painterly rendering and anisotropic diusion (AD) is suggested. Nonlinear partial dierential equations can be used for enhancing the main image structure. When compressing an image, AD, based on the scale-space paradigm, can enhance the basic image features to get visually better quality as Kopilovic et al. show in [48]. Determining the used stroke color based on blurring the underlying area and selecting the most recurring color

20 1.2. NPR/SBR Overview 6 also shows some compatibility with AD because of its integrating nature. Hertzmann et al. [34] introduced a method which can produce ltered images based on training images. They also use lters, e.g. they apply an anisotropic diusion or similar lter (a blurring lter) for preprocessing, in order to maintain good contours but eliminate texture. The goal of this work is to try to propagate a painting's style by painting images with stroke parameters extracted from another image. In [29] Hays et al. present a still and motion picture painting method based on generating meshes of brush strokes with varying spatio-temporal properties. Strokes have location, color, angle and motion properties. By using textures and alpha masks they can mimic dierent styles. Like in Kovács et al. [50], motion data is used to paint video frames, in this case for propagating the stroke objects along motion trajectories, which in some cases can lead to disturbing eects with strokes moving all over the video frames, giving a uid-like feel to the image. In Park et al.'s paper [81] a brush generation technique is presented where spline brush colors are obtained by generating a color palette from selected reference images, strokes follow image edge orientations, and stroke clipping at edges is also used. In the work of Gooch et al. [24] a method was presented that produced a painting-like image composed of strokes. This approach extracts some image features, nds the approximate medial axis of these features, and uses the medial axis to guide brush stroke creation. In [43] Jodoin et al. present a hatching simulation stroke based rendering method, N-grams and texture synthesis tools to generate stroke patterns from a training stroke model and use the generated patterns to render 2D or 3D images and models in hatching style. Santella and DeCarlo presented a method [15] which combines aspects from the approaches of Haeberli [25], Litwinowicz [71] and Hertzmann [31]. They extended their work with a system collecting eye-tracking data from a user in Santella et al.'s [92]. In their system they use curved strokes with single color and paint in details only the picture elements that attracted the user's gaze. We will present a method which has similar goals but does not need external devices, and will control the painting process by automatically estimated relative focus regions [58]. Mignotte in [77] presents a sketching technique based on Bayesian inference and gradient vector eld distribution models combined with a stochastic optimization algorithm. The stroke model used is a B-spline with 12 control points pencil stroke. In [110] Wang et al. presented a video transformation technique based on user performed segmentation on keyframes, interpolating of regions through frames and painterly rendering the segmented areas. Spatio-temporal surfaces are constructed from segmented areas over frames for

21 1.3. New scientic results in this work 7 smoothing using so called edge and stroke sheets. The system in [44], by Kalnins et al., lets the designer directly annotate a 3D model with strokes, creating a subjective 3D NPR representation of a model object. The user chooses a brush style and draws strokes over the model from multiple viewpoints then the system adopts these strokes when rendering the scene to maintain the original look. Another system presented by Xu et al. in [109] is a semi-automatic painting system, where a user with a tablet and a pen can simulate painting eects, the brushes being 3D simulations of real brushes using pen pressure and orientation information to achieve more realistic representations. 1.3 New scientic results in this work As we also have shown in the introduction, the stroke based rendering literature knows a fairly large number of methods and techniques for producing 2D painterly images. Most of these methods use image analysis results on some level of the rendering process. Our point of view and our methods dier from these approaches essentially since we have continuously concentrated on starting from and controlling the painting process by image structure information. That is, while the other methods incorporated image content information on some level, our methods are based on image content and structure analysis from the ground up. Another main dierence that our approaches brought is that we have not stopped at the output of a painterly rendering method, but investigated other possibilities, like storage, encoding and compressing possibilities of generated images, and also scalability and portability issues. We have developed new methods for automatic painterly eect generation, rst by extending stochastic painting, then by introducing scale-space feature controlled painting variations, by investigating optimal stroke set use multilayer painting, by introducing automatic focus based painting, and by various applications of these methods. We will detail the specic dierences and new additions in the respective chapters. We have to clearly state here, that this work has its roots in the stochastic paintbrush transformation method of Szirányi et al. [105], which we shortly introduce in Chapter 2.1. All other parts of the present work are the author's own works and results. Our publications in the eld include the following. In [49,50] we presented our rst steps in cartoon style transformation of image sequences based on painterly transformation of keyframes and optical ow based interframe post-painting. In these works a fairly simple stochastic painting process was used, generating the painted image by randomly placing dierent sizes of rectangular strokes to mimic the model images or video frames. Later starting from [51,52] more emphasis has

22 1.3. New scientic results in this work 8 been added to the coding and compression possibilities of painterly rendered images, showing that ecient lossless encoding can be done when storing the painted images as series of stroke parameters instead of raster images. In [54,55] we investigated the applicability of painterly rendering in regeneration of original paintings, for storage and also for producing dierent reproductions by using various stroke templates and styles to generate other representation of real paintings. In [53] and later in [56] we presented methods combined with scale-space feature extraction where special weighted edge and ridge maps were used to control the painting process, producing a more natural rendering, with less layers. In [60] we proposed to render the paintings into scalable vector graphics (SVG) for more portability and ease of handling. In [61] we introduced a two-layer painterly rendering with location dependent detail control by using our automatic relative focus map extraction method [58]. Generally, comparing dierent stroke-based non-photorealistic rendering methods objectively is a seemingly impossible task. Given all the years of development in the eld, and given such broad surveys like Gooch & Gooch's [23], Strothotte et al.'s [100] or Hertzmann's [36], there is still no objective way to compare two NPR images. This is because of the really wide variety of subjective styles these methods produce, and the really broad range of simulation possibilities. What we can do is comparing the methodologies used in generating a narrow class of NPR images, which in our case is automatic, image feature controlled stroke-based image generation. One approach are comparisons like Tables 1.1 and 2.1 try to show. Oftentimes, there are fairly minor dierences between the methods these renderings use still they can produce endless dierent styles. Table 1.1 (at the end of this chapter) provides a comparison of our and related techniques based on criteria like stroke type, stroke representation or automatism abilities. Related approaches include related 2D SBR methods presented in the previous Overview section, those of Haberli [25], Litwinowicz [71], Curtis et al. [11], Hertzmann [31,33,34], Shiraishi et al. [96], Szirányi et al. [106], DeCarlo et al. [15], Gooch et al. [24] and our approaches [53,54]. Also, the later referenced Table 2.1 (which is at the end of Chapter 2) contains details about the stroke types used in similar SBR methods, and details regarding how these stroke models could be also produced by using a template based multiscale rendering method. That is, to show that SBR methods like the ones we have been working on are dierent from similar methods, still, many other approaches could be simulated by them.

23 1.4. Outline Outline After the introductory role of this chapter, Chapter 2 will present detail on basic stroke based rendering by explaining stochastic painting and the various extensions, optimizations, coding and compression possibilities. Chapter 3 will present scale-space feature controlled painterly rendering. Chapter 4 will present our recent relative focus map extraction technique and its applications in stroke based painting. Chapter 5 contains the presentation of various applications for the previously presented techniques. Chapter 6 has the conclusions followed by the bibliography and the thesis groups. Ref. Stroke type Stroke Auto- Technique representation matism [25] geometrical color, orientation, partial draw an ordered list of geometrical strokes; strokes size, shape, follow the cursor and point-sample the position model then draw a stroke at that position [71] antialiased radius, length, full start at the center of a stroke and grow lines clipped color, position, until edges are met then clip; randomly at edges orientation perturb stroke parameters; orientation following near-constant color [11] no strokes color chosen by partial trying to achieve watercolor style; ordered the user, length, set of washes over a paper, position, orien- containing pigments structured into tation glazes produced by uid simulation [31] long curved B-spline control full use ne strokes only when necessary, strokes, points, brush size, painting layers of strokes upon each gradient based shape, color, other; place strokes on grid points style, orientation thickness, parameters (8+) [96] rectangular position, size, full start with larger and nish with smaller antialiased color, orientation strokes; orientation is based on image strokes moments if the color dierence images [33] short lines color/texture, partial strokes based on contour approximation opacity, shape, with dierent styles, controlled by position, size, style energy functions [34] no strokes, lter descriptors partial use lters to produce articial eects ltering simulating the style of the model [106] rectangular, position, orientation, full coarse-to-ne painting, starting with or user color, size larger and rening with smaller strokes; dened binary orientation determined by a stochastic relabitmaps xation technique; MCMC optimization at the stroke acceptance step [15] following line thickness, partial painting controlled by eye-movement edge-chains, length, color curved strokes [24] central axis of seg- control points of full stroke creation guided by approximate mented areas a B-spline (control feature medial axes; rendering: draw combined into; polygon); scalar parallel lines in the direction of tokens Filbert brush polynom blending the stroke B-spline simulation width values; color [54] user-dened style, position, full coarse-to-ne stochastic placement of grayscale (256 orientation, stroke templates through multiple scales; levels) templates color covering only certain percent of the canvas on a layer then use layer lling; following and aligning to model edges [53] any shape and position, color, full model weighted edges control stroke size of brush orientation orientation and weighted ridges control templates grown stroke scale (meaning stroke line width into lines with radius); only one layer of painting, varying thickness sequential (one coverage then ll the holes) Table 1.1: Short feature list of referred methods.

24 2. Stochastic painterly rendering 10 Chapter 2 Stochastic painterly rendering This chapter presents our basic grounding rendering method that was extended with features and used as a starting point for further development. Thus an automatic, stochastic painting will be presented with details on how the method works and which image features it uses for generating the images. Also, representation, coding and compression of the painted images will be discussed. Regarding the connections with older or similar stroke based rendering techniques, this chapter presents new or enhanced results. First stochastic painting is described, which is where this work has started from. Later painting generation by using freeform stroke templates with grayscale height maps is presented, which is a new addition over previous techniques. It is new in the sense that in previous works, a stroke is either a simple geometrical shape, or it is the result of interactive physical stroke simulation, but here we use a solution that is in the middle: weighted freeform templates can be dened and used in the automatic painting process, thus we have the possibility to generate dierent styles of the same model. Thus the automatic eect generator can have more possibilities. Then, a new mid-process layer interpolation technique is presented, which we used to call color morphology, which is an interpolative layer lling step. This is a fairly straightforward eightdirectional color interpolation method, in order to speed up the generation of coarser layers during the painting process. This is accompanied by an edge following extension, which means that during the painting process the stroke orientations should follow the direction of the nearest edges. The latter technique has been used in the SBR literature, but it is important to introduce it in order to prepare the presentation of our later scale-space structure based painting method. We also present new research results regarding the numbers of layers (or stroke scales) used

25 2.1. Stochastic painting 11 in multilayer painting methods, presenting our results which show that using a higher number of layers than three is usually a waste of time and resources since it cannot generate better results in any sense. The last section of this chapter contains our results regarding a special eld of SBR, which has not been covered, or supercially treated by others, namely issues concerning the storage, compression and scalability aspects of painterly rendered images. 2.1 Stochastic painting The basis of this work was a stochastic painting process [105], where the painted image was built by placing simple geometric stroke shapes with constant color on a virtual canvas until a desired eect was achieved. The strokes' positions were picked randomly over the canvas area, the color of the strokes was taken from the area of the stroke on the model image. Each stroke could have eight orientations as a multiples of π/8. Strokes could be of multiple sizes, each size called a stroke scale. These scales were grouped in stroke sets, and stroke sets used during the painting process were iteratively changed by decreasing order of their sizes. This is where we started from, seeking new possibilities for improvement and creating new techniques. In the following we will introduce our 2D SBR model and show that it can be a comparable general basic model for most methods from the broad literature of SBR. The main steps of this (or such) an algorithm are in Algorithm 2.1, and also in Figure 2.1. load input image; initialize canvas; while there are smaller strokes do repeat select stroke position; select stroke orientation; sample stroke color; place stroke; until stroke set change is required ; switch to smaller stroke set; end Algorithm 2.1: Steps of the plain stochastic painting process. In the above algorithm steps, a stroke set change is required when no improvement (over a threshold) can be made to the canvas with the current stroke scale. That makes the self control of the painting process quite unique since it does not follow strict rules as a stopping condition, but a relative dierence checking between consecutive steps. In the base algorithm, the dierence

26 2.1. Stochastic painting 12 Figure 2.1: Stochastic painting algorithm steps. between the model and the painting is checked at specic intervals (originally after every 5000 placed strokes) by a measure function, which is responsible for stopping the painting process if a goal is achieved. The measure functions can be e.g. PSNR (Peak Signal to Noise Ratio) value related to the model, which should be above a threshold, where where MSE is the mean square error between two images. relative error function F err which is max P SNR = 20log 10 ( ) db (2.1) MSE F err = err i 1 err i err i < ε (2.2) which should be lower than a threshold ε, and where err i is the mean square error between the current painted stage and the model and e i 1 is the same for the previous painted stage. The convergence of the painting process is guaranteed since the process is stopped given

27 2.1. Stochastic painting 13 that a specic db dierence is achieved or the relative change between two consecutive steps is smaller than a threshold. Before the painting process starts, color sampling for the dierent possible stroke sizes is performed as a preprocessing step. This is done by convolving the model image f with all possible stroke patterns with all possible orientations, for obtaining a blurred image series from which color sampling will be performed. b j = f s ij (2.3) where i = 0...stroke set number and j = the orientations, and the blurred series s ij = s i0 cos ( π 8 j) sin ( π 8 j) sin ( π 8 j) cos ( π 8 j) (2.4) where s i0 is the horizontal template in each stroke set. Convolution is done in Fourier space (by multiplying the FFT of the templates with the FFT of the model image) for less computation time, exploiting a very important property of the Fourier transform, that convolution is equivalent to multiplication in Fourier space: f(x) s(x) = F 1 [F (ω)s(ω)] = f(x ζ)s(ζ)dζ (2.5) ζ R N Later during the painting process color sampling will be performed using these pre-generated images: the most frequently occurring color (c) underneath the stroke (s ij ) at a certain position (x, y) is taken, which we call majority sampling: c(s ij, x, y) = argmax hist sij,x,y(t) (2.6) t where hist is the local histogram of colors under the area of the stroke s ij at location (x, y). The parameters describing a stroke are the stroke's id (identies the template and the size), position, orientation and color. Positions are relative, telling the distance from the previous stroke. A very important property is that the size (in bytes) of describing a stroke is always the same, regardless of the size or shape of the used stroke. This means that for higher quality more of the smaller strokes have to be used and the size of the painted output (compressed stroke stream) can increase dramatically. We have to take this in consideration when doing the transformations. The output of the painting process is a series of stroke parameters, which are then used to store the painted image with a lossless compression scheme using Human compression [112].

28 Notes and starter ideas 14 Any other lossless statistical encoder could be used, we chose this because of simplicity and good performance. Of course during the painting process a series of strokes can be even fully covered by following layers. These fully covered strokes are deleted from the stroke stream at a nal stroke-elimination step when we go over the strokes in a backward direction and check whether they are covered by other strokes on a higher hierarchical level of layers. A sample of an image transformed using this technique is shown in Figure 2.2 (the model image is the ower from Figure 2.3. While this technique may seem a bit rough, being stochastic and brute force in its nature, it also allows for a great freedom in the painting process. This means it can be fairly easily extended with relaxation and dierent optimization techniques (as in [106,108]) and other usable extensions can be easily added, as we will show later Notes and starter ideas From the data in Table 1.1 (at the end of Chapter 1) one can realize that while stroke generation and placement methodologies can be quite dierent, stroke representation techniques do not dier that much. This means the main parameters of used strokes in dierent painterly rendering techniques are very similar. In most cases we need to store some stroke style data (e.g. template identication, lter description, etc.), position, orientation and color data, sometimes scale data. In some cases additional data can occur e.g. stroke width variation. When storing the used strokes as spline curves one stroke gets represented not by an identier-position-scale-orientation data quadruple, but with a series of oating point control points. The main idea is that essentially there is not much dierence in the stored data stream representing the painted image (which in our case is a coded stroke stream). Respectively, many stroke shapes and styles can be simulated with other stroke styles and templates of other techniques. As an example long lines even with varying width can easily be substituted with a series of rectangular or similar longish types of stroke templates and converted. This means that the later proposed coding and storage can be used for all other similar painting techniques as well. For a quick overview of possible stroke emulations see Table 2.1 (at the end of this chapter) which suggests possible simulations of other painting techniques by a common multiscale, template-based rendering method, which then could be represented and coded by a common stroke-series based compression scheme which we also use for our methods. In its basic form, stochastic painting with rectangular simulated strokes can produce very

29 Notes and starter ideas 15 Figure 2.2: Sample of painted image (red ower, in Figure 2.3) with 10 layer coarse-to-ne fully stochastic painting using rectangular strokes. Top: rst step (after rst layer), bottom: last step (nal painted image). high quality painterly rendered images, but the stochastic process involves slow computation, and an unpredictable time to complete the painting process. Thus rst we worked on achieving algorithmic speedup while trying to retain the basic properties of the stochastic painting technique.

30 Notes and starter ideas 16 Figure 2.3: Samples of images used during tests (top-to-bottom: buttery, parrot, car, theatre, airplane, lena, red ower). We will present some techniques which improve the algorithm, reduce computation times, and later techniques that change the nature of the painting technique from stochastic to sequential at the same time providing a more natural way of painterly image representation and rendering. We also will provide data regarding the usability of stroke-series based compression for painterly rendered imagery. To summarize the motivations behind improving the transforming the basic stochastic painting idea, while stochastic painting can produce nice eects, it still has its limitations: no consideration of image structure, use of at homogeneous strokes, hard to estimate transformation time, need for complex optimizations, post-processing steps. We sought to alleviate or overcome these limitations.

31 2.2. Painting with templates Painting with templates Rectangular simulated strokes, however easy to handle, did not prove to be enough and able to generate multiple styles of paintings. Thus, we introduced the ability to handle and use any type and kind of stroke shapes, by generating grayscale stroke templates with 256 gray scales, and use them as templates in the painting process. This way many dierent styles can be simulated and interesting pictures can be generated. The stroke templates can be any type of grayscale sample. During the painting process, stroke placement will take place by rotating and shifting a stroke template over the canvas s ij = s i0 cos ( π 8 j) sin ( π 8 j) sin ( π 8 j) cos ( + π 8 j) dx dy (2.7) where s ij is a stroke from set i with orientation j, s i0 is the base template of set i, (dx, dy) are the target stroke center coordinates. The grayscale values of the generated template will be used as they were a height map, i.e. after the overall color of the stroke will be obtained by the previously mentioned majority sampling process (eq. 2.6) and the color of the stroke will be weighted by the grayscale values that the stroke template consists of c sij,x,y(u, v) = c sij,x,y(u, v) w(u, v) ξ (2.8) (u, v) being the position over the stroke area, w the stroke height map, ξ a constant. Thus the color of the placed stroke will vary over its area, depending on the local weight of the template. This method will not reproduce the model image's texture, but this is not its goal either. It should just give us more free hand in producing interesting stroke templates to produce dierent painting eects. For some examples see Figure 2.4. One can even vary the stroke templates during the painting process, thus e.g. starting with a large round stroke shape and nishing with a longish uy shape. Stroke style changes only mean a change of an identication byte in the generated stroke stream. For an example of painterly rendering with a dierent stroke than rectangular see Figure 2.5. Figure 2.4: Some examples of user-dened stroke templates.

32 2.3. Extensions: layer lling, edges 18 Figure 2.5: Sample painted image using stroke style as shown in the upper left corner. Left image: after rst step, right: last step. Figure 2.6: Coarse painting with rectangular (left), uy (middle) and large circular (right) stroke. 2.3 Extensions: layer lling, edges An extension we introduced to our painterly rendering process, which we used to call color morphology, which is a layer lling step based on weighted directional color interpolation, does the following: after a given percent of canvas coverage by a certain stroke layer the process is stopped and uncovered areas are lled by an eight directional distance weighted color interpolation method. The next painting layer takes up from this color-lled stage and continues the painting process. When using multiple layers and stroke scales, switching to a smaller stroke-set occurs when painting ("correcting") the present step does not cause improvement larger than a threshold. If we stop earlier, when a given percent of the canvas area is lled, complete the remaining areas with layer lling and use this lled canvas as a starting point for the next layer, it turns out that a

33 2.3. Extensions: layer lling, edges 19 Figure 2.7: Sample painted by starting with the rst of the two templates shown, and nished with the second one. lesser number of placed strokes is enough to complete the painting. This is mainly because there will be some areas where the interpolative layer lling produces enough color information that the area will not need further stroke renement. From a point C x,y we search in 8 directions (i = 0..7) for the rst occurring color C i. The color at C x,y will be the weighted average of the found colors, where the weights are inversely proportional to the distance of C i. We norm the d i weights so as the sum of weights will be 1. The steps of the interpolation are in Algorithm 2.2. A canvas that is covered 50% with rectangular strokes then lled look like Figure 2.8. A sample image using layer lling and simple edge following stroke orientation can be seen in Figure 2.9. Another extension we introduced is that strokes placed during the painting process shall not have their direction picked stochastically, but from the edge/gradient data of the model image. That means that when placing a stroke on the canvas, the orientation of the stroke will be d=distances of the 8 nearest colored pixels from C x,y ; d c =color values of pixels at respective distance from d; d sum = 7 d[i]; i=0 sd=d sorted in increasing order; for i = do m=position of d[i] in sd; w i =sd[7 m]/d sum the i th weight; c i = d c [i] w i the i th color element; end C x,y = 7 i=0 c i/ 7 i=0 w i; Algorithm 2.2: Steps of the interpolative layer lling extension.

34 2.3. Extensions: layer lling, edges 20 Figure 2.8: Area covered partially (left), then morphed (right). Figure 2.9: Sample of painted image by using interpolative layer lling (only 70% of the canvas is covered) and edge following stroke orientation. a: model image, b: used edge map, c: rst layer, d: rst layer morphed, e: middle step, f : nal painted image. determined from the direction of the nearest edge line. That is, from a stroke place candidate the nearest edge line is searched, its direction is calculated and assigned to the respective stroke.

35 2.4. Optimizing layers and stroke scales 21 This way the produced images will be more coherent, less errors will be introduced by mis-oriented strokes and thus less time will be spent on correcting such mistakes during the painting process. Figure 2.10 shows an example for such generated image. (a) Model and its edge map (b) First, middle and nal steps from the rendering process Figure 2.10: Sample for image painted with combination of edge following stroke orientation and layer lling. 2.4 Optimizing layers and stroke scales We investigated the possibly optimal number of stroke layers for stochastic multilayer/multiresolution painting. We wanted to determine that minimal number of stroke scales which could produce similar quality results as a higher number of layers. The purpose of this research was to reduce transformation time, to produce more compact coded stroke stream on the output and to gather some theoretical ideas about the possibility of stroke scale number reduction. In the original stochastic painting we used a x number of ten scales of strokes for covering the canvas during the painting process. Soon the question arose whether we really needed strokes of so many dierent sizes for optimal coverage. Under "optimal" we mean the optimal compromise between stroke number, quality (db dierence) and time. We conducted experiments concerning

36 2.4. Optimizing layers and stroke scales 22 transformation times and stroke stream compression ratio with dierent numbers of stroke-sets, and also compared the older painting method with our newer approaches. Tests were performed on multiple dierent sized color images. Figure 2.11 contains render runs for a sample image (similarity errors plotted against placed stroke number), showing that using 2-3 stroke scales can produce just the same quality db dierence as when painting with ten scales. The graph in Figure 2.12 shows time required to arrive to the 5000 strokes point in Figure It shows that at the same stroke count painting with 2-3 layers is much faster (all producing PSNR quality within 0.5 db). Figure 2.13 contains some more examples. Figure 2.11: PSNR similarities (see equation 2.1) of painted image plotted against placed stroke numbers (and a magnication of relevant areas): rendering with dierent stroke sets (decreasing from 1 to 10), while morph70 is 3-layer painting combined with interpolative layer lling (with 70% coverage).

37 2.4. Optimizing layers and stroke scales 23 Figure 2.12: Transformation times and PSNR at the 5000 placed strokes point from Figure Using 2-3 layers give better transformation times at the same quality (compared to using more layers). Figure 2.13: Some more examples for painted image quality (vertical) plotted against stroke numbers (horizontal). Figure 2.14 shows (for frames of the Mother and Daughter standard test video, some frames also on Figure 5.6) the rendering time comparison between ten-layer stochastic painting and the extended version with layer lling and using edge orientation for stroke placement. The enhanced version achieves the same quality in less time. Figures 2.15 and 2.16 provide more insight into the experiments concerning nding the

38 2.4. Optimizing layers and stroke scales 24 Figure 2.14: Stochastic 10-layer painting compared to enhanced painting with edge following and layer lling. It is shown (for consecutive frames of the Mother and Daughter standard test video) that for achieving similar quality the new method is much faster. optimum between stroke scales, compression ratios and transformation times. First, render run data regarding PSNR quality (see equation 2.1) against compression ratio (abbreviated as CR) is presented in Figure 2.15 for multiple dierent images and 5 dierent rendering for each image with dierent stroke scales (represented by the roman numbers). The points show the positions where the renders stopped using the relative error checking method. What we should see is that the renders using 2-3 scales (i.e. runs labeled III and I) are always near the optimum in each point cloud. For further details one of the images (buttery, 768x512 pixels, 24 bit color) was picked and more detailed renders were run using the same stroke scales, and stopped at 0.5dB intervals. Data is shown in Figure 2.16 both for time and compression ratio data plotted against PSNR quality. Renders using scales 1-10 (i.e. all the scales) and only 10 (the smallest scale) were included as extreme situations. This extremity clearly shows, as in the upper diagram the 1-10 run requires much more time at every step, on the lower diagram the 10-run produces much bigger code at every step. Therefore, if we discard these two we can clearly see that the run (using stroke scales of 1 : pixels, 5 : 37 9 pixels and 10 : 10 3 pixels) always runs close to the optimum - high compression rate and shorter running time - curves on both the compression ratio and time diagrams. Naturally, if the goal of the rendering process is to produce the most compact stroke-series code, the full 10-layer painting can be used, but knowing that it will take much more time. On the other hand, when time or compression is a lesser issue, using the smallest stroke can lead to the best quality. But on the average, for the same PSNR using 2-3 stroke scales is enough, and the advantage in transformation time can be quite signicant (see upper part in Figure 2.16). For example, when one wishes to obtain 25dB painted quality in the shortest time, one would choose

39 2.5. Scalability, coding, compression, storage 25 Figure 2.15: Rendering of 6 dierent images, all with 5 dierent runs with dierent stroke scales (labeled by roman numbers), plotting compression ratios (CR) against PSNR (see equation 2.1). Points show the positions where the rendering processes converged to. Sample images in Figure 2.3. painting only with the smallest stroke (i.e. 10), knowing that it will produce the largest output. If a somewhat slower convergence can be allowed, painting with three layers can produce the same quality but half the size. Also, starting with a large stroke, then later rening with a smaller one, can signicantly reduce the size of the stroke-stream - for example see the lowest two curves on the lower graph of Figure 2.16: the dierence is that the lowest curve is rendered only with the smallest stroke, the one above uses the largest stroke as a rst step. 2.5 Scalability, coding, compression, storage We have also investigated ecient lossless coding and compression possibilities of stroke based rendering techniques. In stroke based rendering, the emphasis has been on the generated eects, and little care has been about how the painted images should be stored and handled. Images generated with SBR are usually highly susceptible to encoder noise and artifacts, thus in order to retain the painterly quality, one has to store them in a lossless raster image format. But there are usable alternatives to this approach, and this is what we deal with in this section, and later also in Chapters 5.4 and 5.5. In this section the main idea is, that if we store the painted images not as raster images but as a stroke series that it is built of, we will obtain a lossless representation, which is at the same time more compact than losslessly encoding these images with known rater image coders. Later, in

40 2.5. Scalability, coding, compression, storage 26 Figure 2.16: Detailed runs for the buttery image from Figure 2.15, plotting render times and compression ratios (CR) against PSNR (see equation 2.1). With a selected goal e.g. given quality in the shortest time one can select the appropriate rendering mode and vice versa. the above mentioned chapters, we will also introduce vector graphical representation and encoding of painted images, which another step towards lossless, easy to handle and portable representations of these painterly rendered images. Stroke-series representation makes a painted image scale-independent to an extent, in the sense that dierently scaled versions of the painted image can be displayed without re-painting: only the strokes are scaled and the positions adjusted. This is also the case when images are converted and stored as vector graphics (in SVG). The image versions in Figure 2.17 were generated this way, i.e. the smallest one was a painterly generated image, the larger versions are just larger version displays. The scaling of images produced by stroke-based NPR methods has always been a question

41 2.5. Scalability, coding, compression, storage 27 Figure 2.17: Sample showing scalability. The images are all displayed from the same painted stroke-stream. with no denitive answer (e.g. see Chapter 5.3 in [100]). Raster-based interpolation methods are denitely not suitable for creating dierently scaled versions of stroke-generated images, since the easily introduced unpleasant artifacts (blockines, aliasing, etc.). In the case of 3D NPR, when surface points and local textures are available, such scaling is an easier task. But in 2D SBR, scaling is not a trivial task. As we just mentioned above, when we store the painted images as a series of stroke parameters which they are built of, it is easy to scale and shift these stroke parameters before displaying them, thus producing dierently scaled views of the same rendered images. That, of course, does not mean an increase in detail in the painted image. It just means that a larger version can be easily displayed, which will give the same painterly look and feel, when - and this is important - viewed from a greater distance, just as in the case of the painterly rendered images in their original scale. What we will discuss rst in the following is a way to store the stroke series of the generated paintings in a way that makes displaying dierent scale views of the rendered images a transparent

42 2.5. Scalability, coding, compression, storage 28 and easy task. And that is storage in scalable vector graphics representation. Scalable Vector Graphics (SVG) is a standard language for describing 2D vector graphics. The such described graphics les are platform independent and can be manipulated and viewed by a variety of applications and tools, from browser plugins to free and non-free applications. One of the main advantages of storing images as SVG is that these provide scale-independent handling of the included graphics. Our painterly rendered images are represented and stored as a series of strokes. As described earlier, our strokes usually are geometrical shapes which can be easily described as SVG elements. For example a rectangular shape with color at a given position and orientation has the form <rect width="60" height="15" transform="translate(21,128) rotate(22.5,30,7)" fill="#897f0c" /> Many dierent shapes are supported including ellipses, polygons, lines or polygonal lines. For a sample image which has been painterly rendered into SVG and displayed, see Figure See more on the SVG representation in Chapters 5.4 and 5.5. Figure 2.18: Sample for a painterly rendered image displayed from SVG format. Regardless of painting with 10 layers and always covering all the canvas, or painting with less layers with edge following orientations and layer lling after a certain coverage percent, the output always is a sequence of stroke parameters. We code these outputs in a lossless way with

43 2.5. Scalability, coding, compression, storage 29 Human coding to achieve even smaller sizes. In the compression scheme the entire parameter set of all the placed strokes are treated as dierent symbols. That is because at least the position data of the strokes is always dierent, there are no placed strokes with identical parameter sets. Therefore if one would use the entire parameter set of a stroke as a symbol for encoding, there would be as many dierent symbols as the number of placed strokes and ecient coding could not be used. Figure 2.19 shows compression ratios of stroke-series encoding related to lossless image coders for sample images, 2.19(a) also contains data for SVG storage. What can be seen is that coding the painted image by compressing the resulting stroke stream is more ecient than coding the rendered images with usual raster image coders in lossless mode. All sizes are relative to the size of the painted bitmap. Figure 2.20 presents data showing that if we compress the painted output (the stroke-series representation of the painted frames) in our lossless way, than we can achieve at around 3-4 times better compression ratios compared to what we can achieve when compressing the painted output bitmaps with lossless image coders (e.q. Huyuv and JPEG2000 in lossless mode). (a) Figure 2.19: Comparing lossless compression rates of two example rendered image bitmaps by dierent codecs and by stroke-stream compressions. Subgure a) also contains stored in SVG format (last 2 columns). The painting process has a few more properties that make an interesting option. For example the transformation has the ability to decrease errors produced by standard DCT-based techniques. In Figure 2.21 there is an example of an oil painting (JPEG and painted versions), showing that the painting technique smooths the surfaces, preserving contours and main edges, producing a visually (b)

44 2.5. Scalability, coding, compression, storage 30 Figure 2.20: Lossless coding ratios of output stroke-series relative to lossless coded bitmaps of painted images. pleasant output. One should also notice that this "denoising" (more a smoothing) eect which comes with the use of the Painting technique is not a pre-dened goal or strategy we wanted to achieve, it's more a side-eect of the stochastic painting (SPT) method (ergo we don't just paint to lter the noise, the painting itself lters). Figure 2.21: JPEG coded oil painting segment (left) and the JPEG recoded with Paintbrush rendering (right) - showing the ability to mimic stroke styles and noise ltering eects.

45 2.6. Convergence Convergence A question that can pop up from time to time, and it is not a dismissable question, is that what are the convergence properties of the above used techniques for creating painterly rendered images, do they converge to something at all, is their convergence guaranteed? In this chapter we presented stochastic painting methods with certain extensions. That is, at the base of the above methods is a random process, because when strokes are placed on the canvas, their positions are randomly selected. As we already mentioned above, the process can be controlled by either a quality goal specied in db, when the process should stop, or a relative error measure, which means that if a certain change in relative error between two consecutive steps has not occurred, the process should also stop. The combination of these two conditions makes sure the painting process will stop. The time required to achieve a stopping condition depends mostly on the size of the model image and the used stroke scales.

46 2.6. Convergence 32 Stroke type Stroke emulation Approx. coded size [25] geometrical scaled geometrical templates 8 bytes/stroke strokes with sampled color [71] antialiased weighted scaled 8 bytes/stroke/line lines clipped templates aligned at edges to form lines [11] no strokes circular or uy templates 6 bytes/stroke [31] long curved curved strokes ( 10 bytes) strokes, replaced by series of for each gradient based smaller rectangular control point/stroke style, orientation templates [96] rectangular simple scaled 7 bytes/stroke antialiased rectangular templates strokes [33] short lines rectangular or line templates 9 bytes/stroke/line [7] no strokes, use ltering to determine 5 bytes/ltered ltering grayscale stroke template template [34] rectangular, mostly scaled rectangular 7 bytes/stroke or user stroke series dened binary bitmaps [15] following series of smaller 7 bytes/stroke/ edge-chains, geometrical strokes edge-chain curved strokes [24] central axis of seg- circular weighted 6 bytes/template mented areas stroke templates combined into tokens; Filbert brush simulation along tokens [54] user-dened user dened scaled 7 bytes/stroke grayscale (256 weighted templates levels) templates [53] any shape and -//- 6 bytes/stroke/line size of brush templates grown into lines with varying thickness Table 2.1: Possible similar emulations of mentioned stroke representations by a common painting model (multiscale, template-based, stochastic) for using similar compression schemes. Last column shows approximations of bytes needed to store a stroke if simulated by the model.

47 3. Painting with scale-space 33 Chapter 3 Painting with scale-space In this chapter we will present an new automatic painterly rendering approach which - being also partially built upon the previously presented approaches - is a method controlled by automatically extracted scale-space image features. These features are weighted edge and ridge maps extracted by a scale-space approach. Multiple eect generation variations will be presented, some stochastic, some a combination of stochastic methods with structural features, some fully sequential. The question could be asked, why should stochastic elements still sometimes be retained? The answer to that is both simple and complex. During our work we thought that retaining a level of randomness in the painting process retains a bit less articial touch. That is why we also present methods where e.g. the stroke position remains random while size and orientation is taken from local image structure information. In others the stochastic nature can fully be eliminated. Still, these methods are variations based on a common grounding, showing that the approach we took in creating these methods is that of trying to have more free hand in the rendering process. Thus fairly similar approaches can still produce dierent types of images. Optionally retaining randomness in the stroke placement process can give a somewhat more natural feeling, but in the case of other painting parameters like stroke size, orientation, color or weighting, randomness can cause disturbing unnatural eects. These latter parameters are best taken from image structure, color and texture data, which is what we try to include in the presented methods. The novelty of this approach, which dierentiates it from the previous works in the SBR literature and our previous methods, is in that it builds upon image structure information and the control of the whole image generation process is based on these extracted features. Previous

48 3.1. Rendering with multiscale image features 34 techniques have often incorporated some image analysis results in the rendering process (i.e. following edge directions or determining stroke patterns from the underlying image texture), but our approach was new in the sense that it remained an automatic eect generator, it can be fully controlled by image structure information (weighted edge and ridge maps), it can still be combined with all the previous approaches (stochastic or not, using templates or geometric shapes, using multiple layers or not). 3.1 Rendering with multiscale image features We have already seen that dropping the trial-and-error nature of stroke orientation and selecting orientations following edge directions can increase the eciency of the rendering. What we propose is to further increase this eciency by determining both stroke orientation and scale from image structures. For this purpose we use weighted edge and ridge maps of the model images, extracted by a variation of Lindeberg's method [69]. When thinking about the "natural" way of painting, one can hardly say that a painter creates a painting by randomly scattering the same stroke templates over a canvas, although this is a fairly well usable and behaving approximation that can be used for generating articial painterly renderings, as our methods - and others' - have already shown. But, for example, usually a painting starts by roughly sketching the contours, then lling the remaining areas. This idea is where we started from when considering development of a painting process which would stand closer to real life painting in its concepts, and which we will present in this chapter.. The basic idea behind this approach is that we should allow the painting process to be driven by important image features, which give structural information. Here, edges and ridges. The idea is to extract a suitable edge and ridge map from the model image and use them for automatic stroke positioning, scale and orientation determination. The goal was to obtain a technique which would not be stochastic and not need multiple stroke layers, but still remain fully automatic. This method tries to nd main edges by searching for edge positions which produce an accentuate curve in scale-space. Besides this, it provides more than other edge detectors, that is weighting the resulting edges and ridges where the weights reect how relevant the respective curve is.

49 3.1. Rendering with multiscale image features 35 by Given an image f, its Gaussian scale-space representation will be noted as L : R 2 R dened where g : R D R + R is the Gaussian kernel g(x, t) = L( ; t) = g( ; t) f (3.1) P 1 Di=1 x 2 i e 2t (3.2) (2πt) D/2 which will be used to generate the scales used in the detection process, t = σ 2 where t is the so called scale parameter and σ is standard deviation. More specically g((i, j); t) = 1 (i N 2πt e 2 )2 +(j N 2 )2 2t (3.3) where the Gaussian kernel has the dimensions N N (N Z + ) and i, j R [0; N) Z. The L(, t) representations are obtained by applying Gaussian scale-space derivatives, e.g. for two dimensional image f function and x directional derivative: L x ((x, y); t) = ( x g((x, y); t)) f(x, y) (3.4) At each scale level, edges are dened from points at which the gradient magnitude assumes a local maximum in the gradient direction. We use the edge denition when at a given point, if it is an edge-point then the second order derivative (L xx ) is zero and the third order derivative (L xxx ) is negative (for each respective direction, x for horizontal, y for vertical): { L xx = 0 L xxx < 0 We use the following geometrical ridge denition: (3.5) L x = 0 L xx < 0 L xx >= L yy or L y = 0 L yy < 0 L yy >= L xx (3.6) which is the formulation for 2 directions (horizontal and vertical). We extended this formulation with 2 more directions, the two diagonals. Using 10 scales generated with Gaussian convolution with dierent scale parameters we calculate scale-space derivatives. Then we search for all curves connected on the same (maximal) scale and calculate the edge strength formulated as:

50 3.1. Rendering with multiscale image features 36 and the ridge strength as: E = t γ (L 2 x + L 2 y) (3.7) R = t 2γ ((L xx L yy ) 2 + 4L 2 xy) (3.8) The γ normalization value used above is a normalization factor introduced in [1] by Lindeberg, having 1/2 for edges and 3/4 for ridges. Then signicance measures, edge/ridge weights are calculated, which is a line integral of weighted derivatives along the actual curve. S Ce = E(e)de and S Cr = R(r)dr (3.9) e and r being the individual edge and ridge curve points respectively. We store all connected curves on the image and sort them in descending order of their summed weights. At last we display the rst N (usually around 100) curves on an edge/ridge map. For an example of extracted edges and ridges see upper part of Figure 3.2. The edge map is used to determine placed stroke orientations, placing strokes to follow nearby edge directions. The ridge map is used for obtaining stroke size (scale) automatically (simple circular shaped strokes are used). We get the maximal and minimal ridge weight on the ridge map, and project the available stroke sizes over the weight interval in such a way that lowest weight corresponds to smallest stroke scale and highest weight to highest stroke scale. During the painting process, when a stroke needs to be placed its scale is determined by searching for the nearest 8 ridges and calculating the stroke scale by weighting the scales associated to the found ridge weights with the distances they are from the current stroke position. The actual stroke scale will be calculated as where S o = 8 i=1 [ d d i d S ai ] (3.10) S ai = S max [ w a i w min w max w min ] and d= 8 i=1 d i (3.11) where S ai are the weighted distances of the nearest 8 ridge points, the w values are the maximal, minimal and actual weights respectively obtained from the extracted ridge map. The d i values are the real distances of the ridge points in pixels.

51 3.2. Automation by feature control 37 Extending stochastic painting with the above technique can also lead to improvement in eciency. For example, Figure 3.1 shows that using scale-space extracted features to control the orientation and scale of the placed strokes can produce the same PSNR (see equation 2.1) in about half the time in this case. In some cases the dierence can be even higher, e.g. as Figure 3.4 shows for frames of a video sequence. The greatest dierence of the two compared methods is in the following: while in both methods stroke positions are stochastically picked, in stochastic multilayer painting each layer is further rened by a smaller stroke but in feature-controlled painting the orientations and scales of all future placed strokes are predetermined from the edge and ridge maps and all the painting is done in one layer. Figure 3.1: Extending stochastic painting to be controlled by internal image structure improves eciency: same PSNR quality (in this case 27.7dB for the Lena image and 28dB for the ower image in Figure 2.3) can be achieved in about half the time as fully stochastic painting requires. It has to be noted: this approach represents a turning away from stochastic painting techniques, reducing complexity and computational cost of stochastic algorithms, not needing built-in complex extensive optimizations for controlling the stochastic process. The main steps of this technique in Algorithm 3.1. A sample image painted with this technique is shown in Figure Automation by feature control Using the edge and ridge maps extracted by the method from the previous section, in this section we present three methods of painting, each using the information obtained this way to some extent.

52 3.2. Automation by feature control 38 load model image A; initialize canvas C; generate weighted edge and ridge maps E, R; while there are unvisited ridge points do select next ridge point r; calculate stroke color c and width w; search for nearest edge line l in E; draw a stroke from r towards l; end while there remain unlled areas do select next unvisited point p in C; select nearest ridge weights rw in R; interpolate stroke scale at p from rw; search for nearest edge line e from E; draw a stroke from p towards e; end Algorithm 3.1: Steps of the one-layer structure-based painting. Before the painting process starts, we generate the previously mentioned convolutions of the model image with the series of brush strokes, to achieve a blurred sequence which will be used for determining the color of the places strokes. During the painting process, color majority sampling (eq. 2.6) will be performed using these pre-generated images: if we wish to place a stroke on a given position, the blurred image version is taken which belongs to the stroke template, size and orientation in question, colors are counted from under the possible stroke position, and the most frequently occurring color is given to the stroke (eq. 2.6). The main steps of the rst painting approach are shown on the block diagram in Figure 2.1. In this process only the edge information is used, for stroke orientation selection. The steps are as follows: select a stroke set, select random stroke positions, get stroke orientation from the obtained weighted edge map, obtain stroke color, if the placement of the stroke does not induce errors in the image (the placement of the stroke reduces the image error in PSNR between the painting and the model over a threshold) then place the stroke, repeat stroke placement until no improvement over a threshold is obtained, then switch to a smaller stroke set and repeat the whole process until there are no more sets.

53 3.2. Automation by feature control 39 Figure 3.2: Sample of painted image with the multiscale edge/ridge extraction based one layer sequential painterly rendering model. Top: edge map and ridge map used, bottom: rst step, and after closing the holes. Stroke orientations are calculated from the extracted edge maps as described previously (by weighting the calculated orientations of nearby edges). An image generated using this method is shown in Figure 3.3. When using this type of painting, the speedup related to fully stochastic 10 layer painting (which means 10 possible stroke sizes/scales) is shown in Figure 3.4. The second method is stochastic in nature by the means that possible stroke positions are still randomly chosen, but both stroke scale (size) and stroke orientation are obtained from the extracted edge and ridge maps. The diagram of the algorithm is very similar to the previous algorithm (with diagram in Figure 2.1), the only dierence being in the "generate strokes" step. The mains steps are: select random stroke position,

54 3.2. Automation by feature control 40 Figure 3.3: Example of image painted following edge orientations (random stroke positions, sequential 10 layer stroke scales). Figure 3.4: "normal" is the fully stochastic painting method with 10 layers, "espt" is the new method with random stroke positions but following edge orientations when placing the strokes. It shows (for consecutive frames of the Mother and Son qcif video) that for achieving similar PSNR the new method needs much less time. get stroke scale from the weighted ridge map, get stroke orientation from the weighted ridge map, obtain stroke color,

55 3.2. Automation by feature control 41 place the stroke and repeat the process until there is no improvement in error between the model and the painting over a threshold, do post-processing: correct possible errors by a nal overpainting with the nest stroke size (size will be constant, orientation taken from the edge map). Stroke scales are calculated from the extracted ridge maps as described previously (by proportionally mapping the available stroke sizes over the obtained ridge weight minimum-maximum values). An image generated using this way of painting is shown in Figure 3.5. Figure 3.5: Example of image painted following edge orientation and scaling by ridge scales (with random stroke positions). This method is a bridge between stochastic painting and the following method which has no stochastic steps. While it still searches for stroke positions in a random way, placed strokes now follow the main image structure, thus processing time decreases further more, and the number of strokes needed to cover the canvas also drops. The third method we present is totally based and controlled by the edge and ridge maps obtained previously. Its block diagram is shown in Figure 3.6. The main steps are as follows: sequentially search for the next ridge point on the ridge map,

56 3.2. Automation by feature control 42 (a) from that position search for the nearest edge on the edge map, (b) obtain stroke color, get stroke width from the weighted ride map, (c) draw a longish stroke line with width obtained previously until the position of the found edge point, repeat this process until there are no more ridge points, go over the painted image and if there are areas left out (still blank) then do and repeat steps marked a,b,c with a constant stroke width to ll in the missing areas. When painting with this variant a circular stroke template is used with variable radius. Figure 3.6: Main steps of the 3rd painting approach. In the case of this third algorithm the ridge scales are used when determining the width of the stroke line drawn from the actual ridge point till the nearest edge point. The smallest of the available strokes is assigned to the minimum ridge weight, the largest to the maximum ridge weight and the rest are proportionally mapped between the two values. Images generated this technique are shown in Figures 3.7 and 3.8. When painting with the last method, we achieve multiple goals:

57 3.2. Automation by feature control 43 Figure 3.7: Example of image painted following edge orientations, scales determined from ridge scales, stroke positions are sequential. (a) Figure 3.8: Examples of image painted following edge orientations, scales determined from ridge scales, stroke positions are sequential. On a) the main edge contours are also overlaid. the painting process gets a reasonable speedup, because there is no need for stochastic generation of positions, scales or orientations, all of these parameters are automatically obtained from the edge/ridge maps, (b)

58 3.3. Convergence 44 we get a painting process that stands a bit closer to being "natural" in the way that it follows the main image structures. Also, this technique gives space for further development, like adding shading to "hills" (which are spaces between ridges and edges) and "valleys" (which are negative ridges). 3.3 Convergence In this chapter we have dealt with feature controlled painting techniques. These are basically sequential methods, non-stochastic in their nature, thus the question whether these methods will stop or not does not arise. However, when these methods get combined with stochastic elements (e.g. choosing the stroke positions randomly but using the weighted edge and ridge data in the painting process), the same conditions have to be applied as in the case of the stochastic painting processes in Chapter 2. That is, a relative error checking measure is included in the process, in order to stop the painting when no more change has occurred since the last check. But mostly, these feature-based painting variations simply cover the canvas in one or two steps in a single layer.

59 4. Focus maps 45 Chapter 4 Focus maps In this chapter we will introduce our relative focus map extraction technique, and present a painterly rendering method based on focus maps to control the level of detail of painted image regions. We will also propose other applications for the relative focus map extraction in image indexing and video feature extraction which will be detailed in Chapter 5. The presented relative focus map estimation method is a new technique, being the consequence of a new approach to the classical blind deconvolution method of [87]. The novelty of our approach lies in not using the iterative deconvolution formula to deblur images (i.e. reconstruct images from optical type degradation), but to construct a localized version of it and use its features to estimate local relative blurredness of image areas, automatically, and without any a priori information. As we will show, possible application areas can be various, from the painting uses to image and video feature extraction. 4.1 Automatic focus map extraction by blind deconvolution This section's topic is the novel use of blind deconvolution for automatic estimation of focused areas on ordinary images without a priori knowledge about the image or the shooting conditions, to demonstrate its usability and to propose some areas of application. Figure 4.1 shows an introductory example. Deconvolution techniques [2,3841,64,73,87,101] are widely used for reconstruction of images degraded by optics and channel noise. The blurring function which represents the distortion is the so called point spread function (PSF). Blind deconvolution [2,64] is used when there is no, or just estimated knowledge about the distortion. Application areas include tomography [40],

60 4.1. Automatic focus map extraction by blind deconvolution 46 Figure 4.1: Example for an extracted relative focus map (left: input, middle: a coarse 2D map, right: a corresponding class map). microscopy, medical, astronomical areas, aerial imagery [39]. There also exist applications of blind deconvolution for image sharpness evaluation [66]. From blind deconvolution methods we have chosen a Richardson-Lucy [73, 87] based implementation, which has good convergence properties and the iterative process can be controlled for our purposes. We exploit the capabilities within the deconvolution to estimate the blurring imposed on images, seeking to obtain local estimations of the image blurredness and use this information for image area classication. Thus we can obtain a so called region of relevance based segmentation and show that it can also be used for dierent applications, like level of detail controlled painterly rendering, image indexing, search and retrieval on image databases, focus tracking in image sequences, and so on. The method used is similar in goal with other depth map generation techniques [12,18,83] that usually use image series shot with dierent focus settings, which in our case are not available. Depth from defocus techniques are also used to generate three dimensional models of real world objects by taking a series of defocused images [19]. Binary foreground segmentation methods [10] also exist, where a wavelet-based approach is used to segment low depth of eld images. The method uses block mean and wavelet coecient variation for region segmentation, then rening with lower wavelet scales, and texture sharpness is implicitly considered. In our method, the classication feature of the local areas is extracted by localized blind deconvolution. Texture and local image variations are considered implicitly by the deconvolution process and also by using a localized contrast-weighted error classication. This extraction could also be combined with Markov random eld classication in regularization methods or wavelet- or frequency-based feature combinations. Our method tries to be an extraction method for relevant regions, with multiple possible applications like image indexing [57, 62] and surveillance tasks for automatically obtaining the main moving characters for tracking. Here we present new results and applications, with new error

61 Localized deconvolution overview 47 measure and classication, and numerical evaluation. Besides the proposed applications in indexing and feature extraction our approach could also help in foreground/background separation which is usually done by segmentation based on generalized texture parameters and morphology [26]. Also, a common technique for lming closeups and portraits in movies is to capture a close focused sequence of the actor upon a highly blurred background, which could be automatically identied, and used for tracking. In what follows, we will describe the localized blind deconvolution scheme used to obtain the estimations for the local point spread functions and used later in the extraction of the relevant regions. Then we will present the region segmentation process itself, and we will also propose some possible areas of application Localized deconvolution overview The deconvolution we use is a model based on a Bayesian approach with maximum likelihood minimization scheme [10], based on the works of Richardson [87], Lucy [73], Ayers and Dainty [2]. We use a localized scheme, which results in position dependent PSF estimations for local image areas. During the deconvolution process, we obtain PSF estimates which are varying over the set of local regions (i.e. blocks), every region will have it's own PSF. The usability of blind deconvolution for focus map estimation was introduced in [58,59] which we will introduct here in short. Let g(x) = h(x) f(x) be the observed image formed by the convolution of the unknown original image f with the unknown point spread function h. Given the observed image g, we search for the original image f which maximizes the Bayesian probability of observing g given the estimated f of the form P (f i g l ) = [P (g l f i )P (f i )] / j [P (g l f j )P (f j )] (indexes point to image pixels). Using the denition of the conditional probability and the above equation we can write P (f i ) = l P (f i g l ) = l P (f i g l )P (g l ) = l P (g l f i )P (f i )P (g l ) j P (g l f j )P (f j ) (4.1) From this form the following iteration scheme rst introduced by Richardson [87] can be derived P k+1 (f i ) = P k (f i ) l P (g l f i )P (g l ) j P (g l f j )P k (f j ) (4.2) k being the iteration factor, P (f i ) = f i / f, P (g l ) = g l / g and P (g l f i ) = P (h i,l ) = h i,l / h [87] based on the constancy of light energy distribution. Taking an optical/physical point of view, the above form can be conceived as normalized energy distribution [20] instead of probability and by

62 Localized deconvolution overview 48 also substituting h k f k = g k we arrive to the iteration form ] ] g f k+1 = f k [h k = f k [h k ggk f k h k (4.3) A similar iteration scheme for obtaining the point spread function can also be constructed [20,42]. The main steps of the double iteration scheme we use are in Figure 4.2. Figure 4.2: Steps of the iteration. We extended this iteration scheme [58] to make it suitable for localized PSF extraction for our focus area extraction purposes. Thus, the localized deconvolution gives an estimation of the PSF and the original image at every step (k), r denoting the location vector. Convolutions are performed locally around position r, with a region of support T: [ ] f k+1 (r) = f k (r) h k (r) g g k (r) [ h k+1 (r) = h k(r) γ f k (r) g g k (r) ] (4.4) We use the γ weighting in the PSF iteration for constraining the lter values, with the value γ = T g k(r). The double iteration's initial values are for f 0 (T r ) an average gray image with DC value equal to the DC of the observed g image, and for h 0 a constant unity circular guess. Most of the calculations are done in Fourier space, to decrease the computational cost involved with large convolutions. The constraints imposed during the iteration steps are only the most necessary ones, e.g. on the size of the PSF, pixel amplitudes and non-negativity in the space domain and on zero phase in the frequency domain [58]:

63 Error bound 49 in the space domain size of PSF (region of support constraint): spurious elements of h k outside the initial region of support will be zeroed during the iteration pixel amplitude bound 0 f k,r (x) 255 non-negativity for the PSF (h k (r)) and the image (f k ) in the frequency domain: zero phase of h k (r), so as not to induce phase distortion when ltering the image f k Error bound The convergence and stop conditions are important issues. The process is bound by the number of iterations: in each region stop at a given iteration step and calculate the region's local reconstruction error measure. Only a few iterations are needed, just to be able to dierentiate the areas. Starting with f 0, the more in the focus a spot is, the higher the distortion to g is at an early iteration. The localized deconvolution runs on small blocks. Thus the ill-posed [42] iteration process tends to be noisy with higher number of iterations. For the classication we stop at a low iteration count and we needed an error measure which consistently gives dierent values for dierently focused areas, and which is not much aected by the process's noisy nature. In [58] we used simple mean square error (MSE, g g k ) for this purpose, but later experiments have shown that MSE is sensitive to the noise coming from the ill-posedness of the iteration process which often causes uctuations in the classication. Thus we constructed a more stable error measure based on the orthogonality principle [80], considering the independence of noise and the estimated signal. This measure theoretically converges to zero and instead of simple block dierences gives the angle deviation error (ADE) of the measurement and estimation residual error. The main reason ADE proved to be better suited for our classications is that while MSE is a simple dierence measure, which can greatly vary and cannot provide a consistent scale, ADE gives the normalized angle of the reconstruction error which even in such situations provides a stable scale with a zero minimal value. While classication based on MSE values can show uctuations, ADE based classication remains consistent. The ADE measure has the following form: E(g, g k ) = arc sin < g g k, g > g g k g (4.5)

64 Error bound 50 Figure 4.3: Error curves for 8 neighboring blocks (each curve stands for one block) on a blurred texture sample (top) for the same blur with ADE (left), and MSE (right). Ideally, curves of the same measure should remain close to each other. and we use the obtained local angles to distinguish the areas. This measure is new in that it provides a normalized local reconstruction based dierence which is more suitable for our classication purposes. When using g and g k in the error measure in practice, the dierences in convergence of the ADE and MSE are not always evident, although classication results show ADE's superiority. But the convergence of ADE is clear on ground truth data, when we know the original f, we can measure the direct error through comparing f to f k. This case shows that ADE almost always converges to zero while MSE is usually disturbed by the error coming from the low-constrained iterations. E.g. Figure shows an example, where the textured image was blurred uniformly, and eight neighboring areas were iterated. Ideally, all curves of the same measure should remain close to each other (with minor variations because of the local texture dierences), but the example shows the MSE's instability which causes uctuations in the classication. The better performance of ADE on real images is evident from the experimental results, presented later. Thus, the superiority of ADE over MSE comes from the reduction of the uctuation in the relative error values of similarly blurred areas that caused similar areas to be classied as belonging to dierent classes. Simulation and segmentation results are in Figures 4.4, 4.7, 4.8, 4.10 respectively. A sample image showing comparison of extraction with MSE and ADE is in Figure 4.4, for 15 iterations. Extraction with ADE produces a more ne and detailed map.

65 Contrast weighting 51 (a) (b) (c) (d) Figure 4.4: Comparison of a) input image's b) MSE distance based extraction), c) PSF-based classication and d) the new angle deviation error metric. Contrast weighting Eq. 4.5 works well for multitextured images in depth with similar contrast. For improving the robustness of the error deviation (eq. 4.5) based extraction of focused areas for the general case, we use local contrast weighting. This gives the opportunity to assign the appropriate weight to relevant locations. For local contrast measurement we use the conventional contrast denition, with the form: C r (g r ) = g max{x T r } g min{x Tr } g max{x Tr} + g min{x Tr} g max{x Tr},g min{x Tr} being the minimum and maximum local image intensities in region T r at location r. We use the local contrast to calculate the local relative errors, later to be used in the classication: E r (g, g k ) = arc sin < g g k, g > g g k g C r (g r ) max r {C r (g r )} A sample image showing extraction dierences when processing with and without contrast weighting is in Figure 4.5, showing that higher contrast areas get higher weight. (4.6) (4.7) (a) (b) (c) (d) Figure 4.5: Example for focus map extraction with contrast weighting. a) input, b) focus map without contrast weighting, c) focus map with contrast weighting and d) the extracted relative focus surface.

66 Multiresolution, performance 52 Multiresolution, performance The deconvolution and focus extraction process is a quasi-multiresolution technique, in the sense that the processing can be performed from the pixel level through overlapping blocks up to large non-overlapping image blocks. On the pixel level deconvolutions are done at every pixel taking into consideration a local region of support and the focus map will have pixel level values. When using overlapping blocks, we use blocks, with overlapping set to 16 pixels. Figure 4.6 shows an example for pixel-, block- and overlapping block-based maps. (a) (b) (c) (d) Figure 4.6: Example for extraction on the a) input image, with maps on the b) pixel level, c) with overlapping blocks, and d) non-overlapping blocks. The motivation for also investigating block-based approaches was to reduce the computation need of the focus map extraction, since pixel-based map extraction can take almost a minute, processing time required on a image is around 19s for the overlapping- and 5s for the block-based calculations (on a 2Ghz PC). The level of detail and the computation time will depend on the desired map resolution Extraction of Focus Maps The goal of the image classication technique is to extract the focus maps, or so called regions of relevance from images, to automatically determine the areas of images where the attention focuses. To achieve this we use the presented localized blind deconvolution. During the classication step, the locally obtained relative error values (eq. 4.7) are used to separate the areas which are more in focus (c represents the number of classes), with the following linear formula: F (r) = c (E r(g, g k ) min{e r (, )}) max{e r (, )} min{e r (, )} We compared our extraction method to other methods usable for focus area estimation, e.g. gradient- or autocorrelation-based methods. In the present comparisons we consider only the basic image information content and do not exploit any specic a priori knowledge about texture (4.8)

67 Extraction of Focus Maps 53 or shapes. The use of higher order optimizations like wavelet coecients, Markov random elds, texture features, etc. could also be added, but we are considering the basic capabilities of the proposed method herein. Edge content and/or gradient based sharpness measures [17] exploit detections in edge changes for local and global sharpness measures, while autocorrelation methods also can provide local sharpness measure by checking how well neighboring pixels correlate. Practically, in-focus images contain many small correlated areas, having high central peaks. For a quick visual comparison see Figure 4.7, 4.7(b) being the deconvolution-based map. (a) (b) (c) (d) Figure 4.7: Visual comparison for a) input image, for b) deconvolution, c) autocorrelation and d) edge content based extractions. The proposed blind deconvolution based extraction and classication does not require any a priori information about the image contents, giving rened and well scaled relative focus estimations. Depending on edge measurements can give false results e.g. when there is a low blur dierence between image areas, and autocorrelation usually cannot provide enough area discrimination for images with textured content. Figure 4.8 contains samples of focus extraction with our method and the ones mentioned above for visual comparison. The rst (Fig. 4.8(a)) is an example for using the same texture with areas progressively more blurred (numbers show increasing Gaussian blur). The deconvolutionbased method can provide good segmentation and visually distinguishable relative scales of the blurred areas. The second gure (4.8(b)) shows an example where the image is constructed from four dierent textures and the same blur is applied through dierent areas. Our method can both segment the blurred areas from the rest of the image and also provide a relative scaling between the dierent textures, because of the contrast weighting. Figure 4.9 also shows another example for comparison, where texture areas were blurred and the focus maps were extracted. This example also shows the consistency of the deconvolution-based approach (second column).

68 Extraction of Focus Maps 54 (a) Figure 4.8: Example images showing blurred texture discrimination properties of our (top right), autocorrelation (lower left) and edge content based (lower right) methods. a) Top left is the input, numbers show increasingly blurred sub-areas. b) Top left is the input containing blurred areas shown with black borders. (b) Figure 4.9: Columns from left to right for each row: input image with a blurred area, deconvolution based map, autocorrelation based map, edge content based map. Higher intensity means higher detected focusness. In Figure 4.10 numerical evaluation is presented for comparing our deconvolution-based and the other approaches. We used texture-sets [103] of histogram-equalized Brodatz-tiles [8] for the rst two examples. Central areas (with black rectangles) are blurred with changing strength, and the segmentation power of the methods is checked through the masking error. A ground truth mask is generated of the hand-blurred regions, calculating the ratio between the blurred area and the whole image. Then the methods are compared by taking their results and generating a mask containing the most blurred area with the same ratio as in the ground truth. The error metric used

69 Extraction of Focus Maps 55 Figure 4.10: Evaluation (lower is better) of the performance of deconvolution/ autocorrelation/ edge content extraction methods. For deconvolution both the new error measure (deconv) and the mean square error approach (deconv-mse) are investigated. Horizontal axis shows increasing blur, vertical axis shows extraction error relative to the real size of the blurred area. Blurred areas are shown with black borders. Errors larger than 100% mean that areas outside the ground truth area were also falsely detected as being blurred, thus increasing the error. was the ratio between the extracted blurred areas and the ground truth (i.e. the real hand-selected and blurred areas) error(%) = 100 A extracted A real A real (4.9) where lower values mean better extraction. The horizontal axis represent the increasing radius of the applied blur. As the gures show, deconvolution based focus extraction with the ADE measure can give good results even from low blur to high blur and the others can only achieve similar good extraction for high levels of blur, when probably every technique would be able to dierentiate the blurred areas. Also, method can achieve this consistently, proportionally dierentiating blurred areas and identifying areas with the same blur. In Figure 4.9 we also have seen examples of consistent focus map extraction of the deconvolution method, and Figure 4.15 (at the end of this section) adds another set of comparison examples, in this case real life images, to show that the extraction is more consistent than correlation and edge based approaches not just on synthetic images. Figure 4.11 shows examples for extracted areas on images with varying content and texture. Figure 4.12 shows examples of using the focus extraction method to select focused targets in video. It also can track focus changes across a video scene (Figure 4.13). Of course, the question can arise, whether the above presented method could be used on high depth of eld images, where there is no, or just small dierence in blur among the image

70 Extraction of Focus Maps 56 Figure 4.11: Examples for focus extraction on images with various textures (top row: input, bottom row: respective focus maps). Figure 4.12: Two examples of focus extraction on video frames: top row shows the video frames, bottom row shows their respective focus maps (higher intensity means higher focus). areas. The answer to that is that is there is no calculable dierence in blur, the classication will depend on the eventual dierences in contrast of the image areas, since contrast also is part of the error measure used in the classication. If there is no dierence in either blur or contrast, the extracted focus map will of naturally not give any usable information, besides that the image has a at overall focusness of course. Figure 4.14 contains examples for images where there is no really percieavable focus dierence, still the classication can extract some image areas. Figure 4.13: Following the move of focus across a scene (focus shifted from the left of the image to the stairs in the background). Upper row are the input frames, bottom row are their respective focus maps with higher intensity meaning higher focus.

71 Extraction of Focus Maps 57 Figure 4.14: Map extraction results on some high depth of eld images. Figure 4.15: Map extraction results on images with a focused area on blurred background. Columns in each row from left to right: input, deconvolution based map, autocorrelation based map, edge content based map.

72 4.2. Painting with focus extraction Painting with focus extraction The method we describe in this section is a multilayer stroke based rendering technique. It is based on a modied version of the painterly transformation method from [55]. It uses two layers of painting and three stroke scales for generating the painterly output. The idea behind the approach is that usually not all areas and objects on a model image are equally important. Thus if we could generate the painterly output by giving more detail for the important areas and less detail for the less important areas, we could achieve new eects on one hand, and a faster and more compact painting on the other. The main steps of the method are the following: 1. Take input image A, blank canvas C. 2. Generate edge map E of model A for edge orientation data, by scale-space edge extraction [69], for extracting weighted edge map used during the painting process like in [53], as they were described in Chapter Generate relative focus map F of model A for relevant region extraction [58]. 4. Generate segmentation map for extracting similarly colored areas (described below). 5. Generate the background layer with the two large stroke scales (60 15 and 32 8 masks, in this case with rectangular stroke templates), with stroke orientations determined from E (for an example generated this way see Figure 4.17), or by using the colorized segmentation map as low-detail background (an example in Figure 4.19). 6. Rene the areas from F with the small scale stroke (10x3 mask). The input image A will be the model for the rendering process, while the canvas C is a blank image as a starting point. The edge map is generated with scale space edge extraction [69] for obtaining the strongest edges which will give the orientation of the strokes. As in [53] the orientation of the placed strokes will be determined from the nearest weighted edge direction. Thus main contours are preserved and artifacts caused by badly placed strokes are circumvented. For automatically extracting the relevant regions of the model images we use the properties of localized blurring functions obtained with our method based on blind deconvolution, described earlier.

73 4.2. Painting with focus extraction 59 We look at the model image as an original image with some distortion (g model image, f original, h distortion) g = f h and run a few iterations of localized blind deconvolution previously presented for obtaining an estimation of the local variation in focus/blur over the areas of the model image. Then we use the dierences measured by an error function in local blur reconstruction to classify the image areas relative to each other based on the estimated local focus. The main idea of this approach is that local blurring function estimates will vary according to the local blurredness/focusness of the image. For an example see Figure By using this methos we obtain a relative focus map of the model image, which we will use later. Figure 4.16: Samples for relative focus map extraction. Top: inputs and relative focus maps, bottom: relative focus surfaces with four classes.

74 4.2. Painting with focus extraction 60 Figure 4.17: Sample image generation by lling the background layer with larger strokes and rening the foreground with smaller strokes.

75 Color segmentation for quick background generation Color segmentation for quick background generation Our goal with color segmentation is twofold. First, it provides us the a regional segmentation map based on color similarity and at the same time a means to quickly generate low-detail background.. Secondly, to prepare the grounds for later cartoon-style 2D stroke-based rendering. Our approach is based on the hue-intensity based segmentation approach in [72,114]. The main steps of the color segmentation are: 1. Convert the model image's color space from RGB to HSV (hue, saturation, value) x = R G G y = R G B z = R B x u = 4 x+15y+3z y v = 9 x+15y+3z h = tan 1 v n /u n s = 13 u 2 n + vn 2 v = y 16 (4.10) (4.11) (4.12) with v n, u n being normalized for a specic illuminant and normalize the obtained values for the next step: h = 255(h h min /(h max h min ) s = 255s/s max (4.13) v = 255v/v max 2. Anisotropic diusion [84] eliminates small disturbances while enhancing the main structures, thus we use it on the hue and on the value planes (which contain color information which can be reliably clustered) to blur the image by space-variant blur with low eect on contours. For this purpose we use an implementation of the Perona-Malik AD equations; the goal is to produce such a blurred image on which the contours remain clear; the blurred areas will be used in the next steps for the area segmentation and also for colorizing the obtained segmented areas, by taking the majority of the occurring colors - majority sampling (eq. 2.6) - under a given cluster (from the diused image).

76 Color segmentation for quick background generation In this step we need to group similar colors into classes. For this purpose we use K-means clustering [75] to obtain segmentations where those areas are grouped together which are similar to the average over an area. The basis of the clustering is the combined diused hue and value planes. We generate color classes. 4. Extract and assign colors to the classes of the clustering: the most frequent color over the whole extracted class area. This is done by calculating color histograms of the respective cluster, pick the most frequent color and assign it to the pixels belonging to the cluster. For an example see Figure Figure 4.18: Samples for color segmentation and cluster colorization. Left column: input, middle: segmentation into 32 classes, right: colorized clusters.

77 The painting process The painting process The rendering process will construct the painted image basically by two layers: a coarser and a ner layer. Figure 4.19 shows an example for such generated image. Figure 4.19: Sample generated image (from a to e): model, segmentation map for background generation, colorized segmentation map, higher detail area, nal image. Thus after the focus map extraction the low-detail layer is generated. This phase can be achieved by two means: either by covering with larger scaled strokes, or by using the color map obtained by the color segmentation process described above. In either case, after the generated background the relevant regions are rened by the smallest stroke scale. This way the following outputs can be generated:

78 The painting process 64 painted on painted: background and extracted regions both are rendered; painted on original: painted extracted regions are drawn upon the original model; original on painted: the model image contents over the extracted regions are drawn upon the rendered background. The foreground rendering follows the steps outlined earlier. The renement painting process itself is done by placing strokes on the canvas, each stroke having parameters: type (template identication), size, position, orientation (determined from the extracted edge directions) and color (for each stroke the most frequent color under the stroke area). The process is controlled by the relative change in error caused by the placed strokes between the current state of the painting on the canvas and the previous state. States are checked after every thousand placed strokes. If the relative error change between two consecutive states is below a threshold (ε = 5, 0.7, for the respective stroke scales) we switch for the next layer. From the algorithm's point of view it does not matter what kind of articial strokes are used during the painting process. In the present painting simple rectangular stroke templates are used. Figure contains an example for these variations. Figure 4.21 shows example rendered outputs using the technique outlined in this paper (for the painted on painted case). Figure 4.20: Painted outputs. From left to right: input, and focus map; painted object on painted background; painted on original; original on painted. The above presented method builds the painted image by generating a coarser layer and a ner layer on top of it, by using three scales of strokes. Of course, the use of more layer scales could be possible, if more rened transition between the background and the ne layer is desired. But in this method our goal was exactly the production of a ner layer on a coarser one, thus achieving a more cartoonish feel.

79 The painting process 65 Figure 4.21: Rendering samples. Top: input images and their focus maps. Middle row: painted objects on low-detail painted background. Bottom row: painted objects on segmentation-colorized background.

80 5. Applications 66 Chapter 5 Applications This chapter provides a somewhat mixed collection of applications, all of them being applications of previously presented methods: rst about painterly rendering, then about focus map extraction. Of course some closely related applications of the respective techniques have already been presented in the text, here we will show further applications which did not t in or are somewhat farther related to the basic algorithms. 5.1 Creation and coding of stroke-based animations Litwinowitz in his work [72] uses optical ow data to obtain the areas which have to be updated between consecutive frames. The grid points are moved according to the motion data of the optical ow eld, and the strokes are pushed together with the grid points. As a post-processing step, there is an analysis of stroke density, and where strokes are too dense positioned (grid points move closer to each other) some of them are eliminated, generating new grid points and strokes where they become rare. Hertzmann [32] used a so-called cumulative dierence masking technique that is based on frame dierences, and motion data to obtain the areas which have to be updated. The grid points are warped then, using this data, and the areas are over-painted. The strokes are stored as cubic B-splines, are tessellated into triangle meshes and displayed with graphics hardware, obtaining appealing frame rates. This a fairly similar to our approach in [50], the main dierence is in that we did not use grid to align the strokes to, but allowed total freedom in the stroke placement, and using the extracted optical ow elds to obtain the motion areas which we updated between two frames.

81 5.1. Creation and coding of stroke-based animations 67 In Meier's work [76] surfaces are modeled as 3D particle sets, which are attached and stick to surfaces, moving along with them as motion occurs. This way coherence could be obtained when transforming image sequences (highly preferable when generating painted 3D animations). This approach is quite dierent from any other NPR video transformation method, since it is targeted to create coherent 3D NPR animations. In Hays et al.'s [29] a still and motion picture painting method is presented, based on generating meshes of brush strokes with varying spatio-temporal properties. Strokes have location, color, angle and motion properties. They use the motion to move strokes from the previous frame in the motion direction. This leads to a uid-like feeling of the videos, where areas of the frames seem to be in a constant uid-like motion. In our approaches (e.g. [50,54]) we use the motion elds to extract areas where motion has occurred between two frames, and only repaint these motion areas, to avoid such eects. Briey, our 2D video painting technique works as follows: take a keyframe and paint it fully (with the stochastic painting algorithm with extensions). Then do motion estimation to obtain optical ow, and use the motion areas to do a partial repainting on the areas where there was motion, using the optical ow data to specify the stroke orientations on these areas. We do not propagate or move the strokes placed on the previous frame along the motion trajectories as other techniques tried to do, because we did not want to do extra handling for the changes in stroke densities induced by these stroke-movements (we actually do not need to do this, because when repainting the motion areas and do the post-processing which eliminates over-painted and therefore hidden strokes the stroke density becomes generally uniform). Then we compress both the motion data (RLE) and the stroke-data (Human), and go to the next frame. We also do not use grids or special stroke placement prositioning, in order not to introduce any such limitations in the painting process. When doing the painting, we take into account the specied regions of interest (ROI) which can get higher detail in the painting process, the option to follow the edges on the frames (for achieving better contours) and the position of the cuts. ROIs can be dened for a single frame, for a series of frames or for the whole sequence, or even dierent ROIs for all the frames. The transformation be describe in the following is based on a multilayer stochastic approach. Of course, every other method presented in this work could be used for the frame-painting process.

82 Transformation details Transformation details To generate painterly rendered image sequences, our approach was to render the video keyframes, which are at least as frequent as the shot boundaries, then extract optical ow eld motion information from the frames in-between and repaint only the motion areas between two consecutive frames. We tried multiple motion detection algorithms to calculate optical ow (block matching, gradient-based [3, 30, 97] and hierarchical gradient-based [6] methods) to obtain the areas where motion occurs, and every one of them can be used for this purpose. After obtaining every motion eld, all keyframes are fully repainted. The frequency of the keyframes can be arbitrary, but they will be at least at the shot boundaries. Keyframes are important, because they can correct the eventual transformation errors, which are sometimes inevitable since the far from perfect motion detection algorithms. Between the keyframes we use the optical ow data and transform only the motion areas, placing strokes on the moving areas in the direction of the motion (or on edge direction). We use a simple, but fast cut-detection algorithm to detect cuts in the input video, and treat the rst frame after the cut as a keyframe. The cut-detection algorithm works as follows: calculate the summarized pixel dierences between two following frame intensities, for the whole input video or on a frame-window. At the same time we calculate the edge-maps of the frames and the dierences between the edge-maps of consecutive frames. We say we have a transition when the product of pixel- and edge-dierences reaches a threshold (which is the average of the dierenceproducts along the frames multiplied by a constant). This approach results in about 3 5% of cut false-positives, and has moderate ability for fade-in/out and dissolve detection (Figure 5.1). We also tested methods like edge-based contrast feature tracking and YUV-histogram-based methods [Lie99] for transition detection, and the RGB-histogram-based methods in [Han03]. This detection has some positive features: easy to implement, fast, detects cuts and short fades/dissolves correctly. The only drawback shows at longer fades/dissolves. In the worst case the errors induced by possible false transition detection only get corrected at the rst upcoming keyframe. There is also the possibility of specifying ROI (region of interest) areas for single and/or multiple frames, and ROI sequences (sequence of frames which have the same ROI), on single or multiple frames with cursor selection. ROI data (selected rectangular frame regions) are stored as distinct les and get associated with the video at the beginning of the transformation process. Of course the region selection method from Chapter 4 could also be used.

83 Transformation details 69 Figure 5.1: Cut detection used in the video painting process: cuts are detected by thresholding a function which is a weighted product of edge and pixel dierence maps of consecutive frames. The main steps of the video painting algorithm are as follows: 1. Setting of parameters (key-frame frequency, type of motion detection to use and other parameters like block radiuses, search ranges, thresholds, etc.). 2. Cut detection. 3. Edge detection and painterly transformation of frame F (i) to F (i), Human-compressing F (i) and writing of F (i) into the intermediate format. 4. Edge detection and optical ow calculation between F (i) and F (i + 1). 5. Transformation of the areas where motion occurred using the optical ow eld data calculated at the previous step, and the edge data if specied (Figure 5.2). 6. Writing the partially transformed frame data onto F (i), generating F (i + 1) and writing it into the intermediate format along with the motion data. 7. If the next frame will be a key-frame then increase i and jump to step 3, otherwise to step 4.

84 Transformation details 70 (a) (b) (c) Figure 5.2: Painting of videos: painted frame (a), motion eld extracted (b), generated frame from previous frame and motion eld (c). Frame data gets Human-encoded before storing into the output le (which we call mpv, or Motion Painted Video les) and the motion data is run-length encoded [112]. Using the data stored in the output le the painted video can be reconstructed stroke-bystroke, frame-by-frame. The average compression ratios of the output range from 5 to 20%. We have achieved considerable results when transforming cartoons in noise reduction (old cartoons with scratches, salt&pepper noise) and better representation (avoiding the typical artifacts of DCT), with good compression ratios. This could be an alternative way of cartoon representation and storage, but that remains to be seen. As we are talking about a way of generating image sequences that is video coding, we have to talk a bit about controlling the compression process. We cannot provide any simple bit rate controlling possibility, ways of controlling the process are through the: number of stroke-sets used for painting (usually 3 to 10), number of the placed strokes (upper limit), target PSNR (usually around 35dB), keyframe density (keyframes are fully repainted, thus being many times larger than others and thus inuencing the output size as well), relative error threshold (controls switching to a smaller stroke-set). With the number of used stroke-sets the size of the painted output can be inuenced. For example when using more of the smaller strokes the painted quality can be raised but the output size raises also because the amount of data needed to store a stroke if the same for big and small strokes, and we need more strokes to cover the canvas if the stroke's size is small. When changing the relative error threshold we indirectly tell the algorithm to which extent it should allow a strokeset to manipulate the painting. If we allow a greater threshold, the set-changes will occur earlier,

85 Transformation details 71 and the strokes remain more visible at the end. With a smaller threshold the quality of the painting will be higher, but it will certainly contain more strokes. A placed stroke can be described with its parameters (1 identication byte - for the stroke's set and type, 3 bytes of color data - no alpha, 2 or 3 bytes of position data - including horizontal and vertical coordinates). This representation (6 or 7 bytes for a stroke) stays the same for every type of stroke (let that be rectangular, or user-dened). The motion data gets stored in an easy-tohandle form as a frame-sized matrix, where each non-zero data means that there is motion, having one direction out of the 8 possible. The stroke-stream is Human-coded and the motion data is run-length encoded. We have to state here, that decoding (playing) the painted, stroke-series-based output is not a fast process yet (there is no real-time 24 fps decoding). Of course this is no problem is the frames with the painterly eects are raster-encoded into a standard video format for playing. As a possible future development, creating SVG-based vector graphical animations of painterly rendered image sequences is somewhat distant goal. Some work towards achieving this goal have already been done as we will show later this chapter. In Figure 5.3 compression data is shown, demonstrating that - as in the case of still image painting - compressing the rendered frames as a stroke-stream instead of encoding the resulting raster images is a more eective way of storing the painted videos. (a) Figure 5.3: Encoding the painted frames as a stroke stream losslessly is more ecient than encoding them as raster images with typical image coders in lossless mode.(espt stands for enhanced stochastic painting with edge following stroke orientations and layer lling, see Chapter 2). Numbers show average frame sizes for video samples. In Figure 5.4 a) there is data showing the relative sizes of three dierent painted videos when (b)

86 Transformation details 72 changing keyframe density (output fps - frame-per-second - is always the same as of the input fps), not using edge gradient-oriented stroke placement. Connected to these numbers, Figure 5.4 b) shows how transformation times behave when keyframe density is changed. (a) Figure 5.4: a) Variation of output size relative to uncompressed input avi size (in %) with changing keyframe density (every nth frame is keyframe) for videos painted with dierent no. of stroke-sets. b) Relation between coding time and keyframe density for a sample video. When generating the painted frames by extending the basic painterly method with edge direction following stroke orientations and also using the interpolative layer lling extension described in Chapter 2, improvements occur that can make the encoding of the frames more ecient, while not degrading painted image quality. Figure 5.5 contains graphs showing that by using the mentioned extensions in the rendering process more stable frame sizes can be achieved at the same painterly quality. Figures 5.6 and 5.7 contain some examples of rendered video frames. (b) Figure 5.5: Using the enhanced painting process (espt) with layer lling and edge following stroke orientations produces more stable frame sizes at the same quality which is better from the encoding point of view (normal stands for stochastic painting, espt stands for enhanced painting with edge following stroke orientation and layer lling).

87 Transformation details 73 Figure 5.6: Sample painted video frames. Figure 5.7: Sample painted video frames.

88 5.2. Stroke-rendered paintings Stroke-rendered paintings Another area of application for painterly transformation could be representations of real life paintings, as shown in Figure 5.8. Features like sharp edges, ne textures or high contrast transitions can be easily described by the correct templates By transforming these images with espt and losslessly compressing the stroke series we get a representation that can reproduce the painting, with no induced typical compression artifacts - that is besides the painterly nature itself of the transformation process. One has also the possibility to render dierent versions and representations of the original model, by using dierent templates (Figure 5.9(a)). Hertzmann et al. [34] presented a technique with similar goals. Their goal was to generate dierent style representations from a model, where the style's features were extracted from another model. In our approach, we still stick to our goal of having an automatic eect generator process, and we achieve the styles by picking a stroke template for the painting to work with. Both approaches can produce interesting painterly rendered images. Figure 5.8: Rendered oil painting (top:original) (Prole of a Woman / Zorka by József Rippl-Rónai, 1916). Figures 5.9(b) and 5.10 show some model-painted image pairs where the models were images of real paintings. The bottom images are close comparisons of the original and the painted versions. We can generally say that the painterly transformation preserves the painterly style of the model images and at the same time produces an easy to handle, scalable representation.

89 5.2. Stroke-rendered paintings 75 (a) Figure 5.9: a) Rendered painting with uy template. b) Examples of painterly rendered images (from top to bottom, top: model image, bottom: nal and comparison). (b)

90 5.2. Stroke-rendered paintings 76 Figure 5.10: Examples of painterly rendered images (from top to bottom, top: model image, bottom: nal) of real oil paintings.

91 5.3. Realtime painting viewer Realtime painting viewer As we already have mentioned in Chapter 4, our painterly rendered images are built by a series of strokes. These strokes have dierent parameters. In our methods the painted images are stored and encoded as a strokes series representation, thus the format we get is not an easily understandable raster image. Thus we have limited choices when it comes to viewing or handling these encoded images. One of the options one has is to write his own image viewer which we can use to view the stroke series based painterly rendered images. We experimented with an OpenGL based accelerated image viewer for testing purposes. This image viewer could view painterly rendered images which were painted with rectangular strokes, practically by displaying the strokes placed sequentially upon each other on a plane in 3D space. This test software was written in C++ without any optimization whatsoever and was able to produce multiples of realtime framerates on 2-3 year old OpenGL-capable video cards. A shot of this application in action is in Figure Figure 5.11: MPVPlay, a test OpenGL painted image viewer. Of course stroke series based binary encoded images still are hard to handle and no other viewer can handle them. One also can not edit or convert these images. Thus even though the test showed that well performing viewers could be constructed, their use and practicality is somewhat questionable. The next section tries to give an answer to the problems raised regarding portability and ease of handling of these images.

92 5.4. Rendering strokes into scalable vector graphics Rendering strokes into scalable vector graphics For solving the problems regarding portability and ease of handling painterly rendered images we suggest the use of scalable vector graphics (SVG) for describing and storing such images [60]. The SVG [93] syntax is a fairly simple yet feature-rich meta-language, providing means to describe the contents of a graphical vector image by the shapes it consists of, i.e. their geometry, position, orientation, color. There are some predened shapes, but free shapes and contour paths can also be constructed. The image description data itself is a text stream, consisting of the lists of the shapes and their parameters. SVG also supports the specication of multiple layers. In the following we will only discuss those features that are needed for the translation of the painterly rendered data into SVG, for the broad SVG feature set see [93]. For presentation purposes let us consider a case where a binary stroke template is used with rectangular shape, with sizes of 60x15, 32x8 and 10x3 pixels. In this case every element of the resulting stroke stream of the painting process will be a rectangle with its own position, orientation and color parameters. These need to be specied for the SVG representation, e.g.: <rect width="32" height="8" transform="translate(52,98) rotate(45.0,16,4)" fill="#f4028c" /> meaning that a 32x8 rectangle needs to be placed at coordinates (52,98), rotated by 45 degrees around the center of the rectangle, having a purplish colour. Apart from predened shapes, any shape can be user, e.g.: <path d="m L L z" fill="#f4028c" /> The above will produce a purplish colored triangle shape (codes mean: M - move to absolute position, L - line to absolute position, z - end of path). Bezier curves and elliptic curves can also be specied, thus stroke shapes can vary in a broad range. In our case of a series of rectangle strokes, the layers will consist of a series of rect specications, with three layers. Layers are specied with the

93 5.4. Rendering strokes into scalable vector graphics 79 <g id="layer1">... </g> syntax. Then, we encode the SVG text stream with a lossless coder (gzip, rar, etc.). The benet of the SVG representation over the binary stroke series storage is that we do not require a specic viewer application to view the stored paintings since there are many applications on many platforms that can handle SVG data. E.g. even some internet browsers have native SVG handling support. Other than that, it also provides a humanly readable content description instead of binary data. Also, in many cases the vectorial representation is smaller in size than the binary stroke series. As we stated above, the nature of the multilayer coarse-to-ne stroke-based painting makes possible the easy scaling of the painted images. Figure 5.12 contains such an example of a painterly rendered image translated into SVG and then displayed at dierent resolutions. We also present some numerical data showing the compactness of the painterly generated output, be that a binary stroke stream or an SVG translation. Figure 5.13 shows data comparing the compression results of a painted images by general image coders, our earlier painterly rendering method [55] and vectorial [93] representations. Here the point is to nd out how eciently the painterly rendered images can be losslessly coded, thus the output of the painterly transformation process is encoded with dierent coders. The graph shows that the binary stroke-series based representation is always the smallest, the SVG translation falling somewhat behind. Still, on many dierent types of images the SVG representation of the painterly rendered outputs can be more eciently encoded as the binary stroke series. Figure 5.14 contains comparison data of some of such examples (the images producing these results are in Figure 5.15). Other benets of rendering into SVG is that there are many applications that can view or modify such graphics, e.g. modern web browsers can view vectorial images (e.g. open source Firefox browser, Figure 5.16) and a fairly large number of applications can edit such images (e.g. Inkscape open source editor, Figure 5.17). There are also some drawbacks, e.g. it is hard to represent complex strokes in this format, although freeform templates could be constructed by the provided curve functions. Another problematic point is that when the templates do not have constant color, i.e. they are more like texture than color information, these stroke lling information can only be stored as small bitmaps which would make the size of the graphics considerably larger. Thus, should SVG be used for all painterly images, other methods for textural stroke representation should be investigated.

94 5.4. Rendering strokes into scalable vector graphics 80 Figure 5.12: Painterly rendered output as SVG, displayed in dierent sizes.

95 5.4. Rendering strokes into scalable vector graphics 81 Figure 5.13: Storing the painted images as stroke series is more ecient than coding them with image coders. Figure 5.14: In many cases the SVG representation is shorter than the binary stroke sequence.

Enhanced LIC Pencil Filter

Enhanced LIC Pencil Filter Enhanced LIC Pencil Filter Shigefumi Yamamoto, Xiaoyang Mao, Kenji Tanii, Atsumi Imamiya University of Yamanashi {daisy@media.yamanashi.ac.jp, mao@media.yamanashi.ac.jp, imamiya@media.yamanashi.ac.jp}

More information

Non-Photorealistic Rendering

Non-Photorealistic Rendering 15-462 Computer Graphics I Lecture 22 Non-Photorealistic Rendering Pen-and-Ink Illustrations Painterly Rendering Cartoon Shading Technical Illustrations Acknowledgment: Steve Lin April 25, 2002 Frank Pfenning

More information

Non-Photorealistic Rendering

Non-Photorealistic Rendering 15-462 Computer Graphics I Lecture 22 Non-Photorealistic Rendering Pen-and-Ink Illustrations Painterly Rendering Cartoon Shading Technical Illustrations Acknowledgment: Steve Lin April 25, 2002 Frank Pfenning

More information

Tracking Moving Objects In Video Sequences Yiwei Wang, Robert E. Van Dyck, and John F. Doherty Department of Electrical Engineering The Pennsylvania State University University Park, PA16802 Abstract{Object

More information

Visualization and Feature Extraction, FLOW Spring School 2016 Prof. Dr. Tino Weinkauf. Flow Visualization. Image-Based Methods (integration-based)

Visualization and Feature Extraction, FLOW Spring School 2016 Prof. Dr. Tino Weinkauf. Flow Visualization. Image-Based Methods (integration-based) Visualization and Feature Extraction, FLOW Spring School 2016 Prof. Dr. Tino Weinkauf Flow Visualization Image-Based Methods (integration-based) Spot Noise (Jarke van Wijk, Siggraph 1991) Flow Visualization:

More information

A Short Introduction to Computer Graphics

A Short Introduction to Computer Graphics A Short Introduction to Computer Graphics Frédo Durand MIT Laboratory for Computer Science 1 Introduction Chapter I: Basics Although computer graphics is a vast field that encompasses almost any graphical

More information

Graphic Design. Background: The part of an artwork that appears to be farthest from the viewer, or in the distance of the scene.

Graphic Design. Background: The part of an artwork that appears to be farthest from the viewer, or in the distance of the scene. Graphic Design Active Layer- When you create multi layers for your images the active layer, or the only one that will be affected by your actions, is the one with a blue background in your layers palette.

More information

The Scientific Data Mining Process

The Scientific Data Mining Process Chapter 4 The Scientific Data Mining Process When I use a word, Humpty Dumpty said, in rather a scornful tone, it means just what I choose it to mean neither more nor less. Lewis Carroll [87, p. 214] In

More information

Working With Animation: Introduction to Flash

Working With Animation: Introduction to Flash Working With Animation: Introduction to Flash With Adobe Flash, you can create artwork and animations that add motion and visual interest to your Web pages. Flash movies can be interactive users can click

More information

B2.53-R3: COMPUTER GRAPHICS. NOTE: 1. There are TWO PARTS in this Module/Paper. PART ONE contains FOUR questions and PART TWO contains FIVE questions.

B2.53-R3: COMPUTER GRAPHICS. NOTE: 1. There are TWO PARTS in this Module/Paper. PART ONE contains FOUR questions and PART TWO contains FIVE questions. B2.53-R3: COMPUTER GRAPHICS NOTE: 1. There are TWO PARTS in this Module/Paper. PART ONE contains FOUR questions and PART TWO contains FIVE questions. 2. PART ONE is to be answered in the TEAR-OFF ANSWER

More information

MMGD0203 Multimedia Design MMGD0203 MULTIMEDIA DESIGN. Chapter 3 Graphics and Animations

MMGD0203 Multimedia Design MMGD0203 MULTIMEDIA DESIGN. Chapter 3 Graphics and Animations MMGD0203 MULTIMEDIA DESIGN Chapter 3 Graphics and Animations 1 Topics: Definition of Graphics Why use Graphics? Graphics Categories Graphics Qualities File Formats Types of Graphics Graphic File Size Introduction

More information

A Tool for Multimedia Quality Assessment in NS3: QoE Monitor

A Tool for Multimedia Quality Assessment in NS3: QoE Monitor A Tool for Multimedia Quality Assessment in NS3: QoE Monitor D. Saladino, A. Paganelli, M. Casoni Department of Engineering Enzo Ferrari, University of Modena and Reggio Emilia via Vignolese 95, 41125

More information

Face Locating and Tracking for Human{Computer Interaction. Carnegie Mellon University. Pittsburgh, PA 15213

Face Locating and Tracking for Human{Computer Interaction. Carnegie Mellon University. Pittsburgh, PA 15213 Face Locating and Tracking for Human{Computer Interaction Martin Hunke Alex Waibel School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Abstract Eective Human-to-Human communication

More information

Fast and Robust Moving Object Segmentation Technique for MPEG-4 Object-based Coding and Functionality Ju Guo, Jongwon Kim and C.-C. Jay Kuo Integrated Media Systems Center and Department of Electrical

More information

BCC Multi Stripe Wipe

BCC Multi Stripe Wipe BCC Multi Stripe Wipe The BCC Multi Stripe Wipe is a similar to a Horizontal or Vertical Blind wipe. It offers extensive controls to randomize the stripes parameters. The following example shows a Multi

More information

Analecta Vol. 8, No. 2 ISSN 2064-7964

Analecta Vol. 8, No. 2 ISSN 2064-7964 EXPERIMENTAL APPLICATIONS OF ARTIFICIAL NEURAL NETWORKS IN ENGINEERING PROCESSING SYSTEM S. Dadvandipour Institute of Information Engineering, University of Miskolc, Egyetemváros, 3515, Miskolc, Hungary,

More information

B.A IN GRAPHIC DESIGN

B.A IN GRAPHIC DESIGN COURSE GUIDE B.A IN GRAPHIC DESIGN GRD 126 COMPUTER GENERATED GRAPHIC DESIGN I UNIVERSITY OF EDUCATION, WINNEBA DEPARTMENT OF GRAPHIC DESIGN Copyright Acknowledgements The facilitating agent of the course

More information

Texture Screening Method for Fast Pencil Rendering

Texture Screening Method for Fast Pencil Rendering Journal for Geometry and Graphics Volume 9 (2005), No. 2, 191 200. Texture Screening Method for Fast Pencil Rendering Ruiko Yano, Yasushi Yamaguchi Dept. of Graphics and Computer Sciences, Graduate School

More information

Image Compression through DCT and Huffman Coding Technique

Image Compression through DCT and Huffman Coding Technique International Journal of Current Engineering and Technology E-ISSN 2277 4106, P-ISSN 2347 5161 2015 INPRESSCO, All Rights Reserved Available at http://inpressco.com/category/ijcet Research Article Rahul

More information

MODELING AND ANIMATION

MODELING AND ANIMATION UNIVERSITY OF CALICUT SCHOOL OF DISTANCE EDUCATION B M M C (2011 Admission Onwards) VI Semester Core Course MODELING AND ANIMATION QUESTION BANK 1. 2D Animation a) Wire Frame b) More than two Dimension

More information

Course Overview. CSCI 480 Computer Graphics Lecture 1. Administrative Issues Modeling Animation Rendering OpenGL Programming [Angel Ch.

Course Overview. CSCI 480 Computer Graphics Lecture 1. Administrative Issues Modeling Animation Rendering OpenGL Programming [Angel Ch. CSCI 480 Computer Graphics Lecture 1 Course Overview January 14, 2013 Jernej Barbic University of Southern California http://www-bcf.usc.edu/~jbarbic/cs480-s13/ Administrative Issues Modeling Animation

More information

suggestive contours and abstracted shading Daniel Arias

suggestive contours and abstracted shading Daniel Arias suggestive contours and abstracted shading Daniel Arias LINES AND SHADES Shape, volume, shades and texture in drawing lines as nonphotorealistic rendering technique Luis Caballero Suggestive contours Many

More information

m ac romed ia Fl a s h Curriculum Guide

m ac romed ia Fl a s h Curriculum Guide m ac romed ia Fl a s h Curriculum Guide 1997 1998 Macromedia, Inc. All rights reserved. Macromedia, the Macromedia logo, Dreamweaver, Director, Fireworks, Flash, Fontographer, FreeHand, and Xtra are trademarks

More information

Visualization of 2D Domains

Visualization of 2D Domains Visualization of 2D Domains This part of the visualization package is intended to supply a simple graphical interface for 2- dimensional finite element data structures. Furthermore, it is used as the low

More information

This is an example of a fairly state of the art computer generated line drawing. It uses suggestive contours, slightly stylized strokes, and some

This is an example of a fairly state of the art computer generated line drawing. It uses suggestive contours, slightly stylized strokes, and some 1 This is an example of a fairly state of the art computer generated line drawing. It uses suggestive contours, slightly stylized strokes, and some visual emphasis effects, and its a pretty nice, effective

More information

A Learning Based Method for Super-Resolution of Low Resolution Images

A Learning Based Method for Super-Resolution of Low Resolution Images A Learning Based Method for Super-Resolution of Low Resolution Images Emre Ugur June 1, 2004 emre.ugur@ceng.metu.edu.tr Abstract The main objective of this project is the study of a learning based method

More information

3 hours One paper 70 Marks. Areas of Learning Theory

3 hours One paper 70 Marks. Areas of Learning Theory GRAPHIC DESIGN CODE NO. 071 Class XII DESIGN OF THE QUESTION PAPER 3 hours One paper 70 Marks Section-wise Weightage of the Theory Areas of Learning Theory Section A (Reader) Section B Application of Design

More information

TABLE OF CONTENTS. INTRODUCTION... 5 Advance Concrete... 5 Where to find information?... 6 INSTALLATION... 7 STARTING ADVANCE CONCRETE...

TABLE OF CONTENTS. INTRODUCTION... 5 Advance Concrete... 5 Where to find information?... 6 INSTALLATION... 7 STARTING ADVANCE CONCRETE... Starting Guide TABLE OF CONTENTS INTRODUCTION... 5 Advance Concrete... 5 Where to find information?... 6 INSTALLATION... 7 STARTING ADVANCE CONCRETE... 7 ADVANCE CONCRETE USER INTERFACE... 7 Other important

More information

Overview of the Adobe Flash Professional CS6 workspace

Overview of the Adobe Flash Professional CS6 workspace Overview of the Adobe Flash Professional CS6 workspace In this guide, you learn how to do the following: Identify the elements of the Adobe Flash Professional CS6 workspace Customize the layout of the

More information

Video compression: Performance of available codec software

Video compression: Performance of available codec software Video compression: Performance of available codec software Introduction. Digital Video A digital video is a collection of images presented sequentially to produce the effect of continuous motion. It takes

More information

Real-time pedestrian detection in FIR and grayscale images

Real-time pedestrian detection in FIR and grayscale images Real-time pedestrian detection in FIR and grayscale images Dissertation zur Erlangung des Grades eines Doktor-Ingenieurs(Dr.-Ing.) an der Fakultät für Elektrotechnik und Informationstechnik der Ruhr-Universität

More information

Visualizing Data: Scalable Interactivity

Visualizing Data: Scalable Interactivity Visualizing Data: Scalable Interactivity The best data visualizations illustrate hidden information and structure contained in a data set. As access to large data sets has grown, so has the need for interactive

More information

What is Visualization? Information Visualization An Overview. Information Visualization. Definitions

What is Visualization? Information Visualization An Overview. Information Visualization. Definitions What is Visualization? Information Visualization An Overview Jonathan I. Maletic, Ph.D. Computer Science Kent State University Visualize/Visualization: To form a mental image or vision of [some

More information

Blind Deconvolution of Barcodes via Dictionary Analysis and Wiener Filter of Barcode Subsections

Blind Deconvolution of Barcodes via Dictionary Analysis and Wiener Filter of Barcode Subsections Blind Deconvolution of Barcodes via Dictionary Analysis and Wiener Filter of Barcode Subsections Maximilian Hung, Bohyun B. Kim, Xiling Zhang August 17, 2013 Abstract While current systems already provide

More information

Agenda. TreeMaps. What is a Treemap? Basics

Agenda. TreeMaps. What is a Treemap? Basics Agenda TreeMaps What is a Treemap? Treemap Basics Original Treemap Algorithm (Slice-and-dice layout) Issues for Treemaps Cushion Treemaps Squarified Treemaps Ordered Treemaps Quantum Treemaps Other Treemaps

More information

Introduction to Computer Graphics

Introduction to Computer Graphics Introduction to Computer Graphics Torsten Möller TASC 8021 778-782-2215 torsten@sfu.ca www.cs.sfu.ca/~torsten Today What is computer graphics? Contents of this course Syllabus Overview of course topics

More information

Infographics in Illustrator CC

Infographics in Illustrator CC Infographics in Illustrator CC STC 1 STC 2 Background Information graphics or infographics are visual representations of information, data, or knowledge intended to present complex information quickly

More information

ACE: Illustrator CC Exam Guide

ACE: Illustrator CC Exam Guide Adobe Training Services Exam Guide ACE: Illustrator CC Exam Guide Adobe Training Services provides this exam guide to help prepare partners, customers, and consultants who are actively seeking accreditation

More information

A New Robust Algorithm for Video Text Extraction

A New Robust Algorithm for Video Text Extraction A New Robust Algorithm for Video Text Extraction Pattern Recognition, vol. 36, no. 6, June 2003 Edward K. Wong and Minya Chen School of Electrical Engineering and Computer Science Kyungpook National Univ.

More information

A HYBRID APPROACH FOR AUTOMATED AREA AGGREGATION

A HYBRID APPROACH FOR AUTOMATED AREA AGGREGATION A HYBRID APPROACH FOR AUTOMATED AREA AGGREGATION Zeshen Wang ESRI 380 NewYork Street Redlands CA 92373 Zwang@esri.com ABSTRACT Automated area aggregation, which is widely needed for mapping both natural

More information

Freehand Sketching. Sections

Freehand Sketching. Sections 3 Freehand Sketching Sections 3.1 Why Freehand Sketches? 3.2 Freehand Sketching Fundamentals 3.3 Basic Freehand Sketching 3.4 Advanced Freehand Sketching Key Terms Objectives Explain why freehand sketching

More information

Corel Painter 8 Liquid Ink Visual Guide

Corel Painter 8 Liquid Ink Visual Guide Corel Painter 8 Liquid Ink Visual Guide Contents Welcome to Corel Painter 8........... 3 Liquid Ink in Corel Painter 8.......... 4 Liquid Ink Controls in Corel Painter 8...... 7 General Section: Liquid

More information

R Graphics Cookbook. Chang O'REILLY. Winston. Tokyo. Beijing Cambridge. Farnham Koln Sebastopol

R Graphics Cookbook. Chang O'REILLY. Winston. Tokyo. Beijing Cambridge. Farnham Koln Sebastopol R Graphics Cookbook Winston Chang Beijing Cambridge Farnham Koln Sebastopol O'REILLY Tokyo Table of Contents Preface ix 1. R Basics 1 1.1. Installing a Package 1 1.2. Loading a Package 2 1.3. Loading a

More information

Digital Image Processing

Digital Image Processing Digital Image Processing Using MATLAB Second Edition Rafael C. Gonzalez University of Tennessee Richard E. Woods MedData Interactive Steven L. Eddins The MathWorks, Inc. Gatesmark Publishing A Division

More information

Adobe Illustrator CS5 Part 1: Introduction to Illustrator

Adobe Illustrator CS5 Part 1: Introduction to Illustrator CALIFORNIA STATE UNIVERSITY, LOS ANGELES INFORMATION TECHNOLOGY SERVICES Adobe Illustrator CS5 Part 1: Introduction to Illustrator Summer 2011, Version 1.0 Table of Contents Introduction...2 Downloading

More information

ECE 533 Project Report Ashish Dhawan Aditi R. Ganesan

ECE 533 Project Report Ashish Dhawan Aditi R. Ganesan Handwritten Signature Verification ECE 533 Project Report by Ashish Dhawan Aditi R. Ganesan Contents 1. Abstract 3. 2. Introduction 4. 3. Approach 6. 4. Pre-processing 8. 5. Feature Extraction 9. 6. Verification

More information

Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches

Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches PhD Thesis by Payam Birjandi Director: Prof. Mihai Datcu Problematic

More information

Linotype-Hell. Scanned File Size. Technical Information

Linotype-Hell. Scanned File Size. Technical Information Technical Information Scanned File Size Linotype-Hell Once you start scanning images and reproducing them as halftones, you find out very quickly that the file sizes can be enormous. It is therefore important

More information

Optimizing graphic files

Optimizing graphic files Optimizing graphic files Introduction As soon as I started using web-authoring tools, I realized that I should be careful to use graphics on the web. Well-designed graphics usually make the web site more

More information

CREATIVE AND FILE PREPARATION GUIDELINES

CREATIVE AND FILE PREPARATION GUIDELINES Kodak NexPress Dimensional Printing System CREATIVE AND FILE PREPARATION GUIDELINES Take printing to a new level The Kodak NexPress Dimensional Printing System offers the unique ability to create added

More information

Plotting: Customizing the Graph

Plotting: Customizing the Graph Plotting: Customizing the Graph Data Plots: General Tips Making a Data Plot Active Within a graph layer, only one data plot can be active. A data plot must be set active before you can use the Data Selector

More information

Anime Studio Debut vs. Pro

Anime Studio Debut vs. Pro vs. Animation Length 2 minutes (3000 frames) Unlimited Motion Tracking 3 Points Unlimited Audio Tracks 2 Tracks Unlimited Video Tracks 1 Track Unlimited Physics No Yes Poser scene import No Yes 3D layer

More information

ACE: After Effects CC

ACE: After Effects CC Adobe Training Services Exam Guide ACE: After Effects CC Adobe Training Services provides this exam guide to help prepare partners, customers, and consultants who are actively seeking accreditation as

More information

MassArt Studio Foundation: Visual Language Digital Media Cookbook, Fall 2013

MassArt Studio Foundation: Visual Language Digital Media Cookbook, Fall 2013 INPUT OUTPUT 08 / IMAGE QUALITY & VIEWING In this section we will cover common image file formats you are likely to come across and examine image quality in terms of resolution and bit depth. We will cover

More information

Drawing a histogram using Excel

Drawing a histogram using Excel Drawing a histogram using Excel STEP 1: Examine the data to decide how many class intervals you need and what the class boundaries should be. (In an assignment you may be told what class boundaries to

More information

Prerequisite: Completion of ENG 98, if required, and CGA 101 or written permission of instructor

Prerequisite: Completion of ENG 98, if required, and CGA 101 or written permission of instructor Salem Community College Course Syllabus Course Title: Digital Illustration Course Code: CGA 132 Lecture Hours: 2 Laboratory Hours: 2 Credits: 3 Course Description: Digital Illustration will increase student

More information

Template-based Eye and Mouth Detection for 3D Video Conferencing

Template-based Eye and Mouth Detection for 3D Video Conferencing Template-based Eye and Mouth Detection for 3D Video Conferencing Jürgen Rurainsky and Peter Eisert Fraunhofer Institute for Telecommunications - Heinrich-Hertz-Institute, Image Processing Department, Einsteinufer

More information

STATGRAPHICS Online. Statistical Analysis and Data Visualization System. Revised 6/21/2012. Copyright 2012 by StatPoint Technologies, Inc.

STATGRAPHICS Online. Statistical Analysis and Data Visualization System. Revised 6/21/2012. Copyright 2012 by StatPoint Technologies, Inc. STATGRAPHICS Online Statistical Analysis and Data Visualization System Revised 6/21/2012 Copyright 2012 by StatPoint Technologies, Inc. All rights reserved. Table of Contents Introduction... 1 Chapter

More information

A System for Capturing High Resolution Images

A System for Capturing High Resolution Images A System for Capturing High Resolution Images G.Voyatzis, G.Angelopoulos, A.Bors and I.Pitas Department of Informatics University of Thessaloniki BOX 451, 54006 Thessaloniki GREECE e-mail: pitas@zeus.csd.auth.gr

More information

Instructions for Creating a Poster for Arts and Humanities Research Day Using PowerPoint

Instructions for Creating a Poster for Arts and Humanities Research Day Using PowerPoint Instructions for Creating a Poster for Arts and Humanities Research Day Using PowerPoint While it is, of course, possible to create a Research Day poster using a graphics editing programme such as Adobe

More information

Keywords: Image complexity, PSNR, Levenberg-Marquardt, Multi-layer neural network.

Keywords: Image complexity, PSNR, Levenberg-Marquardt, Multi-layer neural network. Global Journal of Computer Science and Technology Volume 11 Issue 3 Version 1.0 Type: Double Blind Peer Reviewed International Research Journal Publisher: Global Journals Inc. (USA) Online ISSN: 0975-4172

More information

Adding Animation With Cinema 4D XL

Adding Animation With Cinema 4D XL Step-by-Step Adding Animation With Cinema 4D XL This Step-by-Step Card covers the basics of using the animation features of Cinema 4D XL. Note: Before you start this Step-by-Step Card, you need to have

More information

Part-Based Recognition

Part-Based Recognition Part-Based Recognition Benedict Brown CS597D, Fall 2003 Princeton University CS 597D, Part-Based Recognition p. 1/32 Introduction Many objects are made up of parts It s presumably easier to identify simple

More information

Proc. First Users Conference of the National Library of Medicine's Visible Human Project, active contours free forms Triangle meshes may be imp

Proc. First Users Conference of the National Library of Medicine's Visible Human Project, active contours free forms Triangle meshes may be imp Proc. First Users Conference of the National Library of Medicine's Visible Human Project, 1996 1 Creation of Finite Element Models of Human Body based upon Tissue Classied Voxel Representations M. Muller,

More information

1. Introduction MINING AND TRACKING EVOLVING WEB USER TRENDS FROM LARGE WEB SERVER LOGS. Basheer Hawwash and Olfa Nasraoui

1. Introduction MINING AND TRACKING EVOLVING WEB USER TRENDS FROM LARGE WEB SERVER LOGS. Basheer Hawwash and Olfa Nasraoui MINING AND TRACKING EVOLVING WEB USER TRENDS FROM LARGE WEB SERVER LOGS Basheer Hawwash and Olfa Nasraoui Knowledge Discovery and Web Mining Lab Dept. of Computer Engineering and Computer Science University

More information

Jiří Matas. Hough Transform

Jiří Matas. Hough Transform Hough Transform Jiří Matas Center for Machine Perception Department of Cybernetics, Faculty of Electrical Engineering Czech Technical University, Prague Many slides thanks to Kristen Grauman and Bastian

More information

Image Processing and Computer Graphics. Rendering Pipeline. Matthias Teschner. Computer Science Department University of Freiburg

Image Processing and Computer Graphics. Rendering Pipeline. Matthias Teschner. Computer Science Department University of Freiburg Image Processing and Computer Graphics Rendering Pipeline Matthias Teschner Computer Science Department University of Freiburg Outline introduction rendering pipeline vertex processing primitive processing

More information

Technical Drawing Specifications Resource A guide to support VCE Visual Communication Design study design 2013-17

Technical Drawing Specifications Resource A guide to support VCE Visual Communication Design study design 2013-17 A guide to support VCE Visual Communication Design study design 2013-17 1 Contents INTRODUCTION The Australian Standards (AS) Key knowledge and skills THREE-DIMENSIONAL DRAWING PARALINE DRAWING Isometric

More information

Intro to Excel spreadsheets

Intro to Excel spreadsheets Intro to Excel spreadsheets What are the objectives of this document? The objectives of document are: 1. Familiarize you with what a spreadsheet is, how it works, and what its capabilities are; 2. Using

More information

CATIA V5 Tutorials. Mechanism Design & Animation. Release 18. Nader G. Zamani. University of Windsor. Jonathan M. Weaver. University of Detroit Mercy

CATIA V5 Tutorials. Mechanism Design & Animation. Release 18. Nader G. Zamani. University of Windsor. Jonathan M. Weaver. University of Detroit Mercy CATIA V5 Tutorials Mechanism Design & Animation Release 18 Nader G. Zamani University of Windsor Jonathan M. Weaver University of Detroit Mercy SDC PUBLICATIONS Schroff Development Corporation www.schroff.com

More information

3D Drawing. Single Point Perspective with Diminishing Spaces

3D Drawing. Single Point Perspective with Diminishing Spaces 3D Drawing Single Point Perspective with Diminishing Spaces The following document helps describe the basic process for generating a 3D representation of a simple 2D plan. For this exercise we will be

More information

Lession: 2 Animation Tool: Synfig Card or Page based Icon and Event based Time based Pencil: Synfig Studio: Getting Started: Toolbox Canvas Panels

Lession: 2 Animation Tool: Synfig Card or Page based Icon and Event based Time based Pencil: Synfig Studio: Getting Started: Toolbox Canvas Panels Lession: 2 Animation Tool: Synfig In previous chapter we learn Multimedia and basic building block of multimedia. To create a multimedia presentation using these building blocks we need application programs

More information

Graphic Design Basics Tutorial

Graphic Design Basics Tutorial Graphic Design Basics Tutorial This tutorial will guide you through the basic tasks of designing graphics with Macromedia Fireworks MX 2004. You ll get hands-on experience using the industry s leading

More information

Clipping Plane. Overview Posterior

Clipping Plane. Overview Posterior Automated 3D Video Documentation for the Analysis of Medical Data S. Iserhardt-Bauer 1, C. Rezk-Salama 2, T. Ertl 1,P. Hastreiter 3,B.Tomandl 4, und K. Eberhardt 4 1 Visualization and Interactive Systems

More information

Vector storage and access; algorithms in GIS. This is lecture 6

Vector storage and access; algorithms in GIS. This is lecture 6 Vector storage and access; algorithms in GIS This is lecture 6 Vector data storage and access Vectors are built from points, line and areas. (x,y) Surface: (x,y,z) Vector data access Access to vector

More information

Help. Contents Back >>

Help. Contents Back >> Contents Back >> Customizing Opening the Control Panel Control Panel Features Tabs Control Panel Lists Control Panel Buttons Customizing Your Tools Pen and Airbrush Tabs 2D Mouse and 4D Mouse Tabs Customizing

More information

QUICK REFERENCE: ADOBE ILLUSTRATOR CS2 AND CS3 SECTION 1: CS3 TOOL BOX: PAGE 2 SECTION 2: CS2 TOOL BOX: PAGE 11

QUICK REFERENCE: ADOBE ILLUSTRATOR CS2 AND CS3 SECTION 1: CS3 TOOL BOX: PAGE 2 SECTION 2: CS2 TOOL BOX: PAGE 11 QUICK REFERENCE, ADOBE ILLUSTRATOR, PAGE 1 QUICK REFERENCE: ADOBE ILLUSTRATOR CS2 AND CS3 CS2 SECTION 1: CS3 TOOL BOX: PAGE 2 SECTION 2: CS2 TOOL BOX: PAGE 11 SECTION 3: GENERAL CONCEPTS: PAGE 14 SELECTING

More information

Graphic Design & PHOTOSHOP COURSE

Graphic Design & PHOTOSHOP COURSE Graphic Design & PHOTOSHOP COURSE WITH KBM MEDIA Web: www.kbmmediasolutions.com Email: info@kbmmediasolutions.com GRAPHIC DESIGN COURSES Illustrator CS5 Fundamental Course Outline Our course Illustrator

More information

Microsoft Word 2010. Quick Reference Guide. Union Institute & University

Microsoft Word 2010. Quick Reference Guide. Union Institute & University Microsoft Word 2010 Quick Reference Guide Union Institute & University Contents Using Word Help (F1)... 4 Window Contents:... 4 File tab... 4 Quick Access Toolbar... 5 Backstage View... 5 The Ribbon...

More information

Otis Photo Lab Inkjet Printing Demo

Otis Photo Lab Inkjet Printing Demo Otis Photo Lab Inkjet Printing Demo Otis Photography Lab Adam Ferriss Lab Manager aferriss@otis.edu 310.665.6971 Soft Proofing and Pre press Before you begin printing, it is a good idea to set the proof

More information

Image Content-Based Email Spam Image Filtering

Image Content-Based Email Spam Image Filtering Image Content-Based Email Spam Image Filtering Jianyi Wang and Kazuki Katagishi Abstract With the population of Internet around the world, email has become one of the main methods of communication among

More information

Deinterlacing/interpolation of TV signals. Elham Shahinfard Advisors: Prof. M. Ahmadi Prof. M. Sid-Ahmed

Deinterlacing/interpolation of TV signals. Elham Shahinfard Advisors: Prof. M. Ahmadi Prof. M. Sid-Ahmed Deinterlacing/interpolation of TV signals Elham Shahinfard Advisors: Prof. M. Ahmadi Prof. M. Sid-Ahmed Outline A Review of some terminologies Converting from NTSC to HDTV; What changes need to be considered?

More information

INTRODUCTION TO RENDERING TECHNIQUES

INTRODUCTION TO RENDERING TECHNIQUES INTRODUCTION TO RENDERING TECHNIQUES 22 Mar. 212 Yanir Kleiman What is 3D Graphics? Why 3D? Draw one frame at a time Model only once X 24 frames per second Color / texture only once 15, frames for a feature

More information

McAFEE IDENTITY. October 2011

McAFEE IDENTITY. October 2011 McAFEE IDENTITY 4.2 Our logo is one of our most valuable assets. To ensure that it remains a strong representation of our company, we must present it in a consistent and careful manner across all channels

More information

Design Elements & Principles

Design Elements & Principles Design Elements & Principles I. Introduction Certain web sites seize users sights more easily, while others don t. Why? Sometimes we have to remark our opinion about likes or dislikes of web sites, and

More information

Universal Simple Control, USC-1

Universal Simple Control, USC-1 Universal Simple Control, USC-1 Data and Event Logging with the USB Flash Drive DATA-PAK The USC-1 universal simple voltage regulator control uses a flash drive to store data. Then a propriety Data and

More information

To determine vertical angular frequency, we need to express vertical viewing angle in terms of and. 2tan. (degree). (1 pt)

To determine vertical angular frequency, we need to express vertical viewing angle in terms of and. 2tan. (degree). (1 pt) Polytechnic University, Dept. Electrical and Computer Engineering EL6123 --- Video Processing, S12 (Prof. Yao Wang) Solution to Midterm Exam Closed Book, 1 sheet of notes (double sided) allowed 1. (5 pt)

More information

animation animation shape specification as a function of time

animation animation shape specification as a function of time animation animation shape specification as a function of time animation representation many ways to represent changes with time intent artistic motion physically-plausible motion efficiency control typically

More information

Introduction. Chapter 1

Introduction. Chapter 1 1 Chapter 1 Introduction Robotics and automation have undergone an outstanding development in the manufacturing industry over the last decades owing to the increasing demand for higher levels of productivity

More information

IE Class Web Design Curriculum

IE Class Web Design Curriculum Course Outline Web Technologies 130.279 IE Class Web Design Curriculum Unit 1: Foundations s The Foundation lessons will provide students with a general understanding of computers, how the internet works,

More information

2013 Getting Started Guide

2013 Getting Started Guide 2013 Getting Started Guide The contents of this guide and accompanying exercises were originally created by Nemetschek Vectorworks, Inc. Vectorworks Fundamentals Getting Started Guide Created using: Vectorworks

More information

Correcting the Lateral Response Artifact in Radiochromic Film Images from Flatbed Scanners

Correcting the Lateral Response Artifact in Radiochromic Film Images from Flatbed Scanners Correcting the Lateral Response Artifact in Radiochromic Film Images from Flatbed Scanners Background The lateral response artifact (LRA) in radiochromic film images from flatbed scanners was first pointed

More information

Mean-Shift Tracking with Random Sampling

Mean-Shift Tracking with Random Sampling 1 Mean-Shift Tracking with Random Sampling Alex Po Leung, Shaogang Gong Department of Computer Science Queen Mary, University of London, London, E1 4NS Abstract In this work, boosting the efficiency of

More information

Guide To Creating Academic Posters Using Microsoft PowerPoint 2010

Guide To Creating Academic Posters Using Microsoft PowerPoint 2010 Guide To Creating Academic Posters Using Microsoft PowerPoint 2010 INFORMATION SERVICES Version 3.0 July 2011 Table of Contents Section 1 - Introduction... 1 Section 2 - Initial Preparation... 2 2.1 Overall

More information

A Proposal for OpenEXR Color Management

A Proposal for OpenEXR Color Management A Proposal for OpenEXR Color Management Florian Kainz, Industrial Light & Magic Revision 5, 08/05/2004 Abstract We propose a practical color management scheme for the OpenEXR image file format as used

More information

CS 4204 Computer Graphics

CS 4204 Computer Graphics CS 4204 Computer Graphics Computer Animation Adapted from notes by Yong Cao Virginia Tech 1 Outline Principles of Animation Keyframe Animation Additional challenges in animation 2 Classic animation Luxo

More information

Silverlight for Windows Embedded Graphics and Rendering Pipeline 1

Silverlight for Windows Embedded Graphics and Rendering Pipeline 1 Silverlight for Windows Embedded Graphics and Rendering Pipeline 1 Silverlight for Windows Embedded Graphics and Rendering Pipeline Windows Embedded Compact 7 Technical Article Writers: David Franklin,

More information

Adobe Certified Expert Program

Adobe Certified Expert Program Adobe Certified Expert Program Product Proficiency Exam Bulletin Adobe Photoshop CS4 Exam # 9A0-094 ACE Certification Checklist The checklist below will help guide you through the process of obtaining

More information

Get The Picture: Visualizing Financial Data part 1

Get The Picture: Visualizing Financial Data part 1 Get The Picture: Visualizing Financial Data part 1 by Jeremy Walton Turning numbers into pictures is usually the easiest way of finding out what they mean. We're all familiar with the display of for example

More information