A Survey of Stroke- Based Rendering

Tutoral A Survey of Stroke- Based Renderng Ths tutoral descrbes stroke-based renderng (SBR), an automatc aroach to creatng nonhotorealstc magery by lacng dscrete elements such as ant strokes or stles. Researchers have roosed many SBR algorthms and styles such as antng, en-and-nk drawng, tle mosacs, stlng, streamlne vsualzaton, and tensor feld vsualzaton. Ths tutoral attemts to make sense of the dsarate work n ths area by Ths tutoral descrbes several creatng a unfed vew of SBR algorthms, whch hels us dentfy the stroke-based renderng (SBR) common elements and the unque deas of each aroach. Moreover, algorthms. SBR s an resentng deas n ths fashon suggests ossbltes for future research. automatc aroach to Fgure 1 shows an SBR algorthm n acton. Startng from a hotograh, a collecton of brush strokes creatng nonhotorealstc are laced n a manner that matches the orgnal hotograh, and then magery by lacng dscrete rendered to have the aearance of elements such as ant an ol antng. 1, Although the detals vary, all SBR strokes or stles. algorthms create mages by lacng strokes accordng to some goals. The most common goal s makng the antng look lke some other mage for examle, n Fgure 1, I wanted to lace colored brush strokes to look lke the cture of the mountan. Another mortant goal s to lmt the number of strokes n some way so that the result wll look lke a antng. Otherwse, the algorthm could use many tny brushstrokes, roducng a good match to the source mage wthout much abstracton, but the result won t look lke a antng. Aaron Hertzmann Unversty of Toronto Fnally, once the algorthm laces the strokes, t can render them n some other form. In Fgure 1, the algorthm ddn t add texture untl after lacng the brush strokes. It comared the source hoto to some ntermedate mage wth a smlfed stroke model. Ths s both for effcency and aesthetc reasons. The man ont s that the fnal renderng may dffer from the way we exressed our goals about the mage. Fgure shows another examle of an automatc vector feld vsualzaton. 3 Here, streamlnes effectvely convey the vector feld s moton. To clearly llustrate the vector feld, the lacements should be laced evenly Fgure b was created wth the goal of makng the blurry verson as close to a constant gray value as ossble. For comarson, Fgure a shows stroke lacements on a regular grd wthout adjustment. Agan, we see that ths streamlne vsualzaton algorthm s an SBR algorthm: t laces strokes (streamlnes) accordng to secfed goals (to follow the vector feld and to match a target tone n the blurred mage). Usually t s not ossble to exactly meet all the goals. Hence, t s useful to have a way of tradng off the goals and quantfyng ther mortance. We can do ths by formalzng an SBR roblem as an objectve functon mnmzaton roblem. An objectve functon s a mathematcal formula that measures how good a renderng s; SBR algorthms attemt to mnmze objectve functons. For examle, t sn t ossble to lace the streamlnes n Fgure to acheve a urely constant tone n the gray mage. Instead, we can use as an objectve functon the devaton of the blurred mage from a constant mage. So far, I ve descrbed two dfferent SBR roblem statements one for anterly renderng and one for vsualzaton but sad nothng of how to desgn algorthms 1 Results of a stroke-based algorthm: source hoto, anted verson, and (c) fnal renderng. (c) 70 July/August 003 Publshed by the IEEE Comuter Socety 07-1716/03/$17.00 003 IEEE

Courtesy of Greg Turk and Davd Banks (c) Examle of an automatc vector-feld vsualzaton: vector feld, fnal renderng, and (c) blurred renderng. for these roblems. Two man aroaches to desgnng SBR algorthms exst. Greedy algorthms greedly lace the strokes to match the target goals. Otmzaton algorthms teratvely lace and then adjust stroke ostons to mnmze the objectve functon. A greedy algorthm roduced Fgure 1, and an otmzaton algorthm roduced Fgure. Ths s a somewhat unusual vew of these algorthms. I have sacrfced chronologcal orderng n ths tutoral n order to resent algorthms n ths way. Haeberl ntroduced both a semautomatc greedy algorthm and an automatc otmzaton algorthm n a semnal aer. 4 The subsequent en-and-nk algorthms develoed together by Mchael Salsbury, Davd Salesn, Georges Wnkenbach, and others demonstrated the otental of hghly automated SBR to create beautful and exressve magery. 5-8 Dgtal ant systems had revously automated some asects of the stroke renderngs, but ddn t automate any stroke lacement choces. Although ths tutoral focuses on the techncal detals of SBR algorthms, t s mortant to remember that they re useless wthout human control. Every asect of the system (ncludng the choce of stroke models, the settng of weght arameters, and the selecton of nut magery) requres aesthetc decsons that only an artst, workng toward some goal, can make. Ideally, a human artst usng the system should have total control over the decsons made. For examle, a user should be able to secfy satally varyng styles to use, so that dfferent renderng styles are used n dfferent arts of the mage, or to secfy ostons of ndvdual strokes. However, one of the great advances n art n the age of dgtal machnes s the ablty to create comlex systems of rocedural art, where the artst doesn t drectly create the fnal work, but rather creates rules accordng to whch fnal decsons are made (although smler rocedural works such as those of John Cage exst wthout comuters). Hence, an artst mght desgn the energy functon, but not necessarly edt every ndvdual mage roduced by the algorthm. In one ossble scenaro, the artwork s creaton mght occur at a tme after the artst s nvolvement. The man goal of SBR algorthms s to rovde rocedural tools that automate arts of the mage creaton rocess, not to relace the artst. Stroke-based renderng Let s begn wth a few defntons. Frst, we need to defne what our strokes can look lke. A stroke s a data (x,y) r structure that can be rendered n the mage lane. A stroke model s a arametrc descrton of strokes, so that dfferent arameter settngs roduce dfferent stroke ostons and aearances. For examle, one form of stlng uses a smle stroke model (see Fgure 3). A stle s a stroke that can be descrbed wth two arameters: the (x, y) oston of the stle n an mage and the radus r of the stle. (As we shall see n the Vorono algorthms secton, other defntons of a stle are ossble.) We create mages by combnng strokes nto an mage structure. An mage structure s a data structure contanng a canvas, defned by a background color or texture, and an ordered lst of strokes, defned by ther arameter settngs. To create an mage, the lst of strokes s rendered by alha-comostng over the background. The background s usually just a sold color or a redefned texture mage. For examle, a PostScrt fle contanng only lne art s an mage structure because t contans a lst of stroke defntons; the data n the fle can be rendered on the screen or on a rnter. Fnally, we use an SBR energy functon to quanttatvely evaluate how good a renderng s. An SBR energy functon s a functon E : I R, where I s the set of ossble mage structures and R s the set of real numbers. An energy functon E(I) takes an mage structure as nut and oututs a number ndcatng the qualty of the mage generally, the goal of an SBR algorthm s to roduce an mage wth the smallest ossble energy. The energy functon s sometmes also called an objectve, cost, or error functon. The term energy comes from analogous uses n hyscs, such as searchng for the mnmum energy confguraton of a set of artcles. SBR algorthms are normally defned n terms of some nut data, usually an nut mage. In most algorthms descrbed here, the energy functon measures how 3 Detal of a stlng stroke model. Indvdual strokes (stles). IEEE Comuter Grahcs and Alcatons 71

Tutoral closely the renderng matches some nut mage. Addtonally, the energy functon encodes trade-offs. For examle, a anterly renderng algorthm takes an nut mage and roduces an mage structure contanng color ant strokes that matches the source mage. However, you could get a erfect match to the source mage by lacng thousands of tny brush strokes, whch would look nearly dentcal to the source mage. You can create a more nterestng antng by addng an abstracton term to the energy functon that s, by assgnng hgher energes to antngs that use less strokes. For examle, the energy functon could be E ( I) = E ( I) + w E ( I) E= I( x, y) S( x, y) match E ( I) = the number of strokes n I abs match abs abs ( x, y) I where w abs s a scalar weght arameter, and E match(i) s the sum-of-squared dfferences between the source mage and the renderng. Ths energy functon has one arameter, w abs. You can control the level of abstracton n the antng style by adjustng arameter values: settng w abs to be small secfes that we want a realstc style (reroducng the orgnal mage as close as ossble); settng w abs to be large secfes an abstract style (caturng the mage wth few strokes). We can then defne an SBR style as a stroke model and energy functon (ncludng arameter settngs) for mage structures. In other words, an algorthm creates an mage n a secfc style by mnmzng the corresondng energy functon. Note that ths s a broadly nclusve noton of style n ths vew, changng the arameters to a gallery effect n an magng tool consttutes changng the style (although not by very much). Ths framework s goal s to rovde a common structure wthn whch you can create and aly many styles. Otmzaton algorthms Two knds of otmzaton algorthms have been aled to SBR. The frst knd, whch I ll call Vorono algorthms, exlots secal roertes of the SBR roblem to erform effcent global udate stes. The second knd, whch I ll call tral-and-error algorthms, assumes no secal structure and erforms heurstcally chosen tests to try to reduce the energy. Generally the Vorono algorthms are effectve and fast, but we can t aly them to all roblems. The tral-and-error algorthms are general-urose, but at the cost of substantal comutaton tmes. (Both of these aroaches have also been called relaxaton algorthms.) Vorono algorthms Vorono algorthms are useful for SBR roblems where the fnal mage wll contan many dentcal nonoverlang strokes and where only stroke densty s constraned. The central dea s to use effcent technques from comutatonal geometry to lace evenly saced strokes nto an mage. Moreover, we can make these technques fast usng grahcs hardware. However, these algorthms don t drectly otmze wth resect to an mage-based metrc (that s, where the renderng should match target tones) snce the energy functon s defned n terms of stroke denstes. Lloyd s method. How can you create a set of evenly saced onts wthn an mage? By usng an teratve otmzaton rocedure, whch requres defnng an arorate energy functon. Let = (x, y) be xel locatons n an mage, and let C be secal ont locatons called centrods; the strokes wll eventually be laced at these locatons. Let L {0,1} be a bnary labelng of xels: f L =1, then the xel has been assgned to centrod. Every xel s assgned to exactly one centrod: = 1 L The goal s to choose both a set of centrods and a labelng that mnmzes the energy functon: E ( I) = L C I = L (( C ) + ( C ) ) where the centrods and labelng are mlctly members of I. (Nonhotorealstc renderng researchers have tycally resented the contnuous verson of ths energy. I refer the dscrete verson because t more closely reflects the roblem actually beng solved. One beneft s that we can rove convergence of the dscrete verson of the algorthm, whereas convergence hasn t been roven for the contnuous verson.) In short, we want every xel to be close to ts assgned centrod. If we knew the set of centrods C n advance, then comutng the otmal labelng would be easy we just assgn every xel to the nearest centrod. The resultng labelng s known as a Vorono dagram (see Fgure 4a) t arttons the lane accordng to whch centrods C are nearest to each ont. From lookng at Fgure 4a, t should be clear that ckng some randomly chosen ont set and then comutng the Vorono dagram doesn t gve a good arrangement of centrods. In fact, we can mrove uon ths set of centrods by adjustng the ont centers to best ft ths artton n other words, by holdng fxed the labelng and otmzng the energy functon wth resect to the centrods. The new otmal centrods are gven by = L C L / x x y y Ths s just the mean of the xel locatons that are assgned to C ; ths formula s easly obtaned by settng E(I)/ C = 0 and solvng for C. We can terate these two stes, whch s known as Lloyd s method: functon LLOYDSMETHOD(n, I): ntalze the centrods C by randomly samlng n onts unformly n the mage I whle the algorthm has not converged reestmate the labelng by 7 July/August 003

1 L 0 reestmate the centrods by C L / return the centrods C Fgure 4b shows the same onts after alyng Lloyd s method. The algorthm converges when the energy doesn t change between stes. However, the algorthm s guaranteed to reduce the energy at every ste before convergence because each ste mnmzes the energy wth resect to some arameters. As a result, the algorthm s guaranteed to converge because the energy decreases at every ste before convergence and because there s a fnte number of ossble labelngs L. Lloyd s method was dscovered searately by the sgnalrocessng communty (where t s known as vector quantzaton) and the machne learnng communty (where t s known as k-means clusterng). Runnng ths otmzaton n software over an entre mage can be qute slow. However, some researchers 9,10 have used grahcs hardware to make the rocess fast. Varatons on Lloyd s method. Now that we have a rocedure for regular lacement of onts, we can easly desgn SBR algorthms on to of t. Perhas the smlest SBR roblem to descrbe s stlng. Deussen et al. 11 resented the frst such method, usng stles to aroxmate gray tones n a target mage (see Fgure 5). In ther method, they manually segment the mage nto dstnct regons. They lace stles evenly wthn each regon, by alyng Lloyd s method to each regon searately. The centrods are ntalzed usng a half-tonng algorthm. Once the centrod locatons are chosen by Lloyd s method, they re relaced wth stles. The sze of each stle s set roortonal to the gray level of the mage underneath t. Secord 1 resents an alternate stlng style and algorthm, by varyng the dot sacng nstead of the dot sze (see Fgure 5d). The dea s to defne a satally varyng densty functon κ() that determnes how dense the stlng should be n dfferent arts of the mage. Ths densty functon s drectly derved from the tones of the target mage T(), that s, κ() = 1 T()/m, where m s the max gray level n T. The new energy functon s E ( I) = L κ( ) C I = L κ( )(( C ) + ( C ) I = argmn C otherwse L x x x y Courtesy of Adran Secord Courtesy of Adran Secord Courtesy of Olver Deussen et al. 4 Vorono dagram of a set of onts C. The mage s arttoned nto a set of regons, one regon for each ont. Each regon contans all xels that are closest to the corresondng ont C. By alyng Lloyd s method, the onts are adjusted so that they le at the centrod of ther regon of the Vorono dagram. The resultng ont set s evenly saced and can be used to secfy regular stroke lacements. (c) 5 Stlng algorthms. Source mages are shown n and (c), results from Deussen et al. 11 are shown n, and from Secord 13 n (d). To match the target gray tones, Deussen et al. s algorthm vares stle sze (keeng stle densty constant), whereas Secord s algorthm vares stle densty. Stle sze s also adjusted as a ostrocess. The eye n the grasshoer mage was manually segmented from the rest of the mage. (d) Followng the same stes resented revously drectly leads to a slghtly dfferent verson of Lloyd s method. The labelng ste s the same, but the centrods are now reestmated as C Lκ( ) / Lκ( ) Ths summaton can be accelerated by recomutng sums of κ(). Ths method gves somewhat sharer mage boundares, snce the stle lacement s drectly affected by the source mage s ntensty. Secord also uses a smler ntalzaton rocedure based on rejecton samlng. IEEE Comuter Grahcs and Alcatons 73

Tutoral (c) 6 Tle mosac results from Hausner. 14 Persectve vew of the vector feld used for the ynyang examle. The vector feld was generated from the heght feld shown. Intal Vorono dagram of randomly laced onts. (c) Fnal tlng. Edges shown n whte are excluded from the otmzaton. (d) Rendered tlng, usng colors from a source mage. (e) Tlng of a Lyban Sbyl mage. (d) (e) Courtesy of Alejo Haunser Secfcally, the algorthm samles ont locatons from a unform dstrbuton and ncludes the samled onts n the ntalzaton wth robablty roortonal to ρ(). Lloyd s method can also create tle mosacs from color source mages. A smle aroach s to create a Vorono dagram of an mage and then color each regon of the mage by the color from the underlyng source mage. 10 However, ths roduces a mosac wth rregular tle shaes. Hausner 14 descrbes two enhancements to ths method (see Fgure 6). Frst, to generate square tle shaes, relace the L norm wth the L 1 norm ( v 1= v x + v y ). Second, to create tlngs wth consstent orentatons (see Fgure 6a), secfy an orentaton feld φ() for the mage. The orentaton of each tle s constraned to match the vector feld: φ =φ(c ). The new energy functon s now I φ( C) 1 E ( I) = L R ( C ) where R φ(c) s a rotaton matrx wth orentaton φ(c ). We can create a new otmzaton rocedure as follows: functon TILEMOSAIC(n, I): ntalze the centrods C by randomly samlng n onts unformly n the mage I whle the algorthm has not converged reestmate the labelng by = R C L C 1 argmn φ( )( ) 1 0 otherwse reestmate the centrods by C L / L return the centrods C Note that Hausner s algorthm s no longer otmal snce the centrod udate ste sn t guaranteed to mrove the energy functon. The algorthm frst selects tle ostons and orentatons, and then colors the tles. The tle ostons and orentatons don t take color nformaton nto account. Nonetheless, the algorthm tends to acheve good results n ractce. Wth Hausner s algorthm, we can aly tlng to manually segmented regons, as before. We can also enhance edges by removng them from the energy functon. Secfcally, onts that le on mage edges aren t ncluded n the labelng or centrod comutaton stes; ths dscourages Vorono regons from straddlng edges. Adjustng the energy functon n varous ways can modfy tle szes and shaes; the resultng roblem s amenable to hardware acceleraton. 14 Tral-and-error algorthms It s dffcult to extend Vorono methods to take color nformaton nto account and handle roblems where strokes mght overla. So far, the only otmzaton methods alcable to these roblems are tral-anderror aroaches, whch you can aly to any SBR roblem. The dea s smle. The algorthm rooses a change to the mage structure. If the roosed change reduces the energy, then the change s ncororated; 74 July/August 003

Courtesy of Paul Haeberl 7 Images comuted usng traland-error algorthms. 4 Overlang rectangular strokes. Vorono dagram of a set of ont centers. otherwse, t s dscarded. The algorthm then reeats. If the roosal mechansm s well desgned, then the algorthm should eventually converge to a low-energy result. However, there are no guarantees that ths wll haen, and, even f t does, the comutaton tme could be substantal. Here s the seudocode for a tral-anderror algorthm: functon TRIALANDERROR(I): I emty mage structure whle not done C SUGGEST() //Suggest change. f (E(C(I)) < E(I)) //Does the change hel? I C(I) //If so, adot t. return I Termnaton condtons are u to the user. For examle, the otmzaton can run for a fxed amount of tme, untl the user s satsfed wth the results, or when only a small orton of the roosals are acceted. Of crtcal mortance s the desgn of a good roosal mechansm. Purely random roosal mechansms can waste substantal tme makng lttle rogress, whereas hand-tuned mechansms quckly rovde better results. Tral-and-error algorthms are closely related to greedy algorthms, snce they both use hand-desgned roosals. The man dfferences are that the tral-and-error algorthms nclude checks to ensure that the roosal actually mroves the mage and that the rocedure can teratvely mrove revous strokes. Hence, usng tral-and-error algorthms frees you from the dffcult task of desgnng a mechansm that always makes good strokes. Haeberl 4 ntroduced the frst tral-and-error algorthm for nonhotorealstc renderng; Fgure 7 shows the results. In each case, a fxed number of strokes are randomly erturbed, and the erturbatons are ket only f the sum-of-squares dfference to the source mage s reduced. Streamlne vsualzaton by tral and error. More recently, Turk and Banks 3 demonstrated a traland-error algorthm for vector feld vsualzaton of streamlnes (see Fgure ). (Jobard and Lefer later descrbed a greedy streamlne lacement algorthm that s much faster than the tral-and-error method. 15 I descrbe Turk and Banks method here for comleteness.) As mentoned n the ntroducton, the roblem s to llustrate a vector feld wth streamlnes for clear vsualzaton of the vector feld. However, a straghtforward aroach to the roblem smly tracng streamlnes from some redetermned startng onts creates rregular streamlne sacng that dstracts from the vector feld s flow. Hence, you need some way of evaluatng the streamlne vsualzaton s qualty and then otmzng for that qualty measure. Turk and Banks roosed blurrng the streamlne renderng, comarng the result to a redefned constant value t. The corresondng energy functon s E ( I) = (( G I)( ) t ) I where (G I)() denotes the blurred verson of the streamlne mage. (Salsbury et al. used a smlar energy functon n an earler study. 5 ) You could also enalze the devatons of the streamlne from the vector feld, snce the goal s to roduce streamlnes that exactly follow the vector feld. Fortunately, the traland-error algorthm enforces ths constrant at every ste, and thus t sn t necessary to nclude t n the energy functon. To aly a tral-and-error algorthm to ths roblem, we must defne the roosal mechansm. Whle a urely random roosal mechansm may decrease the energy n the long run, t wll be far too slow to be ractcal. Hence, Turk and Banks defned many roosal heurstcs desgned to decrease the energy as much as ossble wth each ste. Panterly renderng by tral and error. I ve bult a tral-and-error anterly renderng algorthm, whch I ll brefly descrbe here. 16 At a hgh level, the goal s to seek concse antngs that match a source mage closely and cover the mage wth ant, but use as few strokes as ossble. A brush radus and a lst of control onts defne each brush stroke. The energy functon s IEEE Comuter Grahcs and Alcatons 75

Tutoral (c) 8 Satally varyng style, from Hertzmann. 16 Source mage (courtesy of Phl Greensun; htt://hl.greensun.com). Interactvely anted weght mage (w a). (c) Resultng antng wth the gven weghts. More detal aears near faces and hands. (d) Another choce of weghts; detal s concentrated on the rghtmost fgures. E ( I) = E ( I) + E ( I) + E ( I) E ( I) = w ( ) I( ) S( ) a ( ) I Enstr( I) = wnstr ( number of strokes n I) E ( I) = w ( number of emty xels n I) cov a cov (d) 9 Interactve anterly renderng rocess from Haeberl. 4 a nstr cov 001 IEEE Courtesy of Paul Haeberl Ths energy s a lnear combnaton of three terms. The frst term, E a, measures the xelwse dfferences between the antng and a source mage S. The number of strokes term, E nstr, enalzes the number of strokes. The coverage term, E cov, forces the canvas to be flled wth ant, f desred, by settng w cov to be large. The weghts w are user-defned values. The color dstance reresents Eucldean dstance n RGB sace. The weghts w a() are defned by a weght mage that lets the user secfy satally varyng weghts. The frst two terms of the energy functon quantfy the trade-off between two cometng desres: to closely match the aearance of the source mage and to use as lttle ant as ossble. By adjustng the relatve roorton of w a and w nstr, a user can secfy the relatve mortance of these two desres and thus roduce dfferent antng styles. By default, the value of w a() s ntalzed by a bnary edge mage. If you let the weght vary over the canvas, then you get an effect smlar to havng dfferent energy functons n dfferent arts of the mage (see Fgure 8). The weght mage w a() lets you secfy how much detal s requred n each regon of the mage. You can generate the weght mage automatcally or hand ant t. Ths gves users a hgh level of control wthout requrng them to make every low-level choce. Ths roblem s dffcult to otmze. The search even wth carefully desgned roosal heurstcs can take many hours to run and currently sn t ractcal. However, the algorthm does gve economcal results and substantal hgh-level control to users. Greedy algorthms The most common stroke-based renderng algorthms are greedy: strokes are added to the mage structure n a sngle ass, and strokes are never modfed once created. Greedy algorthms use heurstcs and carefully desgned lacement stes. Ths means that they can quckly roduce hgh-qualty results, but at the cost of flexblty. Greedy algorthms are rarely defned n terms of an energy functon, although one s sometmes mlct. In some stuatons, devsng an arorate energy functon mght be dffcult, but a useful algorthm can be develoed wthout one. Each greedy algorthm oerates by reeatedly lacng strokes and never modfyng them. Consequently, we defne a greedy algorthm by how t makes the followng two choces n the nner loo: Where do we lace the next stroke? What shae wll the next stroke have? Sngle-ont strokes Haeberl 4 descrbes a smle, semautomatc antng algorthm (see Fgure 9). Frst, the user rovdes a source mage. Then the user sees a renderng of the antng, whch s ntally blank. Usng a mouse or tablet, the user clcks and drags wthn the antng area and laces a sngle brush stroke at the locaton of each mouse clck. The system automatcally chooses the color by extractng t from the color of the source mage at that ont and orents the stroke n the drecton of the mage s gradent. Hence, the user decdes where the strokes go and the algorthm decdes what they look lke. The user may set other arameters (such as stroke szes) by adjustng settngs or va ressure on a tablet nterface. Ths system rovdes a fun and easy way to make abstract and attractve versons wthout requrng the user to ossess any drawng sklls. Sngle-layer anterly renderng. Numerous commercal software ackages have ncororated fully automatc versons of Haeberl s algorthm. Ltwnowcz 17 rovdes a comlete descrton of such an algorthm, along wth several enhancements. Ltwnowcz s basc algorthm takes a source mage 76 July/August 003

and orentaton feld as nut and generates a antng wth a set of orented, short, brush strokes. The algorthm laces the brush strokes on a grd n the mage lane, wth randomly erturbed ostons. Each stroke s color comes from the source mage and each stroke s orentaton comes from the orentaton feld. The system draws the strokes n random order, removng regulartes that would aear otherwse. The orentaton feld secfes the strokes desred orentaton and s generated from the source mage n a rerocessng ste. A smle way to generate ths orentaton feld s to set the orentaton φ() at xel to the normal of the mage s gradent ths gves the drecton n whch the mage s most constant. However, n constant regons of the mage, the gradent won t be well defned. The orentatons n these regons can be flled n usng a smoothng algorthm, such as thn-late slne nterolaton. Addtonally, strokes can be cled to edges extracted from the orgnal mages. Ths hels the antng reserve the edges of the orgnal mage more fathfully. Multle-layer anterly renderng. I have develoed an extenson to these algorthms that can create brush strokes wth multle szes. We can motvate the algorthm by observng that an artst often wll begn a antng as a rough sketch and go back later over the antng wth a smaller brush to add detal. Whle much of the motvaton for ths technque doesn t aly to comuter algorthms, t does yeld desrable vsual effects. Ths mage-rocessng algorthm uses fne brush strokes only where necessary to refne the antng and leaves the rest of the antng coarse. Users can also defne where fne strokes are used. The algorthm s smlar to a yramd algorthm, n that you start wth a coarse aroxmaton to the source mage and add rogressve refnements wth smaller brushes. In a sense, ths algorthm greedly otmzes an energy functon that enalzes the dfference between the antng and the source mage and enalzes the number of strokes. The algorthm takes as nut a source mage and a lst of brush szes, whch are exressed as rad r 1 r n. The algorthm then roceeds by antng a seres of layers, one for each radus, from largest to smallest. Generally, t s most useful to use owers of two: r = r 1 1, wth some user-determned value for r 1. The ntal canvas s a constant color mage. I frst create a reference mage for each layer by blurrng the source mage. The reference mage reresents the mage I want to aroxmate by antng wth the current brush sze. The dea s to use each brush to cature only detals that are at least as large as the brush sze. I use a layer subroutne to ant a layer wth brush r, based on the reference mage. Ths rocedure locates areas of the mage that dffer from the reference mage and covers them wth new brush strokes. Areas that match the source mage color to wthn a threshold (T) are left unchanged. The threshold arameter can be ncreased to roduce rougher antngs, or decreased to roduce antngs that closely match the source mage. Blurrng may be erformed by one of several methods. I normally blur by convoluton wth a Gaussan kernel of standard devaton f σr, where f σ s some constant factor. Nonlnear dffuson 18 can be used nstead of a Gaussan blur to roduce slghtly better results near edges, although the mrovement s rarely worth the extra comutaton tme. When seed s essental, I use a summed-area table. Ths entre rocedure reeats for each brush stroke sze. Here s a seudocode summary of the antng algorthm: functon PAINT(I s, //source mage I, //canvas r 1 r n) //brush szes Create a summed-area table A from I s f necessary refresh true foreach brush sze r, from largest to smallest, do Comute a blurred reference mage I r wth blur sze f σr grd r Clear deth buffer foreach oston on a grd wth sacng grd M the regon [ x grd/ x + grd/; y grd/ y + grd/] areaerror M I () I r () f refresh or areaerror > T then arg max M I () I r () PAINTSTROKE(, I,r, I r ) refresh false Each layer s anted usng a smle loo over the mage canvas. The dea s smlar to Ltwnowcz s algorthm. However, we can no longer lace samles smly on a jttered grd, snce ths aroach mght mss shar detals such as lnes and onts that ass between grd onts. Instead, the algorthm searches each grd ont s neghborhood to fnd the nearby ont wth the greatest error and ant at ths locaton. All strokes for the layer are lanned at once and before renderng. Then the strokes are rendered n random order to revent an undesrable aearance of regularty n the brush strokes. In ractce, we can avod the overhead of storng and randomzng a large lst of brush strokes by usng a z-buffer. Each stroke s rendered wth a random z value as soon as t s created. The z-buffer s cleared before each layer. Note that ths mght roduce dfferent results wth sgnfcant transarency, when transarent objects aren t rendered n back-to-front order. Fgure 10 (next age) shows the layers of a antng usng ths algorthm. When aled to a color vector, denotes Eucldean dstance n RGB sace. I also exermented wth the Commsson Internatonale de l Eclarage (CIE, or Internatonal Commsson on Illumnaton) LUV, a ercetually based color sace. Surrsngly, t gave slghtly worse results, but I m not sure why. PAINTSTROKE n the revous code lstng s a generc rocedure that laces a stroke on the canvas begnnng at 1, gven a reference mage and a brush radus. Ths technque focuses attenton on areas of the mage contanng the most detal (hgh-frequency nformaton) by lacng many small brush strokes n these regons. Areas wth lttle detal are anted only wth large brush strokes. Thus, strokes are arorate to the level of detal n the IEEE Comuter Grahcs and Alcatons 77

Tutoral g 1 v 1 v 0 1 g 0 10 Pantng wth three brushes. The nut mage. The remanng mages show the antng after the frst layer (brush radus 8), (c) the second layer (radus 4), (d) the fnal antng (radus ), and (e) the antng wth ant texture added. (Brush strokes from earler layers are stll vsble n the fnal antng.) (c) (d) (e) 0 11 Pantng a brush stroke. 1 A brush stroke begns at a control ont 0 and contnues n drecton v 0, normal to the gradent drecton g 0. From the second ont 1, there are two normal drectons to choose from: θ 1 +π/ and θ 1 π/. I choose v 1 to reduce the stroke curvature. Ths rocedure s reeated to draw the rest of the stroke. The stroke wll be rendered as a cubc B-slne, wth the as control onts. The dstance between control onts s equal to the brush radus. source mage. Ths choce of emhass assumes that detal areas contan the most mortant vsual nformaton, although other choces of emhass are also ossble, such as those secfed by a user or an eye tracker. Long, curved strokes Most real antngs and drawngs use long, curved strokes, nstead of the short strokes that I ve dscussed untl now. Panterly renderng wth long, curved strokes. Ths method can be extended to use long, contnuous curves nstead of short strokes. In my system, I lmt brush strokes to constant color and use mage gradents to gude stroke lacement. The dea s that the strokes wll reresent socontours of the mage wth roughly constant color. My method s to lace control onts for the curve by followng the normal of the gradent drecton. When the color of the stroke s farther from the target color n the reference mage than the antng, the stroke ends at that control ont. My slne lacement algorthm begns at a gven ont n the mage 0, wth a gven brush radus r. The stroke s reresented as a lst of control onts, a color, and a brush radus. Ponts are reresented as floatng-ont values n mage coordnates. I add the control ont 0 to the slne, and use the color of the reference mage at 0 as the color of the slne. I then comute the next ont along the curve. The gradent drecton θ 0 at ths ont s comuted from the Sobel-fltered lumnance of the reference mage. The next ont, 1, s laced n the drecton θ 0 +π/ at a dstance r from 0 (see Fgure 11). You can also use the drecton θ 0 π/; ths choce s arbtrary. I use the brush radus r as the dstance between control onts because r reresents the level of detal I ll cature wth ths brush sze; n ractce, I fnd that ths sze works best. Ths means that large brushes create broad sketches of the mage, whch can be later refned wth smaller brushes. The remanng control onts are comuted by reeat- 0 78 July/August 003

ng ths rocess of movng along the mage and lacng control onts. For a ont, we comute a gradent drecton θ at that ont. There are actually two ossble canddate drectons for the next drecton: θ + π/ and θ π/. I choose the next drecton that leads to the lesser stroke curvature: I ck the drecton v so that the angle between v and v 1 s less than or equal to π/ (see Fgure 11), where v can be (r cos(θ ± π/), r sn(θ ± π/)). The stroke termnates when t reaches the redetermned maxmum stroke length, or the reference mage color at the current control ont dffers from the current stroke color more than t dffers from the current antng at that ont. I find that a ste sze of r works best for caturng the rght level of detal for the brush stroke. We can also exaggerate or reduce the brush stroke curvature by filterng the stroke drectons. The entre stroke lacement rocedure s as follows (note that Yr() s the lumnance channel of Ir, scaled from 0 to 1): functon PAINTSTROKE(0, r, Ir, I) //Arguments: start ont 0, stroke radus (r), //reference mage (Ir), antng so far (I) color Ir(0) K a new stroke wth radus r and color color add ont 0 to K for = 1 to maxstrokelength do //comute mage dervatves g (55 Yr/ x ( 1), 55 Yr/ y ( 1)) //detect vanshng gradent f r g 1 //s gradent tmes length at least a xel? //rotate gradent by 90 degrees v ( gy, gx) (d) (e) Gray tones generated wth a rortzed stroke texture.5 (a-e) The strokes n the texture are rendered n a secfc order that lets dfferent tones be generated from a sngle texture. (f) Image generated from a source hoto usng rortzed stroke textures. (Images courtesy of Mchael Salsbury et al.) Pen-and-nk and other curve tracng algorthms. Many tyes of en-and-nk llustraton greedly otmze stroke lacement. In the smlest case, the goal s to lace en strokes to acheve a desred stroke densty (thus achevng a target tone) wth strokes that trace secfed orentatons. Jobard and Lefer15 descrbe an effcent greedy aroach to ths roblem. They defne a target densty by a desred dstance d between strokes. In a nutshell, ther algorthm conssts of // filter the stroke drecton v fcv + (1 fc)v 1 else f > 1 //contnue n revous stroke drecton v v 1 else return K 1 + rv/ v f > mnstrokelength and Ir ( ) I ( ) < Ir ( ) color return K add to K end for return K then (f) 1 // f necessary, reverse drecton f > 1 and v v 1 < 0 then v v (c) dentfyng seed onts n the mage wth a dstance of at least d from all exstng curves, and tracng a curve from a seed ont along the vector field, untl that curve comes wthn d of another curve. The algorthm contnues untl the mage s densely covered wth strokes. Ths rocedure s guaranteed to roduce curves that trace the vector field and mantan a dstance of at least d from all other curves. Ths algorthm deends on ntellgent heurstcs for gudng the choce of seed onts. A more sohstcated roblem s to use en strokes to llustrate tone, orentaton, and texture. Salsbury et al.5 ntroduced an nteractve tool for lacng en-andnk strokes that convey tone and orentaton. The user secfies the desred tone for a regon and the system automatcally laces strokes to match these tones. Strokes are stored as rortzed stroke textures (see Fgure 1), whch let the system render comlex hatchng atterns whle matchng a desred tone. Salsbury et al.6 descrbe another system that lets the user secfy varyng orentatons for the llustraton as well, thus IEEE Comuter Grahcs and Alcatons 79

Courtesy of Aaron Hertzmann and Dens Zorn Tutoral 13 Pen-and-nk llustraton of a smooth surface, from Hertzmann and Zorn.19 The target orentaton feld and tones are generated automatcally to llustrate the surface. matchng three searate quanttes (tone, texture, and orentaton) wth en-and-nk strokes. You can aly smlar rncles to llustratng surfaces. Tycally, the target tones come from a renderng of the surface and the target stroke orentatons come from orentaton fields defined on the surface. The resultng algorthm s a varaton on revous SBR technques, but wth many adjustments as dctated by the en-and-nk style and 3D model.19 Dens Zorn and I first generated a tone and orentaton field from a 3D model and extended Jobard and Lefer s15 method to hatch an mage of the model (see Fgure 13). Wnkenbach and Salesn7,8 descrbe a system that ales rortzed stroke textures to renderngs of 3D surfaces wth texture, otonally usersuled emhass for dfferent arts of the mage, and user-defined orentaton fields (see Fgure 14). In these methods, the hatchng attemts to match the mage s orentaton, rendered tone, and texture. Lmtatons of energy mnmzaton Concluson Many challenges reman n SBR algorthms. Frst, most of these algorthms are too slow to be useful n an nteractve alcaton, although faster comuters wll lessen the roblem. Second, for most alcatons, artsts and end users need better tools for controllng styles, n between settng arameters of a antng algorthm (whch may gve too lttle control) and antng all strokes manually (whch s very labor ntensve). The work of Kalnns et al.0 gves an excellent examle of artstc control over a secfic tye of SBR. More mortantly, we need to dramatcally exand the range of styles that SBR algorthms can create. Technques u to now have shown the ower of SBR algorthms on relatvely smle styles such as a smle verson of mressonsm. Now, the task s to dscover deeer atterns n artstc styles amenable to mlementaton beyond smle ant-scatterng effects. A fnal and sgnfcant challenge s to create comellng SBR anmaton. Because so few examles exst n tradtonal anmaton to look to for gudance, ths Courtesy of Georges Wnkenbach and Davd Salesn Currently, t seems unlkely that every desrable SBR style can be formulated n the energy functon formu- laton that I resented n the Stroke-based renderng secton. There are several reasons for ths. It s dfficult to cature looseness and sketchness or randomness n an energy functon. Moreover, antng and drawng aren t determnstc rocedures; an artst mght roduce dfferent mages each tme. One way to exress randomness would be to relace the energy functon wth the robablty densty over renderngs. Pantng s then a rocess of samlng from ths densty; the densty would usually be condtoned on the nut data. Several of the greedy aroaches revously descrbed (such as rortzed stroke textures5) and mentoned n the Related Tocs sdebar are dfficult to exress n terms of energy functons. It s sometmes easer to desgn a drect rocedure for a renderng style than to desgn an energy functon, esecally snce desgnng styles s a creatve rocess. Often, we desgn a new algorthm or styles wthout really understandng why they work. Ideally, we should develo addtonal nsght after the fact that lets us convert the drect rocedure to an energy functon. Knowng the energy functon can often gve nsght nto how the drect rocedure works and how to mrove t. However, t bears reeatng that drect rocedures are much faster than otmzaton rocedures. Pen-and-nk llustratons of 3D models, from Wnkenbach and Salesn.7 The left house shows the effect of a user-defned emhass functon; detal s only drawn where secfed by the user. The rght house shows a renderng wth unform emhass. 14 80 July/August 003

requres not just develong new algorthms, but also develong new artstc styles. If we as a feld are successful, we ll have created a new art form that couldn t have exsted wthout comuters. References 1. A. Hertzmann, Panterly Renderng wth Curved Brush Strokes of Multle Szes, Proc. Sggrah 98, ACM Press, 1998,. 453-460.. A. Hertzmann, Fast Pant Texture, Proc. nd Ann. Sym. Non-Photorealstc Anmaton and Renderng (NPAR 00), ACM Press, 00,. 91-96, 161. 3. G. Turk and D. Banks, Image-Guded Streamlne Placement, Proc. Sggrah 96, ACM Press, 1996,. 453-460. 4. P.E. Haeberl, Pant By Numbers: Abstract Image Reresentatons, Comuter Grahcs (Proc. Sggrah 90), vol. 4, ACM Press, 1990,. 07-14. 5. M.P. Salsbury et al., Interactve Pen And Ink Illustraton, Proc. Sggrah 94, ACM Press, 1994,. 101-108. 6. M.P. Salsbury et al., Orentable Textures for Image-Based Pen-and-Ink Illustraton, Proc. Sggrah 97, ACM Press, 1997,. 401-406. 7. G. Wnkenbach and D.H. Salesn, Comuter-Generated Pen-And-Ink Illustraton, Proc. Sggrah 94, ACM Press, 1994,. 91-100. 8. G. Wnkenbach and D.H. Salesn, Renderng Parametrc Surfaces n Pen and Ink, Proc. Sggrah 96, ACM Press, 1996,. 469-476. 9. M. Woo, J. Neder, and T. Davs, OenGL Programmng Gude: The Offcal Gude to Learnng OenGL, nd ed., Addson-Wesley Develoers Press, 1997. 10. K. Hoff III et al., Fast Comutaton of Generalzed Vorono Dagrams usng Grahcs Hardware, Proc. Sggrah 99, ACM Press, 1999,. 77-86. 11. O. Deussen et al., Floatng Ponts: A Method for Comutng Stle Drawngs, Comuter Grahcs Forum, vol. 19, no. 3, Aug. 000. 1. A. Secord, Weghted Vorono Stlng, Proc. nd Ann. Sym. Non-Photorealstc Anmaton and Renderng (NPAR 00), ACM Press, 00,. 7-43. 13. A. Secord, Random Marks on Paer: Non-Photorealstc Renderng wth Small Prmtves, master s thess, Det. of Comuter Scence, Unv. of Brtsh Columba, Oct. 00. 14. A. Hausner, Smulatng Decoratve Mosac, Proc. Sggrah 001, ACM Press, 001,. 573-578. 15. B. Jobard and W. Lefer, Creatng Evenly-Saced Streamlnes of Arbtrary Densty, Proc. 8th Eurograhcs Worksho on Vsualzaton n Scentfc Comutng, Eurograhcs, 1997,. 45-55. 16. A. Hertzmann, Pant by Relaxaton, Comuter Grahcs Int l 001, IEEE CS Press, 001,. 47-54. 17. P. Ltwnowcz, Processng Images and Vdeo for an Imressonst Effect, Proc. Sggrah 97, ACM Press, 1997,. 407-414. Related Tocs Sace lmtatons recluded a comlete survey of stroke-based renderng. You can fnd a detaled lst of references at htt://www. dg.toronto.edu/~hertzman/sbr0. A few of the major related tocs are anmaton and real-tme renderng for creatng stroke-based anmatons, vrtual envronments, and nterfaces; examle-based strokes for creatng llustraton styles from hand-drawn examles; thresholdng algorthms, whch decoule stroke lacement and stroke tones; tensor feld vsualzaton, where stroke-lke elements are used for scentfc vsualzaton of comlex data; hotomosacs and jgsaw mage mosacs, n whch an mage s aroxmated by a collecton of smaller mages; and smulatng stroke texture by numercal smulaton and/or rocedural synthess. 18. P. Perona and J. Malk, Scale-Sace and Edge Detecton Usng Ansotroc Dffuson, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 1, no. 7, July 1990,. 69-639. 19. A. Hertzmann and D. Zorn, Illustratng Smooth Surfaces, Proc. Sggrah 000, ACM Press, 000,. 517-56. 0. R.D. Kalnns et al., WYSIWYG NPR: Drawng Strokes Drectly on 3D Models, ACM Trans. Grahcs, vol. 1, no. 3, 00,. 755-76. Aaron Hertzmann s an assstant rofessor n the Deartment of Comuter Scence at the Unversty of Toronto. Hs research nterests are 3D shae reconstructon, nonhotorealstc renderng, and alcatons of machne learnng to comuter grahcs. Hertzmann receved a PhD n comuter scence from New York Unversty. He s a member of the IEEE. Readers may contact Aaron Hertzmann at the Det. of Comuter Scence, Unversty of Toronto, 10 Kng s College Rd., Room 3303, Toronto, ON, M5S 3G4; hertzman@ dg.toronto.edu. For further nformaton on ths or any other comutng toc, lease vst our Dgtal Lbrary at htt://comuter. org/ublcatons/dlb. IEEE Comuter Grahcs and Alcatons 81