Collaborative Personalization of Iage Enhanceent Juan C. Caicedo Universidad Nacional Bogotá, Colobia jccaicedoru@unal.edu.co Ashish Kapoor Microsoft Research Redond, WA. USA akapoor@icrosoft.co Sing Bing Kang Microsoft Research Redond, WA. USA sbkang@icrosoft.co Abstract While ost existing enhanceent tools for photographs have universal auto-enhanceent functionality, recent research [8] shows that users can have personalized preferences. In this paper, we explore whether such personalized preferences in iage enhanceent tend to cluster and whether users can be grouped according to such preferences. To this end, we analyze a coprehensive data set of iage enhanceents collected fro 336 users via Aazon Mechanical Turk. We find that such clusters do exist and can be used to derive ethods to learn statistical preference odels fro a group of users. We also present a probabilistic fraework that exploits the ideas behind collaborative filtering to autoatically enhance novel iages for new users. Experients show that inferring clusters in iage enhanceent preferences results in better prediction of iage enhanceent preferences and outperfors generic auto-correction tools.. Introduction Despite advances in digital photography that iprove caera functionality taking good quality pictures reain a challenge for casual photographers. Probles include incorrect caera settings and poor lighting conditions. Photos ay be iproved using iage enhanceent tools, but anually retouching every single photograph is infeasible. Auto-enhanceent tools are available (e.g., fro Picasa or Windows Live Photo Gallery), but as Kang et al. [8] have shown, users can have significantly different preferences that are not reflected in such tools. We use Kang et al. s observations to explore a new approach for iage enhanceent. We explore if groups of users have siilar preferences. Our hypothesis is that such clusters do exist and that we can utilize that inforation in order to collaboratively enhance iages. Intuitively, we expect that enhanceent preferences should be siilar for users with siilar taste on siilar iages and consequently we can derive ethods that would let us harness the enhanceent efforts of ultiple users to build predictive odels. Such adaptation of the iage enhanceent proble to the collaborative setting is further strengthened by the ever growing use of web-based photo-sharing systes such as Facebook and Flickr. Adding a personalized iage enhanceent coponent to those sites would allow the syste to suggest appropriate enhanceent paraeters for new iages by utilizing prior enhanceents ade by other users with siilar preferences. Our work has three key contributions. First, we describe a coprehensive user study through Aazon Mechanical Turk in order to collect iage enhanceents fro a large nuber of users. Second, we propose a probabilistic odel that explicitly encodes clusterings of user preferences and derive an efficient inference ethod to collaboratively enhance unseen iages for novel users. Finally, we epirically show that explicitly odeling the clustering of user enhanceent preferences leads to better predictions of enhanceent paraeters than the existing one touch-button coercial auto-enhance functionalities. 2. Previous Work The basic idea of iage-derived priors is to exploit inforation in a predefined iage set to autoatically deterine the ost likely paraeters for enhancing an iage. It has been used for a wide range of tasks including denoising [3], color correction [3], and iage restoration [2]. These approaches use generic data sets with a large nuber of iages that do not provide any inforation about user identity or user preferences. Grabler et al. [4] use saple iage anipulations to learn acros to be applied to iages with siilar content. Also, Joshi et al. [7] narrowed the doain of prior iages to a person s favorite photographs to develop a personal iage enhanceent fraework. In principle, these ethods are user-dependent, but no studies were done to establish if their variance across users is significant. The work closest to ours is that of Kang et al. [8], who proposed a odel for personalization of iage enhanceent that learns user preferences by observing her choices in a training set. They conducted a user study that showed 249
u u x? x 2? 2 iages x n n n users cluster cluster 2 {, ui, uk} { u, u } Iage Enhanceent Paraeter y y n j, p Enhanceent Paraeter 2 y User in Cluster? User in Cluster 2? Building users database Discovering clusters Enhancing new iages Iage 2 y n Figure. Overview of collaborative personalization of iage enhanceent. A database of users is built, with θi u the observed enhanceent vector for user u and iage x i. Enhanceents are not observed for every cobination of iages and users, which is indicated by the sybol? in the atrix. An algorith is used to discover cluster ebership for users, and to infer enhanceent vectors yi c associated with cluster c for each i th iage. New iages are enhanced based on siilarities to existing iages and cluster to which the user belongs. that different users do have different preferences in iage enhanceent, and deonstrated that personalization is an iportant coponent to iprove the subjective quality of iages. Our ethod builds upon both personalization of iage enhanceent and collaborative filtering techniques. For personalization of iage enhanceent, our approach asks users to enhance a set of training iages using an intuitive interface to collect preference inforation. In this portion of our work, we ipleented ethods siilar to those in Kang et al. [8]. However, our work substantially extends that personalization approach to consider the contribution of other users with siilar preferences. Collaborative filtering is an approach to build recoender systes, which analyzes patterns of user interest in products to provide personalized recoendations [9]. Intuitively, collaborative filtering works by building a database of preferences for ites by users, and atching new users against this database to find neighbors with siilar preferences []. Most existing collaborative filtering ethods attept to predict a scalar quantity, and thus are hard to adapt to our scenario where our goal is to predict a vector of enhanceent paraeters. This is the first work that we know of that proposes the idea of using the wisdo of crowds to enhance iages via collaborative filtering. 3. Overview Figure suarizes the flow of tasks towards building a syste for collaborative iage enhanceent. We first acquire a database of iages that are enhanced by ultiple users, and represent the database as a table (leftost of Figure ) whose rows are individual users and coluns are iages. Each entry in the table corresponds to enhanceent paraeters associated with a different user u and iage x i, and are represented by vector θi u. The iages used for training are a fixed set, and are reasonably representative of iages that need enhanceent. Note that this set of representative iages is large; hence, it is unreasonable to expect that each user will provide enhanceent paraeters for all the iages. Instead, every user enhances only a sall subset of iages, and as such, there are quite a few entries in the table that reain unobserved (denoted as? ). Once this database is acquired, we analyze it to learn a statistical odel that explicitly encodes clustering of users. Specifically, we recover groupings of users and estiate the preference paraeters of each individual group for all the representative iages (iddle of Figure ). Once such a statistical odel is learned, we can enhance a new (i.e., unseen) iage for any cluster by considering its siilarity to the iages in the representative set. 4. Building Blocks Prior to running our user study, we need to pick the knobs required to perfor iage enhanceent and select our representative set of training iages. Also in order to effectively perfor collaborative filtering, we require a distance etric between iages that reflects differences in the respective enhanceent operations required on those iages. Enhanceent Operations: We considered two global enhanceent operations: contrast anipulation (S-curve and Gaa correction) and color correction (tint and teperature), which as with Kang et al. [8], are represented by 5 enhanceent paraeters (two paraeters for S-curve, one each for gaa-correction, tint and teperature). Learning a Distance Metric between Iages: One of the key ingredients in our approach is distance (or siilarity) between iages. In particular, given two iages, we want to estiate the extent to which these iages require siilar enhanceent paraeters. We approach this as a etric learning proble, in which a distance between visual features is adapted to include inforation of enhanceent paraeters. For every iage in the photo collection, we extract 6 color histogras fro the RGB channels and the HSV coponents, which are used to build a unique iage descriptor. Forally, let g(x i ) and g(x j ) be visual feature 250
Inforation Gain vectors for two iages x i and x j. We use the Mahalanobis distance in the visual feature space: d A (x i, x j ) = (g(x i ) g(x j )) T A(g(x i ) g(x j )) ().2 0.8 Iage Selection via Sensor Placeent Algorith which is paraeterized by a square atrix A to encode weights associated with different features and correlations aongst the in order to reflect the disparity in the enhanceent paraeter space. We learn the atrix A using an online algorith following the ethodology proposed by Jain et al. [6]. The target distance between any two iages is considered to be the distance between their autoenhanceent paraeters. Specifically, we generate constraints of the for (g(x i ), g(x j ), δ ij ), where δ is the distance between the enhanceent paraeters (obtained fro an auto-enhance utility) of the corresponding iages. This specific online algorith is well suited for our task, because our iage collection has approxiately 00 illion constraints, which akes it non-trivial for other distance etric learning ethods to handle. Representative Iage Selection: Selecting a good set of representative iages is critical. We downloaded a set of real photographs fro Flickr, following a sapling through tie strategy, in which the paraeter date taken is randoly changed [, 5]. The list of keywords includes landscape, nature, people, and sports, aong others, to cover a diverse range of photographs without focusing on particular objects. We filtered out iages that are too sall (iages saller than 800 800 are reoved), blackand-white, and have very little intensity variation (easured by histogra entropy). In the end, our downloaded set consisted of ore than 300,000 photos. Given the set of iages and the distance etric between iages, we need to identify a subset of different iages that represent the original set. This proble was approached as a sensor placeent proble, in which a budget of sensors is available to be located in a space, deciding which positions to chose to axially cover spatial onitoring. In our proble, the sensors are the available iages in the given set where the siilarity (covariance function) is defined by the kernel K ij = exp( d A (x i, x j )/ean(d A (:))). We used the algorith described by Krause et al. [0], in which the eleents that axially reduce the entropy in the original set are iteratively selected. In the Flickr snapshot, we also observed that users tend to upload any siilar iages. We applied the sensor placeent algorith to iages fro the sae user, to keep only the three ost different iages fro each individual. We then cobined these filtered iages and ran the algorith again to select a final photo collection for the user study. We found that with 200 iages, we kept about 84% of the inforation fro a set of approxiately 25,000 candidates. Figure 2 shows how the inforation gain reduces as long as inforative iages are being selected. These 0.6 0.4 0.2 200 iages = 84% inforation (fro 25,000 candidates) 0 0 500 000 500 2000 2500 Nuber of selected iages Figure 2. The iage selection algorith picks iages that provide axiu increase in utual inforation. This results in a decreasing inforation gain in the selected iage set. 200 iages sufficiently represent the visual feature space defined by the etric that we have learned. 5. User Study The goal of this study was to build the user-iage atrix of enhanceent paraeters. We opted for Aazon Mechanical Turk (MT) to engage people in our iage enhanceent task. Each person had to enhance a sall nuber of iages by selecting the ost preferred enhanced version via an interactive web syste. We anaged to collect enhanceent paraeters fro 336 valid users. Two ain criteria guided the design of the syste for iage enhanceent used in this study: () easy of use and (2) web-oriented. For the first criterion, the user interface does not require paraeter tweaking to apply enhanceents. We ipleented a user interface siilar to the one proposed by Kang et al. [8], in which the syste presents a 3 3 atrix, with an enhanced version of the iage in each cell. In the first iteration, the syste presents the original iage in the center cell surrounded by 8 possible initial enhanceents. When the user clicks one of these possible enhanceents, new siilar enhanceents are coputed to replace the current ones. When the user has identified the ost preferred enhanced version, the syste records the enhanceent paraeters that produce that iage and presents another one. Figure 3 shows a screenshot of this user interface. For the second design criterion, the application is required to be a web-oriented syste, to allow people to enhance iages online through MT. We opted to deploy our application using cloud coputing services, which allows our syste to scale up as needed to copute iage enhanceents online. In our user study, we asked each participant to enhance 20 iages, which are randoly assigned fro the collection of 200 photos. The assignent of iages to users attepts to have approxiately the sae nuber of users enhancing each iage. The axiu tie allowed to coplete this task was hour and the reward was 25
Figure 3. Web-based user interface for iage enhanceent. 0.6 0.4 0.2 0 Gender F Age 8-24 25-34 35-44 45-54 55 - older M 40% 30% 20% 0% 0% High School None Basic Education level Soe college Interediate Professional Expertise in Photography Bachelour Masters degree Caera use Ph.D Never Year 3 onths Month Figure 4. Deographics of subjects involved in the MT study. US$.50. The click count, click rate, and tie staps were recorded to identify spaers. Right before starting the task, participants were asked to fill a questionnaire aied to collect deographic inforation, including gender, age bracket, country, and education level. They also have to indicate their level of expertise in photography and frequency of caera use. Figure 4 shows the proportions of these attributes for 336 valid subjects. Most of the subjects are fro USA (43%) and India (4%), with the rest fro ore than 20 other countries. 6. Enhancing Iages Collaboratively Let us denote the collection of all the n iages as X = {x i }. The user study provides us with sets of enhanceent paraeters chosen by all the users. Let θi u be the enhanceent paraeters chosen by user u for iage x i. Because of resource and tie constraints, every user enhances only a sall subset of iages (20 iages out of 200) and consequently there are a lot of iages for which we do not directly observe the enhanceent paraeters corresponding to each user. Our goal is to derive a fraework that can () infer these issing enhanceent paraeters for every user in the study, and ore iportantly, (2) deterine enhanceent paraeters for new iages and users. Our odel is otivated by the ethods in collaborative filtering. In particular, the first key underlying assuption here is that siilar users should have siilar enhanceent Week Daily paraeters for siilar iages. Assuptions like this are also at the heart of any collaborative filtering ethodologies; we could in fact adapt those ethods to infer issing enhanceent paraeters for all the users in the study. However, ost of the work in collaborative filtering focuses on predicting a scalar quantity while our goal is to odel a vector of enhanceent preferences. While off-the-shelf approaches can be individually adapted to each enhanceent paraeter, such siplistic schee ignores the structure of relationships across paraeters and will be sub-optial. In this work, we extend the collaborative filtering to a setting of structured prediction where we jointly predict all the coponents of the enhanceent preferences. Specifically, our odel not only encodes siilarity across iages and users but also odels relationships between different coponents of the enhanceent space. The notion of users siilarity in our syste corresponds to grouping of users into clusters. Thus, users are clustered if they have siilar enhanceent paraeters for all the iages. In the next subsections, we describe () a probabilistic graphical odel for jointly predicting enhanceent preferences that explicitly encodes siilarity across iages and groups users into clusters, (2) an efficient inference algorith, and (3) extensions to do predictions on unseen iages and users that were not part of the user study. 6.. Probabilistic Model for Enhanceents We propose a odel that encodes the dependence of enhanceent paraeters on iage content as well as user preferences. Specifically, given the collection of iages and clustering of users, we assue that there are latent enhanceent preference vectors yi c which correspond to cluster c and iage x i. Further, the enhanceent paraeter vectors we observe in the MT study are siply noisy versions of these latent true enhanceent preferences. Figure 5 illustrates the factor graph corresponding to the proposed odel. The observed enhanceent preferences θi u fro different users are denoted as circles and we introduce a discrete rando variable (squares) h u, u {,.., } for each user that indicates the cluster the user belongs to. We also use Y i to denote collection of all true enhanceent preferences for the i th iage across all the clusterings. The shaded nodes correspond to rando variables that are observed. For exaple, in Figure 5, the enhanceent preferences for user are known for iages 2 and n, while user did not enhance iage 2. This is consistent with our data set where users enhance only a subset of all training iages. The odel also iposes soothness constraints using a GP prior [2] in order to account for the iage content. We use a Gaussian Process prior to enforce the assuption that siilar iages should have siilar preference paraeters. In particular, for each cluster c we induce the GP prior (denoted as GP (c)), where each of the five co- 252
User h θ θ 2 Y Y 2 Y n θ n p θ i Y i, h = φ h (θ i, Y i ) p(y Χ) θ θ 2 User h θ n p θ i Y i, h = φ h (θ i, Y i ) Figure 5. Factor graph depicting the proposed odel. Shaded nodes correspond to observed preferences. Not all observations are available for all the iages and users. Square boxes depict latent rando variables corresponding to cluster ebership. ponents of the latent variables yi c and yc j are assued to be jointly Gaussian with zero ean and the covariance specified using a kernel function applied to x i and x j. Forally, GP (c) 5 p= N (yc (p); 0, K). Here y c (p) is the colun vector of p th coponent of enhanceent preference for all iages corresponding to the cluster c and K is a kernel atrix with K ij = k(x i, x j ) and encodes siilarity between pairs of iages. In this paper, we use K ij = exp( d A (x i, x j )/ean(d A (:))). Note, that the GP prior above does not explicitly encode the relationship between the coponents of the paraeters and we can assue that the coponents of yi c have been whitened (zero ean, with unit covariance) beforehand 2. Also, note that all the diensions of the latent variable Y are coupled in the odel and perforing inference will preserve the relationships between different coponents. Let Θ be all the enhanceent preferences fro all the users and for all the iages. Our proposed odel induces a conditional probability distribution p(θ, Y, h X) using the GP prior p(y X), prior probabilities on the cluster ebership p(h), and the potential ters p(θi u x i, Y i, h u ) that link the latent iage preferences to the ones that are observed. Thus, the conditional distribution induced by our odel can be written as p(θ, Y, h X) = p(y X)p(h)p(Θ X, Y, h) Z = k n GP (c) p(h u ) φ hu (θi u, Y i ), Z c= u= i= This kernel atrix is a positive seidefinite atrix and is akin to the kernel atrix used in classifiers such as SVMs. 2 In this work, we whiten the enhanceent preferences before applying the odel and re-project back to orginal space eventulally which results in preservation of the structure of the paraeter space. where Z is the partition function (noralization ter) and the potential φ hu (θ u i, Y i) corresponding to a user u and iage x i takes the following for: φ hu (θ u i, Y i ) e y hu θ i i u 2 2σ 2. (2) Here, y hu i are the hidden rando variable for the sae cluster as the cluster indicated by h u and the iage x i, and σ 2 is the noise paraeter that deterines how tight the relation between the soothness constraint and the final label is. By changing the value of σ we can ephasize or de-ephasize the effect of the GP prior. In suary, the odel provides a powerful fraework for encoding dependence of enhanceent paraeters on iage content (via the GP prior) as well as the clustering of users and allows us to cobine the prior assuptions with the data that is observed in the MT study. 6.2. Inference in the Model Given the observations Θ o fro the MT study the key task is to infer the posterior distribution p(y, h X, Θ o ) over latent true enhanceent preferences and the clustering ebership for all the users. Perforing exact inference is prohibitive as the joint distribution is a product of a Gaussian (GP prior and the φ( ) potentials) and non-gaussian ters (cluster ebership). We resort to approxiate inference techniques in order to get around this proble. In particular, we perfor an approxiate inference by axiizing the variational lower bound with the assuption that the posterior over the unobserved rando variable Y and h can be factorized: F = Y,h log q(y)q(h) log p(y, h X, Θ o) q(y)q(h) Y,h p(y, h X, Θ o ), where q(y) = k 5 c= p= q(yc (p)) is assued to be a Gaussian distribution and q(h) = u= q(h u) is a discrete joint distribution over the unobserved labels. The approxiate inference algorith ais to copute good approxiations q(y) and q(h) to the real posteriors by iteratively optiizing the above described variational bound. Specifically, given the approxiations q t (Y) and q t (h) fro the t th iteration and assuing unifor prior over p(h) the update rules are: q t+ (y c ) GP (c) t (h u=c) q t+ (h u = c) θ u i Θo [φ c (θ u i, Y i )] q θ u i Θo φ c (θ u i, ean(q t+ (Y i ))). Intuitively, the update of iage enhanceent preferences considers the cluster ebership fro the previous iteration and uses it to decide if a data ter should be included 253
RMSE RMSE in update for each cluster. Siilarly, the update for the posterior over the cluster ebership considers ean enhanceent preferences fro the previous iteration. Thus, starting fro a rando initialization the paraeters and posterior of the cluster eberships are iteratively updated untill convergence. Upon convergence, we obtain posterior distribution of cluster ebership for each user (q(h)) as well the distribution over the true iage enhanceent preferences (q(y)) approxiated as a Gaussian distribution. 6.3. Handling New Iages and Users Besides inferring about the iages in the database, we are also interested in predicting enhanceent preferences for an iage x test that was not in the original training set as well as for a user who was not part of the user study. The ore straightforward case is where we infer enhanceent preferences for a user u who was a part of the user study and had enhanced training iages for us. Here, we siply need to run the inference algorith where we augent the rando variable Y with Y test, the collection of rando variables that represent true enhanceent paraeters for the new test iage across all the clusterings. Thus, the new iage can be enhanced using the ean of inferred enhanceent preferences corresponding to the cluster the user u belongs to. Note that if we have already perfored variational inference on the training data, inference after the augentation only requires one iteration of update equations. This is because the new iage does not introduce any new inforation about clusterings of the user or enhanceent paraeters, hence, aking it fairly efficient. It is trickier if we need to enhance iages for a user who is not part of the study. However, once we know the cluster ebership of the new user, we can siply use the schee entioned above to estiate the enhanceent preferences. Thus, the proble really reduces down to establishing the cluster ebership of the user. To this end, we can eploy several ethods. For exaple, the new user can enhance soe of the iages fro the training set, which in turn will give us evidence about ebership. The user now can be considered a part of the available corpus and inference can be run as before. Alternatively, we can also engage the user in an introductory dialogue where training iages are used; the resulting enhanceents will indicate ebership. In this work, we use the forer approach of asking users to enhance a subset of training iages to test the extension of the syste to new users and iages. 7. Experients and Results We first perfor experients to deterine the correct nuber of clusters using the data collected in the MT study. Train-test splits were created by randoly choosing 0% of the observed enhanceent vectors in the user-iage atrix as test exaples. The other 90% is used to run the in-.248.243.238 Error in Paraeter Space 2 3 4 5 Nuber of Clusters Error in Intensity Space 9.2 9 8.8 8.6 2 3 4 5 Nuber of Clusters Figure 6. Average estiation error on training for different nuber of clusters. ference algorith described in Section 6.2 with σ 2 fixed to 0 3. Evaluating the quality of user adjustents is difficult, since there is no universal agreeent on the perceptual etric. We resorted to the objective easure of intensity difference looking at average Root Mean Square Error (RMSE) in paraeter space (enhanceent vectors) as well as iage intensity space (resulting enhanced iages). Figure 6 highlights the variation of RMSE as we change the nuber of clusters in the odel. Note, that considering only one cluster is equivalent to assuing that all users follow the sae preferences for enhancing iages. We observe that error is highest in both paraeter and intensity spaces when the nuber of clusters is fixed to one and reduces significantly as we start incorporating additional nuber of clusters. This result is consistent with the observations reported by Kang et al. [8], where it was shown that different users do have significanlty different preferences. We also find that the error was iniu in the paraeter space for 3 clusters and pretty close to iniu in the intensity space. Thus, we perfor rest of the analysis and experients by fixing the nuber of clusters to 3. The proposed ethod can be thought of as a clustering algorith with constraints iposed by siilarity in the iage appearance space. Consequently, it is in principle the sae as an EM algorith for learning a Gaussian ixture odel, and it does not prefer that ost users have their own distinct cluster. We think that the increase in error due to further addition of clusters ight be due to the sall nuber of enhanceents operations allowed and we can expect to see different clusterings if additional operations are peritted. Figure 7 shows exaple iages enhanced according to the preferences of users in each of the three clusters, giving a general indication of the preferred appearance in each cluster. While the corrected iages in cluster 2 are slightly ore saturated than the others, iages in cluster 3 see to have ore contrast. A ore quantitative description of the preferred enhanceents in each cluster can be seen in Figure 8, which presents the distribution of the five paraeters as box plots. The box plots are over the eans of the inferred distribution over Y (all 200 iages for the 3 clusters), where the red line in each colun denotes the edian. Note that the largest difference across clusters is for the Power Curve and the S-Curve Inflection Point, suggest- 254
RMSE (Intensity) on Test Set Cluster Cluster 2 Cluster 3 Figure 9. Scatter plot of all iages according to paraeters Power- Curve and S-Curve Inflection Point (which partially control global iage contrast). Cluster in red squares, Cluster 2 in green circles and Cluster 3 in blue triangles. Coparison with Existing Systes Figure 7. Exaple iages enhanced according to the preferences discovered for each cluster. ing that variations in contrast is a doinating factor across the clusters. There are soe differences across the rest of the three paraeters as well suggesting a difference in paraeter choice aongst the users. In the scatter plot shown in 9, each point corresponds to an enhanceent preference color-coded by the cluster to which it was assigned. Note that the points are clustered fairly evenly in this 2D projection of the space, further ephasizing the latent structure which is successfully discovered by the inference algorith. We also analyzed statistics of deographic data across the clusters but the statistics were fairly siilar across the clusters. Next, we evaluate the ability of the proposed odel to predict enhanceent preferences for a new user on unseen iages in a collaborative environent. In this test, 0% of all users were randoly chosen as test subjects and inference was perfored using only the data observed for rest of the users. We siulate the real-life situation where every test user provides evidence about its cluster ebership by enhancing soe iages. Since this work is about collaboratively using inforation about iage enhanceent available fro any different users, we show results on scenarios where a test user has provided such little inforation that personalized odel of [8] cannot learn. We look at the RMS error in intensity space as each user enhances one iage at a tie (chosen randoly) and copare the with results of auto-enhanceent fro Picassa and Windows Live Photogallery. Figure 0 presents the plot of RMSE obtained by the 9.7 9.5 9.3 9. 8.9 8.7 8.5 Our Syste (3 Clusters) Picasa Photogallery 2 3 4 5 6 7 8 9 0 2 3 4 5 6 7 8 9 20 Nuber of Training Iages Enhanced by the Test User Figure 0. Average error of predicting personalized enhanceent paraeters. Auto-enhanceent tools do not learn user preferences. Our approach predicts personalized enhanceents with increasing accuracy as users provide new exaples. syste with increasing nuber of enhanceents available fro the user. We find a consistent reduction in estiation errors as ore enhanceents are added to the personal profile. This is due to the fact that additional enhanced iages provides ore inforation about the cluster ebership of the user, enabling the inference procedure to estiate a better recoendation. We also observe that the perforance of the collaborative approach is far better than the two autoenhanceent tools. While all the three approaches perfor siilarly when there is no evidence about the cluster ebership of the user, we see that the perforance of collaborative strategy greatly iproves as user starts enhancing iages. Also, note that the gains start showing up with as few as 2 iages indicating that strong gains can be obtained with the collaborative enhanceent of iages. Finally, Figure shows several iage exaples with the corresponding enhanced versions produced by Photo- 255
Power S Curve Labda S Curve Inflection Teperature Tint Figure 8. Box plots of values for each paraeter (first three control global contrast, last two control color correction). The x-axis indicates cluster nuber and y-axis corresponds to range of values. Red lines denote the edian, end lines are at the lower and upper quartile values, and crosses beyond the ends of the whiskers are outliers. Original Iage PhotoGallery Picassa Cluster 3 We do not clai that our results extend to a uch larger scale with any ore users and a uch richer set of enhanceent knobs; ore experients are required to substantiate such a clai. Future work includes active selection of iages that would enable better estiation of cluster ebership with fewest iage enhanceents by the user. References Figure. Exaple iages with the corresponding enhanced versions generated by auto-enhanceent tools and the proposed approach for a subject that belongs to cluster 3. Gallery and Picassa. Also, the enhanced version of our ethod, for a subject of the cluster 3 is presented. In general, each tool produces a different enhanceent resulting in a different final look and feel. Notice that the suggestion ade by our ethod has been inferred using the subjective preferences learned fro people in cluster 3, thus it is ore likely to be preferred by a user belonging to that population. 8. Conclusion and Future Work We present a novel approach for iage enhanceent that follows a collaborative fraework to recoend corrections based on user preferences. Instead of building a syste that is trained to enhance iages for a specific user, our technique is a principled and practical way for collaboration at a web scale, and can encopass alternative paraeterizations and error criterion. The ain idea is we prediscover the clusters of personalized enhanceent paraeters through collaborative learning on large iage collections. Then, for a new user, we need only figure out which cluster he/she belongs to. Results of the cluster analysis showed the existence of three ain groups, ainly characterized by differences in contrast preferences. Experiental results indicate that the collaborative enhanceent strategy significantly helps in aking better predictions of enhanceent paraeters than existing one touch-button coercial auto-enhance tools. [] T. S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng. Nus-wide: a real-world web iage database fro national university of singapore. In CIVR 09. ACM, 2009. [2] K. Dale, M. K. Johnson, K. Sunkavalli, W. Matusik, and H. Pfister. Iage restoration using online photo collections. In ICCV. IEEE, 2009. [3] R. Fergus, B. Singh, A. Hertzann, S. T. Roweis, and W. T. Freean. Reoving caera shake fro a single photograph. ACM Trans. Graph., 25(3):787 794, 2006. [4] F. Grabler, M. Agrawala, W. Li, M. Dontcheva, and T. Igarashi. Generating photo anipulation tutorials by deonstration. ACM Trans. Graph., 28(3): 9, 2009. [5] J. Hays and A. A. Efros. Scene copletion using illions of photographs. Coun. ACM, 5(0):87 94, 2008. [6] P. Jain, B. Kulis, I. Dhillon, and K. Grauan. Online etric learning and fast siilarity search. In NIPS, 2008. [7] N. Joshi, W. Matusik, E. H. Adelson, and D. J. Kriegan. Personal photo enhanceent using exaple iages. ACM Trans. Graph., 29(2): 5, 200. [8] S. B. Kang, A. Kapoor, and D. Lischinski. Personalization of iage enhanceent. Coputer Vision and Pattern Recognition Conference, CVPR, 200. [9] Y. Koren, R. Bell, and C. Volinsky. Matrix factorization techniques for recoender systes. IEEE Coputer Journal, pages 30 37, 2009. [0] A. Krause, A. Singh, and C. Guestrin. Near-optial sensor placeents in gaussian processes: Theory, efficient algoriths and epirical studies. J. Mach. Learn. Res., 9:235 284, 2008. [] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Ite-based collaborative filtering recoendation algoriths. ACM, WWW, 200. [2] M. Seeger. Gaussian Processes for achine learning. International Journal of Neural Systes, 4(2), 2004. [3] R. Stanikunas. Investigation of color constancy with a neural network. Neural Networks, 7(3):327 337, April 2004. 256