A Color Placement Support System for Visualization Designs Based on Subjective Color Balance Eric Cooper and Katsuari Kamei College of Information Science and Engineering Ritsumeikan University Abstract: Color placement is a key factor in the effectiveness of design. We develop systems that assist novice designers in placing effective colors. These systems are developed to be broadly applicable to any desktop application and with the ultimate goal of designs that are rated highly for ease-of-use by independent viewers. This paper details the construction of the color conspicuity model for color placement support. This system supports users by giving direct recommendations on color placement for any given color in a design based on the inferred relative conspicuity of every other color in the design. The core of this system is the conspicuity inference system that has been constructed to have a high correlation with ease-of-use evaluation. The system supports designers without interfering, by working with the color choices given. Keywords: color placement, color conspicuity, color design 1 Introduction Color design requires a high degree of color placement expertise. Yet desktop design systems usually provide one or several default palettes for all designs. Although most studies of the subject agree that ease-of-use is the main goal of design and that color placement plays a large role in ease-of-use, there varying opinions on which color placement choices are best. Color placement remains a case-by-case challenge for designers with experience and a great difficulty for those who do not. We develop color systems with the goal of producing systems that will work equally well on any visualization design, in coordination with designers needs. This is an effort to reduce the huge number of human factors involved in color placement (Silverstein) to a few generalized factors, forming a broad base for color placement support. 2 Development Process The goal of this research is to develop systems which can be used in nearly any desktop system and for any visualization design. This broad range requires some limitations of scope. We limit the visualization designs to flatly colored designs. We limit the accessible information to the post-rendered image and the device dependent RGB-derived display values. These are properties available in most programming environments. We do not exclude complex pixel-by-pixel operations such as might be done in image processing but all operations must be done in real-time, as the user places colors. From these inputs, the systems we develop output advice to the user on color placement. Allowing only advice to be given to the user allows the system to work directly with the designers own goals, such as the general use of color in the field and designer or viewer preferences. The final designs, those which were designed with support from the system and those were not, are rated comparatively for ease-of-use by independent viewers in surveys. This development process supports our goals of broad generality for any type of system and any type of visualization, the interaction with the user, as well as the ultimate goals of visualizations that are ease to use.
3 Background 3.1 Conspicuity The system described here is based on the concept of color conspicuity. Conspicuity is the property of an element of the design, in this case each color, to stand out or draw attention. This color support system is designed to use a model of this property alone to provide color support. When properly implemented in an application, the system will use other elements besides conspicuity. However, the exciting results of the current system show that it achieves high correlation with ease-of-use rating using conspicuity alone. Conspicuity is defined by an experiment that gathers data on the phenomenon. The experiment does not present a perfect definition for the phenomenon of conspicuity. Instead, it offers a method of quantifying conspicuity in a way that is both simple and useful. In the color conspicuity survey, a GUI (graphical user interface) displays two rectangular figures on one rectangular ground as shown in Fig. 1. The subject uses a mouse and a pointer on screen to click one or the other of the two figures, whichever the subject sees as most conspicuous. The selected figure becomes one-fifth smaller and the other becomes one-fifth larger. possible. This is the point at which the two figures appear to have equal conspicuity. The responses are calculated as the area of one figure divided by the sum of both areas, so the possible responses are the area ratios 0.1, 0.2, 0.3... 0.9, each calculated as Aa Rab = (1) A + A a where A is the total area of the subscripted plane in pixels. The corresponding definition of conspicuity is as follows may seem a bit awkward: The relative area of one figure two another, on a different colored ground, when both appear to stand out equally. This definition provides a method of quantifying conspicuity that is both simple enough to model and complex enough to involve these necessary aspects: 1) Conspicuity is relative to at least one other color. 2) Conspicuity is relative to ground. 3) Conspicuity is linked to another element of design (in this case, relative area) to have a useful purpose in design. These experiments have provided data for several models of conspicuity, including the one described in the next subsection. This data is in the form of 7,995 responses to the experiment described, for with various combinations of hue, saturation, and lightness, as previously described. The goal of modeling conspicuity as defined here is to construct a system which responds to any individual response on the color balance experiment with a high degree of correlation with actual responses to the experiment. b 3.2 System Objective Fig. 1. The Color Conspicuity Survey Since conspicuity has been shown to be related to relative area (Ashizawa, 1981), the selected figure is expected to lose some conspicuity as its area becomes smaller. The subject repeats these actions until finding the most balanced response of the nine A previous quantification of conspicuity, using the data from the experiments described briefly above, defined a set of rules for related to simple color relationships. Each rule was constructed to quantify conspicuity by linear regression, and each rule had an effectiveness determined by its correlation with the actual responses on the experiments. The sum of each inference multiplied by its effectiveness gave a response very close to the individual responses on the color conspicuity surveys. For all 7,995 responses, this method had a
correlation coefficient of 0.59, fairly high for this somewhat subjective response. An advantage of this rule-based method was that the predictability of the rules could theoretically inform the user of important aspects of conspicuity. Such extrapolation would be necessary for the linguistic advice that had been proposed. However, testing of a prototype showed that the linguistic advice was most probably not going to be useful in color support because it left the user in a familiar place: knowing that there is some problem with the design but not knowing how to fix it. In order for a text-based system to effective, it was determined, the system would practically have to tell the user which colors might lead to greater ease-of-use. For this purpose, the best interface is not the text box, but the palette. A palette-based support system shows the user colors that have a high probability to leading to greater ease-of-use. Generating a set of recommended color changes requires a more effective conspicuity inference than the previous model and something that the previous model did not contain, conspicuity optimums directly linking conspicuity to ease-of-use. The trade-off for providing this increase in efficiency is that elimination of the text-based support allows for so-called "black box" methods of conspicuity. A neural network is a network of nodes that are weighted to provide non-linear modeling of data. A new conspicuity model based on a neural network provides more accurate response to any given color conspicuity survey set of three colors. This neural network is described in the following subsection. The object of the system described in this paper is to give the designer a small set of possible color changes for any given color in a flatly-colored visualization. These recommended colors will be optional and will be varied enough to give the designer a good deal of flexibility. More importantly, the system will infer ease-of-use from each color change and only display the better colors inferred to improve the current overall ease-of-use for the visualization. 3.3 Conspicuity Neural Network We constructed a small neural network to model conspicuity from the experimental data described briefly above. This neural network is central to the design and implementation of the system described below. This section provides a brief description to show how this network fits in with the other parts of the support system. (For a general treatment of neural networks see Van Camp, 1992. This network is built to very similar specifications.) The conspicuity neural network accepts nine inputs, the RGB values of each color in the conspicuity surveys. There is an input later, one hidden layer of nine neural units, and a single output unit. The output of the network, is the inferred ratio of the area of the two figures in the survey. So the data from the experiment trains the network directly. The training algorithm is the back-propagation algorithm, which has been widely applied to a great variety of problems. The basic layout of inputs and outputs is shown in Fig. 2. In this figure, z R, z G, and z B are the RGB values of the ground, and the other inputs are the RGB values for a and b. The output o ( z, a, b) is the networks inference for the relative area of a and b at balance, as described above and the network is trained to output a value close to y i, the actual response from the color balance experiments. Fig. 2. Conspicuity network inputs and outputs. The neural network models the conspicuity responses far better than the previous rule-based
system. It achieves a near perfect correlation with the average score for any given set of three colors and a correlation of 0.72 for any individual response to the 7,995 data sets. 4 System Construction 4.1 Image Storage The image storage in this system is designed to store a simple bitmap image in RGB (red, green and blue color model) integers. In this implementation, images are stored in a method convenient to the scope of this research. Since the research deals only with flatly-colored images, all pixels of one color are placed in a bitmap called a color plane. The color plane format is not rare for image formats. Also, images using this system do not have to be stored in any particular format. An image object or application using this system needs only to construct an image information object, as described here. The image information object is constructed with a top level holding information that changes with color placement, the RGB values of every color plane in the image. The bottom layer holds information that does not change with color placement, the relative size ratios of every color plane. From this simple information, the system must infer ease-of-use of an entire visualization image. 4.2 List Generator Core to the current palette, or to the user's preferred palette. The recommended_list first makes a temporary list of possible color changes from the selected color. The recommended_list object first sends the current color scheme through the generator to compute the current inferred ease-of-use. Here we will describe this method, which is exactly the same for every trial color scheme. The generator forms one color plane triad {a,b,z}, where a is the color plane for which the system is inferring conspicuity, b is another plane, and z is a third plane representing the ground, just as in the color conspicuity surveys. Next, the triad's nine RGB values from image_information go in to the conspicuity neural network to obtain an inferred area ratio between the two colors a and b. The generator does this for planes a and b for every plane z and sums these results for inferred relative area of a for a and b. Next, the generator obtains the actual relative area from image_information and takes the difference. The inferred area ratio of color plane a as it relates to b is calculated by the sum of all weighted outputs of the network for all ground planes z. Summarizing these steps, the inferred ratio K ab for the two color planes is given as K ab = # o(z,a,b) " g z (2) z The core of the conspicuity-based color placement support system is a recommended color change list generator, as shown in Fig. 3. An image information object creates a new instance of the recommended color change list object, recommended_list, by sending the number of the selected color which is to be changed and a reference to itself, the image information. Every recommended_list object has a reference to the color model to be used. Some applications may limit the user's colors to the current color table,
" b q size ( K R ) ab Qa ( size) = (4) n 1 ab where size = { L, S}, R ab is the actual ratio of plane a to plane b as given for the conspicuity surveys, K ab is as given above and q size is the best conspicuity difference described in the following section. There are two possible reasons for different optimums for large and small figures, one being that conspicuity inference in this system relies heavily on size in area and there may be a discrepancy here. Another quite likely reason is that large figures, for example background colors and large areas must be inconspicuous. The best values are set less than zero for the large figures, at q L =-0.13 and q S =0.18 for small figures. These best values are constants which give the direct connection to ease-of use. The method for determining these best values and the preliminary evaluation of the system are discussed briefly in the following subsection. The total conspicuity differences are weighted and added to two totals, one for large color planes and another for small planes. Fig. 4 shows the subsets that weight the planes by size, using the same total area ratio that was used to calculate ground. Fig. 3. Generating the recommended_list. Where o(z,a,b) is the output of the conspicuity network. g z is a weight for how much plane z acts as a ground to planes a and b and is given as Az g z = (3) A i where A i is the area in pixels of the subscripted color plane. This is the ratio of plane z to the total area of the visualization. Next the generator calculates total conspicuity differences Q al and Q as for large planes and small planes, respectively. i Fig. 4. The weight of a plane according to size. Finally, the two best totals are combined to form one total and the whole ease-of-use score for the visualization. The list generator exchanges the selected color with a color from the trial color list and calculates an
ease-of-use inference Q a from for the modified color a in the new trial scheme. Trial color selections with a lower score (closer to the best conspicuity ) are added to the recommended color list. The list may be returned to the caller, the module or program that requested it, in order of increasing total weighted difference from the best values so the caller may adjust its size appropriately. When the list is complete, the caller (a palette, color picker, color support system) uses this list to create a new palette of recommended colors. 4.3 Best Conspicuity As described above, the best values for conspicuity are determined beforehand by testing conspicuity values for their direct correlation with ease-of-use. To set these best values, we use data from previous color placement support testing. The method of evaluation, the type of visualizations, and the scores have been detailed in previous work (Cooper, Kamei, 1999). Here we provide a description of the type of evaluation data that forms the basis of the conspicuity optimums. On an earlier prototype placement system, twenty novice designers placed colors for ten visualizations each. These visualizations were chosen to represent a variety of typical flatly-colored visualizations as might be created on a desktop computer. The types of visualization are shown in Table 1. Table 1. Ten visual objects evaluated type of visual object planes line graph with vertical indices 8 line graph with five variables 7 quartile box-plot 5 stacked totals bar-plot 6 scatter plot with one index 5 satellite weather map 7 apartment floor plan 6 wire molecule projection 4 CAD contour drawing 4 3D wire and shadow graph 8 The resulting 200 designs were evaluated by comparison in sets of ten, by ten viewers each. With this comparison method, the viewer first browses a set of ten designs. Then reduced versions of the designs appear on the screen and the viewer uses the GUI to manipulate the order in which the reduced versions appear. At any time, the viewer is able to access the full-sized versions. After determining the order of descending ease-of-use, the viewer clicks a GUI button and the survey automatically records the response. We tune the best values in the system described above to have high correlation with the ease-of-use scores. Previous systems were set to a conspicuity value of zero. Using conspicuity alone, the previous system was able to achieve a correlation with ease-of-use of 0.089. To improve this, the optimums q L and q S, mentioned in the previous section were found by testing values from maximum of 1.0 to minimum of -1.0. These best values allow the system to predict ease-of-use with a correlation coefficient of -0.48 for large planes and of 0.25 for small planes. The relationship is negative because as the conspicuity value differs from the optimum, the ease-of-use score drops. These correlations are much higher than those of the previous implementations that also used only conspicuity inference. These conspicuity values seem to be quantifying other factors of ease-of-use such as visibility and contrast. This gives the system a much more powerful and broad base, one that is an effective improvement of conspicuity alone. 5 Conclusions We constructed a system that generates a small palette of recommended colors for a selected color in a flatly-colored visualization. At the core of this system is a neural network that effectively models conspicuity better than any previous system. The other powerful component is a conspicuity optimizing process that links conspicuity values directly to ease-of-use scores.
The main characteristics of the evaluation system are comparative and subjective ease-of-use evaluation. Since we aim to create systems which assist the user in finding the best color scheme for a particular design, comparative evaluation is the most suitable. Subjective ease-of-use evaluation is prone to questions of validity. The relationship between these subjective evaluations and actual ease-of-use must be investigated in future work. This system is designed to help designers find colors schemes that viewers rate highly for ease-of-use and this limited goal is satisfied by subjective evaluation. The use of small palette to give advice seems simple enough. This simplicity is deceptive because we do not have any idea how the novice designers and those with more experience will react to the system. We certainly would be disappointed if this support system was considered another unnecessary feature to be turned off in the application preferences. The system requires rigorous testing to ensure the interaction produces easy to use visualization images, as outlined in the development process. Color placement support in one form or another will eventually make its way into the features of every application concerned with design. This highly-generalized research provides a base on which more targeted systems can be constructed to assist designers in one of the most difficult tasks they face. References Silverstein, L. Human Factors for Color Display Systems: Concepts, Methods, and Research. Color and the Computer. Orlando, FL: Academic Press; 1987. pp. 27-61. Cooper, E, Kamei, K. (1999). Development of a Color Balance Support System. Journal of Human Interface Society;. Vol. 1, No. 4, pp. 73-80 E. Cooper and K. Kamei (2002). A Study of Color Conspicuity for Ease-Of-Use Inference in Visualization, Color Research and Application, Vol. 27, No. 2, pp. 74-82. Ashizawa, I (1981). Size Effect in Color Conspicuity. Journal of the Color Science Association of Japan;. Vol. 18, No. 3. pp. 200-204. Van Camp, D (1992). The Amateur Scientist: Neurons for Computers. Scientific American; September 1992 pp. 125-12