CLOUD CHARACTERIZATION USING LOCAL TEXTURE INFORMATION. Antti Isosalo, Markus Turtinen and Matti Pietikäinen

CLOUD CHARACTERIZATION USING LOCAL TEXTURE INFORMATION Antti Isosalo, Markus Turtinen and Matti Pietikäinen Machine Vision Group Department of Electrical and Information Engineering University of Oulu, Finland {isosaloa,dillian,mkp}@ee.oulu.fi ABSTRACT In this paper, local texture information is used to classify clouds in the sky views. The proposed system is designed for practical meteorology using a camera directed to horizon. The goal is to study how local texture measures, namely local binary patterns (LBP) and local edge patterns (LEP), can distinguish different cloud genres. The approach indicated very good performance in a five class classification experiment. The results were promising also in a more difficult case where all the ten cloud genres and clear sky were considered. As known, land-based cloud classification resolves many problems which are present in satellite images. 1. INTRODUCTION Weather has become a topic of interest in unexpected levels and not the least because of the feared climate change. The need for comprehensive and consistent cloud observations is highly important. Normally clouds are being detected by human observers, satellites, humidity soundings, ceilometers and pyrometers. Nevertheless of continuous observation of clouds and sky, the automatic analysis and weather prediction is still an open issue. This is reasonable because giving a description of the cloudy sky is a very difficult problem. From the practical meteorological point of view, cloud altitude and general cloud appearance combined to other indicators such as air temperature and pressure are equally important to create reliable weather predictions. This makes automatic description of the weather more difficult. Despite the possible challenges, there is a clear requirement for automatic cloud classification. Machine vision provides an interesting set of tools for cloud analysis. Typical vision-based approaches use satellite images [1, 2]. Though functional systems exist, there are some limitations related to this kind of approach that lower the performance. The problems are mainly caused by thick cloud layers that prevent the camera eye from seeing what is hidden behind the topmost (e.g. high or middle Fig. 1. An example of an all-sky image. clouds) opaque (non-see-through) layer. In this work, the land-based imaging is used instead. The land-based cloud classification faces similar problems caused by low or middle clouds, but separate clouds and cloud types can be more easily distinguished from the earth surface. Automatic cloud classification using land-based imaging has been a topic of some research. Peura et al. [3] used all-sky imager data (see Fig. 1) using a wide-angle optics and extracted features from the images using different convolution masks. They classified cloud images into predefined categories using k-nn classifier. Similarly, Buch and Sun [4] used whole-sky imager (WSI) and tried to classify each pixel in the image with a tree-based approach. They used texture, position information and pixel brightness features in their system. In this paper, instead of using a wholesky imager, a camera directed to the horizon is used. The camera was attached to a weather radar tower positioned over a hill. When observing the sky from a bit higher than the earth surface, the only limiting factor is the meteorological visibility not landscape or vegetation. Traditional weather and cloud observations (see URL: http://www.metoffice.gov.uk/publications/clouds) are made specifically from earth surface, so this must be kept in

mind when using this kind of arrangement. With the kind of images used in this research, vertical dimensions appear nicely when clouds are viewed from the side. This opens more possibilities to detect Cumulonimbus clouds Cumulonimbus clouds are normally associated with thunderstorms and showers of rain. When the camera is directed to horizon we do not need to rely on darkness to detect thick clouds, as it is affected by the amount and the thickness of cloud layers above and overall cloudiness. There are also less undesired reflections by default in this imagery than the all-sky images sadly contain. In this paper, cloud classification is considered as a texture characterization problem. Cloud texture appearance is modelled with local texture measures using the local binary pattern (LBP) methodology [5]. LBP is invariant against any monotonic changes in the gray-scale and has proven to be very robust in numerous real-world texture analysis tasks. The robustness in varying illumination conditions makes the LBP attractive for cloud texture classification as well. Instead of being a pure texture statistic, LBP has turned out to be more general visual descriptor [6]. It combines the microtextons and shape of texture elements in a compact way. This is useful in cloud image analysis where the shapes of the cloud are in important role. In this paper, different versions of the LBP and its extensions are systematically experimented in the cloud classification application. In addition, the local edge patterns (LEP) [7] based on the LBP are also experimented to confirm the hypotheses of the usefulness of the local texture information in cloud classification. 2. PROPOSED APPROACH The strength of texture is that it is based on instinctive impression of visual similarity. For one s part, the feature extraction method and the metric used in comparing feature vectors play a key role in achieving comparative classification results. The local image features extracted with LBP or LEP represent wide variety of different microtextons. Statistical properties of such texture elements are able to encode huge amount of information about the image content. In cloud image analysis application LBP and LEP can be used to describe the general appearance of the cloud as well as the shape and structures of different clouds. Typically different cloud genres can be visually distinguished using these characteristics. 2.1. Local Binary Patterns The local binary pattern (LBP) operator is a theoretically simple, yet very powerful and gray-scale invariant method of analyzing textures [5]. In practice, the LBP operator combines characteristics of statistical and structural texture analysis: it describes the texture with micro-primitives, often called textons, and their statistical placement rules. LBP is very easy to calculate. For each pixel in an image, a binary code is produced by thresholding its neighborhood (8 pixels) with the value of the center pixel. A histogram is then constructed to collect up the occurrences of different binary patterns representing different types of curved edges, spots, flat areas etc. In this study, the most common variants of LBP were used [6]. LBP-features are extracted from gray-scale images. With multi-scale LBP u2 8,1+16,2+24,3 there were need for scaling the pictures as described in [8]. With multi-scale LBPF u2 8,1+8,2+8,3 [9], where F stands for filtering, Gaussian low-pass filtering was used. In the multi-scale approach the resulting feature vector is concatenated from histograms obtained by different radii and/or number of samples in a circular neighborhood. For example in multi-resolution LBP u2 8,1+16,2+24,3 the first operator uses a neighborhood radius of one pixel and eight neighborhood samples, the second operator uses two as the neighborhood radius, and considers 16 neighborhood samples and the third operator uses three as the neighborhood radius, and considers 24 neighborhood samples u2 stands for uniform codes (at most two one-to-zero or zero-to-one transitions in the circular binary code) as presented in [5]. For classification with the LBP, a k-nn classifier with the χ 2 -distance measure was used. 2.2. Local Edge Patterns The LEP is similar to the LBP, but it is calculated from the edge images [7, 10]. The motivation of using LEP is its robustness in translation, rotation and scale of textures. In addition, it is known that the shapes of cloud edges provide valuable supporting information to both traditional and automated cloud classification. These are automatically incorporated in the LEP features. Fig. 2 shows how the LEP features are extracted. In practice, the feature histogram has twice as many bins as the LBP because the edge pixels have to be considered separately. Before the LEP features can be calculated, some preprocessing is needed. The threshold to be used with LEP in the edge image creation proposed by Yao and Chen [10] was too high for our application. Linked to the same problem, some normalization was also needed to minimize the effects brought by sunlight. Therefore a normalization procedure, say modified FCF-method, presented by Liao and Chen [7] was utilized it is based on FCF-method proposed by Finlayson et al. [11]. There was some loss of information with the normalization method, but within limits of acceptance. Then the luminance component representing gray-scale information is obtained using the equation

a block from an edge image 1 1 1 1 1 0 0 0 1 1 2 4 128 256 0 0 0 16 LEP mask 1 2 4 128 256 8 64 32 16 multiplication LEP code for the center pixel in an edge image block: 1 + 2 + 4 + 16 + 128 + 256 = 407 Fig. 2. LEP feature extraction. Y n = 0.299R n + 0.587G n + 0.114B n, where R n, G n, and B n are the normalized RGB components. Also an image specific (adaptive) threshold was used to produce the binary image from which edges were detected using the Sobel method (which suits well for the task [10]). Histograms consisting feature distributions obtained by different versions of LEP, namely LEPSEG and LEPINV, were calculated as described in [10]. In the actual experiments we decided to make concatenated versions of LEP- SEG and LEPINV. It provided the best results in terms of mean classification rate. Only the tails of LEPSEG and LEPINV were used. The tail consists of bins that originate from those center pixels of the edge image block that are edge pixels. The result is referred as LEP 8,1 in Table 1. The same lookup tables were used with LEPINV as with the LBP to achieve codes that meet different conditions, here rotation invariance. Yao and Chen used histogram intersection as similarity measure. We have used the χ 2 -distance with k-nn, since distance functions can be used to achieve the same goal. It also helps us to perform comparison between the two methods, LBP and LEP. In the classification, each cloud texture sample was modelled with its own feature histogram. For actual classification, a k-nn classifier with χ 2 -distance measure was used. At this point, no other classifiers were utilized. Classification was done by dividing the whole data into separate training and testing sets. Tests were repeated several times to increase statistical accuracy and reliability. Results are presented as confusion matrices and also by mean classification rate, the rate of correctly classified samples, related to the repetitions of the test. In detail, the mean classification rate is the mean of all calculated classification rates attained during the mentioned iterations. Classification rate is the sum of diagonal values of the confusion matrix divided by the number of the samples that are currently tried to classify. The results are represented percentages. The weakness of k-nn classifier concerning high dimensional feature vectors is noted, but the fight against the curse of dimensionality was out of the scope of this research. 3. EXPERIMENTS This section describes our test arrangements and provides results obtained with different LBP and LEP operators. 3.1. The Data Used The Finnish Meteorological Institute (FMI) has 10 (year 2006) similar cameras in active use providing sky images similar to the ones used in this work (see URL: http://www.vimpeli.fi/default.aspx?id=408363). The camera, Ikegami ICD-848P, is equipped with normal optics and it employs a high-sensitivity CCD-sensor with about 440,000 pixels. The original images (704 by 576 pixels and 16 million colors) are stored in JPEG-format before they are transmitted to FMI s server. An example image is shown in Fig. 3. 2.3. Classification Fig. 3. An example of the original images used in our experiments. The camera is static and therefore it was very straightforward to crop the images in our disposal in a way that only

Table 1. The mean classification rates obtained with different LBP and LEP operators Method Bins 11 classes 5 classes LPB 8,1 256 49% 69% LPB ri 8,1 36 37% 64% LPB u2 8,1 59 49% 70% LPB riu2 8,1 10 38% 60% LBP u2 8,1+16,2+24,3 857 49% 71% LBP riu2 8,1+16,2+24,3 54 42% 65% MS-LBPF u2 8,1+8,2+8,3 177 49% 68% MS-LBP u2 8,1+16,2+24,3 2571 57% 80% MS-LBP riu2 8,1+16,2+24,3 162 47% 76% LEP 8,1 292 32% 52% Fig. 4. An example of the cropped input image and the samples created from it using the labels shown in the middle. the effective area was left in the images, e.g. clouds and sky. At this point, the resulting images were saved in such a format that no more information is lost, since the original images were only available on JPEG-format as mentioned. We selected 623 images (out of over 10000) for the labeling procedure. The labeling was done manually using a simple MATLAB-tool. Doing so, 479 samples were produced for the experiments. The distribution of samples is the following: 23 Stratus (St), 71 Stratocumulus (Sc), 27 Cumulus (Cu), 29 Cumulonimbus (Cb), 9 Nimbostratus (Ns), 42 Altostratus (As), 68 Altocumulus (Ac), 2 Cirrocumulus (Cc), 65 Cirrus (Ci), 33 Cirrostratus (Cs) and 109 sky samples. A sample of cropped input image and result images (obtained by applying the results of the labeling process to original image) in one example case are shown in Fig. 4. Because of the fact that one the authors had more than adequate weather observation training and operational experience on traditional cloud classification, we were able to carry out the labeling without outside assistance. The task was time to time quite technical without the help of cloud height information, but not so tedious as often described. Possibility of misclassification still exists. 3.2. Tests and Results Features were extracted globally throughout the input image the black areas, seen in sample image (see Fig. 4) presenting the extraction of samples, are discarded in the calculation. The current implementations of multi-scale LBPF and multi-scale LBP, that are well equipped to capture largescale structures, were time consuming to calculate. Results were promising and therefore worth to wait. Two different classification set-ups were used. In the first, all the ten cloud categories and the sky were tried to be classified. In the second set-up, five categories (St, Cu, Ac, Ci, and clear sky) similarly to [4] were considered to perform some comparison. The results obtained with different LBP and LEP operators are collected in Table 1. It shows the mean classification rate for each texture operator in both classification tasks. In addition it shows the number of bins used in each feature histogram. The prefix MS in the table refers to multi-scale approach. Though it is agreed that edges contain a considerable amount of information to distinguish different clouds automatically, LEP did not solve the problem quite the way we hoped. There are, for instance, found edges from clear sky and almost transparent high clouds are occasionally left undetected by the edge detector. Even though improvement in resolution in the imagery might in itself help, features are not working as desired. The threshold value used in the pre-processing of images before calculating LEP is very important in LEP feature extraction. It will require more research on what kind of pre-processing should be used with LEP. Again, attached to the low resolution, the fibrous structure of high clouds were occasionally out of reach for LEP. Smooth clouds (like Altostratus and Cirrostratus) with edges in infinity were similarly problematic. We have selected here representative classification results. Figs. 5 and 6 represent confusion matrices obtained by extracting features with LBP u2 8,1-operator. Figs. 7 and 8 illustrate classification results achieved by performing feature extraction with multi-scale LBP u2 8,1+16,2+24,3. Numerical values contained by the confusion matrices are percentage values. LBP is able to extract well discriminative features also

St Sc Cu Cb Ns As Ac Cc Ci Cs Sky St 20 6 0 8 10 6 2 0 2 7 1 Sc 23 36 26 16 5 13 19 0 12 2 0 Cu 1 10 35 18 0 0 2 0 0 0 0 Cb 1 5 19 31 0 6 5 44 4 2 0 Ns 11 2 0 0 40 6 1 0 0 2 0 As 11 11 3 5 29 32 9 0 2 10 2 Ac 2 19 9 14 13 9 43 0 14 3 0 Cc 0 0 0 0 0 0 1 0 2 0 0 Ci 11 8 6 7 0 10 14 56 50 21 7 Cs 11 3 0 0 2 4 12 0 7 32 6 Sky 10 0 1 1 2 7 0 0 6 19 83 St Sc Cu Cb Ns As Ac Cc Ci Cs Sky St 24 7 2 5 13 1 3 0 3 6 1 Sc 14 37 20 10 12 9 20 0 9 1 0 Cu 4 10 44 23 0 10 4 0 1 0 0 Cb 8 7 18 40 1 10 4 0 1 0 0 Ns 6 2 0 0 23 5 1 0 0 1 0 As 14 7 3 6 35 41 5 0 3 16 0 Ac 0 19 7 10 13 5 54 0 44 7 0 Cc 0 0 0 0 0 0 0 0 3 0 0 Ci 22 10 6 3 0 9 7 100 68 16 2 Cs 3 1 0 2 3 12 3 0 7 49 2 Sky 5 0 0 0 0 5 1 0 2 3 95 Fig. 5. The confusion matrix obtained with ten cloud genres and the sky class using LBP u2 8,1 features. St Cu Ac Ci Sky St 31 9 10 9 1 Cu 10 64 6 1 0 Ac 19 17 60 20 1 Ci 22 9 22 61 9 Sky 17 1 3 9 89 Fig. 7. The confusion matrix obtained with ten cloud genres and the sky class using LBP u2 8,1+16,2+24,3 features. St Cu Ac Ci Sky St 36 12 9 8 1 Cu 8 70 5 3 0 Ac 21 8 70 7 0 Ci 32 10 15 78 2 Sky 3 0 1 4 97 Fig. 6. The confusion matrix obtained with the four cloud genres and the sky class using LBP u2 8,1 features. from thin and smooth clouds. Understandable confusion can be observed in couple of situations. Certain forms of Altocumulus and Stratocumulus clouds with similar kind of appearance are misclassified in some cases they are both layered clouds, Stratocumulus being broader. Cloud base height information would bring solution to this problem. The fact that Cirrocumulus is classified as Cirrus, as it is in the confusion matrix in Fig. 7, is quite satisfactory. The Cirrocumulus is often quite a short term form. Here the misclassification is tied to the fact that there were only 2 samples of this cloud genre zero samples for training and two for testing for example. Cumulonimbus (incus) consists of veil-like top (formed of ice crystals) that resembles (and also acts as origin of thick) high clouds. There are, of course, Cumulus like forms in Cumulonimbus. Cumulonimbus clouds evolve from Cumulus clouds, but have more massive vertical dimension. These facts indicate that LBP features have no significant problems. These are more Fig. 8. The confusion matrix obtained with the four cloud genres and the sky class using LBP u2 8,1+16,2+24,3 features. modelling related problems. LBP works quite well as a feature extractor in the cloud classification problem. The problems that arise from the number of samples per class appear especially in the five class classification experiment. Filtering removes some unwanted elements from images. Down side is that some useful elements are also faded away due to the resolution. Usually, processing with multiple scales helps in incorporating more information about the texture and makes the approach more robust to scale variations as well. The multi-scale LBP features indicated very promising results in the cloud texture classification. This is reasonable because clouds appear in very broad scales in the images. 4. CONCLUSIONS Two things, clouds and texture, are brought here together either of them have a definition with unanimous acceptance.

Texture is a salient elementary feature in natural scene images. Here it manages to distinguish different cloud genres with very good performance by harnessing the characterization powers of local patterns. The approach indicated a very good performance in the classification experiment, where five classes (four cloud genres and clear sky) were considered. The results were promising also in the more difficult case where all the ten cloud genres and clear sky were considered. It is, though, safe to say that the cloud characterization problem can not be solved entirely by using just textural features no matter how large amount of human labeled training data is available. Peura et al. [3] were able to classify all cloud genres except Cumulonimbus with a good accuracy from all-sky imagery reaching classification rate of 65% (correct classifications). The performance that they have achieved is mostly due to their extensive and well-thought-out set of features. As their imaging setup, sample set and generalization used to present the results of the classification are so different from our approach, it is difficult to make direct comparison. The features that Peura et al. have developed could also be applied in our research, but it was decided to limit to local texture information only. The aim was to propose new features to supplement the existing ones. Our feature extractor could be a worthy addition to those presented in [3]. Buch and Sun [4] reported a overall misclassification rate of 39%. With that, we are able compete with our most efficient methods in terms of classification rate. Surprisingly, the uniform sky texture was not that easy to distinguish as it would be expected. The use of adaptive color or texture based segmentation might help to segment the clouds automatically from the sky views. However, better quality images would be more than welcome in our approach anyway. It would be reasonable to suggest that cloud images, for a more comprehensive study, should be obtained simultaneously with weather observations, and classified manually immediately after taking them. Cloud height information (Ceilometer data) should also be collected at the same time as well to make verification of the labels given later on. It might be useful to create natural language descriptions automatically from sky images, e.g. descriptions of regions in an image. Each region could, for example, get the label that best describes that area, e.g. which cloud class has the majority over others in the section that is currently being processed. The fact that the classification results will remain quite local (with this kind of arrangement) is not a disbenefit other variables like temperature and air pressure are also measured locally throughout the globe. Though, a better sky coverage using, for example, rotating camera would be of interest in the future. Acknowledgements The Finnish Meteorological Institute is acknowledged for providing the sky images. 5. REFERENCES [1] A. Visa, K. Valkealahti, J. Iivarinen, and O. Simula, Experiences from operational cloud classifier based on self-organizing map, in Application of Artificial Neural Networks, SPIE 2243, 1994, pp. 484 495. [2] A. Visa and J. Iivarinen, Evolution and evaluation of a trainable cloud classifier, IEEE Transactions on Geoscience and Remote Sensing, vol. 35, no. 5, pp. 1307 1315, September 1997. [3] M. Peura, A. Visa, and P. Kostamo, A new approach to land-based cloud classification, in International Conference on Pattern Recognition, 1996, pp. 143 147. [4] K. Buch and C. Sun, Cloud classification using whole-sky imager data, in 9th Symposium on Meteorological Observations and Instrumentation, 1995, pp. 353 358. [5] T. Ojala, M. Pietikäinen, and T. Mäenpää, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 7, pp. 971 987, 2002. [6] T. Mäenpää and M. Pietikäinen, Texture analysis with local binary patterns, in C. Chen and P. Wang (eds.) Handbook of Pattern Recognition and Vision. 2005, pp. 197 216, World Scientific, 3rd Edition. [7] C-J. Liao and S-Y. Chen, Complementary retrieval for distorted images, Pattern Recognition, vol. 35, no. 8, pp. 1705 1722, August 2002. [8] M. Turtinen and M. Pietikäinen, Contextual analysis of textured scene images, in British Machine Vision Conference, 2006, pp. 849 858. [9] T. Mäenpää and M. Pietikäinen, Multi-scale binary patterns for texture analysis, in 13th Scandinavian Conference on Image Analysis, 2003, pp. 885 892. [10] C-H. Yao and S-Y. Chen, Retrieval of translated, rotated and scaled color textures, Pattern Recognition, vol. 36, no. 4, pp. 913 929, April 2003. [11] G. Finlayson, S. Chatterjee, and B. Funt, Color angular indexing, in European Conference on Computer Vision (ECCV), 1996, pp. 16 27.