Design of Efficient Algorithms for Image Compression with Application to Medical Images

Transcription

1 Design of Efficient Algorithms for Image Compression with Application to Medical Images Ph.D. dissertation Alexandre Krivoulets IT University of Copenhagen February 18, 2004

2

3 Abstract This thesis covers different topics on design of image compression algorithms. The main focus in this work is development of efficient entropy coding algorithms, development of optimization techniques for context modeling (with respect to the minimum code length) and application of the methods in the design of an algorithm for compression of medical images. Specifically, we study entropy coding methods based on a technique of binary decomposition of source symbols. We show that the binarization allows to fit a parametric probability distribution model to a source coding algorithm, e.g., arithmetic coding, reducing the number of coding parameters to that of the distribution model. Context modeling is an essential part of an image coding algorithm, which basically defines its compression performance. In the thesis, we describe a unified approach to this problem based on statistics learning methods and the minimum description length principle. In particular, we present a design of optimized models using context quantization and initialization techniques. The optimization allows to find a model, which yields the minimum code length for some set of training data samples. Entropy coding and context modeling methods are applied for developing a compression algorithm intended for medical images. The algorithm allows for progressive near-lossless coding and is based on lossy plus refinement layered approach. We show that this method results in a better compression performance and image quality for large distortion values compared with the recently adopted standard JPEG-LS for lossless and near-lossless image compression. We also investigate a possibility of image reconstruction with the minimum mean squared error criterion within the proposed framework. i

4 ii

5 Contents Contents iii 1 Introduction Motivation and main goals Previous work About this thesis Source coding and image compression Information sources Source coding Arithmetic coding Universal source coding Compression of images Source coding via binary decomposition Introduction The binary decomposition technique On redundancy of binary decomposition Binary decomposition of FSM sources Binary decomposition and universal coding Applications of binary decomposition Introduction Generalized two-sided geometric distribution Efficient coding of sources with the GTSGD Experimental results On redundancy of rice coding Summary Context modeling for image compression Introduction Context formation Context model optimization Context initialization Context quantization Summary iii

6 iv CONTENTS 6 Optimization in the JPEG2000 standard Introduction Context modeling in JPEG High-order context modeling Experimental results Summary Hierarchical modeling Introduction Two-stage quantization Tree-structured quantization Optimal tree pruning Experimental results Summary Compression of medical images Introduction Embedded near-lossless quantization Entropy coding of the refinement layers Experimental results Reconstruction with minimum MSE criterion Summary Bibliography 89 A Bit rates for test sets of medical images 95 B PSNR for test sets of medical images 117

7 Chapter 1 Introduction 1.1 Motivation and main goals Computer imaging plays a significant role in many areas ranging from consumer digital photo albums to remote earth sensing. The growing production of images and demands to their quality require high performance compression methods for efficient transmission, storage and archival. Most image compression algorithms can be viewed as consisting of an image transformation, which performs decomposition of an image into a sequence of descriptors, followed by entropy coding of the descriptors. Entropy coding, in essence, performs the compression. Prediction, discrete cosine and wavelet transforms are examples of image decomposition, whereas prediction errors and transform coefficients are examples of the descriptors. They constitute an information source for the entropy coder, which encodes the sequence of source symbols according to some source model. This model is normally designed off-line using some assumptions on the data to be coded. The better the model approximates statistical properties of the data, the higher compression can be achieved. The lower bound on the compression performance is defined by the entropy of the source. The design of efficient models is a task of primary interest for any compression algorithm. Most image compression algorithms use the universal coding approach, where the model has fixed structure and unknown parameters. The parameters are estimated on the fly during encoding. This approach allows to adopt the model to the data statistics, which may vary for different data. On the other hand, the price for that adaptivity is the increase of the code length due to the necessity of implicit or explicit transmission of information on the model parameters (the so called model cost). The higher the model order, the more parameters it involves, and the higher the model cost. Thus, there is a trade-off between the order of the model and the overhead data needed to specify the model parameters. The use of domain knowledge of the data allows for reduction of the overhead information. Finding optimal solutions to this problem is one of the goals of this thesis. In this connection, we investigate the technique of source coding via binary decomposition of symbols. The decomposition can be used as a mean for model cost 1

8 2 CHAPTER 1. INTRODUCTION reduction and/or for efficient model optimization. Another goal is the design of an efficient algorithm for compression of medical images. The design of such an algorithm differs from the design of general purpose image compression methods due to some specific requirements. The main concern in medical imaging is the quality of the reconstructed image. There are three kinds of image compression methods: lossless, lossy and near-lossless. Lossless compression methods perfectly reconstruct an image, but they compress data at the lowest degree. Lossy compression techniques are usually used for images intended for human observation in general-purpose computer systems. Such methods are based on the fact that the human visual system does not perceive the high frequency spatial components of the image, and those components may be removed without any visible degradation. Thus, introducing distortion in the reconstructed data, one can achieve higher compression, while preserving its visual information content. Lossy methods have the main advantage that they can compress images at the highest degree. However, the distortion of the data is undesirable in some applications, e.g., in image analysis, object recognition and some others. Near-lossless compression is a lossy image coding algorithm, where the distortion is specified by the maximum allowable absolute difference (the tolerance value) between pixel values of the original and the reconstructed images. The method allows for a rigid and robust control of errors while achieving reasonable compression performance. For that reason near-lossless compression seems to be an attractive method for medical images. In our work, we develop an efficient algorithm, which allows for progressive near-lossless compression up to the lossless mode. The extended functionality and makes the algorithm more suitable for real applications. Compression of medical images also allows for efficient use of the similarity of statistical properties of the data to design high-order source models and thus, to achieve better compression performance. 1.2 Previous work Image coding originates from the communication theory established in 1948 by Shannon in his famous paper [50]. Yet, it took years before the methods developed in this work could be used in practical algorithms for image compression. In the recent decades, image compression has been extensively studied by many researchers. The first lossy and lossless still image coding standard JPEG [33] appeared by the end of 1980 s and had a remarkable performance at that time. The standard uses a block discrete cosine transform (DCT). The DCT was shown to be very close to the optimal decomposition for the class of natural images, performing almost perfect decorrelation of image pixels, see, e.g., [35]. The standard uses an ad-hoc Huffman or arithmetic entropy coding of transform coefficients. The version with arithmetic coding exploits binarization of the coefficients and a simple heuristic context modeling. The JPEG standard remains the most popular image compression algorithm. Introduction of a (discrete) wavelet transform (DWT) in the middle of 1980 s [20,

9 1.3. ABOUT THIS THESIS 3 30, 26] launched a new era in image compression. Besides being good for decorrelation of image pixels, the new transform provides more functionality to the compression algorithm, such as embedded coding, where any initial part of the compressed bit stream can be used to reconstruct the image with the quality in proportion to the length of this part. A benchmark method called the embedded zerotree wavelet (EZW) coding was introduced in [51] and further developed in the SPIHT (Set Partitioning in Hierarchical Trees) algorithm [47]. The use of integer-to-integer (reversible) wavelet transform [9] allowed for progressive coding up to the lossless image reconstruction [8, 48, 3, 64]. Development of context modeling techniques and rate-distortion optimization lead to a remarkable improvement of compression efficiency of algorithms based on DWT. The most sophisticated methods are the high-order embedded entropy coder of wavelet coefficients (ECECOW) [64], embedded block coding with optimized trancation (EBCOT) [55], and the compression with reversible embedded wavelets (CREW) algorithm [69], just to name a few. The algorithm EBCOT was adopted as a basis for the new standard for still image coding JPEG2000 [56]. The best compression results for lossless image compression were obtained by algorithms based on predictive coding and context modeling of prediction residuals. For example, algorithm CALIC (Context-based, Adaptive, Lossless Image Codec) [68] uses a sophisticated context modeling and a special kind of prediction (it exploits non-linear, gradient adaptive predictor) to achieve the best compression performance among practical coders. The performance of CALIC turns out to be even better than the UCM (Universal Context Modeling) method [57] developed on the basis of a universal source coding algorithm with provable asymptotic optimality. The ideas from CALIC were employed in the LOCO-I (LOw COmplexity LOssless COmpression for Images) algorithm [59], which became the new standard for lossless image coding JPEG-LS [60]. The standard also supports near-lossless compression. Near-lossless compression first appeared in [10] as a method allowing for much better compression performance compared to the lossless coding at the price of a very small and controllable distortion of the image. It was further elaborated in [65] to achieve higher compression efficiency. The fist method allowing progressive coding was proposed in [5]. The major improvements in compression performance are due to development of new techniques for context modeling, such as context quantization [67], and the use of parametrized probability distribution models (LOCO-I). 1.3 About this thesis The thesis covers two major problems in image compression: entropy coding and context modeling. For efficient entropy coding we studied the use of a binary decomposition technique with application to coding of sources with parametrized distribution. For design of high-order context models we developed optimization methods based on training data samples. The proposed solutions are tested in a series of experiments and applied in the design

10 4 CHAPTER 1. INTRODUCTION of an algorithm for near-lossless compression primarily intended for coding medical images. The thesis is organized such that most chapters are self-contained is the sense that they cover different topics, yet in the same framework. The material of the thesis is based on the following papers: 1. A. Krivoulets, On redundancy of coding using binary tree decomposition, in Proc IEEE Int. Workshop on Inf. Theory, p. 200, Oct., Bangalore, India, (Chapter 3) 2. V.F. Babkin, A.G. Krivoulets, On coding of sources with Laplacian distribution, in Proc. of Popov s Society Conference, p. 254, May, 2000, Moscow, Russia. (in Russian) (Chapter 4) 3. A. Krivoulets, On coding of sources with two sided geometric distribution using binary decomposition, in Proc Data Compression Conf., p. 459, Snowbird, UT, Apr., (Chapter 4) 4. A. Krivoulets, Efficient entropy coding for image compression, IT University of Copenhagen, Tech. report TR , February, (Chapter 4) 5. A. Krivoulets, Fast and efficient coding of low entropy sources with two-sided geometric distribution, in Proc. of the 2nd European workshop on Advanced video-based surveillance systems, pp , Kingston, UK, Sep., (Chapter 4) 6. A. Krivoulets On redundancy of Rice coding, IT University of Copenhagen, Tech. report TR , September, (Chapter 4) 7. A. Krivoulets, X. Wu, and S. Forchhammer, On optimality of context modeling for bit-plane entropy coding in the JPEG2000 standard, In Proc. VLBV03 Workshop, pp Madrid, Spain, Sep., 2003, (LNCS 2849) (Chapters 5,6) 8. A. Krivoulets and X. Wu, Hierarchical modeling via optimal context quantization, in Proc. 12th International Conference on Image Analysis and Processing, pp , Mantova, Italy, Sep., (Chapter 7) 9. A. Krivoulets, A method for progressive near-lossless image compression, in Proc. ICIP2003, vol. 2, pp , Barcelona, Spain, Sep., (Chapter 8) 10. A. Krivoulets, Progressive near-lossless coding of medical images, in Proc. 3d International Symposium on Image and Signal Processing and Analysis, vol. 1, pp , Rome, Italy, Sep., (Chapter 8) A brief description of the chapters is as follows. Chapter 2 describes the main concepts of image compression algorithms and introduces the framework of our research. In this chapter, we describe basic principles and the place of entropy coding and context modeling in image compression algorithms.

11 1.3. ABOUT THIS THESIS 5 In Chapter 3, we discuss a technique of binary decomposition of source symbols as an efficient mean for entropy coding and present some properties of the technique. Some applications of the technique are presented and discussed in Chapter 4. In this chapter, we introduce a probabilistic model of sources, which often occurs in image compression, and describe efficient methods of coding of such sources using the binarization technique. Binarization allows to reduce the number of coding parameters, which is usually equal to the size of the source alphabet, to the number of probability distribution parameters, which normally is much lower. Binarization also simplifies context model optimization described in the next chapter. Chapter 5 presents basic principles of high-order context modeling for image compression. The chapter covers two main topics of the context model design: context formation and context model optimization. The latter is based on context initialization and quantization based on some prior statistics on the data. In Chapter 6, we describe an application of the high-order context modeling techniques developed in Chapter 5 to optimization of context models adopted in the JPEG2000 standard. The models are used in the bit-plane entropy coding of wavelet transform coefficients and basically define the compression performance of the standard. In this chapter we demonstrate almost optimality of the models adopted in JPEG2000 with the given context template. Chapter 7 extends ideas of the context quantization technique to build a hierarchical set of models intended for a better fit to the actual data. Finally, in Chapter 8, we present an algorithm for near-lossless compression intended for medical images, which allows for progressive coding and reconstruction. In the algorithm design, we exploited the methods developed in Chapters 3,4, and 5. We show, that the resulting algorithm allows for more efficient coding (in terms of both compression performance and functionality) than the recently adopted standard JPEG- LS for lossless and near-lossless image compression.

12 6 CHAPTER 1. INTRODUCTION

13 Chapter 2 Source coding and image compression In this chapter we introduce the basic theory, concepts and definitions of source coding and consider a general structure of image compression algorithms. This background will be used throughout the thesis. 2.1 Information sources By an information source we assume a mechanism generating discrete random variables u from a countable (often finite, but maybe infinite) set A = {a 0, a 1,...,a m 1 }. The set A is called the source alphabet and m = A defines the alphabet size (hereafter, denotes cardinality of a set or the length of a string). Let u t be a random variable generated at the time instance t. A string of source symbols of length n (a source message) is denoted as u n 1 = u 1 u 2 u 3...u n. The empty string u 0 1 is denoted as. Let An be the set of all possible messages of length n in the alphabet A: A n = {u n 1 }. The simplest source model generates an independent and identically distributed (i.i.d.) symbols according to the probability distribution P(A) = {p(a), a A}, which defines the model parameters (probabilities of source symbols). Such a model is called the memoryless source. The distribution P(A) does not depend on the past symbols. A general source model is a finite state machine (FSM) model, which is defined by the (finite) set of states S = {s}, the source alphabet A, the set of conditional probability distributions {P s (A), s S}, P s (A) = {p(a s), a A}, and the initial state s 1. At each instance of time t the source generates a symbol u t A and changes its state from s t to s t+1 according to the state transition rule s t+1 = F(s t, u t 1 ). (2.1) The function F( ) will be referred to as a model structure. The set of probability distributions {P s (A), s S} specifies the set of model parameters. A message u n 1 generated by an FSM source can be decomposed into S subsequences generated at the states s S, which are the i.i.d. sequences drawn according to the distribution P s (A). 7

14 8 CHAPTER 2. SOURCE CODING AND IMAGE COMPRESSION If the state s t+1 is uniquely defined by the previous state s t and the symbol u t generated at this state, i.e., s t+1 = F(s t, u t ), (2.2) then the model is called a Markov source. The structure function F( ) of a Markov source can be described by a directed graph of state transitions, where nodes define the states and edges correspond to the source symbols and define state transitions. Let P ij be the probability of entering the state s = j from the state s = i: P ij = Pr(s t+1 = j s t = i). (2.3) If the state transition rule F( ) is such that the probability (2.3) depends only on the previous state, i.e., Pr(s t+1 s t, s t 1,...) = Pr(s t+1 s t ), then the sequence of the states form a homogeneous Markov chain. If the state is uniquely defined by the o last source symbols u t t o+1, then the source is called an o-order Markov chain. For the o-order Markov chain model we have: and where p(a s) = p(a u t t o+1 ) s t+1 = F(u t t o+1), F(u t t o+1 ) : ut t o+1 s {1, 2,..., Ao }. The FSM model is a good approximation to most practical sources, therefore it is widely used in compression algorithms. The main property of an information source is its entropy defined as where and p(u n 1 ) is the probability of the string un 1. 1 H = lim n n H(An ), (2.4) H(A n ) = p(u n 1 ) log 2 p(u n 1 ), (2.5) p(u n 1 ) An 2.2 Source coding Let B = {0, 1} be a binary alphabet and B be a set of all words in the alphabet B. Source coding is a mapping, which to each source message u n 1 A n assigns a codeword ϕ(u n 1 ) B such that u n 1 can be uniquely reconstructed. A set of codewords {ϕ(u n 1 ), un 1 An } is called a prefix code if no one codeword is a prefix of any other in the set. The use of prefix code guarantees the unique decipherability.

15 2.2. SOURCE CODING 9 The codeword lengths ϕ( ) of any set of prefix codes must satisfy the Kraft inequality: u n 1 An 2 ϕ(un 1 ) 1. (2.6) Conversely, if the set of codeword lengths satisfies (2.6), then there exists a prefix code with these codeword lengths. The set Q(A n ) = {q(u n 1 ) = 2 ϕ(un 1 ), u n 1 An } is called the coding probability distribution on A n. Since the codeword lengths are of main concern, this distribution is often useful in the source coding analysis. Given the source model, the key challenges in source coding are the choice of the codeword lengths and the construction of the codewords. Normally the codewords are chosen to minimize the description length ϕ(u n 1), meaning data compression. Thus, the terms source coding and data compression are often interchanged in the literature, even though source coding has a broader meaning. In the future, we will follow the tradition and use both terms in the same sense. Let L = p(u n 1 ) ϕ(un 1 ) (2.7) u n 1 An be the average code length for the set A n. The source coding theorem ([16], Theorem 3.3.1) establishes the lower bound on L for lossless coding L H(A n ), (2.8) with equality iff ϕ(u n 1) = log 2 p(u n 1). The theorem also states that there exists a prefix code such that L < H(A n )+1. Thus, the minimum possible code length for the message u n 1 is defined by ϕ(u n 1) = log 2 p(u n 1). (2.9) This quantity is called the self-information of the message u n 1. This is the basic idea of source coding: data compression is possible by assigning shorter codewords to more probable messages (symbols) and longer codewords to less probable messages (symbols). Maximum compression is achieved by choosing the codeword lengths equal to minus the logarithm of the probability of a message. The main property of a code is its redundancy. There are different redundancy measures defined in source coding. The two measures, which will be used later in the thesis, are the average redundancy and the individual redundancy R a = L H(A n ), (2.10) R i = ϕ(u n 1) + log 2 p(u n 1), (2.11) respectively. The task of a codeword construction can be solved by using arithmetic coding.

16 10 CHAPTER 2. SOURCE CODING AND IMAGE COMPRESSION 2.3 Arithmetic coding Arithmetic coding is a method for sequential calculation of the codeword ϕ(u n 1) for the source message u n 1. It is based on unpublished Elias algorithm described by Abramson [2] and Jelinec [22]. First practical implementations are due to Rissanen [43] and Pasco [32], who solved the finite precision problem and Witten et al. [61] who made it popular by publishing the C-code of their implementation. In arithmetic coding, the codeword is recursively calculated as a cumulative coding probability of the string u n 1: ϕ(u n 1 ) = a<u 1 q(a ) + q(u 1 ) a<u 2 q(a u 1 ) + + q(u n 1 1 ) a<u n q(a u n 1 1 ), (2.12) where {q t (a u t 1 ), a A, t = 0, 1,..., n 1} is a sequence of conditional coding probability distributions satisfying q t (a u t 1) = q(ut 1 a) q(u t 1), (2.13) such that q t (a u t 1) 1, (2.14) a A n 1 q(u n 1 ) = t=0 q(u t+1 u t 1 ). (2.15) The calculations are assumed to have infinite precision resulting in ideal arithmetic coding. The main property of ideal arithmetic coding is the following theorem, which is a slightly modified version of Theorem 1 from [52]. Theorem 1. Given a sequence of coding distributions {q t (a u t 1), a A, t = 1, 2,..., n} satisfying (2.13), (2.14), an arithmetic coder achieves codeword lengths The codewords form a prefix code. ϕ(u n 1) < log 2 q(u n 1) + 2, u n 1 A n. (2.16) In practice, calculations are performed with finite precision. In this case, estimates on upper bounds of the coding redundancy are defined in the following theorem. Theorem 2. Let r bits be used in a binary representation of the coding probabilities q(a ), a A, and g r + 2-bit registers for calculations. Then R a mn(r + log 2 e)2 (g 2) + 2, (2.17) R i g + (n 1)2 r g. (2.18)

17 2.4. UNIVERSAL SOURCE CODING 11 ENCODER DECODER u ϕ(u) u Arithmetic Arithmetic coder coder q(u t+1 u t 1) q(u t+1 u t 1) Source model Source model Figure 2.1: A block-chart of source coding using arithmetic coder. This theorem essentially is a compilation of Theorem 4 form [46], which establishes the bound for R a, and the estimate on R i given in [41]. It is clear from the theorem, that even with finite precision arithmetic implementation, the coding redundancy is usually negligible. Using arithmetic coding, we can separate the problem of assigning the coding probabilities q(u t+1 u t 1 ), according to some chosen criterion (different criteria will be considered in the next section), from the codeword construction. Source coding using arithmetic coder is schematically represented on Figure 2.1 [1], where the source model does the job of sequential assigning the coding probabilities q(u t+1 u t 1 ) and the arithmetic coder performs sequential calculation of the corresponding code word. 2.4 Universal source coding If the source model is known, then the choice of the coding distribution q(a ) = p(a ), a A allows to calculate a codeword for the message u n 1 using arithmetic coding exceeding the ideal by at most 2 bits (disregarding the arithmetic precision problem). However in practice, the source parameters (or even the underlying source model) are usually not known in advance or may vary for different data. In universal coding, it is assumed that the source belongs to some predefined set of models Ω = {ω}. A code is designed to perform well for all, or most of the models in the set. The set can be just a parametric probabilistic set of sources having the same model structure (e.g., Markov source with the same state transition rule) or may be a double mixture of sources with different structure and a set of parameters [11]. Let ϕ Ω (A n ) B be a prefix code on A n used for all sources in the set Ω and R{ϕ Ω (A n )} be a measure of performance (redundancy) of the code such that R{ϕ Ω (A n )} 0, (2.19) with equality when the set Ω contains only a single element, i.e., when the source is

18 12 CHAPTER 2. SOURCE CODING AND IMAGE COMPRESSION known (the source structure and the parameters are given). Thus, the redundancy of universal coding is only due to the lack of knowledge about the source. Let ϕ Ω = arg inf R{ϕ Ω(A n )} (2.20) ϕ and R = R{ ϕ Ω (A n )}. (2.21) The code ϕ Ω is said to be universal w.r.t the model set Ω if R 0 (2.22) as n [11]. There are two main measures used in universal coding, which are based on the average and individual redundancy. Let and R n a(ϕ Ω, ω) = p(u n 1 ω) An p(u n 1 ω) ϕ Ω (u n 1) H(A n ω) (2.23) Ri n (ϕ Ω, ω) = ϕ Ω (u n 1 ) + log 2 p(u n 1 ω) (2.24) be the average and individual redundancy of the code ϕ Ω for the source ω, respectively. Then the measures of universal coding are defined by the maximum average and individual R a {ϕ Ω (A n 1 )} = sup n Rn a(ϕ Ω, ω) (2.25) ω Ω R i {ϕ Ω (A n 1 )} = sup n Rn i (ϕ Ω, ω) (2.26) per-symbol redundancy. In general, the use of different criteria results in different codes. The criterion (2.26) was first proposed in [53, 54]. Clearly, it is stronger than (2.25) and a code ϕ Ω with good properties according to the criterion (2.26) also implies good behavior w.r.t. the criterion (2.25). The lower bound on the convergence (2.22) w.r.t. to both criteria for the parametric sets of memoryless and FSM sources is defined by [44, 52] ω Ω R K 2n log 2 n + O ( ) 1, (2.27) n where K is the number of free parameters. For memoryless sources K = m 1, and for FSM sources with S states K = S (m 1) (m is the alphabet size). Coding of an FSM source using the sequential coding probability distribution q t+1 (a s) = ϑ(a uts 1 (s)) + α a (s) t s + a A α a(s), (2.28)

19 2.5. COMPRESSION OF IMAGES 13 combined with arithmetic coding, where ϑ(a u ts 1 (s)) is the number of occurrences of the symbol a in the subsequence u ts 1 (s) corresponding the state s, and t s is the length of the subsequence, allows to achieve the optimal convergence rate (2.27). The values {α a (s) > 0, a A, s S} define prior distributions on the source parameters {P s (A), s S}. If nothing is known about P(A) s, then the best choice is α a (s) = 1, a A, s S, see, e.g., [23, 54]. 2 For memoryless sources, (2.28) is reduced to 2.5 Compression of images q t+1 (a) = ϑ(a ut 1 ) + α a t + a A α. (2.29) a A gray-scale digital image 1 is a 2-dimensional array of bounded integer values (image pixels) v[y, x], 1 x X <, 1 y Y <, where y and x define the row and the column coordinates, respectively. The values X and Y define the size of an image. The range of pixel s values is normally defined in terms of the number of bits k required to represent all image pixels. Thus, for a k-bit image, the pixel s value lies in the range 2 v[y, x] [0, 2 k 1]. Most general purpose images use 8-bit representation. Medical images often use bit representation. An example of an image is shown on Figure 2.2, where the small squares represent pixels and the grey scale reflects the pixel s value. In its original representation, an image takes X Y k bits. Image compression aims at reducing this number by using source coding techniques. An image compression system can be described in terms of four functional blocks: image transformation, quantization, encoder model (not to be confused with the source model introduced in Section 2.2), and entropy coder, as it is shown on Figure 2.3, where the mandatory part is the entropy coder. (In general, the image pixels can be fed directly to the entropy coder, however, in most applications this approach would lead to inferior compression performance due to significant spatial correlation of the image pixels.) The image transform part converts an image into a sequence of descriptors being another (abstract) representation of the image. The most used transformation techniques are prediction and a discrete orthogonal transform. The transformations use the fact that image pixels typically have substantial correlation with their neighbors. Predictive coding exploits an auto-regressive model, where the weighted past image pixels is used to predict the value of the next pixel. The difference values (residuals) between the estimated (predicted) and the original pixel values define the sequence of descriptors. The idea of an orthogonal transformation is that the image is represented as a linear combination of some basis functions (waveforms). The transform coefficients are the weighting factors to these functions. If the transform coefficients are real values, then quantization of the coefficients is a necessary step of image coding. 1 In the thesis, we will deal only with 2-D images. 2 Without loss of generality, we assume that pixels can take only non-negative values.

20 14 CHAPTER 2. SOURCE CODING AND IMAGE COMPRESSION x v[y, x] y Figure 2.2: Example of a gray-scale 8-bit image of size pixels. The zero-order entropy of prediction residuals and (quantized) transform coefficients is much lower than that of image pixels. Furthermore, using orthogonal transformation, an image can be represented by a few transform coefficients. All that significantly reduces the amount of storing or transmitting data. The most used kinds of orthogonal transformation for compression are the discrete cosine and wavelet transforms (DCT and DWT, respectively) [36]. Basis functions of the DCT are asymptotically optimal for decorrelation of the 1-st order 1-dimensional stationary Markov process. Nevertheless, 2-D separable transformation still has a good decorrelation property and allows for a high compression capability (retaining most information in a few transform coefficients). There exist algorithms for fast calculation. The DCT is used in many image and video compression algorithms. Examples are the still image coding standard JPEG (Joint Picture Expert Group) [33], and the video coding standard MPEG (Motion Picture Expert Group) [25, 49]. The wavelet transform has a superior energy compaction due to comparatively short basis functions. It fits better to a non-stationary signal like real images. Yet, the main advantage of using DWT is its inherited ability for multiresolution representation of the signal, which adds an additional useful functionality to a compression algorithm. One more benefit of the DWT is that there are reversible integer-to-integer transformation algorithms [9]. Using embedded quantization of the transform coefficients,

21 2.5. COMPRESSION OF IMAGES 15 Original Image Encoder Compressed Data Encoder Image Transform Quantization Descriptors Encoder Source Entropy Model Symbols Coder Figure 2.3: An image compression system. such a transformation allows for an efficient progressive quality and spatial resolution representation up to perfect reconstruction of the original image. The DWT is exploited in the new standard for compression of still images JPEG2000 [56]. Objective and subjective tests show that the DWT achieves higher compression performance then the DCT. However, the DWT is normally more time and memory consuming. The transformation produces almost uncorrelated data and makes further quantization step more efficient, ultimately resulting in a better compression. Quantization of the descriptors (prediction errors, transform coefficients) introduces distortion into the reconstructed image and results in lossy compression. Lossy compression allows for higher compression rates than the lossless coding. Using an appropriate quantization rule, the quality of the reconstructed image and the compression rate can be efficiently managed and controlled. The theoretical principles of such a trade-off are given in the rate-distortion theory, see, e.g., [7, 16]. Quantization of the DCT or DWT transform coefficients yields an approximation of the original image in L 2 sense. Quantization of prediction errors results in the so called near-lossless compression, where the distortion is specified by the maximum absolute difference between the original and the reconstructed image pixels. In the literature it is also called L constrained lossy coding [65]. The encoder model maps the descriptors into source symbols for the entropy coder. It may be done explicitly or implicitly. An example of explicit mapping is the (old) JPEG standard. In JPEG, the sequences of quantized high-frequency transform coefficients are converted into blocks, which are then coded using Huffman code. Bit-plane coding of wavelet transform coefficients in the JPEG2000 standard is an example of implicit mapping, where the sequence of binary symbols is an information source for the entropy coder. In some coders the descriptors themselves (transform coefficients or prediction residuals) constitute input symbols for the entropy coder.

22 16 CHAPTER 2. SOURCE CODING AND IMAGE COMPRESSION The transformation alone does not perform the data compression. On the contrary, it often performs an expansion of the data, since the transform coefficients, even quantized, may take more bits for their representation than the pixels themselves. The encoder model just translates the descriptors into a more convenient representation for the entropy coder and it does not change the amount of description information. The compression is performed solely by the entropy coder. That is why developing an efficient entropy coder is an important step in the design of an image compression algorithm.

23 Chapter 3 Source coding via binary decomposition In this chapter, we study source coding using binary decomposition of source symbols and derive some properties. 3.1 Introduction In Chapter 2 we introduced a general structure of image compression algorithms consisting of image transformation and entropy coding of description symbols (descriptors). The sequence of descriptors constitutes an information source for the entropy coder. In general, the definition of the information source in compression algorithms (the encoder model) is left to the algorithm designer. It can be a sequence of the descriptors itself. The source symbols can also be represented by blocks of the descriptors, like it is implemented in the JPEG standard for coding the AC coefficients of the DCT [33]. The grouping of symbols is called alphabet extension [42]. It is commonly used in combination with a Huffman coding. However, it was shown in [42] that any source can be coded without alphabet extension using an arithmetic coder. An information source can also be reduced to a binary source via binary decomposition of source symbols. Binary decomposition combined with a binary arithmetic coding is a well known method for coding m-ary sources, see, e.g., [34, 24, 21]. The use of binary decomposition of (non-binary) source symbols has a number of advantages over conventional m-ary coding. Binary arithmetic coding is much simpler for hardware and software implementations than an m-ary arithmetic coding, especially if m 2. On the other hand, one has to encode more than one binary event for each input symbol. However, using an appropriate tree for decomposition and a binary coding technique, the method may result in a faster and/or more efficient entropy coding. Another advantage of using binarization is the possibility of easy optimization of context models for conditional entropy coding, as will be described in Chapter 5. In this chapter, we formally introduce the technique and present some properties. 17

24 18 CHAPTER 3. SOURCE CODING VIA BINARY DECOMPOSITION 3.2 The binary decomposition technique Let us suppose a memoryless source with an alphabet A = {a 0, a 1,...,a m 1 }, and a probability distribution of source symbols P(A) = {p(a), a A}. Let B = {1, 0} denote the binary alphabet and let A n, B n be sets of all words of length n in the alphabets A and B, respectively. As usually, a source message of length n is denoted as u n 1 = u 1u 2...u n, u A. Let T m be a set of proper and complete binary trees with m terminating nodes (leaves) χ 0, χ 1,..., χ m 1 and Λ = {η j, j = 0, 1,...,m 2} be a set of m 1 internal nodes of a tree τ T m. Let the tree τ be assigned to the source A in such a way, that to each symbol a k, k = 0, 1,..., m 1, of the source, there corresponds a leaf χ k of the tree τ: τ : a k χ k. (3.1) Then the source symbol a k is represented by a string of binary decisions, which is a path from the root to the leaf χ k. The m-ary (memoryless) source, generating a message u n 1 of length n in the alphabet A, can now be considered as a binary Markov source modeled by the tree τ generating a binary sequence b 1 b 2...b n = b n 1 of length n, where b B. Each node η Λ Bn of the tree corresponds to a state of the Markov source and the tree uniquely defines a corresponding directed graph of state transitions. The initial state is defined by the root node. At each state (node), the source generates symbol 0 or 1 with the probability distribution p(0 η) and changes its state (maybe to the same state). The distributions {p(0 η), η Λ} are uniquely defined by the probability distribution of the alphabet P(A). Such a binary source can be encoded by conventional methods of coding of Markov sources. Given the model and the initial state, the binary sequence b n 1 is decomposed into m 1 subsequences {b n η 1 (η), η Λ}, generated at each state. The subsequences in essence are the sequences of i.i.d. binary symbols drawn with the probability p(0 η). They can be effectively encoded using some kind of binary coding techniques, e.g., arithmetic coding [61] or Golomb run-length coding [19]. (Note, that for an arithmetic coder there is no need for an explicit decomposition of the binary sequence into the subsequences corresponding to the states. The arithmetic coder just uses the conditional probabilities p(0 η) to calculate a codeword for the whole binary sequence.) The main parameter of a decomposition tree is the average number of binary coding operations per source symbol n = a A p(a)n(a), (3.2) where n(a) is the number of binary decisions required to code a (the length of the binary path to the symbol a). This parameter defines the coding efficiency (in terms of both redundancy and speed, as will be discussed in the following sections). The minimum n is achieved when the tree is a Huffman tree for this source. The use of a

25 3.3. ON REDUNDANCY OF BINARY DECOMPOSITION 19 binary decomposition assumes that the decomposition tree is fixed 1 during encoding, and compression is performed by a binary coding technique. 3.3 On redundancy of binary decomposition In this section, we derive an upper bound estimate on the coding redundancy using the binarization technique. We define a redundancy R as the difference between the average code length per source symbol and the entropy of the source: R = 1 p(u n 1 n ) ϕ(un 1 ) H, (3.3) u n 1 An where ϕ(u n 1 ) denotes the codeword length for the message un 1 (see Chapter 2) and H = a A p(a) log p(a). Let us allow the use of different binary coding techniques for each node. (For example, if some subsequence has probability of a binary symbol close to 0.5, then, for the sake of simplicity of implementation, this subsequence may be put directly into the output bit stream). If the probabilities of source symbols are unknown, then the subsequences can be coded using some kind of universal or adaptive coding technique (e.g., binary arithmetic coding with the symbol probability estimate given by (2.29). Let n η be the length of a binary sequence, generated at the state η. We define an average coding redundancy per binary symbol at the state η as ρ η = 1 n η b n η 1 Bn η p(b n η 1 ) ϕ b (b n η 1 ) H η, (3.4) where ϕ b (b n η 1 ) is the code length of a binary string bn η 1, generated with the probability p(b n η 1 ) and H η is the entropy of binary symbols generated at the state η: H η = p(0 η) log 2 p(0 η) (1 p(0 η)) log 2 (1 p(0 η)). The following theorem establishes the relationship between the decomposition tree τ, the (redundancy of) coding techniques used at the nodes, and the resulting redundancy R. The theorem also helps to pose some interesting properties of the method, which are stated as Corollaries 1 4. Theorem 3. Let the source generate a message of length n of symbols from an alphabet A = {a 0, a 1,...,a m 1 }. Let this message be sequentially coded using the binary tree decomposition technique described above. Then the redundancy is defined as R = n η Λ π η ρ η, (3.5) 1 Otherwise, one would come to a (dynamic) Huffman coding technique and there would be no need for binary coding (at least for sources with an entropy larger than 1 bit).

26 20 CHAPTER 3. SOURCE CODING VIA BINARY DECOMPOSITION where n and ρ η are defined by (3.2), (3.4) and π η = n η / nn can be viewed as a probability of the state η. Remark: In this formula, n and π η are defined by the decomposition tree, whereas ρ η is determined by binary coding techniques, used at the nodes. Proof. The proof is straightforward. We rewrite (3.5) as follows: = n η Λ π η = n η Λ = 1 n π η 1 n η η Λ 1 n η R = n η Λ π η ρ η b n η 1 Bn η b n η 1 Bn η b n η 1 Bn η p(b n η 1 ) ϕ b (b n η 1 ) + H η p(b n η 1 ) ϕ b (b n η 1 ) n η Λ π η H η p(b n η 1 ) ϕ b (b n η 1 ) n η Λ π η H η The first term in the last equality is the average code length and the second defines the entropy of the input source. Corollary 1. If for η Λ ρ η = ρ, then R = nρ. Corollary 2. If r-bits registers are used to represent probabilities and an arithmetic coder with registers of size g r + 2 is used to code binary symbols at the nodes, then R < 2 n(r + log 2 e)2 (g 2) + 2/n. (3.6) Proof. It was proven in [46], that the per-symbol redundancy for an m-ary arithmetic coder is upper bounded by the inequality ρ < m(r + log 2 e)2 (g 2). (3.7) For binary coding m = 2, and two additional bits are required to terminate the message. This yields (3.6). Corollary 3. If the decomposition tree is a Huffman tree, r-bits registers are used to represent probabilities and a binary arithmetic coder with registers of size g r + 2 is used to code at the nodes, then R < (H + 1)(r + log 2 e)2 (g 2) + 2/n (3.8)

27 3.4. BINARY DECOMPOSITION OF FSM SOURCES 21 Proof. The proof follows immediately from the fact that for the Huffman tree n < H + 1. (3.9) This corollary is essential for coding low entropy sources (H < 1) with large alphabets (m 2). In this case, the use of a Huffman tree results in the redundancy of order O(H + 1), independent on the alphabet size, whereas for an m-ary arithmetic coding the redundancy is of order O(m), see (3.7). Finally, it is always possible to choose such a tree, that n log 2 m. Corollary 4. If the decomposition tree is such that n log 2 m, r-bits registers are used to represent probabilities and a binary arithmetic coder with registers of size g r + 2 is used to code at the nodes, then R < 2(log 2 m + 1)(r + log 2 e)2 (g 2) + 2/n Thus, for sources with an alphabet of size m > 4, the upper bound on the coding redundancy using binary decomposition combined with binary arithmetic coding is lower than that of m-ary arithmetic coding (see (3.7)). 3.4 Binary decomposition of FSM sources An alphabet binarization can be as well used in coding FSM sources. Let an FSM source be defined by the alphabet A, a set of states S = {s}, the initial state s 1 and a state transition rule. In this case, each state is assigned a decomposition tree τ s T m with the corresponding set of states (decision nodes) Λ s = {η s }. In general, the decomposition trees τ s can be different (and may be also optimized) for each state. At each state the source generates a binary string of decisions corresponding to a source symbol. The resulting binary sequence can be modeled as a double FSM source with the set of states, which is the product of the sets: S = Λ s S. 3.5 Binary decomposition and universal coding Consider a memoryless m-ary source. If the source parameters P(A) = {p(a), a A} are not known, then the subsequences of binary decisions can be coded using a universal coding approach. In this case, the decision probabilities can be estimated according to (2.28), where s corresponds to a node η. The use of a binary decomposition technique does not increase the stochastic complexity of the source model, since the number of parameters is retained the same. An m-ary memoryless source and its binary counterpart have m 1 free parameters (in the latter case this is the number of internal nodes of the decomposition tree). This concerns FSM sources as well, where the number of parameters is (m 1) S.

28 22 CHAPTER 3. SOURCE CODING VIA BINARY DECOMPOSITION Thus, binarization of source symbols at least does not make a compression algorithm worse in terms universal coding redundancy (2.25) or (2.26) (due to the lack of knowledge on the parameters). Asymptotically this redundancy is the same. Yet, for finite length messages, the redundancy may be even less, resulting in a better compression performance. We will show it in the following example. Let us assume a memoryless m-ary source, and let the output message u n 1 of length n > 0 be coded by an ideal m-ary arithmetic coder using adaptive coding probability calculation defined by (2.29) with the parameter vector α a = 1, a A. Then, the 2 code length is defined as ϕ(u n 1 ) = Γ ( ) m 2 π m 2 Γ ( n + m 2 = n H + m 1 2 ) ( Γ ϑ(a u n 1 ) + 1 ) 2 a A log 2 n + O 1 (1). where Γ( ) is a Gamma function, ϑ(a u n 1) denotes the symbol counts and (3.10) H = a A ϑ(a u n 1) n log 2 ϑ(a u n 1) n (3.11) is the empirical entropy of the message. Let the same message be coded using the binary decomposition technique combined with an ideal binary arithmetic coder. Then the code length is the sum of the codes of binary sub-strings, corresponding to the decomposition tree nodes: ϕ(u n 1) = η Λ ϕ(b nη 1 (η)) = η Λ ( n η Hη + 1 ) 2 log 2 n η + O η (1). (3.12) Thus ϕ(u n 1 ) = n η Hη log 2 n η + O 2 (1) (3.13) η Λ where n η denotes lengths of the binary sub-strings, H η = ϑ(0 unη 1 ) n η log 2 ϑ(0 u nη 1 ) n η η Λ ϑ(1 unη 1 ) n η is the empirical entropy of the binary sub-strings and log 2 ϑ(1 u nη 1 ) n η (3.14) O 2 (1) = η Λ O η (1). (3.15) It can be shown that n H = η Λ n η Hη. (3.16)