# Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay

Size: px
Start display at page:

Download "Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay"

Transcription

1 Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding In our study so far we have seen that Huffman code is optimal and it has an expected length within one bit of the entropy of the source. Redundancy which is defined as the difference between the average length of the code, and the entropy of the source for the Huffman codes can be reduced by blocking symbols together. Blocking two symbols together means alphabet size goes from m to m square where m is the size of the alphabet. So, if m is equal to 3 then when we block two symbols together the new alphabet size will become 9. This size is not an excessive burden for most of applications. However, if the probabilities of the symbols or letters of the source are unbalanced then it would require blocking many more symbols together before the redundancy is reduced to an accepted level. So, as we block more and more symbols together the size of the alphabet grows exponentially and in such a situation the Huffman coding schemes become impractical. Therefore, it is mandatory in such situations to look for other techniques than Huffman coding. Let us try to clear these concepts with an help of an example. (Refer Slide Time: 02:36)

2 So, let us consider a source which puts out identically independent distributed letters from the alphabet consisting of three symbols. Let us assume the probability model for this source given as probability of S 2 is equal to 0.02 and probability of S 3 is equal to Now, the entropy for this source can be calculated and this is equal to bits per symbol. If we considers symbols individually than we can find the Huffman code for this source as S 1, the code word is 0, for S 2 the code word is 11 and for the S 3 the code word is given as 10. Now, the average length for this code is equal to 1.05 bits per symbol. Redundancy which is defined as the difference between the average length minus the entropy in this case will be equal to bits per symbol. So, this is, this entropy, this redundancy is about 213 percent of the entropy. This means that the, that to code this sequence we would need more than twice the number of bits promised by the entropy. Now, same shows if we block the symbols in blocks of two and then code, we can find the Huffman code for the second extension of the source and that is as shown here. (Refer Slide Time: 05:16) So, we take the second extension of the source, we calculate the probabilities of the new symbols and we can find the code, Huffman code for this new source. And for this new source we find out that the average length comes out to be in terms of the original source as bits per symbol. So, redundancy is equal to bits per symbol. Still this redundancy is 72 percent of the entropy.

3 (Refer Slide Time: 06:01) But what this example shows is that by continuing to block symbols together we find that redundancy drops to acceptable values when we block eight symbols together. The corresponding alphabet size for this level of blocking is 6561 which is quite excessive. So, a code of this size is impractical for a number of reasons. First of all memory may not be available for many applications. Now, even if it were possible to design reasonably efficient encoders, but decoding a Huffman code of this size would be highly inefficient and time consuming. Now, next reason for not using this kind of code is if there were some perturbations in the source model then we would have a major impact on the efficiency of the code. (Refer Slide Time: 07:26)

4 So, with this example we can reach to these conclusions that for small source alphabets we have efficient coding only if we use long blocks of source symbols. That is only if we go for higher extension of the source. If you want to achieve high efficiency or reduce the redundancy in the code then we have to use long blocks in order to achieve an expected length per symbol close to the entropy of the source. Now, this is, therefore it is desirable to have an efficient coding procedures that works for long blocks of source symbols. Unfortunately, Huffman coding is not ideal for this situation since Huffman coding is a bottom up procedure that requires the calculation of the probabilities of all source sequences of a particular block length and the construction of the corresponding complete code tree. We are then limited to using that block length. A better scheme is one which can be easily extended to longer block lengths without having to redo all the calculations. And one such way to achieve this is to use arithmetic coding, but before we study arithmetic coding let us have a look at a simplified version of arithmetic coding which is based on what is known as Shannon-Fano-Elias coding. Then we will extend the concepts of Shannon-Fano-Elias coding to arithmetic coding. (Refer Slide Time: 09:36) So, we will have a look at Shannon-Fano-Elias coding. We have seen that the set of lengths given as l x is equal to log of 1 by p x can be used to construct a uniquely decodable code for the source. Now, we will consider a simple constructive procedure which uses the cumulative distribution function to allot code words. And this is the idea which is used in Shannon-Fano-

5 Elias coding. Before we begin development of arithmetic coding or Shannon-Fano-Elias coding let us established some notation. Recall that a random variable maps the outcomes or set of outcomes of an experiment to values on the real number line. For example, in a coin tossing experiment the random value variable could map a head to 0. So, I could have a mapping as head to zero and a tail to one or I could use a mapping which is head has some number let us say and tail as minus 152. So, to use this technique we need to map the source symbols or letters from the source to numbers. For convenience in the discussion to follow we will use the following mapping. The mapping is given by X and it maps the source symbol or source letter as equal to i where Si belongs to source alphabet. So, your S consists of source letters. This is a discrete source and X is a random variable. Now, when we use this kind of mapping given a probability model P for the source we also have a probability density function for the random variable X that is once we specify the source model. I can get my probability distribution function for the random variable for the discrete random variable as probability of X equal to i is equal to probability of Si, and this for our convenience will indicate it as P suffix i. So, probability model has been given to us as P of S 1 P of S 2 P of S m. Once, we are given the probability distribution function for the discrete random variable X, or you can say also probability mass function for the discrete random variable X, then we can define the cumulative distribution function. (Refer Slide Time: 14:25)

6 So, we will define cumulative distribution function that is c d f as probability of X is equal to k. k ranges from1 to i and this by definition is P k k equal to 1 to i. Now, this function is illustrated in the figure here. So, I have my eye on x axis and so on y axis I have my c d f and on x axis the random variable. So, this is your c d f. The minimum value for this is 0 and the maximum value corresponds to 1. Now, let us consider the modified c d f as (Refer Slide Time: 16:24) So, consider the modified c d f as F of X i, we will use tilde notation for modified c d f is equal to k is equal to 1i minus 1 plus Pi by 2 where this modified c d f denotes the sum of the probabilities of all symbols less than i plus half the probability of the symbol i. Since, the random variable is discrete the cumulative distribution function consists of steps of size Pi. The value of the function of this modified that is modified F X i tilde is the midpoint of the step corresponding to i. Now, since all the probability are positive what this implies that F X j is not equal to F X k if j is not equal to k. And hence, we can determine i if we know FX tilde i. So, merely look at that graph of the c d f and find the corresponding i. So, because this is unique by looking at the graph we can once the F Xi is been given we can find out what is the value of i. So, what this implies that the value of modified c d f can be used as a code for i. Now, in general this value is a real number and which is expressible in a practical scenario only when finite number of bits. So, it is not efficient to use the exact value of this modified c d f as the code word for i. We will have to

7 use an approximate value for this modified c d f. So, if we use approximate value the next question is what is the required accuracy? So, let us assume. (Refer Slide Time: 20:04) So, let us assume that we round off F tilde to l i bits and this rounding off is denoted by lower floor of F tilde l i. So, what it means that we use the first l i bits of F tilde as a code word for i. So, this is will be acting as a code word for i. Now, by definition of rounding off by we have F tilde minus lower floor F tilde l i less than 1 by 2 raise to l i. This is by the definition of rounding off operation. Now, if we use, if we assume that we are going to choose l i given by the relationship as log of 1 by Pi, upper floor of that plus 1 then we can show that 1 by 2 raise to l i is less than and this is equal to So, if you use l i given by this expression then it is very easy to show that this inequality is valid. And therefore, what it means that this quantity out here lies within the step corresponding to i. Thus, l i bits suffice to describe i. Now, in addition to requiring that the code word identifies the corresponding letter or symbol we also require the set of code words to be prefix free.

8 (Refer Slide Time: 24:02) To check whether the code is prefix free we consider each code word which is of the form Z 1 Z 2 up to Z l i to represent not a single point, but the interval given by So, each code word represents the interval given here. So, the code word is prefix free if and only if you can prove the intervals corresponding to code words are disjoint. Now, let us verify the code is prefix free. Now, the interval corresponding two any code word has a length 2 raised to minus l i which we have shown is less than half the height of the step corresponding to i by equation one. By equation one, we have shown that this interval is less than half the height of the step corresponding to i. So, what this implies is that the lower end of the interval is in the lower half of the step. Thus, the upper end of the interval lies below the top of the step. And therefore, the interval corresponding to any code words lies entirely within the step corresponding to that symbol in the cumulative distribution function. Therefore, the intervals corresponding to different code words are disjoint and the code is prefix free. Note, that this procedure does not require the symbols to be ordered in terms of probability. Now, since we use the length which is given by l i is equal to log of 1 by P upper floor of this plus 1 to represent the ith letter. The expected length of the code can be shown to be as equal to Pi l i summation over i is less than or equal to entropy of the source plus 2. So, the average length of the code obtain from Shannon-Fano-Elias coding will be within two bits of the

9 entropy of the source. Now, let us look at some of the examples to understand this coding scheme. (Refer Slide Time: 28:48) Let me first consider an example where the probabilities of the letters of the source are dyadic. So, let us assume that we have a source consisting of four source symbols. They are being identified by this mapping as with Pi given as 0.250, 0.500, , So, for this we can construct c d f as follows and we can construct the modified c d f as Now, we will represent this in modified c d f in binary. We decided the number of bits for representing this code words by this l i which is given as the upper floor log of 1 by Pi plus1 and which in this case is 3244, and the code words for this can be written as 001,10,1101, these are obtained from the binary representation. So, this forms the code based on Shannon-Fano-Elias coding. And if we calculate average length, average length for this code turns out to be 2.75 bits, while the entropy for this source is equal to 1.75 bits per symbol. The Huffman code for this case achieves the entropy bound. Now, looking at the code word it is obvious that there is some kind of inefficiency. For example, the last bit of the last two code words can be omitted, but if you remove the last bit from all the code words the code is no longer prefix free. Now, this we have seen for the case where the distribution is dyadic.

10 (Refer Slide Time: 32:50) But if the distribution is not dyadic then let us look at that example. So, the case where the distribution is not dyadic I have a source consisting of five source symbols with the probabilities Pi given as follows. So, for this we can construct my c d f that is and 1 and for the same source we can construct the modified c d f as Now, if you look at the lengths which is given by log of 1 by Pi upper floor of this plus 1, this comes out to be So, the binary representation of this would be, binary representation will be given 0.001, 0.011, , 0011 keeps on recurring. So, we put a bar out here. For this case we find again this is recovering, recurring, so we put a bar and finally for Now, the code words will be decided by this l i, so this will be 001. So, this will be my code words 001, 011, , , Now, the above code is 1.2 bits longer on the average than the Huffman code for this source. Now, we will extend the concept of Shannon-Fano-Elias coding and describe a computationally efficient algorithm for encoding and decoding call arithmetic coding. In order to find a Huffman code for a particular sequence of length m, we need code words for all possible sequences of length m. This fact causes exponential growth in the size of the code word. Therefore, we need a way of assigning code words to particular sequences without having to generate code for all sequences of that length. The arithmetic coding technique fulfills this requirement. Let us have a look at the basics of arithmetic coding.

11 (Refer Slide Time: 37:01) In arithmetic coding a unique identifier or a tag is generated for the sequence to be encoded. This tag corresponds to a binary fraction which becomes the binary code for the sequence. Impact is the generation of the tag and the binary code are the same process. However, the arithmetic coding approach is easier to understand if we conceptually divide the approach in two phases. So, in the first phase a unique identifier or tag is generated for a given sequence of symbols. And in the second phase this tag is given a unique binary code word. Now, a unique arithmetic code word can be generated for a sequence of length m. m is the size of the alphabet without the need for generating code words for all sequences of length m. This is unlike the situation for Huffman codes. In order to generate a Huffman code for sequence of length m where the code is not a concatenation of the code words for the individual symbols, we need to obtain the Huffman codes for all sequences of length m. So, let us look at the coding of a sequence in arithmetic coding. So, to code a sequence we follow the following procedure. In order to distinguish a sequence of symbols from another sequence of symbols we need to tag it with a unique identifier.

12 (Refer Slide Time: 40:23) One possible set of tag for sequences of symbols are the numbers in the unit interval0 to 1.Because the number of numbers in the unit interval is infinite, it should be possible to assign a unique tag to each distinct sequence of symbols. Now, in order to do this we need a function that will map sequences of symbols into the unit interval. A function that maps random variables and sequences of random variables into the unit interval is the cumulative distribution function. So, let us use this function in developing the arithmetic code. So, the next thing is how do we generate a tag? The procedure for generating the tag works by reducing the size of the interval in which the tag resides as more and more elements of the sequence are received. So, the first thing is we start out by dividing the unit interval into subintervals of the forms F Xi minus 1 and F Xi for i equal to 12 m where m is the size of my source alphabet. Now, because the minimum value of the c d f is 0 and the maximum value is 1 this exactly partitions the unit interval. We associate the sub interval F Xi minus 1, F Xi with the symbol Si. The appearance of the first symbol in the sequence restricts the interval containing the tag to one of these sub intervals. Suppose, the first symbol was S k then the interval containing the tag will be the sub interval as given here. Now, this sub interval is now partitioned in exactly the same proportions as the original interval.

13 (Refer Slide Time: 43:57) What it means that j th interval corresponding to the symbol S j is given by, on having received S k plus So, if the second symbol in the sequences S j, then the interval containing the tag value becomes as shown here. Each succeeding symbol causes the tag to be restricted to a sub interval that is further partitioned in the same proportion. This process can be more clearly understood through an example. (Refer Slide Time: 45:42) So, let us take an example of a source consisting of three symbols with the probability model as given here. And we will use the mapping which we discussed for Shannon-Fano-Elias

14 coding as shown here. Therefore, F X1 is equal to 0.7, F X2 is equal to 0.8 and F X3 is equal to 1. Now, this partitions the unit interval as shown in the figure. So, this gets partition unit interval as shown here. Now, if the, let us assume that the first symbol is S 1 then tag lies in the interval between 0 and 0.7. (Refer Slide Time: 47:32) So, the tag will lie in the interval given by 0.0, 0.7. If the first symbol S 2 then the tag will lie in the interval given by 0.7, 0.8 and if the tag lie, if the first symbol is S 3 then the tag will lie in the interval 0.8 to 1.0. Now, once the interval containing the tag has been determined the rest of the unit interval is discarded. And this restricted interval is again divided in the same proportion as the original interval. So, if you assume that the first symbol was S 1 the tag would be containing the interval given by this. This sub interval is then divided in exactly the same proportion as the original interval yielding, the sub intervals from 0.0 to 0.49, 0.49 to 0.56 and 0.56 to 0.7. So, the first partition as before corresponds to the symbol S 1, the second partition corresponds to symbol S 2 and the third partitions correspond to symbol S 3, suppose the second symbol in the sequence is S 2. The tag value is then restricted to lie in the interval between 0.49 and We now partition this interval in the same proportion as the original interval to obtain the sub intervals as shown here. So, 0.49 to will correspond to S 1, to will correspond S 2 and finally, to will correspond to S 3. Now, again if the third symbol is S 3 the tag will be

15 restricted to this interval which is between to 0.560, which can be subdivided further as shown here. Notice, that the appearance of each new symbol restricts the tag to a sub interval that is disjoined from any other sub interval that may have been generated using the process. For the sequence beginning with S 1, S 2 and S 3 by that time the third symbol S 3 is received the tag has been restricted to the interval to If the third symbol had been S 1 instead of S 3 the tag would have resided in the interval between to 0.539, which is disjoint from the sub interval to Now, even if the two sequences are identical from this point onward one starting with a S 1, S 2 and S 3 and other beginning with S 1, S 2 and S 1 that tag interval for the two sequences will always be disjoint. We will study the tag generation process mathematically starting with the sequences of length 1, and then we will extend this approach to longer sequences by imposing, what is known as lexicographic ordering on the sequences.

### Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur

Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur Module No. #01 Lecture No. #15 Special Distributions-VI Today, I am going to introduce

### Gambling and Data Compression

Gambling and Data Compression Gambling. Horse Race Definition The wealth relative S(X) = b(x)o(x) is the factor by which the gambler s wealth grows if horse X wins the race, where b(x) is the fraction

### arxiv:1112.0829v1 [math.pr] 5 Dec 2011

How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly

### Discrete Mathematics and Probability Theory Fall 2009 Satish Rao, David Tse Note 13. Random Variables: Distribution and Expectation

CS 70 Discrete Mathematics and Probability Theory Fall 2009 Satish Rao, David Tse Note 3 Random Variables: Distribution and Expectation Random Variables Question: The homeworks of 20 students are collected

### Chapter 4 Lecture Notes

Chapter 4 Lecture Notes Random Variables October 27, 2015 1 Section 4.1 Random Variables A random variable is typically a real-valued function defined on the sample space of some experiment. For instance,

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete

### Arithmetic Coding: Introduction

Data Compression Arithmetic coding Arithmetic Coding: Introduction Allows using fractional parts of bits!! Used in PPM, JPEG/MPEG (as option), Bzip More time costly than Huffman, but integer implementation

### 9.2 Summation Notation

9. Summation Notation 66 9. Summation Notation In the previous section, we introduced sequences and now we shall present notation and theorems concerning the sum of terms of a sequence. We begin with a

### Estimating the Average Value of a Function

Estimating the Average Value of a Function Problem: Determine the average value of the function f(x) over the interval [a, b]. Strategy: Choose sample points a = x 0 < x 1 < x 2 < < x n 1 < x n = b and

### Information, Entropy, and Coding

Chapter 8 Information, Entropy, and Coding 8. The Need for Data Compression To motivate the material in this chapter, we first consider various data sources and some estimates for the amount of data associated

### Math Review. for the Quantitative Reasoning Measure of the GRE revised General Test

Math Review for the Quantitative Reasoning Measure of the GRE revised General Test www.ets.org Overview This Math Review will familiarize you with the mathematical skills and concepts that are important

### Cryptography and Network Security. Prof. D. Mukhopadhyay. Department of Computer Science and Engineering. Indian Institute of Technology, Kharagpur

Cryptography and Network Security Prof. D. Mukhopadhyay Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module No. # 01 Lecture No. # 12 Block Cipher Standards

### REPEATED TRIALS. The probability of winning those k chosen times and losing the other times is then p k q n k.

REPEATED TRIALS Suppose you toss a fair coin one time. Let E be the event that the coin lands heads. We know from basic counting that p(e) = 1 since n(e) = 1 and 2 n(s) = 2. Now suppose we play a game

### Load Balancing and Switch Scheduling

EE384Y Project Final Report Load Balancing and Switch Scheduling Xiangheng Liu Department of Electrical Engineering Stanford University, Stanford CA 94305 Email: liuxh@systems.stanford.edu Abstract Load

### Review Horse Race Gambling and Side Information Dependent horse races and the entropy rate. Gambling. Besma Smida. ES250: Lecture 9.

Gambling Besma Smida ES250: Lecture 9 Fall 2008-09 B. Smida (ES250) Gambling Fall 2008-09 1 / 23 Today s outline Review of Huffman Code and Arithmetic Coding Horse Race Gambling and Side Information Dependent

### In this chapter, you will learn improvement curve concepts and their application to cost and price analysis.

7.0 - Chapter Introduction In this chapter, you will learn improvement curve concepts and their application to cost and price analysis. Basic Improvement Curve Concept. You may have learned about improvement

### Factoring & Primality

Factoring & Primality Lecturer: Dimitris Papadopoulos In this lecture we will discuss the problem of integer factorization and primality testing, two problems that have been the focus of a great amount

### Ex. 2.1 (Davide Basilio Bartolini)

ECE 54: Elements of Information Theory, Fall 00 Homework Solutions Ex.. (Davide Basilio Bartolini) Text Coin Flips. A fair coin is flipped until the first head occurs. Let X denote the number of flips

### Probability Interval Partitioning Entropy Codes

SUBMITTED TO IEEE TRANSACTIONS ON INFORMATION THEORY 1 Probability Interval Partitioning Entropy Codes Detlev Marpe, Senior Member, IEEE, Heiko Schwarz, and Thomas Wiegand, Senior Member, IEEE Abstract

### (Refer Slide Time: 01:11-01:27)

Digital Signal Processing Prof. S. C. Dutta Roy Department of Electrical Engineering Indian Institute of Technology, Delhi Lecture - 6 Digital systems (contd.); inverse systems, stability, FIR and IIR,

### Useful Number Systems

Useful Number Systems Decimal Base = 10 Digit Set = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} Binary Base = 2 Digit Set = {0, 1} Octal Base = 8 = 2 3 Digit Set = {0, 1, 2, 3, 4, 5, 6, 7} Hexadecimal Base = 16 = 2

### Soil Dynamics Prof. Deepankar Choudhury Department of Civil Engineering Indian Institute of Technology, Bombay

Soil Dynamics Prof. Deepankar Choudhury Department of Civil Engineering Indian Institute of Technology, Bombay Module - 2 Vibration Theory Lecture - 8 Forced Vibrations, Dynamic Magnification Factor Let

### Broadband Networks. Prof. Dr. Abhay Karandikar. Electrical Engineering Department. Indian Institute of Technology, Bombay. Lecture - 29.

Broadband Networks Prof. Dr. Abhay Karandikar Electrical Engineering Department Indian Institute of Technology, Bombay Lecture - 29 Voice over IP So, today we will discuss about voice over IP and internet

parent ROADMAP MATHEMATICS SUPPORTING YOUR CHILD IN HIGH SCHOOL HS America s schools are working to provide higher quality instruction than ever before. The way we taught students in the past simply does

### CORRELATED TO THE SOUTH CAROLINA COLLEGE AND CAREER-READY FOUNDATIONS IN ALGEBRA

We Can Early Learning Curriculum PreK Grades 8 12 INSIDE ALGEBRA, GRADES 8 12 CORRELATED TO THE SOUTH CAROLINA COLLEGE AND CAREER-READY FOUNDATIONS IN ALGEBRA April 2016 www.voyagersopris.com Mathematical

### Revised Version of Chapter 23. We learned long ago how to solve linear congruences. ax c (mod m)

Chapter 23 Squares Modulo p Revised Version of Chapter 23 We learned long ago how to solve linear congruences ax c (mod m) (see Chapter 8). It s now time to take the plunge and move on to quadratic equations.

### 10.2 Series and Convergence

10.2 Series and Convergence Write sums using sigma notation Find the partial sums of series and determine convergence or divergence of infinite series Find the N th partial sums of geometric series and

### Project and Production Management Prof. Arun Kanda Department of Mechanical Engineering Indian Institute of Technology, Delhi

Project and Production Management Prof. Arun Kanda Department of Mechanical Engineering Indian Institute of Technology, Delhi Lecture - 9 Basic Scheduling with A-O-A Networks Today we are going to be talking

### NUMBER SYSTEMS. William Stallings

NUMBER SYSTEMS William Stallings The Decimal System... The Binary System...3 Converting between Binary and Decimal...3 Integers...4 Fractions...5 Hexadecimal Notation...6 This document available at WilliamStallings.com/StudentSupport.html

### Coding and decoding with convolutional codes. The Viterbi Algor

Coding and decoding with convolutional codes. The Viterbi Algorithm. 8 Block codes: main ideas Principles st point of view: infinite length block code nd point of view: convolutions Some examples Repetition

### COMP 250 Fall 2012 lecture 2 binary representations Sept. 11, 2012

Binary numbers The reason humans represent numbers using decimal (the ten digits from 0,1,... 9) is that we have ten fingers. There is no other reason than that. There is nothing special otherwise about

### (Refer Slide Time: 02:17)

Internet Technology Prof. Indranil Sengupta Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture No #06 IP Subnetting and Addressing (Not audible: (00:46)) Now,

### We can express this in decimal notation (in contrast to the underline notation we have been using) as follows: 9081 + 900b + 90c = 9001 + 100c + 10b

In this session, we ll learn how to solve problems related to place value. This is one of the fundamental concepts in arithmetic, something every elementary and middle school mathematics teacher should

### The Basics of Interest Theory

Contents Preface 3 The Basics of Interest Theory 9 1 The Meaning of Interest................................... 10 2 Accumulation and Amount Functions............................ 14 3 Effective Interest

### The Classes P and NP

The Classes P and NP We now shift gears slightly and restrict our attention to the examination of two families of problems which are very important to computer scientists. These families constitute the

### K-Cover of Binary sequence

K-Cover of Binary sequence Prof Amit Kumar Prof Smruti Sarangi Sahil Aggarwal Swapnil Jain Given a binary sequence, represent all the 1 s in it using at most k- covers, minimizing the total length of all

### AP CALCULUS AB 2009 SCORING GUIDELINES

AP CALCULUS AB 2009 SCORING GUIDELINES Question 5 x 2 5 8 f ( x ) 1 4 2 6 Let f be a function that is twice differentiable for all real numbers. The table above gives values of f for selected points in

### E3: PROBABILITY AND STATISTICS lecture notes

E3: PROBABILITY AND STATISTICS lecture notes 2 Contents 1 PROBABILITY THEORY 7 1.1 Experiments and random events............................ 7 1.2 Certain event. Impossible event............................

### International Journal of Advanced Research in Computer Science and Software Engineering

Volume 3, Issue 7, July 23 ISSN: 2277 28X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Greedy Algorithm:

### Breaking The Code. Ryan Lowe. Ryan Lowe is currently a Ball State senior with a double major in Computer Science and Mathematics and

Breaking The Code Ryan Lowe Ryan Lowe is currently a Ball State senior with a double major in Computer Science and Mathematics and a minor in Applied Physics. As a sophomore, he took an independent study

### Sums of Independent Random Variables

Chapter 7 Sums of Independent Random Variables 7.1 Sums of Discrete Random Variables In this chapter we turn to the important question of determining the distribution of a sum of independent random variables

### Image Compression through DCT and Huffman Coding Technique

International Journal of Current Engineering and Technology E-ISSN 2277 4106, P-ISSN 2347 5161 2015 INPRESSCO, All Rights Reserved Available at http://inpressco.com/category/ijcet Research Article Rahul

### Lecture 13 - Basic Number Theory.

Lecture 13 - Basic Number Theory. Boaz Barak March 22, 2010 Divisibility and primes Unless mentioned otherwise throughout this lecture all numbers are non-negative integers. We say that A divides B, denoted

### Department of Mathematics, Indian Institute of Technology, Kharagpur Assignment 2-3, Probability and Statistics, March 2015. Due:-March 25, 2015.

Department of Mathematics, Indian Institute of Technology, Kharagpur Assignment -3, Probability and Statistics, March 05. Due:-March 5, 05.. Show that the function 0 for x < x+ F (x) = 4 for x < for x

### Maximum Likelihood Estimation

Math 541: Statistical Theory II Lecturer: Songfeng Zheng Maximum Likelihood Estimation 1 Maximum Likelihood Estimation Maximum likelihood is a relatively simple method of constructing an estimator for

### College of information technology Department of software

University of Babylon Undergraduate: third class College of information technology Department of software Subj.: Application of AI lecture notes/2011-2012 ***************************************************************************

### Lecture 2: Universality

CS 710: Complexity Theory 1/21/2010 Lecture 2: Universality Instructor: Dieter van Melkebeek Scribe: Tyson Williams In this lecture, we introduce the notion of a universal machine, develop efficient universal

### Probability: Terminology and Examples Class 2, 18.05, Spring 2014 Jeremy Orloff and Jonathan Bloom

Probability: Terminology and Examples Class 2, 18.05, Spring 2014 Jeremy Orloff and Jonathan Bloom 1 Learning Goals 1. Know the definitions of sample space, event and probability function. 2. Be able to

### Linear Codes. Chapter 3. 3.1 Basics

Chapter 3 Linear Codes In order to define codes that we can encode and decode efficiently, we add more structure to the codespace. We shall be mainly interested in linear codes. A linear code of length

### TCOM 370 NOTES 99-4 BANDWIDTH, FREQUENCY RESPONSE, AND CAPACITY OF COMMUNICATION LINKS

TCOM 370 NOTES 99-4 BANDWIDTH, FREQUENCY RESPONSE, AND CAPACITY OF COMMUNICATION LINKS 1. Bandwidth: The bandwidth of a communication link, or in general any system, was loosely defined as the width of

### RN-coding of Numbers: New Insights and Some Applications

RN-coding of Numbers: New Insights and Some Applications Peter Kornerup Dept. of Mathematics and Computer Science SDU, Odense, Denmark & Jean-Michel Muller LIP/Arénaire (CRNS-ENS Lyon-INRIA-UCBL) Lyon,

### 1 Construction of CCA-secure encryption

CSCI 5440: Cryptography Lecture 5 The Chinese University of Hong Kong 10 October 2012 1 Construction of -secure encryption We now show how the MAC can be applied to obtain a -secure encryption scheme.

### South Carolina College- and Career-Ready (SCCCR) Algebra 1

South Carolina College- and Career-Ready (SCCCR) Algebra 1 South Carolina College- and Career-Ready Mathematical Process Standards The South Carolina College- and Career-Ready (SCCCR) Mathematical Process

### Section 1.3 P 1 = 1 2. = 1 4 2 8. P n = 1 P 3 = Continuing in this fashion, it should seem reasonable that, for any n = 1, 2, 3,..., = 1 2 4.

Difference Equations to Differential Equations Section. The Sum of a Sequence This section considers the problem of adding together the terms of a sequence. Of course, this is a problem only if more than

### Cost Model: Work, Span and Parallelism. 1 The RAM model for sequential computation:

CSE341T 08/31/2015 Lecture 3 Cost Model: Work, Span and Parallelism In this lecture, we will look at how one analyze a parallel program written using Cilk Plus. When we analyze the cost of an algorithm

### Find-The-Number. 1 Find-The-Number With Comps

Find-The-Number 1 Find-The-Number With Comps Consider the following two-person game, which we call Find-The-Number with Comps. Player A (for answerer) has a number x between 1 and 1000. Player Q (for questioner)

### Notes 11: List Decoding Folded Reed-Solomon Codes

Introduction to Coding Theory CMU: Spring 2010 Notes 11: List Decoding Folded Reed-Solomon Codes April 2010 Lecturer: Venkatesan Guruswami Scribe: Venkatesan Guruswami At the end of the previous notes,

### Learning in Abstract Memory Schemes for Dynamic Optimization

Fourth International Conference on Natural Computation Learning in Abstract Memory Schemes for Dynamic Optimization Hendrik Richter HTWK Leipzig, Fachbereich Elektrotechnik und Informationstechnik, Institut

### Digital System Design Prof. D Roychoudhry Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur

Digital System Design Prof. D Roychoudhry Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture - 04 Digital Logic II May, I before starting the today s lecture

### 4. Continuous Random Variables, the Pareto and Normal Distributions

4. Continuous Random Variables, the Pareto and Normal Distributions A continuous random variable X can take any value in a given range (e.g. height, weight, age). The distribution of a continuous random

### 1 The Brownian bridge construction

The Brownian bridge construction The Brownian bridge construction is a way to build a Brownian motion path by successively adding finer scale detail. This construction leads to a relatively easy proof

### Computer Networks and Internets, 5e Chapter 6 Information Sources and Signals. Introduction

Computer Networks and Internets, 5e Chapter 6 Information Sources and Signals Modified from the lecture slides of Lami Kaya (LKaya@ieee.org) for use CECS 474, Fall 2008. 2009 Pearson Education Inc., Upper

### Cryptography and Network Security Prof. D. Mukhopadhyay Department of Computer Science and Engineering Indian Institute of Technology, Karagpur

Cryptography and Network Security Prof. D. Mukhopadhyay Department of Computer Science and Engineering Indian Institute of Technology, Karagpur Lecture No. #06 Cryptanalysis of Classical Ciphers (Refer

### MATH 132: CALCULUS II SYLLABUS

MATH 32: CALCULUS II SYLLABUS Prerequisites: Successful completion of Math 3 (or its equivalent elsewhere). Math 27 is normally not a sufficient prerequisite for Math 32. Required Text: Calculus: Early

### Simple Random Sampling

Source: Frerichs, R.R. Rapid Surveys (unpublished), 2008. NOT FOR COMMERCIAL DISTRIBUTION 3 Simple Random Sampling 3.1 INTRODUCTION Everyone mentions simple random sampling, but few use this method for

### Probability and Statistics Vocabulary List (Definitions for Middle School Teachers)

Probability and Statistics Vocabulary List (Definitions for Middle School Teachers) B Bar graph a diagram representing the frequency distribution for nominal or discrete data. It consists of a sequence

### Solutions to Homework 10

Solutions to Homework 1 Section 7., exercise # 1 (b,d): (b) Compute the value of R f dv, where f(x, y) = y/x and R = [1, 3] [, 4]. Solution: Since f is continuous over R, f is integrable over R. Let x

### (Refer Slide Time: 01.26)

Discrete Mathematical Structures Dr. Kamala Krithivasan Department of Computer Science and Engineering Indian Institute of Technology, Madras Lecture # 27 Pigeonhole Principle In the next few lectures

### 4.3 Lagrange Approximation

206 CHAP. 4 INTERPOLATION AND POLYNOMIAL APPROXIMATION Lagrange Polynomial Approximation 4.3 Lagrange Approximation Interpolation means to estimate a missing function value by taking a weighted average

### Summation Algebra. x i

2 Summation Algebra In the next 3 chapters, we deal with the very basic results in summation algebra, descriptive statistics, and matrix algebra that are prerequisites for the study of SEM theory. You

### Irrational Numbers. A. Rational Numbers 1. Before we discuss irrational numbers, it would probably be a good idea to define rational numbers.

Irrational Numbers A. Rational Numbers 1. Before we discuss irrational numbers, it would probably be a good idea to define rational numbers. Definition: Rational Number A rational number is a number that

### MODELING RANDOMNESS IN NETWORK TRAFFIC

MODELING RANDOMNESS IN NETWORK TRAFFIC - LAVANYA JOSE, INDEPENDENT WORK FALL 11 ADVISED BY PROF. MOSES CHARIKAR ABSTRACT. Sketches are randomized data structures that allow one to record properties of

### The Union-Find Problem Kruskal s algorithm for finding an MST presented us with a problem in data-structure design. As we looked at each edge,

The Union-Find Problem Kruskal s algorithm for finding an MST presented us with a problem in data-structure design. As we looked at each edge, cheapest first, we had to determine whether its two endpoints

### Encoding Text with a Small Alphabet

Chapter 2 Encoding Text with a Small Alphabet Given the nature of the Internet, we can break the process of understanding how information is transmitted into two components. First, we have to figure out

### Lectures 5-6: Taylor Series

Math 1d Instructor: Padraic Bartlett Lectures 5-: Taylor Series Weeks 5- Caltech 213 1 Taylor Polynomials and Series As we saw in week 4, power series are remarkably nice objects to work with. In particular,

### Reading 13 : Finite State Automata and Regular Expressions

CS/Math 24: Introduction to Discrete Mathematics Fall 25 Reading 3 : Finite State Automata and Regular Expressions Instructors: Beck Hasti, Gautam Prakriya In this reading we study a mathematical model

### THE FUNDAMENTAL THEOREM OF ARBITRAGE PRICING

THE FUNDAMENTAL THEOREM OF ARBITRAGE PRICING 1. Introduction The Black-Scholes theory, which is the main subject of this course and its sequel, is based on the Efficient Market Hypothesis, that arbitrages

### 6.4 Normal Distribution

Contents 6.4 Normal Distribution....................... 381 6.4.1 Characteristics of the Normal Distribution....... 381 6.4.2 The Standardized Normal Distribution......... 385 6.4.3 Meaning of Areas under

### HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!

Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following

### Analysis of Algorithms, I

Analysis of Algorithms, I CSOR W4231.002 Eleni Drinea Computer Science Department Columbia University Thursday, February 26, 2015 Outline 1 Recap 2 Representing graphs 3 Breadth-first search (BFS) 4 Applications

### A simple criterion on degree sequences of graphs

Discrete Applied Mathematics 156 (2008) 3513 3517 Contents lists available at ScienceDirect Discrete Applied Mathematics journal homepage: www.elsevier.com/locate/dam Note A simple criterion on degree

### MAS108 Probability I

1 QUEEN MARY UNIVERSITY OF LONDON 2:30 pm, Thursday 3 May, 2007 Duration: 2 hours MAS108 Probability I Do not start reading the question paper until you are instructed to by the invigilators. The paper

### Lecture 18: Applications of Dynamic Programming Steven Skiena. Department of Computer Science State University of New York Stony Brook, NY 11794 4400

Lecture 18: Applications of Dynamic Programming Steven Skiena Department of Computer Science State University of New York Stony Brook, NY 11794 4400 http://www.cs.sunysb.edu/ skiena Problem of the Day

### Lecture 3: Finding integer solutions to systems of linear equations

Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture

### Lecture 4: BK inequality 27th August and 6th September, 2007

CSL866: Percolation and Random Graphs IIT Delhi Amitabha Bagchi Scribe: Arindam Pal Lecture 4: BK inequality 27th August and 6th September, 2007 4. Preliminaries The FKG inequality allows us to lower bound

### Entropy and Mutual Information

ENCYCLOPEDIA OF COGNITIVE SCIENCE 2000 Macmillan Reference Ltd Information Theory information, entropy, communication, coding, bit, learning Ghahramani, Zoubin Zoubin Ghahramani University College London

### Application. Outline. 3-1 Polynomial Functions 3-2 Finding Rational Zeros of. Polynomial. 3-3 Approximating Real Zeros of.

Polynomial and Rational Functions Outline 3-1 Polynomial Functions 3-2 Finding Rational Zeros of Polynomials 3-3 Approximating Real Zeros of Polynomials 3-4 Rational Functions Chapter 3 Group Activity:

### An example of a computable

An example of a computable absolutely normal number Verónica Becher Santiago Figueira Abstract The first example of an absolutely normal number was given by Sierpinski in 96, twenty years before the concept

### 1 Approximating Set Cover

CS 05: Algorithms (Grad) Feb 2-24, 2005 Approximating Set Cover. Definition An Instance (X, F ) of the set-covering problem consists of a finite set X and a family F of subset of X, such that every elemennt

### IAM 530 ELEMENTS OF PROBABILITY AND STATISTICS INTRODUCTION

IAM 530 ELEMENTS OF PROBABILITY AND STATISTICS INTRODUCTION 1 WHAT IS STATISTICS? Statistics is a science of collecting data, organizing and describing it and drawing conclusions from it. That is, statistics

### 1 Review of Least Squares Solutions to Overdetermined Systems

cs4: introduction to numerical analysis /9/0 Lecture 7: Rectangular Systems and Numerical Integration Instructor: Professor Amos Ron Scribes: Mark Cowlishaw, Nathanael Fillmore Review of Least Squares

### U.C. Berkeley CS276: Cryptography Handout 0.1 Luca Trevisan January, 2009. Notes on Algebra

U.C. Berkeley CS276: Cryptography Handout 0.1 Luca Trevisan January, 2009 Notes on Algebra These notes contain as little theory as possible, and most results are stated without proof. Any introductory

### Random variables, probability distributions, binomial random variable

Week 4 lecture notes. WEEK 4 page 1 Random variables, probability distributions, binomial random variable Eample 1 : Consider the eperiment of flipping a fair coin three times. The number of tails that

### 3. INNER PRODUCT SPACES

. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.

### Unit 1 Number Sense. In this unit, students will study repeating decimals, percents, fractions, decimals, and proportions.

Unit 1 Number Sense In this unit, students will study repeating decimals, percents, fractions, decimals, and proportions. BLM Three Types of Percent Problems (p L-34) is a summary BLM for the material

Academic Standards for Grades Pre K High School Pennsylvania Department of Education INTRODUCTION The Pennsylvania Core Standards in in grades PreK 5 lay a solid foundation in whole numbers, addition,

### Chapter 4 One Dimensional Kinematics

Chapter 4 One Dimensional Kinematics 41 Introduction 1 4 Position, Time Interval, Displacement 41 Position 4 Time Interval 43 Displacement 43 Velocity 3 431 Average Velocity 3 433 Instantaneous Velocity

### Linear Equations and Inequalities

Linear Equations and Inequalities Section 1.1 Prof. Wodarz Math 109 - Fall 2008 Contents 1 Linear Equations 2 1.1 Standard Form of a Linear Equation................ 2 1.2 Solving Linear Equations......................