03. Boltzmann Entropy, Gibbs Entropy, Shannon Information. I. Entropy in Statistical Mechanics.

Transcription

1 03. Boltzmann Entropy, Gibbs Entropy, Shannon Information. I. Entropy in Statistical Mechanics. Goal: To explain the behavior of macroscopic systems in terms of the dynamical laws governing their microscopic consituents. In particular: To provide a micro-dynamical explanation of the 2nd Law. 1. Boltzmann's Approach. Consider different "macrostates" of a gas: Ludwig Boltzmann Why does the gas prefer to be in the equilibrium macrostate (last one)?

2 Suppose the gas consists of N identical particles governed by Hamilton's equations of motion (the micro-dynamics). Def. 1. A microstate X of a gas is a specification of the position (3 values) and momentum (3 values) for each of its N particles.! Ω = phase space = 6N-dim space of all possible microstates.! Ω E = region of Ω that consists of all microstates with constant energy E. Hamiltonian dynamics maps initial microstate X i to final microstate X f. Can 2nd Law be explained by recourse to this dynamics? X i X f Ω E

3 Def. 2. A macrostate Γ of a gas is a specification of the gas in terms of macroscopic properties (pressure, temperature, volume, etc.). Relation between microstates and macrostates: Macrostates supervene on microstates!! To each microstate there corresponds exactly one macrostate.! Many distinct microstates can correspond to the same macrostate. So: Ω E is partitioned into a finite number of regions corresponding to macrostates, with each microstate X belonging to one macrostate Γ(X). microstates macrostate Ω E Γ(X) X

4 Claim #1: Different macrostates have vastly different sizes. Def. 3. The Boltzmann Entropy is defined by S B (Γ(X)) = k log Γ(X) Γ(X) = the volume of Γ(X). So: S B (Γ(X)) is a measure of the size of Γ(X). Claim #2: The equilibrium macrostate Γ eq is vastly larger than any other macrostate (in other words, S B obtains its maximum value for Γ eq ). Γ eq Ω E very small nonequilibrium macrostates Thus: S B increases over time because, for any initial microstate X i, the dynamics will map X i into Γ eq very quickly, and then keep it there for an extremely long time.

5 Two Ways to Explain the Approach to Equilibrium: (a) Appeal to Typicality Claim: A system approaches equilibrium because equilibrium microstates are typical and nonequilibrium microstates are atypical. Why? For large N, Ω E is almost entirely filled up with equilibrium microstates. Hence they are "typical".! But: What is it about the dynamics that evolves atypical states to typical states? - "If a system is in an atypical microstate, it does not evolve into an equilibrium microstate just because the latter is typical." (Frigg 2009.)! Need to identify properties of the dynamics that guarantee atypical states evolve into typical states.! And: Need to show that these properties are typical. - Ex: If the dynamics is chaotic (in an appropriate sense), then (under certain conditions), any initial microstate X i will quickly be mapped into Γ eq and remain there for long periods of time. (Frigg 2009.)

6 (b) Appeal to Probabilities Claim: A system approaches equilibrium because it evolves from states of lower toward states of higher probability, and the equilibrium state is the state of highest probabililty. Associate probabilities with macrostates: the larger the macrostate, the greater the probability of finding a microstate in it. "In most cases, the initial state will be a very unlikely state. From this state the system will steadily evolve towards more likely states until it has finally reached the most likely state, i.e., the state of thermal equilibrium."

7 P 6 w 1 w 2 w 3 P 89 Arrangement #1: state of P 6 in w 1, state of P 89 in w 3, etc. Ω µ! Start with the 6-dim phase space Ω µ of a single particle. (Ω E = N copies of Ω µ ) Partition Ω µ into l cells w 1, w 2,..., w l of size δw. A state of an N-particle system is given by N points in Ω µ. point in Ω µ = singleparticle microstate. Def. 4. An arrangement is a specification of which points lie in which cells.

8 P 89 w 1 w 2 w 3 P 6 Arrangement #1: state of P 6 in w 1, state of P 89 in w 3, etc. Arrangement #2: state of P 89 in w 1, state of P 6 in w 3, etc. Distribution: (1, 0, 2, 0, 1, 1,...) Ω µ! Takes form (n 1, n 2,..., n l ), where n j = # of points in w j. Start with the 6-dim phase space Ω µ of a single particle. (Ω E = N copies of Ω µ ) Partition Ω µ into l cells w 1, w 2,..., w l of size δw. A state of an N-particle system is given by N points in Ω µ. Def. 4. An arrangement is a specification of which points lie in which cells. Def. 5. A distribution is a specification of how many points (regardless of which ones) lie in each cell. point in Ω µ = singleparticle microstate. Note: More than one arrangement can correspond to the same distribution.

9 How many arrangements G(D i ) are compatible with a given distribution D i = (n 1, n 2,..., n l )? Answer: Check: G(D i ) = N! n 1!n 2!!n l!! Let D 1 = (N, 0,..., 0) and D 2 = (N 1, 1, 0,..., 0). n! = n(n 1)(n 2)#1 = number of sequences that n distinguishable objects can be arranged. 0! = 1! G(D 1 ) = N!/N! = 1. (Only one way for all N particles to be in w 1.)! G(D 2 ) = N!/(N 1)! = N(N 1)(N 2)#1/(N 1)(N 2)#1 = N. (There are N different ways w 2 could have one point in it; namely, if P 1 was in it, or if P 2 was in it, or if P 3 was in it, etc...) "The probability of this distribution [D i ] is then given by the number of permutations of which the elements of this distribution are capable, that is by the number [G(D i )]. As the most probable distribution, i.e., as the one corresponding to thermal equilibrium, we again regard that distribution for which this expression is maximal..."

10 Note: Each distribution D i corresponds to a macrostate Γ Di. Why? Because a system's macroscopic properties (volume, pressure, temp, etc) only depend on how many particles are in particular microstates, and not on which particles are in which microstates. What is the size of this macrostate?! Note: A point in Ω E corresponds to an arrangement in Ω µ.! And: The size of a macrostate Γ Di in Ω E is given by the number of points it contains (i.e., the number of arrangements compatible with D i ) multiplied by a volume element of Ω E.! And: A volume element of Ω E is given by N copies of a volume element δw of Ω µ. So: The size of Γ Di is Γ Di = number of arrangements compatible with D i $ volume element of Ω E = G(D i ) δw N Thus: The Boltzmann entropy of Γ Di is given by: S B (Γ Di ) = k log(g(d i ) δw N ) = k log(g(d i )) + Nk log(δw) = k log(g(d i )) + const. "... S B is a measure of how much we can infer about the arrangement of a system on the basis of its distribution. The higher S B, the less information a distribution conveys about the arrangement of the system." (Frigg & Werndl 2011)

11 S B (Γ Di ) = k log(g(d i )) + const. N! = k log n 1!n 2!!n l! +const. Stirling's approx: logn! nlogn n n n l = N = k log(n!) k log(n 1!)... k log(n l!) + const. (Nk logn N ) (n 1 k logn 1 n 1 )... (n l logn l n l ) + const. l = k n j logn j j=1 Let: p j = n j /N = +const. probability of finding a randomly chosen microstate in cell w j Then: S B (Γ Di ) = Nk p j log p j l j=1 +const. The biggest value of S B is for the distribution D i for which the p j 's are all 1/l; i.e., the n j 's are all N/l, which is the equilibrium distribution. Boltzmann (1872) shows: The D i that maximizes S B is characterized by n j = µe λε j (Maxwell equilibrium distribution, where ε j = energy of microstates in cell w j ).

12 Relation between Boltzmann Entropy S B and Thermodynamic Entropy S T First: Let's derive the Maxwell-Boltzmann equilibrium distribution. Assume: A system of N weakly interacting particles described by: n j = N j ε j n j = U j n j = # states in cell w j N = total # particles ε j = energy associated with states in w j U = total internal energy Recall: S B (n j ) = (Nklog N N ) k (n j log n j n j ) + const. j d d So: S B = 0 k (n j logn j n j ) dn j dn + 0 j Or: ds B = k j = k (log n j + n j /n j 1) j = k j j log n j log n j dn j

13 Now: S B takes its maximum value for the values n j * that solve: ds B = k j log n j* dn j = 0 small changes to S B due only to small changes dn j! subject to the constraints on the small changes dn j : dn = dn j = 0 j du = ε j dn j = 0 j Note: Can add arbitrary multiples of the constraints to our equation and still get zero result: ds B = j ( k log n j * + α + βε j )dn j = 0 Or: k log n j * + α + βε j = 0 for appropriate choices of α and β! Now solve for n j* : n j * = e (α + βε j )/k Maxwell-Boltzmann equilibrium distribution for weakly interacting, distinguishable particles(the most probable distribution, corresponding to the largest macrostate).!

14 What is α? Substitute n j * into n j = N and get: N = e (α + βε j )/k = e α/k e βε j /k. Or: α = k log (N / e βε j /k ) α is a normalization constant that enforces correct particle number N More importantly: What is β? Consider: Small changes in internal energy of a reversible process: Macroscopic point of view du = δq dw = TdS T PdV Microscopic point of view du = d( ε j n j ) = ε j dn j + n j dε j Note: If PdV = n j dε j, then ds T = (1/T) ε j dn j Recall: ds B (n j* ) = k log n j* dn j = k (α + βε j )/k dn j, since log(e (α + βε j )/k ) = (α + βε j )/k = β ε j dn j, since kα dn j = 0 So: For the equilibrium distribution n j*, S B = S T provided β = 1/T.

15 Macroscopic point of view du = δq dw = TdS T PdV Microscopic point of view du = d( ε j n j ) = ε j dn j + n j dε j Recap: ds B (n j* ) = β ε j dn j So: For the equilibrium distribution n j*, S B = S T provided β = 1/T. What this shows: For a reversible process involving a large number of distinguishable particles characterized by their positions and velocities, it is consistent to identify the Boltzmann entropy with the thermodynamic entropy. But: Are we forced to?! S T measures absolute changes in heat per temperature of a reverisble process.! For thermally isolated processes, S T absolutely increases or remains constant.! S B measures how likely a given distribution of states occurs.! No absolute law that requires S B to increase or remain constant.

16 2. Gibbs' Approach. Boltzmann's Schtick: analysis of a single system. Each point of phase space Ω represents a possible state of the system. Willard Gibbs Gibbs' Schtick: analysis of an ensemble of infinitely many systems. Each point of phase space Ω represents a possible state of one member of the ensemble. The state of the entire ensemble is given by a density function ρ(x, t) on Ω. ρ(x, t)dx is the number of systems in the ensemble whose states lie in the region (x, x + dx). p t (R) = ρ(x,t)dx = R probability at time t of finding the state of a system in region R The Gibbs Entropy is then given by: S G (ρ) = k ρ(x,t)log(ρ(x,t))dx Ω

17 Interpretive Issues: (1) Why do low-probability states evolve into high-probability states? Characterizations of the dynamics are, again, required to justify this. (2) How are the probabilities to be interpreted? (a) Ontic probabilities = properties of physical systems Long run frequencies? Single-case propensities? (b) Epistemic probabilities = measures of degrees of belief Objective (rational) degrees of belief? Subjective degrees of belief?

18 II. Entropy in Classical Information Theory. Goal: To construct a measure for the amount of information associated with a message. The amount of info gained from the reception of a message depends on how likely it is. The less likely a message is, the more info gained upon its reception! Let X = {x 1, x 2,..., x l } = set of l messages. Def. 1: A probability distribution P = (p 1, p 2,..., p l ) on X is an assignment of a probability p j = p(x j ) to each message x j. Claude Shannon Recall: This means p j 0 and p 1 + p p l = 1.

19 Def. 2: A measure of information for X is a real-valued function H(X) : {prob. distributions on X} R, that satisfies:! Continuity. H(p 1,..., p l ) is continuous.! Additivity. H(p 1 q 1,..., p l q l ) = H(P) + H(Q), for probability distributions P, Q.! Monoticity. Info increases with l for uniform distributions: If m > l, then H(Q) > H(P), for any P = (1/l,..., 1/l) and Q = (1/m,..., 1/m).! Branching. H(p 1,..., p l ) is independent of how the process is divided into parts.! Bit normalization. The average info gained for two equally likely messages is one bit: H(½, ½) = 1. Claim (Shannon 1949): There is exactly one function that satisfies these criteria; namely, the Shannon Entropy (or Shannon Information): H(X) = l j=1 p j log 2 p j! H(X) is maximal for p 1 = p 2 =... = p l = 1/l.! H(X) = 0 just when one p j is 1 and the rest are 0.! Logarithm is to base 2: log 2 x = y x = 2 y. Bit normalization requires: If X = {x 1, x 2 }, and P = (½, ½), then H(X) = 1. Note: H(X) = (½log½ + ½log½) = log2. And: log2 = 1 if and only if log is to base 2. In what sense is this a measure of information?

20 1. H(X) as Maximum Amount of Message Compression Let X = {x 1,..., x l } be a set of letters from which we construct the messages. Suppose the messages have N letters a piece. The probability distribution P = (p 1,..., p l ) is now over the letter set. What this means:! Each letter x i has a probability of p i of occuring in a message.! In other words: A typical message will contain p 1 N occurrences of x 1, p 2 N occurrences of x 2, etc. Thus: So: log 2 The number of distinct typical messages The number of distinct typical messages N! = (p 1 N )!(p 2 N )!!(p l N )! N! = log 2 (p 1 N )!(p 2 N )!!(p l N )! Let's simplify the RHS...

21 N! log 2 (p 1 N )!(p 2 N )!!(p l N )! = log 2(N!) {log 2 ((p 1 N )!)+...+ log 2 ((p l N )!)} (N log 2 N N) {(p 1 N log 2 p 1 N p 1 N )+...+(p l N log 2 p l N p l N )} = N {log 2 N 1 p 1 log 2 p 1 p 1 log 2 N + p 1... p l log 2 p l p l log 2 N + p l } l = N p j log 2 p j j=1 = NH(X) Thus: log 2 The number of distinct typical messages = NH(X) So: The number of distinct typical messages = 2 NH(X)

22 So: There are only 2 NH(X) typical messages with N letters. This means, at the message level, we can encode them using only NH(X) bits. Check: 2 possible messages require 1 bit: 0, 1. 4 possible messages require 2 bits: 00, 01, 10, 11. etc. Now: At the letter level, how many bits are needed to encode a message of N letters drawn from an n-letter alphabet? First: How many bits are needed to encode each letter in an n-letter alphabet? n = #letters x = #bits per letter% 2 letters 1 bit: 0, 1 4 letters 2 bits: 00, 01, 10, 11 8 letters 3 bits: 000, 001, 010, 011, 100, 101, 110, 111 So: n = 2 x, so x = log 2 n. Note: log 2 n bits per letter entails N log 2 n bits for a sequence of N letters. Thus: Instead of requiring N log 2 n bits to encode our messages, we can get by with only NH(X) bits. Thus: H(X) represents the maximum amount that (typical) messages drawn from a given set of letters can be compressed.

23 Ex: Let X = {A, B, C, D} (n = 4) Then: We need log 2 4 = 2 bits per letter. For instance: A = 00, B = 01, C = 10, D = 11. So: We need 2N bits to encode a message with N letters. Now: Suppose the probabilities for each letter to occur in a typical N-letter message are the following: p A = 1/2, p B = 1/4, p C = p D = 1/8 Then: The number of such typical N-letter messages is: NH(X) = N ( 1 log log log log ) = 1.75N Thus: Instead of requiring 2N bits to encode the N messages, we can get by with only 1.75N. Note: If all letters are equally likely (the equilibrium distribution), then p A = p B = p C = p D = 1/4. And: ( ) = 2N. NH(X) = N 1 4 log log log log 2 1 4

24 2. H(X) as a Measure of Uncertainty Suppose P = (p 1,..., p l ) is a probability distribution over a set of values {x 1,..., x l } of a random variable X. Def. 1. The expected value E(X) of X is given by E(X) = l p j x j. j=1 Def. 2. The information gained if X is measured to have the value x j is given by log 2 p j.! Motivation: The greater p j is, the more certain x j is, and the less information should be associated with it. Then the expected value of log 2 p j is just the Shannon information: E( log 2 p j ) = What this means: l j=1 p j log 2 p j = H(X) H(X) tells us our expected information gain upon measuring X.

25 3. Conditional Entropy Def. 1: A communication channel is a device with a set of input states X = {x 1,..., x l } that are mapped to a set of output states Y = {y 1,..., y l }. Def. 2: Given probability distributions p(x i ) and p(y i ) on X and Y, and a joint distribution p(x i y i ), the conditional probability p(x i y j ) of input x i, given output y j, is defined by: p(x i y j ) = p(x i y j ) p(y j ) Def. 3: The conditional Shannon Entropy of X given Y is given by: H(X Y ) = p(y j ) p(x i y j )log 2 p(x i y j ) What this means: j i H(X Y) measures the average uncertainty about the value of an input, given a particular output value.

26 Suppose we use our channel a very large number N of times.! Then: There will be 2 NH(X) typical input strings.! And: There will be 2 NH(Y) typical output strings.! Thus: There will be 2 NH(X Y) typical strings of X and Y values. So: # of typical input strings that could result in a given output Y = 2NH (X Y ) 2 NH (Y ) = 2 N (H (X Y ) H (Y )) Claim: H(X Y) = H(Y) + H(X Y) So: # of typical input strings that could result in a given output Y NH (X Y ) = 2 If one is trying to use a noisey channel to send a message, then the conditional entropy specifies the # of bits per letter that would need to be sent by an auxiliary noiseless channel in order to correct all the errors due to noise.

27 Check Claim: H(X Y ) = p(x i y j )log 2 [p(x i y j )] i,j = p(x i y j )log 2 [p(y j )p(x i y j )] i,j = p(x i y j )log 2 [p(y j )] p(x i y j )log 2 [p(x i y j )] i,j = p(y j )log 2 [p(y j )] p(y j )p(x i y j )log 2 [p(x i y j )] j i,j i,j = H(Y )+ H(X Y ) p(x i y j ) = p(y j ) i

28 Interpretive Issues: 1. How should the probabilities p(x i ) be interpreted? Emphasis is on uncertainty: The information content of a message x i is a function of how uncertain it is, with respect to the receiver. So: Perhaps the probabilities are epistemic. In particular: p(x i ) is a measure of the receiver's degree of belief in the accuracy of message x i. But: The probabilities are set by the nature of the source. If the source is not probabilistic, then p(x i ) can be interpreted epistemically. If the source is inherently probabilistic, then p(x i ) can be interpreted as the ontic probability that the source produces message x i.

29 2. How is Shannon Information/Entropy related to other notions of entropy? Thermodynamic entropy: Boltzmann entropy: ΔS = S f S i = S B (Γ Di ) = Nk l R j=1 i f δq R T p j ln p j +const. Gibbs entropy: S G (ρ) = k Ω ρ(x,t)ln(ρ(x,t))dx Shannon entropy: H(X) = l j=1 p j log 2 p j Can statistical mechanics be given an information-theoretic foundation? Can the 2nd Law be given an information-theoretic foundation?

30 Relation between Boltzmann Entropy S B and Shannon Entropy H: Boltzmann gas N = # of single-particle microstates. (n 1,..., n l ) = l-cell distribution. (p 1,..., p l ) = probabilty distribution over cells. p j = n j /N = probability that a microstate occurs in cell w j. Np j = # of microstates in cell w j. S B (Γ Di ) = Nk l j=1 p j ln p j +const. Size of macrostate corresponding to (p 1,..., p l ). Shannon message N = # of letters in message. {x 1,..., x l } = l-letter alphabet. (p 1,..., p l ) = probabilty distribution over letters. p j = n j /N = probability that x j occurs in a given message. Np j = # of x j 's in message. H(X) = l j=1 p j log 2 p j Maximum amount an N-length message drawn from {x 1,..., x l } can be compressed.