Information Theory (2)

Transcription

1 Some Applications in Computer Science Jean-Marc Vincent MESCAL-INRIA Project Laboratoire d Informatique de Grenoble Universities of Grenoble, France Jean-Marc.Vincent@imag.fr LICIA - joint laboratory LIG -UFRGS This work was partially supported by CAPES/COFECUB 1 / 16

2 Modeling Application SYSTEM Modelling Environment description Workload generator Model description algorithms, automata, functions,... Simulation/Execution Kernel Simulation/Execution control Output analysis EVALUATION RESULTS 3 / 16

3 Workload generation problem (2) WORKLOAD GENERATOR Load profiler Random seed Structure profile Amount Random Generator Types Quantitative profile Amount Size INJECTOR KERNEL Temporal behaviour Distribution Elapsed time STATE OF THE SYSTEM Generated load - sequence of events - marks on events 4 / 16

4 Observations (example) Consider a Web server, 4 types of requests A, B, C, D. Basic Question Build a stochastic model of the workload : Without any other assumptions, we assume the workload be: - a stochastic sequence of independent and uniformly distributed random variables on {A, B, C, D} Some more assumptions For each type of request Type of request A B C D Average processing time (s) Observation : The average processing time of N requests is NT with T = 6 Build a stochastic model of the workload 5 / 16

5 MaxEnt Principle Observation Partial information on the system state X : Function Φ and we observe Eφ(X). Examples : φ(x) = 1 normalization; φ(x) = x average state; φ(x) = log(x) order of magnitude; φ(x) = 1 X>α threshold constraint... The total information on the system is given by C = {(φ j, a j ), j = 0, 1,..., m}, Eφ j (x) = a j, Convention : φ 0 = 1 and a 0 = 1 7 / 16

6 MaxEnt Principle Principle Under the set of constraints C = {(φ j, a j ), j = 0, 1,..., m}, model the system by the distribution maximizing H(X) = i p i ( log 2 p i ) under C. Minimize the a priori information inside the model description Uniform distribution is the 0-knowledge hypothesis Computable form of the distribution Danger : such a distribution may not exist 8 / 16

7 Application of MaxEnt Principle Maximize entropy of the model under constraints (Φ 0 ) p A + p B + p C + p D = 1, (Φ 1 ) 10p A + 4p B + 3p C + p D = 6. Using Lagrange multipliers and g(p, λ 0, λ 1 ) = p i ( log p i ) + λ 0 ( p i 1) + λ 1 ( T i p i T ). g p i = log p i 1 + λ 0 + λ 1 T i, p i = 2 λ 0 1+λ 1 T i. 9 / 16

8 Using constraints Ti 2 λ 1T i 2 λ 1 T i = T. Application of MaxEnt Principle λ 1 = Type of request A B C D Average processing time (s) Probability ,15 10 / 16

9 Classical laws Integer valued variable on {0, 1, 2,, n} No constraints : uniform distribution φ 1 (x) = x average constraint p i = 1 ρ 1 ρ n+1 ρi, Geometric distribution Extends to N 11 / 16

10 Continuous Variables Continuous Variable Entropy For X random variable with density f X we define H(X) = ( log f X (x))f X (x)dx. The properties of H are almost the same as for discrete distributions (positivity fails) Gibbs distributions A random variable has a Gibbs distribution when the density has the form m f (x) = exp λ j φ j (x), j=0 play a central role in statistics exponential, Gaussian,... are Gibbs distributions 13 / 16

11 Optimal distributions Gibbs densities are MaxEnt distributions Suppose that there is a Gibbs distribution of X satisfying the constraints C = {(φ j, a j ), j = 1,..., m}. then X has the maximum entropy distribution under C. Uniform distribution is MaxEnt under constraint X [a, b] Exponential distribution is MaxEnt under constraint EX = m Normal distribution is MaxEnt under constraint EX = m and Var X = σ 2 Poisson process, Markov process, etc. 14 / 16

12 For modeling systems Synthesis establish the knowledge on your system parameters (fixed or variable) establish what is really random establish the knowledge on the random part (put the constraints) apply the MaxEnt principle (use independence if there are no correlations) generate/analyse your workload Generalization Combinatorial structures Non uniforme reference distribution (Gaussian)... References Cover, T. and Thomas, J. (2006), Elements of Information Theory 2nd Edition, Wiley-Interscience E. T. Jaynes, Information Theory and Statistical Mechanics, Physical Review, vol. 106, no. 4, pp ; May 15, / 16