Basic Descriptive Statistics & Probability Distributions



Similar documents
Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur

Math Quizzes Winter 2009

BNG 202 Biomechanics Lab. Descriptive statistics and probability distributions I

Discrete Mathematics and Probability Theory Fall 2009 Satish Rao, David Tse Note 13. Random Variables: Distribution and Expectation

Mathematical Expectation

Chicago Booth BUSINESS STATISTICS Final Exam Fall 2011

Exploratory Data Analysis. Psychology 3256

Lecture 14. Chapter 7: Probability. Rule 1: Rule 2: Rule 3: Nancy Pfenning Stats 1000

Statistics and Random Variables. Math 425 Introduction to Probability Lecture 14. Finite valued Random Variables. Expectation defined

How to Win the Stock Market Game

Probability and Statistics Vocabulary List (Definitions for Middle School Teachers)

Problem sets for BUEC 333 Part 1: Probability and Statistics

Betting systems: how not to lose your money gambling

9. Sampling Distributions

Geostatistics Exploratory Analysis

Module 3: Correlation and Covariance

FACT A computer CANNOT pick numbers completely at random!

Center: Finding the Median. Median. Spread: Home on the Range. Center: Finding the Median (cont.)

PITFALLS IN TIME SERIES ANALYSIS. Cliff Hurvich Stern School, NYU

If A is divided by B the result is 2/3. If B is divided by C the result is 4/7. What is the result if A is divided by C?

Exploratory Data Analysis

Financial Mathematics and Simulation MATH Spring 2011 Homework 2

Chapter 4 Lecture Notes

consider the number of math classes taken by math 150 students. how can we represent the results in one number?

The Binomial Distribution

Summary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4)

International Statistical Institute, 56th Session, 2007: Phil Everson

An Introduction to Basic Statistics and Probability

WHERE DOES THE 10% CONDITION COME FROM?

6.042/18.062J Mathematics for Computer Science December 12, 2006 Tom Leighton and Ronitt Rubinfeld. Random Walks

STA 130 (Winter 2016): An Introduction to Statistical Reasoning and Data Science

13.0 Central Limit Theorem

Institute of Actuaries of India Subject CT3 Probability and Mathematical Statistics

2DI36 Statistics. 2DI36 Part II (Chapter 7 of MR)

Math Review. for the Quantitative Reasoning Measure of the GRE revised General Test

Statistics 104: Section 6!

4. Continuous Random Variables, the Pareto and Normal Distributions

THE ROULETTE BIAS SYSTEM

Chapter 16. Law of averages. Chance. Example 1: rolling two dice Sum of draws. Setting up a. Example 2: American roulette. Summary.

CA200 Quantitative Analysis for Business Decisions. File name: CA200_Section_04A_StatisticsIntroduction

E3: PROBABILITY AND STATISTICS lecture notes

The Normal Approximation to Probability Histograms. Dice: Throw a single die twice. The Probability Histogram: Area = Probability. Where are we going?

MONEY MANAGEMENT. Guy Bower delves into a topic every trader should endeavour to master - money management.

Polynomials and Factoring. Unit Lesson Plan

Discrete Math in Computer Science Homework 7 Solutions (Max Points: 80)

MA 1125 Lecture 14 - Expected Values. Friday, February 28, Objectives: Introduce expected values.

Probability and Expected Value

Descriptive Statistics. Purpose of descriptive statistics Frequency distributions Measures of central tendency Measures of dispersion

The Math. P (x) = 5! = = 120.

Curriculum Map Statistics and Probability Honors (348) Saugus High School Saugus Public Schools

Recursive Estimation

6 PROBABILITY GENERATING FUNCTIONS

The overall size of these chance errors is measured by their RMS HALF THE NUMBER OF TOSSES NUMBER OF HEADS MINUS NUMBER OF TOSSES

Multivariate Analysis of Variance (MANOVA): I. Theory

For a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )

A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution

Descriptive Statistics

Stat 20: Intro to Probability and Statistics

Statistics 100A Homework 3 Solutions

A Coefficient of Variation for Skewed and Heavy-Tailed Insurance Losses. Michael R. Powers[ 1 ] Temple University and Tsinghua University

Measures of Central Tendency and Variability: Summarizing your Data for Others

WEEK #22: PDFs and CDFs, Measures of Center and Spread

Experimental Designs (revisited)

Example: Find the expected value of the random variable X. X P(X)

The Assumption(s) of Normality

Algebra I Vocabulary Cards

The Binomial Probability Distribution

Math 431 An Introduction to Probability. Final Exam Solutions

3: Summary Statistics

STT315 Chapter 4 Random Variables & Probability Distributions KM. Chapter 4.5, 6, 8 Probability Distributions for Continuous Random Variables

Section 7C: The Law of Large Numbers

6.4 Normal Distribution

Quantitative Methods for Finance

Def: The standard normal distribution is a normal probability distribution that has a mean of 0 and a standard deviation of 1.

Chapter 5: Normal Probability Distributions - Solutions

Data Analysis Tools. Tools for Summarizing Data

Recall this chart that showed how most of our course would be organized:

Ch. 3.1 # 3, 4, 7, 30, 31, 32

Contemporary Mathematics Online Math 1030 Sample Exam I Chapters No Time Limit No Scratch Paper Calculator Allowed: Scientific

Lecture 1: Review and Exploratory Data Analysis (EDA)

Statistics Review PSY379

Using Excel (Microsoft Office 2007 Version) for Graphical Analysis of Data

Intro to Data Analysis, Economic Statistics and Econometrics

Chapter 5. Discrete Probability Distributions

A Quick Algebra Review

DESCRIPTIVE STATISTICS. The purpose of statistics is to condense raw data to make it easier to answer specific questions; test hypotheses.

Lecture 3: Continuous distributions, expected value & mean, variance, the normal distribution

business statistics using Excel OXFORD UNIVERSITY PRESS Glyn Davis & Branko Pecar

Question: What is the probability that a five-card poker hand contains a flush, that is, five cards of the same suit?

Economics 1011a: Intermediate Microeconomics

Mind on Statistics. Chapter 8

A New Interpretation of Information Rate

Texas Hold em. From highest to lowest, the possible five card hands in poker are ranked as follows:

You flip a fair coin four times, what is the probability that you obtain three heads.

Characteristics of Binomial Distributions

AMS 5 CHANCE VARIABILITY

How To Write A Data Analysis

MAS108 Probability I

UNIT I: RANDOM VARIABLES PART- A -TWO MARKS

Solution Let us regress percentage of games versus total payroll.

Transcription:

Basic Descriptive Statistics & Probability Distributions Scott Oser Lecture #2 Physics 509 1

Outline Last time: we discussed the meaning of probability, did a few warmups, and were introduced to Bayes theorem. Today we cover more basics. Basic descriptive statistics Covariance and correlation Properties of the Gaussian distribution The binomial distribution Application of binomial distributions to sports betting The multinomial distribution Physics 509 2

Basic Descriptive Statistics WHAT IS THIS DISTRIBUTION? Physics 509 3 Often the probability distribution for a quantity is unknown. You may be able to sample it with finite statistics, however. Basic descriptive statistics is the procedure of encoding various properties of the distribution in a few numbers.

The Centre of the Data: Mean, Median, & Mode Mean of a data set: N x= 1 N i=1 x i Mean of a PDF = expectation value of x x dx P x x Median: the point with 50% probability above & 50% below. (If a tie, use an average of the tied values.) Less sensitive to tails! Mode: the most likely value 4

Variance V & Standard Deviation (a.k.a. RMS) Variance of a distribution: V x = 2 = dx P x x 2 V x = dx P x x 2 2 dx P x x 2 dx P x = x 2 2 = x 2 x 2 Variance of a data sample (regrettably has same notation as variance of a distribution---be careful!): V x = 2 = 1 N i x i x 2 =x 2 x 2 An important point we'll return to: the above formula underestimates the variance of the underlying distribution, since it uses the mean calculated from the data instead of the true mean of the true distribution. V x = 2 = 1 x N 1 i x 2 V x = 2 = 1 i N x i 2 i This is unbiased if you must estimate the mean from the data. Use this if you know the true mean of the underlying distribution. Physics 509 5

FWHM & Quartiles/Percentiles FWHM = Full Width Half Max. It means what it sounds like--- measure across the width of a distribution at the point where P(x)=(1/2)(P max ). For Gaussian distributions, FWHM=2.35 Quartiles, percentiles, and even the median are rank statistics. Sort the data from lowest to highest. The median is the point where 50% of data are above and 50% are below. The quartile points are those at which 25%, 50%, and 75% of the data are below that point. You can also extend this to percentile rank, just like on a GRE exam. FWHM or some other width parameter, such as 75% percentile data point 25% data point, are often robust in cases where the RMS is more sensitive to events on tails. Physics 509 6

Higher Moments Of course you can calculate the r th moment of a distribution if you really want to. For example, the third central moment is called the skew, and is sensitive to the asymmetry of the distribution (exact definition may vary---here's a unitless definition): skew= = 1 x N 3 i x 3 i Kurtosis (or curtosis) is the fourth central moment, with varying choices of normalizations. For fun you are welcome to look up the words leptokurtotic and platykurtotic, but since I speak Greek I don't have to. Warning: Not every distribution has well-defined moments. The integral or sum will sometimes not converge! Physics 509 7

A bad distribution: the Cauchy distribution Consider the Cauchy, or Breit-Wigner, distribution. Also called a Lorentzian. It is characterized by its centroid M and its FWHM. P x, M = 1 2 x M 2 /2 2 A Cauchy distribution has infinite variance and higher moments! Unfortunately the Cauchy distribution actually describes the mass peak of a particle, or the width of a spectral line, so this distribution actually occurs! Physics 509 8 Cauchy (black) vs. Gaussian (red)

Covariance & Correlation The covariance between two variables is defined by: cov x, y = x x y y = xy x y This is the most useful thing they never tell you in most lab courses! Note that cov(x,x)=v(x). The correlation coefficient is a unitless version of the same thing: cov x, y = x y If x and y are independent variables (P(x,y) = P(x)P(y)), then cov x, y = dx dy P x, y xy dx dy P x, y x dx dy P x, y y = dx P x x dy P y y dx P x x dy P y y = 0 Physics 509 9

More on Covariance Correlation coefficients for some simulated data sets. Physics 509 10 Note the bottom right---while independent variables must have zero correlation, the reverse is not true! Correlation is important because it is part of the error propagation equation, as we'll see.

Variance and Covariance of Linear Combinations of Variables Suppose we have two random variable X and Y (not necessarily independent), and that we know cov(x,y). Consider the linear combinations W=aX+bY and Z=cX+dY. It can be shown that cov(w,z)=cov(ax+by,cx+dy) = cov(ax,cx) + cov(ax,dy) + cov(by,cx) + cov(by,dy) = ac cov(x,x) + (ad + bc) cov(x,y) + bd cov(y,y) = ac V(X) + bd V(Y) + (ad+bc) cov(x,y) Special case is V(X+Y): V(X+Y) = cov(x+y,x+y) = V(X) + V(Y) + 2cov(X,Y) Very special case: variance of the sum of independent random variables is the sum of their individual variances! Physics 509 11

Gaussian Distributions By far the most useful distribution is the Gaussian (normal) distribution: P x, = 1 1 2 2 e x 2 2 Mean =, Variance= 2 Note that width scales with. Area out on tails is important---use lookup tables or cumulative distribution function. In plot to left, red area (>2 ) is 2.3%. 68.27% of area within 1 95.45% of area within 2 99.73% of area within 3 90% of area within 1.645 95% of area within 1.960 99% of area within 2.576 Physics 509 12

Why are Gaussian distributions so critical? They occur very commonly---the reason is that the average of several independent random variables often approaches a Gaussian distribution in the limit of large N. Nice mathematical properties---infinitely differentiable, symmetric. Sum or difference of two Gaussian variables is always itself Gaussian in its distribution. Many complicated formulas simplify to linear algebra, or even simpler, if all variables have Gaussian distributions. Gaussian distribution is often used as a shorthand for discussing probabilities. A 5 sigma result means a result with a chance probability that is the same as the tail area of a unit Gaussian: 2 5 dt P t =0, =1 This way of speaking is used even for non-gaussian distributions! Physics 509 13

Why you should be very careful with Gaussians.. The major danger of Gaussians is that they are overused. Although many distributions are approximately Gaussian, they often have long non-gaussian tails. While 99% of the time a Gaussian distribution will correctly model your data, many foul-ups result from that other 1%. It's usually good practice to simulate your data to see if the distributions of quantities you think are Gaussian really follow a Gaussian distribution. Common example: the ratio of two numbers with Gaussian distributions is itself often not very Gaussian (although in certain limits it may be). Physics 509 14

A slightly non-trivial example Two measurements (X & Y) are drawn from two separate normal distributions. The first distribution has mean=5 & RMS=2. The second has mean=3 & RMS=1. The correlation coefficient of the two distributions is =-0.5. What is the distribution of the sum Z=X+Y? Physics 509 15

A slightly non-trivial example Two measurements (X & Y) are drawn from two separate normal distributions. The first distribution has mean=5 & RMS=2. The second has mean=3 & RMS=1. The correlation coefficient of the two distributions is =-0.5. What is the distribution of the sum Z=X+Y? First, recognize that the sum of two Gaussians is itself Gaussian, even if there is a correlation between the two. To see this, imagine that we drew two truly independent Gaussian random variables X and W. Then we could form a linear combination Y=aX+bW. Y would clearly be Gaussian, although correlated with X. Then Z=X+Y=X+aX+bW=(a+1)X+bW is the sum of two truly independent Gaussian variables itself. So Z must be a Gaussian. Physics 509 16

A slightly non-trivial example Two measurements (X & Y) are drawn from two separate normal distributions. The first distribution has mean=5 & RMS=2. The second has mean=3 & RMS=1. The correlation coefficient of the two distributions is =-0.5. What is the distribution of the sum Z=X+Y? Now, recognizing that Z is Gaussian, all we need to figure out are its mean and RMS. First the mean: X Y = dx dy P X,Y X Y = dx dy P X,Y X dx dy P X,Y Y = X Y This is just equal to 5+3 = 8. Physics 509 17

A slightly non-trivial example Two measurements (X & Y) are drawn from two separate normal distributions. The first distribution has mean=5 & RMS=2. The second has mean=3 & RMS=1. The correlation coefficient of the two distributions is =-0.5. What is the distribution of the sum Z=X+Y? Now for the RMS. Use V(Z)=cov(Z,Z)=cov(X+Y,X+Y) V(Z) = cov(x,x) + 2 cov(x,y) + cov(y,y) = x 2 + 2 x y + y 2 = (2)(2) + 2(2)(1)(-0.5) + (1)(1) = 3 So Z is a Gaussian with mean=8 and RMS of =sqrt(3) Physics 509 18

Binomial Distributions Many outcomes are binary---yes/no, heads/tails, etc. Ex. You flip N unbalanced coins. Each coin has probability p of landing heads. What is the probability that you get m heads (and N-m tails)? The binomial distribution: P m p, N = p m 1 p N m N! m! N m! First term: probability of m coins all getting heads Second term: probability of N-m coins all getting tails Third term: number of different ways to pick m different coins from a collection of N total be to heads. Physics 509 19

Binomial distributions P m p, N = p m 1 p N m N! m! N m! Mean = Np Variance = Np(1-p) Notice that the mean and variance both scale linearly with N. This is understandable---flipping N coins is the sum of N independent binomial variables. When N gets big, the distribution looks increasingly Gaussian! 20

But a binomial distribution isn't a Gaussian! Gaussian approximation fails out on the tails... Physics 509 21

More on the binomial distribution In the limit of large Np, Gaussian approximation is decent so long as P(m=0) P(m=N) 0, provided you don't care much about tails. Beware a common error: =sqrt(np(1-p)), not =sqrt(m)=sqrt(np). The latter is only true if p 1. The error is not always just the simple square root of the number of entries! Use a binomial distribution to model most processes with two outcomes: Detection efficiency (either we detect or we don't) Cut rejection Win-loss records (although beware correlations between teams that play in the same league) Physics 509 22

An example from the world of sports... Consider a best-of-seven series... the first team to win four games takes the prize. We have a model which predicts that Team A is favoured in any game with p=0.6. What is the probability that A wins the series? How could we approach this problem? Physics 509 23

Best of 7 series: brute force Easiest approach may be simply to list the possibilities: A. Win in 4 straight games. Probability = p 4 B. Win in 5 games. Four choices for which game the team gets to lose. Probability = 4p 4 (1-p) C. Win in 6 games. Choose 2 of the previous five games to lose. Probability = C(5,2)p 4 (1-p) 2 = 10p 4 (1-p) 2 D. Win in 7 games. Choose 3 of the previous six games to lose. Probability = C(6,3)p 4 (1-p) 3 = 20p 4 (1-p) 3 Prob p = p 4 1 4 1 p 10 1 p 2 20 1 p 3 Physics 509 24

Best of 7 series: outcomes Prob p = p 4 1 4 1 p 10 1 p 2 20 1 p 3 Symmetry evident between p and 1-p, which makes good logical sense For p=0.6, probability of series win is only 71% Physics 509 25

Best of 7 series: online betting studies Efficient market hypothesis: if a market mis-estimates a risk, smart investors will figure this out and bet accordingly, driving the odds back to the correct value. There is significant evidence that this hypothesis (almost) holds in many real-life markets. See A Random Walk Down Wall Street for details.* Does this work for online sports betting? * Warning: reading this book may endanger your career in physics by getting you interested in quantitative analysis of financial markets. Physics 509 26

Best of 7 series: online betting studies I got interested in this during the 2006 baseball playoffs, as my beloved Cardinals came very close to collapsing entirely, yet went on to win the World Series. I used a coin flip model to predict series odds: All games treated as independent, with equal probability. In simplest case, assume p=0.5 More complicated case: using Bill James' Pythagorean Theorem to predict winning percentage of matchup: Runs Scored 2 p= Runs Scored 2 Runs Allowed 2 Physics 509 27

My brother is stupid. Younger brothers always are. He objected to my coin flip model: Assigning 50/50 odds is ludicrous when you know the Astros will start Clemens. If you want to estimate p, you should only look at recent records to estimate odds. How dare he deny my math! But the proof is in the pudding... Physics 509 28

What do the markets say? Odds for St. Louis to win NL Central title (going into final weekend): Coin flip model says 74.6%. Betting market said 74%. After St. Louis loses some ground to Houston, coin flip model says 59%. The betting markets predicted 61%. My brother's recency prior for p predicts 45%. Going into next to last day of season, my coin flip model says 89% for Cards. Betting market is mixed: odds for Cards to win are all over the map, but odds for Houston to win is right at 11%. Opportunity for arbitrage. Last day: coin flip and markets both predict ~93% Physics 509 29

In the middle of the first round of playoffs Coin Flip Betting Markets St Louis over San Diego 68.8% 66.1% Dodgers over Mets 50.0% 44.6% Twins over A's 31.3% 36.0% Yanks over Detroit 68.8% 85.5% One way to view the betting market odds is actually as an estimator of the p value for a matchup. For example, the market felt (wrongly) that Detroit was badly overmatched. In the end, CARDS WIN! Physics 509 30

Negative binomial distribution In a regular binomial distribution, you decide ahead of time how many times you'll flip the coin, and calculate the probability of getting k heads. In the negative binomial distribution, you decide how many heads you want to get, then calculate the probability that you have to flip the coin N times before getting that many heads. This gives you a probability distribution for N: P N k, p = N 1 k 1 pk 1 p N k Physics 509 31

Multinomial distribution We can generalize a binomial distribution to the case where there are more than two possible outcomes. Suppose there are k possible outcomes, and we do N trials. Let n i be the number of times that the i th outcome comes up, and let p i be the probability of getting outcome i in one trial. The probability of getting a certain distribution of n i is then: P n 1, n 2,...,n k p 1... p k = N! n 1! n 2!...n k! p n 1 n 1 p2 2... n k pk Note that there are important constraints on the parameters: k i p i =1 k i n i =N Physics 509 32

What is the multinomial distribution good for? Any problem in which there are several discrete outcomes (binomial distribution is a special case). Note that unlike the binomial distribution, which basically predicts one quantity (the number of heads---you get the number of tails for free), the multinomial distribution is a joint probability distribution for several variables (the various n i, of which all but one are independent). If you care about just one of these, you can marginalize over the other (sum them over all of their possible values) to get the probability distribution for the one you care about. This obviously will have a binomial distribution. A common application: binned data! If you sample independent trials from a distribution and bin the results, the numbers you predict for each bin follow the multinomial distribution. Physics 509 33

Dealing with binned data Very often you're going to deal with binned data. Maybe there are too many individual data points to handle efficiently. Maybe you binned it to make a pretty plot, then want to fit a function to the plot. Some gotchas: Nothing in the laws of statistics demands equal binning. Consider binning with equal statistics per bin. Beware bins with few data points. Many statistical tests implicitly assume Gaussian errors, which won't hold for small numbers. General rule of thumb: rebin until every bin has >5 events. Always remember that binning throws away information. Don't do it unless you must. Try to make bin size smaller than any relevant feature in the data. If statistics don't permit this, then you shouldn't be binning, at least for that part of the distribution. Physics 509 34