# Statistical foundations of machine learning

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Machine learning p. 1/45 Statistical foundations of machine learning INFO-F-422 Gianluca Bontempi Département d Informatique Boulevard de Triomphe - CP 212

2 Machine learning p. 2/45 Testing hypothesis Hypothesis testing is the second major area of statistical inference. A statistical hypothesis is an assertion or conjecture about the distribution of one or more random variables. A test of a statistical hypothesis is a rule or procedure for deciding whether to reject the assertion on the basis of the observed data. The basic idea is formulate some statistical hypothesis and look to see if the data provides any evidence to reject the hypothesis.

3 Machine learning p. 3/45 An hypothesis testing problem Consider the model of the traffic in the boulevard. Suppose that the measures of the inter-arrival times are D N = {10, 11, 1, 21, 2,... } seconds. Can we say that the mean inter-arrival time θ is different from 10? Consider the grades of two different school sections. Section A had {15, 10, 12, 19, 5, 7}. Section B had {14, 11, 11, 12, 6, 7}. Can we say that Section A had better grades than Section B? Consider two protein coding genes and their expression levels in a cell. Are the two genes differentially expressed? A statistical test is a procedure that aims to answer such questions.

4 Machine learning p. 4/45 Types of hypothesis We start by declaring the working (basic, null) hypothesis H to be tested, in the form θ = θ 0 or θ ω Θ, where θ 0 or ω are given. The hypothesis can be Simple. It fully specifies the distribution of z. Composite. It partially specifies the distribution of z. Example: if D N constitutes a random sample of size N from N(µ, σ 2 ) the hypothesis H : µ = µ 0, σ = σ 0, (with µ 0 and σ 0 known values) is simple while the hypothesis H : µ = µ 0 is composite since it leaves open the value of σ in (0, ).

5 Machine learning p. 5/45 Types of statistical test Suppose we have collected N samples D N = {z 1,...,z N } from a distribution F z and we have declared a null hypothesis H about F. Three are the most common types of statistical test: Pure significance test: data D N are used to assess the inferential evidence against H. Significance test: the inferential evidence against H is used to judge whether H is inappropriate. In other words it is a rule for rejecting H. Hypothesis test: data D N are used to assess the hypothesis H against a specific alternative hypothesis H. In other words this is a rule for rejecting H in favour of H.

6 Machine learning p. 6/45 Pure significance test Suppose that the null hypothesis H is simple. Let t(d N ) be a statistic such that the larger its value the more it casts doubt on H. The quantity t(d N ) is called test statistic or discrepancy measure. Let t N = t(d N ) the value of t calculated on the basis of the sample data D N. Let us consider the p-value quantity p = Prob {t(d N ) > t N H} If p is small the sample data D N are highly inconsistent with H and p (significance probability or significance level ) is the measure of such inconsistency.

7 Machine learning p. 7/45 Some considerations p is the proportion of situations under the hypothesis H where we would observe a degree of inconsistency at least to the extent represented by t N. t N is the observed value of the statistic for a given D N. Different D N yield different values of p (0, 1). it is essential that the distribution of t(d N ) under H is known. We cannot say that p is the probability that H is true but better that p is the probability that the dataset D N is observed given that H is true Open issues 1. What if H is composite? 2. how to choose t(d N ).

8 Machine learning p. 8/45 Tests of significance Suppose that the value p is known. If p is small either a rare event has occured or perhaps H is not true. Idea: if p is less than some stated value α, we reject H. We choose a critical level α, we observe D N and we reject H at level α if P {t(d N ) > t N H) α This is equivalent to choose some critical value t α and we reject H if t N > t α. We obtain two regions in the space of sample data: critical region S 0 where if D N S 0 we reject H. non-critical region S 1 where the sample data D N gives us no-reason to reject H on the basis of the level-α test.

9 Machine learning p. 9/45 Some considerations The principle is that we will accept H unless we witness some event that has sufficiently small probability of arising when H is true. If H were true we could still obtain data in S 0 and consequently wrongly reject H with probability Prob {D N S 0 H} = Prob {t(d N ) > t α H} = α The significance level α provides an upper bound to the maximum probability of incorrectly rejecting H. The p-value is the probability that the test statistic is more extreme than its observed value. The p-value changes with the observed data (i.e. it is a random variable) while α is a level fixed by the user.

10 Machine learning p. 10/45 Standard normal distribution 1 Normal distribution function (µ=0, σ=1) 0.4 Normal density function (µ=0, σ=1) Remember that z This means that, if z N(0, 1), then Prob {z z 0.05 } = 0.05 and also that For a generic z N(µ, σ 2 ) Prob { z z 0.05 } = = 0.1 Prob { z µ σ z 0.05 } = = 0.1

11 Machine learning p. 11/45 TP: example Let D N consist of N independent observations of x N(µ, σ 2 ), with known variance σ 2. We want to test the hypothesis H : µ = µ 0 with µ 0 known. Consider as test statistic t(d N ), the quantity ˆµ µ 0 where ˆµ is the sample average estimator. If H is true we know that ˆµ N(µ 0, σ 2 /N). Let us calculate the value t(d N ) = ˆµ µ 0 and assume that the rejection region is S 0 = { ˆµ µ 0 ˆµ µ 0 > t α }. Let us put a significance level α = 10% = 0.1. This means that t α should satisfy Prob {t(d N ) > t α H} = Prob { ˆµ µ 0 > t α H} = Prob {(ˆµ µ 0 > t α ) (OR) (ˆµ µ 0 < t α ) H} = 0.1

12 Machine learning p. 12/45 TP: example (II) For a normal variable x N(µ, σ 2 ) Prob {x µ > 1.645σ} = 1 F x (1.645σ) = 0.05 and consequently Prob {x µ > 1.645σ (OR) x µ < 1.645σ} = = 0.1 It follows that being ˆµ N(µ 0, σ 2 /N) (i.e. ˆµ µ 0 σ/ N N(0, 1)) once we put t α = 1.645σ/ N we have Prob { ˆµ µ 0 > t α H} = 0.1 and that the critical region is S 0 = { D N : ˆµ µ 0 > 1.645σ/ } N

13 Machine learning p. 13/45 TP: example (III) Suppose that σ = 0.1 and that we want to test if µ = µ 0 = 10 with a significance level 10%. After N = 6 observations we have D N = {10, 11, 12, 13, 14, 15}. On the basis of the dataset we compute ˆµ = = 12.5 and t(d N ) = ˆµ µ 0 = 2.5 Since t α = / 6 = , and t(d N ) > t α, the observations D N are in the critical region. The hypothesis is rejected.

14 Machine learning p. 14/45 Hypothesis testing: types of error So far we considered a single hypothesis. Let us now consider two alternative hypothesis: H and H. Type I error. It is the error we make when we reject H if it is true. Significance level represents the probability of making the type I error. Type II error. It is the error we make when we accept H if it is false. In order to define this error, we are forced to declare an alternative hypothesis H as a formal definition of what is meant by H being false. The probability of type II error is the probability that the test leads to acceptance of H when in fact H prevails. When the alternative hypothesis is composite, there is no unique Type II error.

15 Machine learning p. 15/45 An analogy Consider the analogy with a murder trial, where we have as suspect Mr. Bean. The null hypothesis H is Mr. Bean is innocent. The dataset is the amount of evidence collected by the police against Mr. Bean. The Type I error is the error that we make if, being Mr. Bean innocent, we send him to penalty death. The Type II error is the error that we make if, being Mr. Bean guilty, we acquit him.

16 Machine learning p. 16/45 Hypothesis testing Suppose we have some data {z 1,...,z N } F from a distribution F. H and H represent two hypotheses about F. On the basis of the data, one is accepted and one is rejected. Note that the two hypotheses have different philosophical status (asymmetry). H is a conservative hypothesis, not to be rejected unless evidence is clear. This means that a type I error is more serious than a type II error (benefit of the doubt). It is often assumed that F belongs to a parametric family F(z, θ). The test on F becomes a test on θ. A particular example of hypothesis test is the goodness of fit test where we test H : F = F 0 against H : F F 0.

17 Machine learning p. 17/45 The five steps of hypothesis testing 1. Declare the null (e.g. H: honest student) and the alternative hypothesis ( H: cheat student) 2. Choose the numeric value of the type I error (e.g. the risk I want to run). 3. Choose a procedure to obtain test statistic (e.g. number of similar lines). 4. Determine the critical value of the test statistic (e.g. 4 identical lines) that leads to a rejection of H. This is done in order to ensure the Type I error defined in Step Obtain the data and determine whether the observed value of the test statistic leads to an acceptation or rejection of H.

18 Machine learning p. 18/45 Quality of the test Suppose that N students took part to the exam, N N did not copy, N P copied, ˆNN were considered not guilty and passed the exam ˆNP were considered guilty and rejected F P honest students were refused F N cheat students passed.

19 Machine learning p. 19/45 Confusion matrix Then we have Not refused Refused H: Not guilty student (-) T N F P N N H: Guilty student (+) F N T P N P ˆN N ˆNP N F P is the number of False Positives and the ratio F P /N N represents the type I error. F N is the number of False Negatives and the ratio F N /N P represents the type II error.

20 Machine learning p. 20/45 Specificity and sensitivity Specificity: the ratio (to be maximized) SP = T N F P + T N = T N N N = N N F P N N = 1 F P N N, 0 SP 1 It increases by reducing the number of false positive. Sensitivity: the ratio (to be maximized) SE = T P T P + F N = T P N P = N P F N N P = 1 F N N P, 0 SE 1 It increases by reducing the number of false negatives and corresponds to the power of the test (i.e. it estimates the quantity 1-Type II error).

21 Machine learning p. 21/45 Specificity and sensitivity (II) There exists a trade-off between these two quantities. In the case of a test who return always H (e.g. very kind professor) we have ˆN P = 0, ˆN N = N, F P = 0, T N = N N and SP = 1 but SE = 0. In the case of a test who return always H (e.g. very suspicious professor) we have ˆN P = N, ˆN N = 0, F N = 0, T P = N P and SE = 1 but SP = 0.

22 Machine learning p. 22/45 False Positive and False Negative Rate False Positive Rate: FPR = 1 SP = 1 T N F P + T N = F P F P + T N = F P N N, 0 FPR 1 It decreases by reducing the number of false positive and estimates the Type I error. False Negative Rate FNR = 1 SE = 1 T P T P + F N = F N T P + F N = F N N P 0 FPR 1 It decreases by reducing the number of false negative.

23 Machine learning p. 23/45 Predictive value Positive Predictive value: the ratio(to be maximized) PPV = T P T P + F P = T P ˆN P, 0 PPV 1 Negative Predictive value: the ratio (to be maximized) PNV = T N T N + F N = T N ˆN N, 0 PNV 1 False Discovery Rate: the ratio (to be minimized) FDR = F P T P + F P = F P ˆN P = 1 PPV, 0 FDR 1

24 Machine learning p. 24/45 Receiver Operating Characteristic curve The Receiver Operating Characteristic (also known as ROC curve) is a plot of the true positive rate (i.e. sensitivity or power) against the false positive rate (Type I error) for the different possible decision thresholds of a test. Consider an example where t + N(1, 1) and t N( 1, 1). Suppose that the examples are classed as positive if t > THR and negative if t < THR, where THR is a threshold. If THR =, all the examples are classed as positive: TN = FN = 0 which implies SE = T P N P = 1 and FPR = F P F P +T N = 1. If THR =, all the examples are classed as negative: TP = FP = 0 which implies SE = 0 and FPR = 0.

25 Machine learning p. 25/45 ROC curve SE FPR R script roc.r

26 Machine learning p. 26/45 Choice of test The choice of test and consequently the choice of the partition {S 0, S 1 } is based on two steps 1. Define a significance level α, that is the probability of type I error Prob {reject H H} = Prob {D N S 0 H} α that is the probability of incorrectly rejecting H 2. Among the set of tests {S 0, S 1 } of level α, choose the test that minimizes the probability of type II error Prob { accept H H } = Prob { D N S 1 H } that is the probability of incorrectly accepting H. This is equivalent to look for maximizing the power of the test Prob { reject H H } = Prob { D N S 0 H } = 1 Prob { D N S 1 H } which is the probability of correctly rejecting H. The higher the power, the better!

27 Machine learning p. 27/45 TP example Consider a r.v. z N(µ, σ 2 ), where σ is known and a set of N iid observations are given. We want to test the null hypothesis µ = µ 0 = 0, with α = 0.1 Consider the 3 critical regions S 0 1. ˆµ µ 0 > 1.645σ/ N 2. ˆµ µ 0 > 1.282σ/ N 3. ˆµ µ 0 < 0.126σ/ N For all these tests Prob {D N S 0 H} α, hence the significance level is the same. However if H : µ = 10 the type II error of the three tests is significantly different. What is the best one?

28 Machine learning p. 28/45 µ: H S TP example (II) µ: H On the left: distribution of the test statistic ˆµ if H : µ 0 = 0 is true. On the right: distribution of the test statistic ˆµ if H : µ 1 = 10 is true. The interval marked by S 1 denotes the set of observed ˆµ values for which H is accepted (non-critical region). The interval marked by S 0 denotes the set of observed ˆµ values for which H is rejected (critical region). The area of the black pattern region on the right equals Prob {D N S 0 H}, i.e. the probability of rejecting H when H is true (Type I error). The area of the grey shaded region on the left equals the probability of accepting H when H is false (Type II error). S

29 Machine learning p. 29/45 TP example (III) S 1 µ: H 0 10 S 0 S 1 µ: H On the left: distribution of the test statistic ˆµ if H : µ 0 = 0 is true. On the right: distribution of the test statistic ˆµ if H : µ 1 = 10 is true. The two intervals marked by S 1 denote the set of observed ˆµ values for which H is accepted (non-critical region). The interval marked by S 0 denotes the set of observed ˆµ values for which H is rejected (critical region). The area of the pattern region equals Prob {D N S 0 H}, i.e. the probability of rejecting H when H is true (Type I error). Which area corresponds to the probability of the Type II error?

30 Machine learning p. 30/45 Type of parametric tests Consider random variables with a parametric distribution F(, θ). One-sample vs. two-sample: in the one-sample test we consider a single r.v. and we formulate hypothesis about its distribution. In the two-samples test we consider 2 r.v. z 1 and z 2 and we formulate hypothesis about their differences/similarities. Simple vs composite: the test is simple if H describes completely the distributions of the involved r.v. otherwise it is composite. Single-sided (or one-tailed) vs Two-sided (or two-tailed): in the single-sided test the region of rejection concerns only one tail of the distribution of the null distribution. This means that H indicates the predicted direction of the difference (e.g. H : θ > θ 0 ). In the two-sided test, the region of rejection concern both tails of the null distribution. This means that H does not indicate the predicted direction of the difference (e.g. H : θ θ 0 ).

31 Machine learning p. 31/45 Example of parametric test Consider a parametric test on the distribution of a gaussian r.v., and suppose that the null hypothesis is H : θ = θ 0 where θ 0 is given and represents the mean. The test is one-sample and composite. In order to know whether it is one or two-sided we have to define the alternative configuration: if H : θ < θ 0 the test is one-sided down, if H : θ > θ 0 the test is one-sided up, if H : θ θ 0 the test is double-sided.

32 Machine learning p. 32/45 z-test (one-sample and one-sided) Consider a random sample D N x N(µ, σ 2 ) with µ unknown et σ 2 known. STEP 1: Consider the null hypothesis and the alternative (composite and one-sided) H : µ = µ 0 ; H : µ > µ0 STEP 2: fix the value α of the type I error. STEP 3: choose a test statistic: If H is true then the distribution of ˆµ is N(µ 0, σ 2 /N). This means that the variable z is z = (ˆµ µ 0) N N(0, 1) σ It is convenient to rephrase the test in terms of the test statistic z.

33 Machine learning p. 33/45 z-test (one-sample and one-sided) (II) STEP 4: determine the critical value for z. We reject the hypothesis H is rejected if z N > z α where z α is such that Prob {N(0, 1) > z α } = α. Ex: for α = 0.05 we would take z α = since 5% of the standard normal distribution lies to the right of R command: z α =qnorm(alpha,lower.tail=false) STEP 5: Once the dataset D N is measured, the value of the test statistic is z N = (ˆµ µ 0) N σ

34 Machine learning p. 34/45 TP: example z-test Consider a r.v. z N(µ, 1). We want to test H : µ = 5 against H : µ > 5 with significance level Supose that the data is D N = {5.1, 5.5, 4.9, 5.3}. Then ˆµ = 5.2 and z N = (5.2 5) 2/1 = 0.4. Since this is less than z α = 1.645, we do not reject the null hypothesis.

35 Machine learning p. 35/45 Two-sided parametric tests Assumption: all the variables are normal! Name one/two sample known H H z-test one σ 2 µ = µ 0 µ µ 0 z-test two σ1 2 = σ2 2 µ 1 = µ 2 µ 1 µ 2 t-test one µ = µ 0 µ µ 0 t-test two µ 1 = µ 2 µ 1 µ 2 χ 2 -test one µ σ 2 = σ0 2 σ 2 σ0 2 χ 2 -test one σ 2 = σ0 2 σ 2 σ0 2 F-test two σ1 2 = σ2 2 σ1 2 σ2 2

36 Machine learning p. 36/45 Student s t-distribution If x N(0, 1) and y χ 2 N are independent then the Student s t-distribution with N degrees of freedom is the distribution of the r.v. z = x y/n We denote this with z t N. If z 1,...,z N are i.i.d. N(µ, σ 2 ) then N(ˆµ µ) ŜS/(N 1) = N(ˆµ µ) ˆσ t N 1

37 Machine learning p. 37/45 t-test: one-sample and two-sided Consider a random sample from N(µ, σ 2 ) with σ 2 unknown. Let H : µ = µ 0 ; H : µ µ0 Let t(d N ) = T = 1 N 1 N(ˆµ µ0 ) = (ˆµ µ 0) N i=1 (z i ˆµ) 2 ˆσ 2 N a statistic computed using the data set D N.

38 Machine learning p. 38/45 t-test: one-sample and two-sided (II) It can be shown that if the hypothesis H holds, T T N 1 is a r.v. with a Student distribution with N 1 degrees of freedom. The size α t-test consists in rejecting H if T > k = t α/2,n 1 where t α/2,n 1 is the upper α point of a T -distribution on N 1 degrees of freedom, i.e. Prob { t N 1 > t α/2,n 1 } = α/2. where t N 1 T N 1. In other terms H is rejected when T is large. R command: t α/2,n 1 =qt(alpha/2,n-1,lower.tail=true)

39 Machine learning p. 39/45 TP example Does jogging lead to a reduction in pulse rate? Eight non jogging volunteers engaged in a one-month jogging programme. Their pulses were taken before and after the programme pulse rate before pulse rate after decrease Suppose that the decreases are samples from N(µ, σ 2 ) for some unknown σ 2. We want to test H : µ = µ 0 = 0 against H : µ 0 with a significance α = We have N = 8, ˆµ = 2.75, T = 1.263, t α/2,n 1 = Since T t α/2,n 1, the data is not sufficient to reject the hypothesis H. In other terms we have not enough evidence to show that there is a reduction in pulse rate.

40 Machine learning p. 40/45 The chi-squared distribution For a N positive integer, a r.v. z has a χ 2 N distribution if z = x x 2 N where x 1,x 2,...,x N are i.i.d. random variables N(0, 1). The probability distribution is a gamma distribution with parameters ( 1 2 N, 1 2 ). E[z] = N and Var[z] = 2N. The distribution is called a chi-squared distribution with N degrees of freedom.

41 Machine learning p. 41/45 χ 2 -test: one-sample and two-sided Consider a random sample from N(µ, σ 2 ) with µ known. Let H : σ 2 = σ0; 2 H : σ2 σ0 2 Let ŜS = i (z i µ) 2. It can be shown that if H is true then ŜS/σ2 0 χ 2 N The size α χ 2 -test rejects H if ŜS/σ2 0 < a 1 or ŜS/σ2 0 > a 2 where Prob {ŜS σ 2 0 < a 1 } + Prob {ŜS σ 2 0 > a 2 } = α If µ is unknown, you must 1. replace µ with ˆµ in the quantity ŜS 2. use a χ 2 N 1 distribution.

42 Machine learning p. 42/45 t-test: two-samples, two-sided Consider two r.v.s x N(µ 1, σ 2 ) and y N(µ 2, σ 2 ) with the same variance. Let DN x and Dy M two independent sets of samples. We want to test H : µ 1 = µ 2 against H : µ 1 µ 2. Let ˆµ x = N i=1 x i N, SS x = N (x i ˆµ x ) 2, ˆµ y = i=1 M i=1 y i M, SS y = M (y i ˆµ y ) 2 i=1 Once defined the statistic T = ( 1 M + 1 N ˆµ x ˆµ y ) ( SS x +SS y M+N 2 ) T M+N 2 it can be shown that a test of size α rejects H if T > t α/2,m+n 2

43 Machine learning p. 43/45 F-distribution Let x χ 2 M and y χ2 N be two independent r.v.. A r.v. z has a F-distribution F m,n with M and N degrees of freedom if If z F M,N then 1/z F N,M. If z T N then z 2 F 1,N. z = x/m y/n

44 Machine learning p. 44/45 F-distribution 0.9 F M,N density: M=20 N=10 1 F M,N cumulative distribution: M=20 N= R script s_f.r.

45 Machine learning p. 45/45 F-test: two-samples, two-sided Consider a random sample x 1,...,x M from N(µ 1, σ 2 1) and a random sample y 1,...,y N from N(µ 2, σ 2 2) with µ 1 and µ 2 unknown. Suppose we want to test Let us consider the statistic H : σ 2 1 = σ 2 2; H : σ 2 1 σ 2 2 f = ˆσ2 1 ˆσ 2 2 = ŜS 1/(M 1) ŜS 2 /(N 1) σ2 1 χ2 M 1 /(M 1) σ2 2χ2 N 1 /(N 1) = σ2 1 σ2 2 F M 1,N 1 It can be shown that if H is true, the ratio f has a F-distribution F M 1,N 1 We reject H if the ratio f is large, i.e. f > F α,m 1,N 1 where if z F M 1,N 1. Prob {z > F α,m 1,N 1 } = α

### Hypothesis Testing Level I Quantitative Methods. IFT Notes for the CFA exam

Hypothesis Testing 2014 Level I Quantitative Methods IFT Notes for the CFA exam Contents 1. Introduction... 3 2. Hypothesis Testing... 3 3. Hypothesis Tests Concerning the Mean... 10 4. Hypothesis Tests

### Null Hypothesis Significance Testing Signifcance Level, Power, t-tests Spring 2014 Jeremy Orloff and Jonathan Bloom

Null Hypothesis Significance Testing Signifcance Level, Power, t-tests 18.05 Spring 2014 Jeremy Orloff and Jonathan Bloom Simple and composite hypotheses Simple hypothesis: the sampling distribution is

### 9-3.4 Likelihood ratio test. Neyman-Pearson lemma

9-3.4 Likelihood ratio test Neyman-Pearson lemma 9-1 Hypothesis Testing 9-1.1 Statistical Hypotheses Statistical hypothesis testing and confidence interval estimation of parameters are the fundamental

### Quantitative Biology Lecture 5 (Hypothesis Testing)

15 th Oct 2015 Quantitative Biology Lecture 5 (Hypothesis Testing) Gurinder Singh Mickey Atwal Center for Quantitative Biology Summary Classification Errors Statistical significance T-tests Q-values (Traditional)

### Chapter 8 Introduction to Hypothesis Testing

Chapter 8 Student Lecture Notes 8-1 Chapter 8 Introduction to Hypothesis Testing Fall 26 Fundamentals of Business Statistics 1 Chapter Goals After completing this chapter, you should be able to: Formulate

### Hypothesis Testing or How to Decide to Decide Edpsy 580

Hypothesis Testing or How to Decide to Decide Edpsy 580 Carolyn J. Anderson Department of Educational Psychology University of Illinois at Urbana-Champaign Hypothesis Testing or How to Decide to Decide

### 22. HYPOTHESIS TESTING

22. HYPOTHESIS TESTING Often, we need to make decisions based on incomplete information. Do the data support some belief ( hypothesis ) about the value of a population parameter? Is OJ Simpson guilty?

Chapter 8 Hypothesis Tests Chapter Table of Contents Introduction...157 One-Sample t-test...158 Paired t-test...164 Two-Sample Test for Proportions...169 Two-Sample Test for Variances...172 Discussion

### HYPOTHESIS TESTING: POWER OF THE TEST

HYPOTHESIS TESTING: POWER OF THE TEST The first 6 steps of the 9-step test of hypothesis are called "the test". These steps are not dependent on the observed data values. When planning a research project,

### Chapter 8. Hypothesis Testing

Chapter 8 Hypothesis Testing Hypothesis In statistics, a hypothesis is a claim or statement about a property of a population. A hypothesis test (or test of significance) is a standard procedure for testing

### Statistical Inference and t-tests

1 Statistical Inference and t-tests Objectives Evaluate the difference between a sample mean and a target value using a one-sample t-test. Evaluate the difference between a sample mean and a target value

### Tutorial 5: Hypothesis Testing

Tutorial 5: Hypothesis Testing Rob Nicholls nicholls@mrc-lmb.cam.ac.uk MRC LMB Statistics Course 2014 Contents 1 Introduction................................ 1 2 Testing distributional assumptions....................

### Inferential Statistics

Inferential Statistics Sampling and the normal distribution Z-scores Confidence levels and intervals Hypothesis testing Commonly used statistical methods Inferential Statistics Descriptive statistics are

### Statistiek I. t-tests. John Nerbonne. CLCG, Rijksuniversiteit Groningen. John Nerbonne 1/35

Statistiek I t-tests John Nerbonne CLCG, Rijksuniversiteit Groningen http://wwwletrugnl/nerbonne/teach/statistiek-i/ John Nerbonne 1/35 t-tests To test an average or pair of averages when σ is known, we

### Section 7.1. Introduction to Hypothesis Testing. Schrodinger s cat quantum mechanics thought experiment (1935)

Section 7.1 Introduction to Hypothesis Testing Schrodinger s cat quantum mechanics thought experiment (1935) Statistical Hypotheses A statistical hypothesis is a claim about a population. Null hypothesis

### Hypothesis testing S2

Basic medical statistics for clinical and experimental research Hypothesis testing S2 Katarzyna Jóźwiak k.jozwiak@nki.nl 2nd November 2015 1/43 Introduction Point estimation: use a sample statistic to

### Two-Sample T-Tests Allowing Unequal Variance (Enter Difference)

Chapter 45 Two-Sample T-Tests Allowing Unequal Variance (Enter Difference) Introduction This procedure provides sample size and power calculations for one- or two-sided two-sample t-tests when no assumption

### Module 7: Hypothesis Testing I Statistics (OA3102)

Module 7: Hypothesis Testing I Statistics (OA3102) Professor Ron Fricker Naval Postgraduate School Monterey, California Reading assignment: WM&S chapter 10.1-10.5 Revision: 2-12 1 Goals for this Module

### Two-Sample T-Tests Assuming Equal Variance (Enter Means)

Chapter 4 Two-Sample T-Tests Assuming Equal Variance (Enter Means) Introduction This procedure provides sample size and power calculations for one- or two-sided two-sample t-tests when the variances of

### Hypothesis Testing. Bluman Chapter 8

CHAPTER 8 Learning Objectives C H A P T E R E I G H T Hypothesis Testing 1 Outline 8-1 Steps in Traditional Method 8-2 z Test for a Mean 8-3 t Test for a Mean 8-4 z Test for a Proportion 8-5 2 Test for

### THE FIRST SET OF EXAMPLES USE SUMMARY DATA... EXAMPLE 7.2, PAGE 227 DESCRIBES A PROBLEM AND A HYPOTHESIS TEST IS PERFORMED IN EXAMPLE 7.

THERE ARE TWO WAYS TO DO HYPOTHESIS TESTING WITH STATCRUNCH: WITH SUMMARY DATA (AS IN EXAMPLE 7.17, PAGE 236, IN ROSNER); WITH THE ORIGINAL DATA (AS IN EXAMPLE 8.5, PAGE 301 IN ROSNER THAT USES DATA FROM

### How to Conduct a Hypothesis Test

How to Conduct a Hypothesis Test The idea of hypothesis testing is relatively straightforward. In various studies we observe certain events. We must ask, is the event due to chance alone, or is there some

### Probability and Statistics Lecture 9: 1 and 2-Sample Estimation

Probability and Statistics Lecture 9: 1 and -Sample Estimation to accompany Probability and Statistics for Engineers and Scientists Fatih Cavdur Introduction A statistic θ is said to be an unbiased estimator

### Chapter 2. Hypothesis testing in one population

Chapter 2. Hypothesis testing in one population Contents Introduction, the null and alternative hypotheses Hypothesis testing process Type I and Type II errors, power Test statistic, level of significance

### 3.4 Statistical inference for 2 populations based on two samples

3.4 Statistical inference for 2 populations based on two samples Tests for a difference between two population means The first sample will be denoted as X 1, X 2,..., X m. The second sample will be denoted

### Statistiek (WISB361)

Statistiek (WISB361) Final exam June 29, 2015 Schrijf uw naam op elk in te leveren vel. Schrijf ook uw studentnummer op blad 1. The maximum number of points is 100. Points distribution: 23 20 20 20 17

### Inferences About Differences Between Means Edpsy 580

Inferences About Differences Between Means Edpsy 580 Carolyn J. Anderson Department of Educational Psychology University of Illinois at Urbana-Champaign Inferences About Differences Between Means Slide

### Chapter 1 Hypothesis Testing

Chapter 1 Hypothesis Testing Principles of Hypothesis Testing tests for one sample case 1 Statistical Hypotheses They are defined as assertion or conjecture about the parameter or parameters of a population,

### Data Analysis. Lecture Empirical Model Building and Methods (Empirische Modellbildung und Methoden) SS Analysis of Experiments - Introduction

Data Analysis Lecture Empirical Model Building and Methods (Empirische Modellbildung und Methoden) Prof. Dr. Dr. h.c. Dieter Rombach Dr. Andreas Jedlitschka SS 2014 Analysis of Experiments - Introduction

### Chapter 7. Section Introduction to Hypothesis Testing

Section 7.1 - Introduction to Hypothesis Testing Chapter 7 Objectives: State a null hypothesis and an alternative hypothesis Identify type I and type II errors and interpret the level of significance Determine

### Two-sample hypothesis testing, I 9.07 3/09/2004

Two-sample hypothesis testing, I 9.07 3/09/2004 But first, from last time More on the tradeoff between Type I and Type II errors The null and the alternative: Sampling distribution of the mean, m, given

### General Procedure for Hypothesis Test. Five types of statistical analysis. 1. Formulate H 1 and H 0. General Procedure for Hypothesis Test

Five types of statistical analysis General Procedure for Hypothesis Test Descriptive Inferential Differences Associative Predictive What are the characteristics of the respondents? What are the characteristics

### Single sample hypothesis testing, II 9.07 3/02/2004

Single sample hypothesis testing, II 9.07 3/02/2004 Outline Very brief review One-tailed vs. two-tailed tests Small sample testing Significance & multiple tests II: Data snooping What do our results mean?

### MATH 10: Elementary Statistics and Probability Chapter 9: Hypothesis Testing with One Sample

MATH 10: Elementary Statistics and Probability Chapter 9: Hypothesis Testing with One Sample Tony Pourmohamad Department of Mathematics De Anza College Spring 2015 Objectives By the end of this set of

### LAB 4 INSTRUCTIONS CONFIDENCE INTERVALS AND HYPOTHESIS TESTING

LAB 4 INSTRUCTIONS CONFIDENCE INTERVALS AND HYPOTHESIS TESTING In this lab you will explore the concept of a confidence interval and hypothesis testing through a simulation problem in engineering setting.

### Chapter 9: Hypothesis Testing Sections

Chapter 9: Hypothesis Testing Sections Skip: 9.2 Testing Simple Hypotheses Skip: 9.3 Uniformly Most Powerful Tests Skip: 9.4 Two-Sided Alternatives 9.5 The t Test 9.6 Comparing the Means of Two Normal

### Introduction to Hypothesis Testing

I. Terms, Concepts. Introduction to Hypothesis Testing A. In general, we do not know the true value of population parameters - they must be estimated. However, we do have hypotheses about what the true

### Principles of Hypothesis Testing for Public Health

Principles of Hypothesis Testing for Public Health Laura Lee Johnson, Ph.D. Statistician National Center for Complementary and Alternative Medicine johnslau@mail.nih.gov Fall 2011 Answers to Questions

### Lesson 1: Comparison of Population Means Part c: Comparison of Two- Means

Lesson : Comparison of Population Means Part c: Comparison of Two- Means Welcome to lesson c. This third lesson of lesson will discuss hypothesis testing for two independent means. Steps in Hypothesis

### Introduction to Hypothesis Testing. Point estimation and confidence intervals are useful statistical inference procedures.

Introduction to Hypothesis Testing Point estimation and confidence intervals are useful statistical inference procedures. Another type of inference is used frequently used concerns tests of hypotheses.

### Example Hypotheses. Chapter 8-2: Basics of Hypothesis Testing. A newspaper headline makes the claim: Most workers get their jobs through networking

Chapter 8-2: Basics of Hypothesis Testing Two main activities in statistical inference are using sample data to: 1. estimate a population parameter forming confidence intervals 2. test a hypothesis or

### Sampling and Hypothesis Testing

Population and sample Sampling and Hypothesis Testing Allin Cottrell Population : an entire set of objects or units of observation of one sort or another. Sample : subset of a population. Parameter versus

### Hypothesis Testing (unknown σ)

Hypothesis Testing (unknown σ) Business Statistics Recall: Plan for Today Null and Alternative Hypotheses Types of errors: type I, type II Types of correct decisions: type A, type B Level of Significance

### Name: Date: Use the following to answer questions 3-4:

Name: Date: 1. Determine whether each of the following statements is true or false. A) The margin of error for a 95% confidence interval for the mean increases as the sample size increases. B) The margin

### Chapter 9: Hypothesis Testing Sections

Chapter 9: Hypothesis Testing Sections 9.1 Problems of Testing Hypotheses Skip: 9.2 Testing Simple Hypotheses Skip: 9.3 Uniformly Most Powerful Tests Skip: 9.4 Two-Sided Alternatives 9.6 Comparing the

### An Introduction to Statistics Course (ECOE 1302) Spring Semester 2011 Chapter 10- TWO-SAMPLE TESTS

The Islamic University of Gaza Faculty of Commerce Department of Economics and Political Sciences An Introduction to Statistics Course (ECOE 130) Spring Semester 011 Chapter 10- TWO-SAMPLE TESTS Practice

### Analysis of numerical data S4

Basic medical statistics for clinical and experimental research Analysis of numerical data S4 Katarzyna Jóźwiak k.jozwiak@nki.nl 3rd November 2015 1/42 Hypothesis tests: numerical and ordinal data 1 group:

### Simple Linear Regression Inference

Simple Linear Regression Inference 1 Inference requirements The Normality assumption of the stochastic term e is needed for inference even if it is not a OLS requirement. Therefore we have: Interpretation

### Testing a claim about a population mean

Introductory Statistics Lectures Testing a claim about a population mean One sample hypothesis test of the mean Department of Mathematics Pima Community College Redistribution of this material is prohibited

### Chapter 5: Basic Statistics and Hypothesis Testing

Chapter 5: Basic Statistics and Hypothesis Testing In this chapter: 1. Viewing the t-value from an OLS regression (UE 5.2.1) 2. Calculating critical t-values and applying the decision rule (UE 5.2.2) 3.

### Null Hypothesis H 0. The null hypothesis (denoted by H 0

Hypothesis test In statistics, a hypothesis is a claim or statement about a property of a population. A hypothesis test (or test of significance) is a standard procedure for testing a claim about a property

### HypoTesting. Name: Class: Date: Multiple Choice Identify the choice that best completes the statement or answers the question.

Name: Class: Date: HypoTesting Multiple Choice Identify the choice that best completes the statement or answers the question. 1. A Type II error is committed if we make: a. a correct decision when the

### Sample Size and Power in Clinical Trials

Sample Size and Power in Clinical Trials Version 1.0 May 011 1. Power of a Test. Factors affecting Power 3. Required Sample Size RELATED ISSUES 1. Effect Size. Test Statistics 3. Variation 4. Significance

### Class 19: Two Way Tables, Conditional Distributions, Chi-Square (Text: Sections 2.5; 9.1)

Spring 204 Class 9: Two Way Tables, Conditional Distributions, Chi-Square (Text: Sections 2.5; 9.) Big Picture: More than Two Samples In Chapter 7: We looked at quantitative variables and compared the

### Nonparametric Methods for Two Samples. Nonparametric Methods for Two Samples

Nonparametric Methods for Two Samples An overview In the independent two-sample t-test, we assume normality, independence, and equal variances. This t-test is robust against nonnormality, but is sensitive

### Chapter 7 Part 2. Hypothesis testing Power

Chapter 7 Part 2 Hypothesis testing Power November 6, 2008 All of the normal curves in this handout are sampling distributions Goal: To understand the process of hypothesis testing and the relationship

### Chapter 9, Part A Hypothesis Tests. Learning objectives

Chapter 9, Part A Hypothesis Tests Slide 1 Learning objectives 1. Understand how to develop Null and Alternative Hypotheses 2. Understand Type I and Type II Errors 3. Able to do hypothesis test about population

### Chapter 11-12 1 Review

Chapter 11-12 Review Name 1. In formulating hypotheses for a statistical test of significance, the null hypothesis is often a statement of no effect or no difference. the probability of observing the data

### Develop hypothesis and then research to find out if it is true. Derived from theory or primary question/research questions

Chapter 12 Hypothesis Testing Learning Objectives Examine the process of hypothesis testing Evaluate research and null hypothesis Determine one- or two-tailed tests Understand obtained values, significance,

### 7 Hypothesis testing - one sample tests

7 Hypothesis testing - one sample tests 7.1 Introduction Definition 7.1 A hypothesis is a statement about a population parameter. Example A hypothesis might be that the mean age of students taking MAS113X

### Mind on Statistics. Chapter 12

Mind on Statistics Chapter 12 Sections 12.1 Questions 1 to 6: For each statement, determine if the statement is a typical null hypothesis (H 0 ) or alternative hypothesis (H a ). 1. There is no difference

### Probability of rejecting the null hypothesis when

Sample Size The first question faced by a statistical consultant, and frequently the last, is, How many subjects (animals, units) do I need? This usually results in exploring the size of the treatment

### Study Guide for the Final Exam

Study Guide for the Final Exam When studying, remember that the computational portion of the exam will only involve new material (covered after the second midterm), that material from Exam 1 will make

### Null Hypothesis Significance Testing Signifcance Level, Power, t-tests. 18.05 Spring 2014 Jeremy Orloff and Jonathan Bloom

Null Hypothesis Significance Testing Signifcance Level, Power, t-tests 18.05 Spring 2014 Jeremy Orloff and Jonathan Bloom Simple and composite hypotheses Simple hypothesis: the sampling distribution is

### Recall this chart that showed how most of our course would be organized:

Chapter 4 One-Way ANOVA Recall this chart that showed how most of our course would be organized: Explanatory Variable(s) Response Variable Methods Categorical Categorical Contingency Tables Categorical

### HYPOTHESIS TESTING WITH SPSS:

HYPOTHESIS TESTING WITH SPSS: A NON-STATISTICIAN S GUIDE & TUTORIAL by Dr. Jim Mirabella SPSS 14.0 screenshots reprinted with permission from SPSS Inc. Published June 2006 Copyright Dr. Jim Mirabella CHAPTER

### Outline of Topics. Statistical Methods I. Types of Data. Descriptive Statistics

Statistical Methods I Tamekia L. Jones, Ph.D. (tjones@cog.ufl.edu) Research Assistant Professor Children s Oncology Group Statistics & Data Center Department of Biostatistics Colleges of Medicine and Public

### AP Statistics 2002 Scoring Guidelines

AP Statistics 2002 Scoring Guidelines The materials included in these files are intended for use by AP teachers for course and exam preparation in the classroom; permission for any other use must be sought

### Design and Analysis of Equivalence Clinical Trials Via the SAS System

Design and Analysis of Equivalence Clinical Trials Via the SAS System Pamela J. Atherton Skaff, Jeff A. Sloan, Mayo Clinic, Rochester, MN 55905 ABSTRACT An equivalence clinical trial typically is conducted

### Module 5 Hypotheses Tests: Comparing Two Groups

Module 5 Hypotheses Tests: Comparing Two Groups Objective: In medical research, we often compare the outcomes between two groups of patients, namely exposed and unexposed groups. At the completion of this

### 6. Statistical Inference: Significance Tests

6. Statistical Inference: Significance Tests Goal: Use statistical methods to check hypotheses such as Women's participation rates in elections in France is higher than in Germany. (an effect) Ethnic divisions

### UCLA STAT 13 Statistical Methods - Final Exam Review Solutions Chapter 7 Sampling Distributions of Estimates

UCLA STAT 13 Statistical Methods - Final Exam Review Solutions Chapter 7 Sampling Distributions of Estimates 1. (a) (i) µ µ (ii) σ σ n is exactly Normally distributed. (c) (i) is approximately Normally

### Multiple Hypothesis Testing: The F-test

Multiple Hypothesis Testing: The F-test Matt Blackwell December 3, 2008 1 A bit of review When moving into the matrix version of linear regression, it is easy to lose sight of the big picture and get lost

### Chapter 7 Notes - Inference for Single Samples. You know already for a large sample, you can invoke the CLT so:

Chapter 7 Notes - Inference for Single Samples You know already for a large sample, you can invoke the CLT so: X N(µ, ). Also for a large sample, you can replace an unknown σ by s. You know how to do a

### HYPOTHESIS TESTING (ONE SAMPLE) - CHAPTER 7 1. used confidence intervals to answer questions such as...

HYPOTHESIS TESTING (ONE SAMPLE) - CHAPTER 7 1 PREVIOUSLY used confidence intervals to answer questions such as... You know that 0.25% of women have red/green color blindness. You conduct a study of men

### HYPOTHESIS TESTING (ONE SAMPLE) - CHAPTER 7 1. used confidence intervals to answer questions such as...

HYPOTHESIS TESTING (ONE SAMPLE) - CHAPTER 7 1 PREVIOUSLY used confidence intervals to answer questions such as... You know that 0.25% of women have red/green color blindness. You conduct a study of men

### AP STATISTICS 2009 SCORING GUIDELINES (Form B)

AP STATISTICS 2009 SCORING GUIDELINES (Form B) Question 5 Intent of Question The primary goals of this question were to assess students ability to (1) state the appropriate hypotheses, (2) identify and

### Chapter III. Testing Hypotheses

Chapter III Testing Hypotheses R (Introduction) A statistical hypothesis is an assumption about a population parameter This assumption may or may not be true The best way to determine whether a statistical

### HYPOTHESIS TESTING AND TYPE I AND TYPE II ERROR

HYPOTHESIS TESTING AND TYPE I AND TYPE II ERROR Hypothesis is a conjecture (an inferring) about one or more population parameters. Null Hypothesis (H 0 ) is a statement of no difference or no relationship

### Introduction to Hypothesis Testing OPRE 6301

Introduction to Hypothesis Testing OPRE 6301 Motivation... The purpose of hypothesis testing is to determine whether there is enough statistical evidence in favor of a certain belief, or hypothesis, about

### Introduction. Hypothesis Testing. Hypothesis Testing. Significance Testing

Introduction Hypothesis Testing Mark Lunt Arthritis Research UK Centre for Ecellence in Epidemiology University of Manchester 13/10/2015 We saw last week that we can never know the population parameters

### CHAPTER 11 SECTION 2: INTRODUCTION TO HYPOTHESIS TESTING

CHAPTER 11 SECTION 2: INTRODUCTION TO HYPOTHESIS TESTING MULTIPLE CHOICE 56. In testing the hypotheses H 0 : µ = 50 vs. H 1 : µ 50, the following information is known: n = 64, = 53.5, and σ = 10. The standardized

### Homework 6 Solutions

Math 17, Section 2 Spring 2011 Assignment Chapter 20: 12, 14, 20, 24, 34 Chapter 21: 2, 8, 14, 16, 18 Chapter 20 20.12] Got Milk? The student made a number of mistakes here: Homework 6 Solutions 1. Null

### Chapter 4: Statistical Hypothesis Testing

Chapter 4: Statistical Hypothesis Testing Christophe Hurlin November 20, 2015 Christophe Hurlin () Advanced Econometrics - Master ESA November 20, 2015 1 / 225 Section 1 Introduction Christophe Hurlin

### Introduction to. Hypothesis Testing CHAPTER LEARNING OBJECTIVES. 1 Identify the four steps of hypothesis testing.

Introduction to Hypothesis Testing CHAPTER 8 LEARNING OBJECTIVES After reading this chapter, you should be able to: 1 Identify the four steps of hypothesis testing. 2 Define null hypothesis, alternative

### Selected Nonparametric and Parametric Statistical Tests for Two-Sample Cases 1

Selected Nonparametric and Parametric Statistical Tests for Two-Sample Cases The T-statistic is used to test differences in the means of two groups. The grouping variable is categorical and data for the

### Hypothesis Testing COMP 245 STATISTICS. Dr N A Heard. 1 Hypothesis Testing 2 1.1 Introduction... 2 1.2 Error Rates and Power of a Test...

Hypothesis Testing COMP 45 STATISTICS Dr N A Heard Contents 1 Hypothesis Testing 1.1 Introduction........................................ 1. Error Rates and Power of a Test.............................

### Nonparametric Test Procedures

Nonparametric Test Procedures 1 Introduction to Nonparametrics Nonparametric tests do not require that samples come from populations with normal distributions or any other specific distribution. Hence

### SAMPLING & INFERENTIAL STATISTICS. Sampling is necessary to make inferences about a population.

SAMPLING & INFERENTIAL STATISTICS Sampling is necessary to make inferences about a population. SAMPLING The group that you observe or collect data from is the sample. The group that you make generalizations

### Statistics - Written Examination MEC Students - BOVISA

Statistics - Written Examination MEC Students - BOVISA Prof.ssa A. Guglielmi 26.0.2 All rights reserved. Legal action will be taken against infringement. Reproduction is prohibited without prior consent.

### Power and Sample Size Determination

Power and Sample Size Determination Bret Hanlon and Bret Larget Department of Statistics University of Wisconsin Madison November 3 8, 2011 Power 1 / 31 Experimental Design To this point in the semester,

### Statistical Significance and Bivariate Tests

Statistical Significance and Bivariate Tests BUS 735: Business Decision Making and Research 1 1.1 Goals Goals Specific goals: Re-familiarize ourselves with basic statistics ideas: sampling distributions,

### Statistical Inference

Statistical Inference Idea: Estimate parameters of the population distribution using data. How: Use the sampling distribution of sample statistics and methods based on what would happen if we used this

### Hypothesis Testing Summary

Hypothesis Testing Summary Hypothesis testing begins with the drawing of a sample and calculating its characteristics (aka, statistics ). A statistical test (a specific form of a hypothesis test) is an

### Hypothesis Testing and Confidence Interval Estimation

Biostatistics for Health Care Researchers: A Short Course Hypothesis Testing and Confidence Interval Estimation Presented ed by: Susan M. Perkins, Ph.D. Division of Biostatistics Indiana University School

### 1 SAMPLE SIGN TEST. Non-Parametric Univariate Tests: 1 Sample Sign Test 1. A non-parametric equivalent of the 1 SAMPLE T-TEST.

Non-Parametric Univariate Tests: 1 Sample Sign Test 1 1 SAMPLE SIGN TEST A non-parametric equivalent of the 1 SAMPLE T-TEST. ASSUMPTIONS: Data is non-normally distributed, even after log transforming.

### Lecture 13 More on hypothesis testing

Lecture 13 More on hypothesis testing Thais Paiva STA 111 - Summer 2013 Term II July 22, 2013 1 / 27 Thais Paiva STA 111 - Summer 2013 Term II Lecture 13, 07/22/2013 Lecture Plan 1 Type I and type II error

### Chapter 9: Hypothesis Testing Sections

Chapter 9: Hypothesis Testing Sections - we are still here Skip: 9.2 Testing Simple Hypotheses Skip: 9.3 Uniformly Most Powerful Tests Skip: 9.4 Two-Sided Alternatives 9.5 The t Test 9.6 Comparing the