# Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn

Save this PDF as:

Size: px
Start display at page:

Download "Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn"

## Transcription

1 Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn Gordon K. Smyth & Belinda Phipson Walter and Eliza Hall Institute of Medical Research Melbourne, Vic 352, Australia Published October 21; with corrections 9 February 211 Abstract Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which p-values are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation p-values published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign p-values. Although the p-value understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact p-values when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice. Keywords: permutation test, Monte Carlo test, p-values, multiple testing, microarry Statistical Applications in Genetics and Molecular Biology 9 (21), No. 1, Article 39. 1

3 finds nearly 8 genomic articles, likely an underestimate of the number of relevant articles. Permutations are almost always randomly drawn rather than exhaustively enumerated. We were motivated to write this article because of the number of times we have seen permutation p-values computed incorrectly in the genomic literature. Many published algorithms and software tools return permutation p-values that are really estimates of p-values, i.e., ˆp-values. In most cases, published p-values obtained from permutation tests are understated by about 1/m, where m is the number of permutations. The situation is similar when Monte Carlo simulation is used to assign p-values. Of particular concern is the tendency of many public software packages to return ˆp-values that can be exactly zero. It is obvious that a permutation test could never yield an exactly zero p-value if all permutations were exhaustively enumerated, and it makes no inferential sense to assert that the p-value can be reduced to zero by considering only a subset of the permutations. Most concerning of all is the potential for multiple testing procedures to be applied to ˆp-values that are exactly zero. This has the potential to lead to grossly misleading results at the family-wise level, in which the results are judged to be significant at fantastically strong levels, even when all of the null hypotheses are true. This article contains only a modest amount of novel statistical theory. We provide a new computational solution for evaluating exact p-values when permutations are drawn at random. Our main aim is to clarify the need to evaluate exact p-values, and to make the relevant results more accessible to the genomic community. 2 Estimating p-values tends to increase the type I error rate Randomization tests are used in the following context. A test statistic t has been calculated from the data at hand. We would like to use the null hypothesis tail probability p = P(T > t) of the observed test statistic as the p-value, but this probability cannot be directly computed. In a general sense, the purpose of randomization tests is to estimate p. The purpose of this section is to point out that p-value calculation must take into account not just the estimate of p but also the margin of error of this estimate. Simply replacing p with an unbiased estimator can result in a failure to correctly control the type I error rate. If the test statistic t is continuously distributed, then p will be uniformly distributed on (,1) under the null hypothesis. To control the type I error rate at level α, if p was known, the null hypothesis would be rejected when p α. In other words, P(p α) = α under the null hypothesis. Now suppose that ˆp is an unbiased estimator of p. It is often true that P( ˆp α) > α, because ˆp < α occurs with positive probability even when p > α. In general P( ˆp α) = 1 P( ˆp α p )d p. To consider a concrete case, suppose that m Monte Carlo data sets are generated under the null hypothesis, and suppose that ˆp = B/m where B is the number of Monte Carlo test statistics greater than or equal to t. Then m ˆp is binomially distributed with size m and success probability p. Plainly, ˆp is an unbiased estimator of p. Furthermore, ˆp converges to p as 3

4 the number of Monte Carlo data sets m. This is all conditional on the observed data however. To compute the size of the resulting Monte Carlo hypothesis test, we have to average over all possible data sets that might have been observed. Since m ˆp p is binomial and p is uniform, the unconditional distribution of m ˆp is a special case of beta-binomial. In fact, it is easy to show that ˆp follows a discrete uniform distribution on the values,1/m,2/m,...,1: P( ˆp = b/m) = = = 1 1 P( ˆp = b/m p )d p ( ) m p b (1 p ) m b d p b 1 m + 1 for all b =,1,...,m. Hence the type I error rate of the Monte Carlo test that arises from using ˆp as a working p-value is P( ˆp α) = mα + 1 m + 1. This expression can be greater or less than the nominal size α. The discreteness of ˆp however ensures that P( ˆp α) > α for most values of α near zero, and P( ˆp α) < α for most values of α near 1 (Figure A). Since α.5 for most experiments, the true type I error rate P( ˆp α) exceeds α for most practical values of m and α. The type I error rate always exceeds α when α = i/m for some integer i, or when α < 1/(m + 1). In fact P( ˆp α) is never less than 1/(m + 1) regardless of how stringent the desired rate α is chosen. This demonstrates an unintuitive principle. Inserting an unbiased estimator of a p-value for the true p-value does not generally lead to a test of the same size. Rather, the type I error rate of the resulting randomized test is usually greater than that which would arise from using the true p-value. To achieve a randomized test of the correct size, we must in effect substitute for the true p-value an estimator with positive bias. The moral is that p-values must be directly evaluated as probabilities, not treated as if they were unknown parameters to be estimated in a statistical model. 3 Under-estimating p-values is dangerous in a multiple testing context If only one hypothesis is to be tested, mis-calculating a p-value probability by a small amount will often be of little consequence. In genomic research, however, it is typically the case that many tests are to be conducted. When the number of tests is large, any systematic underestimation of p-values can lead to dangerously wrong conclusions at the family-wise level. Suppose for example that 5, tests are to be conducted, one for each transcript in a transcriptome perhaps, and suppose that the resulting p-values are to be adjusted by Bonferroni or Holm s methods (Holm, 1979). If each p-value is under-estimated by just 1/5,, then it is highly likely that at least one null hypothesis will be rejected at any reasonable significance 4

5 Actual type I error rate Nominal type I error rate Figure A: Type I error rate, P( ˆp α), arising from using ˆp as a working p-value (y-axis) versus the nominal size α (x-axis) when m = 2. The equality line, P( ˆp α) = α, is shown in blue. The type I error rate is generally above the nominal rate for α <.5. level, even if none of the null hypotheses are false. In this case the family-wise type I error rate is radically wrong, even though the type I error rate is only very slightly under-estimated for each individual test. This situation can arise easily if a permutation test is conducted for each gene on a microarray platform. To give an example, suppose that permutation ˆp-values are computed from m = 1 sample permutations for each of 3, genes, where the total number of distinct permutations possible is much larger than 1. Suppose that a working p-value is computed for each gene as ˆp = B/m, where B is the number of permutations yielding a test statistic at least as extreme as that observed from the original data. Clearly ˆp = B/m is an unbiased estimate of the tail probability for the test statistic under the null distribution induced by permutating samples. Yet then the number of exact zero values will be around 3, even when there are no differentially expressed genes. These 3 zero values will remain statistically significant regardless of multiple testing procedure or significance level applied. The basic problem here is that the family-wise error rate depends on the relative accuracy on the p-values, rather than on additive errors. For the smallest p-values, the error may be large in relative terms even when it is small in absolute terms. When a non-zero p-value is mis-estimated as zero, the relative error is 1%. 5

6 4 Exact p-values for Monte Carlo tests It is helpful to consider first the case of Monte Carlo tests, which have much in common with permutations tests, but are mathematically simpler. Suppose t obs is the observed test statistic for a given hypothesis for a given experiment. The basic assumption of Monte Carlo tests is that it is possible to generate independent random datasets under the null hypothesis H. Typically, this is done by simulation using pseudo-random numbers. Suppose m such datasets are simulated under the null hypothesis, each yielding a distinct test statistic t sim. In principle, the ideal p-value is p = P(t sim t obs ) but this is unknown. To compute p exactly it would be necessary to generate an infinite number of simulated datasets, but only a finite number m are available. For simplicity, we assume that the test statistic t is continuous, so that p is uniform on the unit interval and all the t sim are distinct. Let B be the number of times out of m that t sim t obs. The previous section showed that using the unbiased estimator ˆp = B/m in place of p leads to an invalid test that does not correctly control the type I error rate at the required level. A more valid approach is to compute the tail probability directly for the Monte Carlo results. From the point of view of the randomization test, the test statistic is B rather than t obs, and the required tail probability is P(B b). The previous section showed that, under the null hypothesis, the marginal distribution of B over all possible data sets is discrete uniform on the integers from,...,m. Hence the exact Monte Carlo p-value is p u = P(B b) = b + 1 m + 1. In effect, the randomization test replaces the unknown tail probability p with a biased estimator (b + 1)/(m + 1), the amount of positive bias being just enough to allow for the uncertainty of estimation and to produce a test with the correct size. This p-value calculation is presented in a number of textbooks on randomization tests including Davison and Hinkley (1997), Manly (1997) and Edgington and Onghena (27). 5 Sampling permutations without replacement Now we consider permutation tests, a type of re-sampling randomization test. The basic setup for a permutation test is as follows. We observe data pairs (y i,g i ), i = 1,...,y n, where the y i are independent data observations and the g i are labels that classify the observations into two or more groups. The aim of a permutation test is to test the association between y and g. A suitable test statistic t is computed from the (y i,g i ). This might be a two-sample t- statistic, for example, but it could in principle be any statistic designed to test for differences between the data groups. Write t obs for the observed value of the statistic. The idea of a permutation test is to assign a p-value to t obs by randomly permuting the class labels g i a large number of times. Then the empirical distribution of the test statistics t perm from the permuted data estimates the null distribution of the test statistic. The null hypothesis H 6

7 being tested is that the y i are independently and identically distributed, versus an alternative hypothesis in which the distribution of y i differs between groups. For simplicity, we will assume that the y i are all distinct, so that permutation yields distinct values of the test statistic unless the partition of y values into groups is identical. We assume that there are m t possible distinct values of the test statistic, not counting the original value, and that all are equally likely to arise under the permutation distribution. The original idea of permutation tests was that all possible permutations would be enumerated (Fisher, 1935). However m t can be very large indeed for moderate to large sample sizes, so that complete enumeration may be impractical. In this situation, it is common practice to examine a random subset of the possible permutations. It is important to distinguish two possible scenarios. The random permutations may be drawn without replacement, so that all permutations are distinct. Or permutations may be drawn with replacement, so that the same permutation may be used more than once. We consider the first scenario now. The second scenario is the topic of the next section. Suppose that a random sample m m t permutations is drawn, without repetition, such that each permutation yields a distinct test statistic t perm, different from the original statistic. Then the situation is essentially the same as for Monte Carlo tests considered in the previous section. The argument used in the previous section to derive the Monte Carlo p-value applies again, showing that the exact permutation p-value is p u = (b + 1)/(m + 1), where b is the number of t perm greater than t obs. This result includes the situation in which all m t possible distinct permutations are evaluated. The exhaustive permutation p-value is p t = (b t + 1)/(m t + 1), where b t is the total number of distinct statistic values exceeding t obs. 6 Sampling permutations with replacement 6.1 Exact p-values We now consider the case in which permutations are randomly drawn with replacement. Each permutation is randomly generated independently of previous permutations or of the original data configuration. This is by far the most common way that permutation tests are used in practice, especially in the genomic literature. This scenario requires careful treatment from a mathematical point of view, because of the possibility that the random permutations may include repetitions of the same permutations and indeed of the original data. Suppose that an independent random sample of m permutations is drawn with replacement. The resulting test statistics t perm may include repeat values, including possibly the original observed value t obs. The exact p-value is now slightly less than (b + 1)/(m + 1), because of the possibility that the original data is included at least once as one of the random permutations. Write B for the number of permutations out of m yielding test statistics at least as extreme as t obs. We wish to evaluate the exact permutation p-value p e = P(B b) for any given b. We do so using a formal conditional approach. Write B t for the total number of possible distinct statistic values exceeding t obs, again an 7

8 unknown quantity, and note p t = (B t + 1)/(m t + 1). Although p t is in principle the ideal permutation p-value, it is unknown in the scenario we now consider. If the null hypothesis is true, then B t follows a discrete uniform distribution on the integers,...,m t. Conditional on B t = b t, B follows a binomial distribution with size m and probability p t. Hence p e = m t P(B b B t = b t )P(B t = b t H ) = 1 m t b t = m t + 1 F(b;m, p t ) (1) b t = where F is the cumulative probability function of the binomial distribution. This expression is essentially the same as Proposition 1 of Dwass (1957), although Dwass did not seem to intend it for practical computational use. Although not very difficult to derive, this exact p-value has not been repeated to our knowledge elsewhere in the literature. 6.2 Computation and approximation Equation (1) provides a formula for the exact p-value, but the summation may not be practical to evaluate explicitly if m t is very large. Hence we explore an approximation that can be evaluated quickly. Replacing the summation with an integral, and using a continuity correction, yields p e = 1.5/(m t +1) 1 F(b;m, p t )d p t F(b;m, p t )d p t.5/(mt +1) F(b;m, p t )d p t = b + 1.5/(mt+1) m + 1 F(b;m, p t )d p t. (2) This shows that p e < p u = (b + 1)/(m + 1), and also that p e converges to p u as m t increases. The last integral in equation (2) is easily evaluated using numerical integration algorithms. We have found it convenient to use Gaussian quadrature. The valid but conservative p-value p u has been recommended by a number of authors including Davison and Hinkley (1997), Manly (1997) and Ernst (24). The extent to which p u over-estimates the exact p-value depends on m t. When m t = 1, p u over-estimates the.5 p-value by about 2%, but smaller p-values are over-estimated by greater relative amounts (Figure 1b). When m t is small, p u can be very conservative (Figure 1a). 6.3 Specific scenarios The number m t of possible distinct values of the test statistic is not simply a function of the number of permutations, but also depends on the form of the test statistic and on the sample sizes. This is because different permutations can lead to the same test statistic, especially for two-sided tests with balanced samples. Consider a comparison between two groups of sizes n 1 and n 2. In this situation it is usual to sample permutations by randomly assigning n 1 of the indices i to group 1 and the 8

9 (a) m t = 126 (b) m t = 1 estimate/true p value p =.5 estimate/true p value p = log 1 (true p value) log 1 (true p value) Figure 1: Ratio of the upper bound p u = (b+1)/(m+1) to the exact p-value. Panel (a) is for m t = 126 and (b) is for m t = 1. The number of permutations is m = 1 in each case, and values are plotted for all possible values of b. remaining n 2 to group 2. If the test statistic is one-sided, or if n 1 n 2, then each permutation will generally yield a distinct test statistic. In that case, the number of distinct values of the permuted statistics is ( ) n1 + n 2 m t + 1 =. For example, a one-sided test with n 1 = n 2 = 5 gives m t + 1 = ( 1) 5 = 252. The integral approximation (2) provides an excellent approximation to the exact p-values (Table 1). If a two-sided test statistic is used, for example the absolute value of a t-statistic, and if the two groups are of equal size n 1 = n 2, then each permutation gives the same value of the test statistic as the corresponding permutation with the two groups reversed. In this case, the total number of distinct test statistics possible is m t + 1 = ( n 1 +n 2 ) n 1 /2. If there are three or more groups of unequal sizes then m t +1 is the multinomial coefficient ( k ) m t + 1 = j=1 n j n 1... n k where k is the number of groups. If some of the group sizes n j are equal, then some permutations will in general give equal test statistics, so the calculaton of m t will need to consider the particular configuration of samples sizes. n 1 7 Discussion and recommendations The approach posited in Section 5, of drawing permutations without repeating previous values of the test statistic, turns out to be a surprisingly difficult combinatorial programming 9

10 Table 1: Exact and approximate p-values for a one-sided test between two groups of size five using m = 1 random permutations. Number of Estimated Exact Integral approximated Upper bound t sim t obs ˆp-value p-value p-value p u exercise. Hence this approach is seldom used so much so that we have never seen it in the genomic literature. Nevertheless, this is the superior approach to permutation tests. Sampling without replacment gives rise to a simple easily understood exact p-value in the form of p u. Even more importantly, it is possible to show that sampling permutations without replacement gives more statistical power than sampling with replacement for any given number of permutations m m t. Our first recommendation then is that more attention be given to this problem by computer scientists and combinatorialists, to provide biostatisticians with efficient numerical algorithms to sample permutations without replacement. Our second recommendation is that unbiased estimators of p-values such as ˆp should not be used, because they can t be guaranteed to control the type I error rate. Although the risk involved in using ˆp in place of the exact p-value may be modest in many applications, there is no justification for using an invalid p-value when a valid p-value, in the form of p u, is readily available for no extra cost. Our third recommendation is that users might consider using the exact p-value p e, at the cost of a trivial amount of extra computation. Our algorithm to compute p e is implemented as the function permp in the software package statmod for R available from r-project.org. The exact value p e gives a worthwhile gain in statistical power over p u when the number of permutations m performed is a more than negligible proportion of the total number possible m t. A referee asked us to warn users that the number of permutations m should not be chosen too small. While this is true, the urgent need to avoid small m disappears when the exact p-value p e is used in place of ˆp, because p e ensures a valid statistical test regardless of sample size or the number of permutations. When exact p-values are used, the only penalty for choosing m small is a loss of statistical power to reject the null hypothesis. This is as it should be: more permutations should generally provide greater power. The same referee asked us to comment on the difference between permuting genes and permuting samples. One obvious difference between genes and samples is that there are generally more of the former than the latter, so that m t is likely to be astronomically large 1

11 when genes are permuted. This implies that p u is likely to be a good approximation for p e in this context. We have avoided discussing gene permutation in this article however because gene permutation has another obvious problem. Permutation assumes that the items being permuted are statistically independent, whereas genes usually cannot be assumed independent in genomic applications (Goeman and Bühlman, 27; Wu et al, 21). In general, we recommend permuting samples rather than permuting genes when there is a choice. Finally, it is worthwhile mentioning that there are alternatives to permutation that alleviate the problem of small m t when the number of samples is not large (Dørum, 29; Wu et al, 21). References Barnard, G.A. (1963) Discussion of The spectral analysis of point processes by MS Bartlett. J.R. Statist. Soc. B, 25, 294. Davison, A.C. and Hinkley, D.V. (1997) Bootstrap methods and their application, Cambridge University Press, United Kingdom. Dwass, M. (1957) Modified randomization tests for nonparametric hypotheses. Ann. Math. Statist., 28, Dørum, G., Snipen, L., Solheim, M., and Saebo, S. (29). Rotation testing in gene set enrichment analysis for small direct comparison experiments. Stat. Appl. Genet. Mol. Biol, 8, Article 34. Efron, B. (1982) The jackknife, the bootstrap and other resampling plans, Society for Industrial and Applied Mathematics, England. Efron, B. and Tibshirani, R.J.(1993) An introduction to the bootstrap, Chapman & Hall, New York. Edgington, E.S. and Onghena, P. (27) Randomization tests, Fourth Edition, Chapman & Hall/CRC, Boca Raton, FL. Ernst, M.D. (24) Permutation methods: a basis for exact inference. Statistical Science, 19, Fisher, R.A. (1935) The Design of Experiments, 3rd Edition, Oliver & Boyd, London. Goeman, J.J. and Bühlmann, P. (27) Analyzing gene expression data in terms of gene sets: methodological issues. Bioinformatics, 23, Good, P. (1994) Permutation tests: a practical guide to resampling methods for testing hypotheses, Springer-Verlag New York, New York. Higgins, J.J. (24) An introduction to modern nonparametric statistics, Brooks/Cole, Pacific Grove, CA. 11

12 Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, Manly, B.F.J. (1997) Randomization, bootstrap and Monte Carlo methods in biology, Second Edition, Chapman & Hall, London. Mielke, P.W. and Berry, K.J. (21) Permutation tests: a distance function approach, Springer- Verlag New York, New York. Onghena, P. and May, R.B. (1995) Pitfalls in computing and interpreting randomization test p values: A commentary on Chen and Dunlap. Behaviour Research Methods, Instruments, & Computers, 27, Wu, D., Lim, E., Vaillant, F., Asselin-Labat, M., Visvader, J.E. and Smyth, G.K. (21) ROAST: rotation gene set tests for complex microarray experiments. Bioinformatics, 26,

### Multiple Testing. Joseph P. Romano, Azeem M. Shaikh, and Michael Wolf. Abstract

Multiple Testing Joseph P. Romano, Azeem M. Shaikh, and Michael Wolf Abstract Multiple testing refers to any instance that involves the simultaneous testing of more than one hypothesis. If decisions about

### The Variability of P-Values. Summary

The Variability of P-Values Dennis D. Boos Department of Statistics North Carolina State University Raleigh, NC 27695-8203 boos@stat.ncsu.edu August 15, 2009 NC State Statistics Departement Tech Report

### MINITAB ASSISTANT WHITE PAPER

MINITAB ASSISTANT WHITE PAPER This paper explains the research conducted by Minitab statisticians to develop the methods and data checks used in the Assistant in Minitab 17 Statistical Software. One-Way

### Testing: is my coin fair?

Testing: is my coin fair? Formally: we want to make some inference about P(head) Try it: toss coin several times (say 7 times) Assume that it is fair ( P(head)= ), and see if this assumption is compatible

### NCSS Statistical Software

Chapter 06 Introduction This procedure provides several reports for the comparison of two distributions, including confidence intervals for the difference in means, two-sample t-tests, the z-test, the

### Recall this chart that showed how most of our course would be organized:

Chapter 4 One-Way ANOVA Recall this chart that showed how most of our course would be organized: Explanatory Variable(s) Response Variable Methods Categorical Categorical Contingency Tables Categorical

### Bootstrapping Big Data

Bootstrapping Big Data Ariel Kleiner Ameet Talwalkar Purnamrita Sarkar Michael I. Jordan Computer Science Division University of California, Berkeley {akleiner, ameet, psarkar, jordan}@eecs.berkeley.edu

### NCSS Statistical Software

Chapter 06 Introduction This procedure provides several reports for the comparison of two distributions, including confidence intervals for the difference in means, two-sample t-tests, the z-test, the

### Two-Sample T-Tests Assuming Equal Variance (Enter Means)

Chapter 4 Two-Sample T-Tests Assuming Equal Variance (Enter Means) Introduction This procedure provides sample size and power calculations for one- or two-sided two-sample t-tests when the variances of

### Two-Sample T-Tests Allowing Unequal Variance (Enter Difference)

Chapter 45 Two-Sample T-Tests Allowing Unequal Variance (Enter Difference) Introduction This procedure provides sample size and power calculations for one- or two-sided two-sample t-tests when no assumption

### t Tests in Excel The Excel Statistical Master By Mark Harmon Copyright 2011 Mark Harmon

t-tests in Excel By Mark Harmon Copyright 2011 Mark Harmon No part of this publication may be reproduced or distributed without the express permission of the author. mark@excelmasterseries.com www.excelmasterseries.com

### Test of Hypotheses. Since the Neyman-Pearson approach involves two statistical hypotheses, one has to decide which one

Test of Hypotheses Hypothesis, Test Statistic, and Rejection Region Imagine that you play a repeated Bernoulli game: you win \$1 if head and lose \$1 if tail. After 10 plays, you lost \$2 in net (4 heads

### Name: Date: Use the following to answer questions 3-4:

Name: Date: 1. Determine whether each of the following statements is true or false. A) The margin of error for a 95% confidence interval for the mean increases as the sample size increases. B) The margin

### False Discovery Rates

False Discovery Rates John D. Storey Princeton University, Princeton, USA January 2010 Multiple Hypothesis Testing In hypothesis testing, statistical significance is typically based on calculations involving

### Tutorial 5: Hypothesis Testing

Tutorial 5: Hypothesis Testing Rob Nicholls nicholls@mrc-lmb.cam.ac.uk MRC LMB Statistics Course 2014 Contents 1 Introduction................................ 1 2 Testing distributional assumptions....................

### From the help desk: Bootstrapped standard errors

The Stata Journal (2003) 3, Number 1, pp. 71 80 From the help desk: Bootstrapped standard errors Weihua Guan Stata Corporation Abstract. Bootstrapping is a nonparametric approach for evaluating the distribution

### Sample Size and Power in Clinical Trials

Sample Size and Power in Clinical Trials Version 1.0 May 011 1. Power of a Test. Factors affecting Power 3. Required Sample Size RELATED ISSUES 1. Effect Size. Test Statistics 3. Variation 4. Significance

### Checklists and Examples for Registering Statistical Analyses

Checklists and Examples for Registering Statistical Analyses For well-designed confirmatory research, all analysis decisions that could affect the confirmatory results should be planned and registered

### , for x = 0, 1, 2, 3,... (4.1) (1 + 1/n) n = 2.71828... b x /x! = e b, x=0

Chapter 4 The Poisson Distribution 4.1 The Fish Distribution? The Poisson distribution is named after Simeon-Denis Poisson (1781 1840). In addition, poisson is French for fish. In this chapter we will

### tests whether there is an association between the outcome variable and a predictor variable. In the Assistant, you can perform a Chi-Square Test for

This paper explains the research conducted by Minitab statisticians to develop the methods and data checks used in the Assistant in Minitab 17 Statistical Software. In practice, quality professionals sometimes

### Inferential Statistics

Inferential Statistics Sampling and the normal distribution Z-scores Confidence levels and intervals Hypothesis testing Commonly used statistical methods Inferential Statistics Descriptive statistics are

### CONTENTS OF DAY 2. II. Why Random Sampling is Important 9 A myth, an urban legend, and the real reason NOTES FOR SUMMER STATISTICS INSTITUTE COURSE

1 2 CONTENTS OF DAY 2 I. More Precise Definition of Simple Random Sample 3 Connection with independent random variables 3 Problems with small populations 8 II. Why Random Sampling is Important 9 A myth,

### The Bonferonni and Šidák Corrections for Multiple Comparisons

The Bonferonni and Šidák Corrections for Multiple Comparisons Hervé Abdi 1 1 Overview The more tests we perform on a set of data, the more likely we are to reject the null hypothesis when it is true (i.e.,

### The Assumption(s) of Normality

The Assumption(s) of Normality Copyright 2000, 2011, J. Toby Mordkoff This is very complicated, so I ll provide two versions. At a minimum, you should know the short one. It would be great if you knew

### Life Table Analysis using Weighted Survey Data

Life Table Analysis using Weighted Survey Data James G. Booth and Thomas A. Hirschl June 2005 Abstract Formulas for constructing valid pointwise confidence bands for survival distributions, estimated using

### Hypothesis Testing Level I Quantitative Methods. IFT Notes for the CFA exam

Hypothesis Testing 2014 Level I Quantitative Methods IFT Notes for the CFA exam Contents 1. Introduction... 3 2. Hypothesis Testing... 3 3. Hypothesis Tests Concerning the Mean... 10 4. Hypothesis Tests

### 4. Introduction to Statistics

Statistics for Engineers 4-1 4. Introduction to Statistics Descriptive Statistics Types of data A variate or random variable is a quantity or attribute whose value may vary from one unit of investigation

### Fairfield Public Schools

Mathematics Fairfield Public Schools AP Statistics AP Statistics BOE Approved 04/08/2014 1 AP STATISTICS Critical Areas of Focus AP Statistics is a rigorous course that offers advanced students an opportunity

### Quantitative Methods for Finance

Quantitative Methods for Finance Module 1: The Time Value of Money 1 Learning how to interpret interest rates as required rates of return, discount rates, or opportunity costs. 2 Learning how to explain

### Classical Hypothesis Testing and GWAS

Department of Computer Science Brown University, Providence sorin@cs.brown.edu September 15, 2014 Outline General Principles Classical statistical hypothesis testing involves the test of a null hypothesis

### Inequality, Mobility and Income Distribution Comparisons

Fiscal Studies (1997) vol. 18, no. 3, pp. 93 30 Inequality, Mobility and Income Distribution Comparisons JOHN CREEDY * Abstract his paper examines the relationship between the cross-sectional and lifetime

### AP Statistics 2010 Scoring Guidelines

AP Statistics 2010 Scoring Guidelines The College Board The College Board is a not-for-profit membership association whose mission is to connect students to college success and opportunity. Founded in

### Permutation Tests for Comparing Two Populations

Permutation Tests for Comparing Two Populations Ferry Butar Butar, Ph.D. Jae-Wan Park Abstract Permutation tests for comparing two populations could be widely used in practice because of flexibility of

### 9-3.4 Likelihood ratio test. Neyman-Pearson lemma

9-3.4 Likelihood ratio test Neyman-Pearson lemma 9-1 Hypothesis Testing 9-1.1 Statistical Hypotheses Statistical hypothesis testing and confidence interval estimation of parameters are the fundamental

### Multiple-Comparison Procedures

Multiple-Comparison Procedures References A good review of many methods for both parametric and nonparametric multiple comparisons, planned and unplanned, and with some discussion of the philosophical

### SAMPLE SIZE ESTIMATION USING KREJCIE AND MORGAN AND COHEN STATISTICAL POWER ANALYSIS: A COMPARISON. Chua Lee Chuan Jabatan Penyelidikan ABSTRACT

SAMPLE SIZE ESTIMATION USING KREJCIE AND MORGAN AND COHEN STATISTICAL POWER ANALYSIS: A COMPARISON Chua Lee Chuan Jabatan Penyelidikan ABSTRACT In most situations, researchers do not have access to an

### Tests for Two Proportions

Chapter 200 Tests for Two Proportions Introduction This module computes power and sample size for hypothesis tests of the difference, ratio, or odds ratio of two independent proportions. The test statistics

### Introduction to. Hypothesis Testing CHAPTER LEARNING OBJECTIVES. 1 Identify the four steps of hypothesis testing.

Introduction to Hypothesis Testing CHAPTER 8 LEARNING OBJECTIVES After reading this chapter, you should be able to: 1 Identify the four steps of hypothesis testing. 2 Define null hypothesis, alternative

### ESTIMATING COMPLETION RATES FROM SMALL SAMPLES USING BINOMIAL CONFIDENCE INTERVALS: COMPARISONS AND RECOMMENDATIONS

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 49th ANNUAL MEETING 200 2 ESTIMATING COMPLETION RATES FROM SMALL SAMPLES USING BINOMIAL CONFIDENCE INTERVALS: COMPARISONS AND RECOMMENDATIONS Jeff

### On Small Sample Properties of Permutation Tests: A Significance Test for Regression Models

On Small Sample Properties of Permutation Tests: A Significance Test for Regression Models Hisashi Tanizaki Graduate School of Economics Kobe University (tanizaki@kobe-u.ac.p) ABSTRACT In this paper we

### Monte Carlo testing with Big Data

Monte Carlo testing with Big Data Patrick Rubin-Delanchy University of Bristol & Heilbronn Institute for Mathematical Research Joint work with: Axel Gandy (Imperial College London) with contributions from:

### Exact Nonparametric Tests for Comparing Means - A Personal Summary

Exact Nonparametric Tests for Comparing Means - A Personal Summary Karl H. Schlag European University Institute 1 December 14, 2006 1 Economics Department, European University Institute. Via della Piazzuola

### The Bootstrap. 1 Introduction. The Bootstrap 2. Short Guides to Microeconometrics Fall Kurt Schmidheiny Unversität Basel

Short Guides to Microeconometrics Fall 2016 The Bootstrap Kurt Schmidheiny Unversität Basel The Bootstrap 2 1a) The asymptotic sampling distribution is very difficult to derive. 1b) The asymptotic sampling

### A study on the bi-aspect procedure with location and scale parameters

통계연구(2012), 제17권 제1호, 19-26 A study on the bi-aspect procedure with location and scale parameters (Short Title: Bi-aspect procedure) Hyo-Il Park 1) Ju Sung Kim 2) Abstract In this research we propose a

### International Journal of Information Technology, Modeling and Computing (IJITMC) Vol.1, No.3,August 2013

FACTORING CRYPTOSYSTEM MODULI WHEN THE CO-FACTORS DIFFERENCE IS BOUNDED Omar Akchiche 1 and Omar Khadir 2 1,2 Laboratory of Mathematics, Cryptography and Mechanics, Fstm, University of Hassan II Mohammedia-Casablanca,

### 1.2 Statistical testing by permutation

Statistical testing by permutation 17 Excerpt (pp. 17-26) Ch. 13), from: McBratney & Webster (1981), McBratney et al. (1981), Webster & Burgess (1984), Borgman & Quimby (1988), and François-Bongarçon (1991).

### Multivariate Analysis of Ecological Data

Multivariate Analysis of Ecological Data MICHAEL GREENACRE Professor of Statistics at the Pompeu Fabra University in Barcelona, Spain RAUL PRIMICERIO Associate Professor of Ecology, Evolutionary Biology

### Central Limit Theorem and Sample Size. Zachary R. Smith. and. Craig S. Wells. University of Massachusetts Amherst

CLT and Sample Size 1 Running Head: CLT AND SAMPLE SIZE Central Limit Theorem and Sample Size Zachary R. Smith and Craig S. Wells University of Massachusetts Amherst Paper presented at the annual meeting

### Hypothesis Testing. Chapter Introduction

Contents 9 Hypothesis Testing 553 9.1 Introduction............................ 553 9.2 Hypothesis Test for a Mean................... 557 9.2.1 Steps in Hypothesis Testing............... 557 9.2.2 Diagrammatic

Statistics Graduate Courses STAT 7002--Topics in Statistics-Biological/Physical/Mathematics (cr.arr.).organized study of selected topics. Subjects and earnable credit may vary from semester to semester.

### Uncertainty quantification for the family-wise error rate in multivariate copula models

Uncertainty quantification for the family-wise error rate in multivariate copula models Thorsten Dickhaus (joint work with Taras Bodnar, Jakob Gierl and Jens Stange) University of Bremen Institute for

### A direct approach to false discovery rates

J. R. Statist. Soc. B (2002) 64, Part 3, pp. 479 498 A direct approach to false discovery rates John D. Storey Stanford University, USA [Received June 2001. Revised December 2001] Summary. Multiple-hypothesis

### X X X a) perfect linear correlation b) no correlation c) positive correlation (r = 1) (r = 0) (0 < r < 1)

CORRELATION AND REGRESSION / 47 CHAPTER EIGHT CORRELATION AND REGRESSION Correlation and regression are statistical methods that are commonly used in the medical literature to compare two or more variables.

### LAB 4 INSTRUCTIONS CONFIDENCE INTERVALS AND HYPOTHESIS TESTING

LAB 4 INSTRUCTIONS CONFIDENCE INTERVALS AND HYPOTHESIS TESTING In this lab you will explore the concept of a confidence interval and hypothesis testing through a simulation problem in engineering setting.

### Non-Inferiority Tests for One Mean

Chapter 45 Non-Inferiority ests for One Mean Introduction his module computes power and sample size for non-inferiority tests in one-sample designs in which the outcome is distributed as a normal random

### Combining information from different survey samples - a case study with data collected by world wide web and telephone

Combining information from different survey samples - a case study with data collected by world wide web and telephone Magne Aldrin Norwegian Computing Center P.O. Box 114 Blindern N-0314 Oslo Norway E-mail:

### Hypothesis tests, confidence intervals, and bootstrapping

Hypothesis tests, confidence intervals, and bootstrapping Business Statistics 41000 Fall 2015 1 Topics 1. Hypothesis tests Testing a mean: H0 : µ = µ 0 Testing a proportion: H0 : p = p 0 Testing a difference

### Confidence Intervals for Cp

Chapter 296 Confidence Intervals for Cp Introduction This routine calculates the sample size needed to obtain a specified width of a Cp confidence interval at a stated confidence level. Cp is a process

### Confidence Intervals for Cpk

Chapter 297 Confidence Intervals for Cpk Introduction This routine calculates the sample size needed to obtain a specified width of a Cpk confidence interval at a stated confidence level. Cpk is a process

### Task force on quality of BCS data. Analysis of sample size in consumer surveys

Task force on quality of BCS data Analysis of sample size in consumer surveys theoretical considerations and factors determining minimum necessary sample sizes, link between country size and sample size

### AP Statistics 2007 Scoring Guidelines

AP Statistics 2007 Scoring Guidelines The College Board: Connecting Students to College Success The College Board is a not-for-profit membership association whose mission is to connect students to college

### Sociology 6Z03 Topic 15: Statistical Inference for Means

Sociology 6Z03 Topic 15: Statistical Inference for Means John Fox McMaster University Fall 2016 John Fox (McMaster University) Soc 6Z03: Statistical Inference for Means Fall 2016 1 / 41 Outline: Statistical

### Computational Statistics and Data Analysis

Computational Statistics and Data Analysis 53 (2008) 17 26 Contents lists available at ScienceDirect Computational Statistics and Data Analysis journal homepage: www.elsevier.com/locate/csda Coverage probability

### MCQ TESTING OF HYPOTHESIS

MCQ TESTING OF HYPOTHESIS MCQ 13.1 A statement about a population developed for the purpose of testing is called: (a) Hypothesis (b) Hypothesis testing (c) Level of significance (d) Test-statistic MCQ

### Hypothesis tests: the t-tests

Hypothesis tests: the t-tests Introduction Invariably investigators wish to ask whether their data answer certain questions that are germane to the purpose of the investigation. It is often the case that

### SAS Software to Fit the Generalized Linear Model

SAS Software to Fit the Generalized Linear Model Gordon Johnston, SAS Institute Inc., Cary, NC Abstract In recent years, the class of generalized linear models has gained popularity as a statistical modeling

### AP Statistics 2013 Scoring Guidelines

AP Statistics 2013 Scoring Guidelines The College Board The College Board is a mission-driven not-for-profit organization that connects students to college success and opportunity. Founded in 1900, the

### Cross Validation techniques in R: A brief overview of some methods, packages, and functions for assessing prediction models.

Cross Validation techniques in R: A brief overview of some methods, packages, and functions for assessing prediction models. Dr. Jon Starkweather, Research and Statistical Support consultant This month

: Table of Contents... 1 Overview of Model... 1 Dispersion... 2 Parameterization... 3 Sigma-Restricted Model... 3 Overparameterized Model... 4 Reference Coding... 4 Model Summary (Summary Tab)... 5 Summary

### MAT140: Applied Statistical Methods Summary of Calculating Confidence Intervals and Sample Sizes for Estimating Parameters

MAT140: Applied Statistical Methods Summary of Calculating Confidence Intervals and Sample Sizes for Estimating Parameters Inferences about a population parameter can be made using sample statistics for

### General Procedures for Testing Hypotheses

Chapter 19 General Procedures for Testing Hypothesies 297 CHAPTER 19 General Procedures for Testing Hypotheses Introduction Skeleton Procedure for Testing Hypotheses An Example: Can the Bio-Engineer Increase

### 7 Hypothesis testing - one sample tests

7 Hypothesis testing - one sample tests 7.1 Introduction Definition 7.1 A hypothesis is a statement about a population parameter. Example A hypothesis might be that the mean age of students taking MAS113X

### WISE Power Tutorial All Exercises

ame Date Class WISE Power Tutorial All Exercises Power: The B.E.A.. Mnemonic Four interrelated features of power can be summarized using BEA B Beta Error (Power = 1 Beta Error): Beta error (or Type II

### Statistical foundations of machine learning

Machine learning p. 1/45 Statistical foundations of machine learning INFO-F-422 Gianluca Bontempi Département d Informatique Boulevard de Triomphe - CP 212 http://www.ulb.ac.be/di Machine learning p. 2/45

### NCSS Statistical Software. One-Sample T-Test

Chapter 205 Introduction This procedure provides several reports for making inference about a population mean based on a single sample. These reports include confidence intervals of the mean or median,

### QUANTITATIVE METHODS BIOLOGY FINAL HONOUR SCHOOL NON-PARAMETRIC TESTS

QUANTITATIVE METHODS BIOLOGY FINAL HONOUR SCHOOL NON-PARAMETRIC TESTS This booklet contains lecture notes for the nonparametric work in the QM course. This booklet may be online at http://users.ox.ac.uk/~grafen/qmnotes/index.html.

### Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization. Learning Goals. GENOME 560, Spring 2012

Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization GENOME 560, Spring 2012 Data are interesting because they help us understand the world Genomics: Massive Amounts

### Asymmetry and the Cost of Capital

Asymmetry and the Cost of Capital Javier García Sánchez, IAE Business School Lorenzo Preve, IAE Business School Virginia Sarria Allende, IAE Business School Abstract The expected cost of capital is a crucial

### BA 275 Review Problems - Week 6 (10/30/06-11/3/06) CD Lessons: 53, 54, 55, 56 Textbook: pp. 394-398, 404-408, 410-420

BA 275 Review Problems - Week 6 (10/30/06-11/3/06) CD Lessons: 53, 54, 55, 56 Textbook: pp. 394-398, 404-408, 410-420 1. Which of the following will increase the value of the power in a statistical test

### Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay

Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding

### Raimund Alt (Autor) Multiple Hypotheses Testing in the Linear Regression Model with Applications to Economics and Finance

Raimund Alt (Autor) Multiple Hypotheses Testing in the Linear Regression Model with Applications to Economics and Finance https://cuvillier.de/de/shop/publications/2694 Copyright: Cuvillier Verlag, Inhaberin

### Sequences with MS

Sequences 2008-2014 with MS 1. [4 marks] Find the value of k if. 2a. [4 marks] The sum of the first 16 terms of an arithmetic sequence is 212 and the fifth term is 8. Find the first term and the common

### 5.1 Identifying the Target Parameter

University of California, Davis Department of Statistics Summer Session II Statistics 13 August 20, 2012 Date of latest update: August 20 Lecture 5: Estimation with Confidence intervals 5.1 Identifying

### arxiv:1112.0829v1 [math.pr] 5 Dec 2011

How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly

### 1 Limiting distribution for a Markov chain

Copyright c 2009 by Karl Sigman Limiting distribution for a Markov chain In these Lecture Notes, we shall study the limiting behavior of Markov chains as time n In particular, under suitable easy-to-check

### CHAPTER 11 CHI-SQUARE: NON-PARAMETRIC COMPARISONS OF FREQUENCY

CHAPTER 11 CHI-SQUARE: NON-PARAMETRIC COMPARISONS OF FREQUENCY The hypothesis testing statistics detailed thus far in this text have all been designed to allow comparison of the means of two or more samples

### Lecture 3 : Hypothesis testing and model-fitting

Lecture 3 : Hypothesis testing and model-fitting These dark lectures energy puzzle Lecture 1 : basic descriptive statistics Lecture 2 : searching for correlations Lecture 3 : hypothesis testing and model-fitting

### Fractionally integrated data and the autodistributed lag model: results from a simulation study

Fractionally integrated data and the autodistributed lag model: results from a simulation study Justin Esarey July 1, 215 Abstract Two contributions in this issue, Grant and Lebo (215) and Keele, Linn

### CONTINGENCY TABLES ARE NOT ALL THE SAME David C. Howell University of Vermont

CONTINGENCY TABLES ARE NOT ALL THE SAME David C. Howell University of Vermont To most people studying statistics a contingency table is a contingency table. We tend to forget, if we ever knew, that contingency

### reductio ad absurdum null hypothesis, alternate hypothesis

Chapter 10 s Using a Single Sample 10.1: Hypotheses & Test Procedures Basics: In statistics, a hypothesis is a statement about a population characteristic. s are based on an reductio ad absurdum form of

### Chapter 8 Introduction to Hypothesis Testing

Chapter 8 Student Lecture Notes 8-1 Chapter 8 Introduction to Hypothesis Testing Fall 26 Fundamentals of Business Statistics 1 Chapter Goals After completing this chapter, you should be able to: Formulate

### INTRODUCTORY STATISTICS

INTRODUCTORY STATISTICS FIFTH EDITION Thomas H. Wonnacott University of Western Ontario Ronald J. Wonnacott University of Western Ontario WILEY JOHN WILEY & SONS New York Chichester Brisbane Toronto Singapore

### Tests for One Proportion

Chapter 100 Tests for One Proportion Introduction The One-Sample Proportion Test is used to assess whether a population proportion (P1) is significantly different from a hypothesized value (P0). This is

### Comparison of frequentist and Bayesian inference. Class 20, 18.05, Spring 2014 Jeremy Orloff and Jonathan Bloom

Comparison of frequentist and Bayesian inference. Class 20, 18.05, Spring 2014 Jeremy Orloff and Jonathan Bloom 1 Learning Goals 1. Be able to explain the difference between the p-value and a posterior

### Math Review. for the Quantitative Reasoning Measure of the GRE revised General Test

Math Review for the Quantitative Reasoning Measure of the GRE revised General Test www.ets.org Overview This Math Review will familiarize you with the mathematical skills and concepts that are important

### 1. How different is the t distribution from the normal?

Statistics 101 106 Lecture 7 (20 October 98) c David Pollard Page 1 Read M&M 7.1 and 7.2, ignoring starred parts. Reread M&M 3.2. The effects of estimated variances on normal approximations. t-distributions.

### Sampling and Hypothesis Testing

Population and sample Sampling and Hypothesis Testing Allin Cottrell Population : an entire set of objects or units of observation of one sort or another. Sample : subset of a population. Parameter versus

### Power and Sample Size. In epigenetic epidemiology studies

Power and Sample Size In epigenetic epidemiology studies Overview Pros and cons Working examples Concerns for epigenetic epidemiology Definition Power is the probability of detecting an effect, given that