# Machine Learning. Topic: Evaluating Hypotheses

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Machine Learning Topic: Evaluating Hypotheses Bryan Pardo, Machine Learning: EECS 349 Fall 2011

2 How do you tell something is better? Assume we have an error measure. How do we tell if it measures something useful? To measure intelligence, which is a better? {grades, IQ, salary} If it is useful, how precise/unbiased/noisy is it? How much of a difference in the measure is required to say things two things are truly different? Maria s IQ is 103. Bob s is 101. Does that make her smarter?

3 What s a useful measure for a Classifier (Decision tree) An idea: Count how often the classifier is wrong Regressor (Linear regression) An idea: the distance between predicted values and observed values Probability Mass (or density) Estimator Pick the distribution that maximizes the likelihood of the data? Pick the distribution that looks the most reasonable? Ranker (like a search engine) : Rank order? User interface widget: User satisfaction?

4 Definitions of Error error D (h) is the true error of hypothesis h with respect to the target function f and data distribution D. It is the probability h will misclassify an instance drawn at random according to D. error s (h) is the sample error of hypothesis h with respect to the target function f and data sample set S. It is the proportion of examples in S that h misclassifies.

5 True Error vs Sample Error Misclassified: h(x)!= f(x) Correctly classified: h(x) = f(x) error D (h) = 0.5 error s (h) = 0.2 Our sample

6 Sample Error: It s all we have Generally, we never know the true error error D (h). We only get to see the sample error error S (h). How well does the sample error estimate the true error? Can we set conditions for our experiment so that we can get an estimate that is good enough for our needs?

7 Problems Estimating Error BIAS: If S is the training set, error s (h) is optimistically biased. For an unbiased estimate we need a validation set that was not used in training. error s (h) = 0 Training set

8 Problems Estimating Error Variance: Even without bias, error s (h) may still vary from error D (h) error s1 (h) = 0.2 error s2 (h) = 0.71 Sample S 1 Sample S 2

9 Experiment 1. Choose sample S of size n using distribution D 2. Measure error s (h) Question: What can we conclude about error D (h) from error s (h)? Answer: That s what we re here to learn today.

10 Coin flips Assume an unbiased coin X that takes two values {0,1}. Let K be the number we get if we flip the coin n times and add up the values of all the flips. What is the expected value of K? Assume n = 5 How likely is K to be 0? How likely is K to be n/2? What distribution models this?

11 Some definitions A Bernoulli Trial is an experiment whose outcome is random (and has one of two outcomes (e.g. heads or tails). Think of it is a Boolean random variable, X. A set of random {X 1, X 2, X n }variables is independent and identically distributed (IID) if all variables in the set are mutually independent and all are governed by the same probability distribution D.

12 Back to the coin example If all coin flips use the same coin, we assume that they are IID Bernoulli trials This is modeled by the Binomial Distribution P(K = k) = B(n, k, p) =! # " n k \$ & p k (1 p) n k % Here, K is the summed value of the n coin flips and p is the probability of heads. (recall this->)! # " n k \$ & = % n! k!(n k)!

13 The Normal Distribution As n goes to infinity, the Normal distribution approximates the Binomial distribution, if you set the standard deviation s and mean m correctly. P(K = k) = B(n, k, p) N(µ,σ 2 ) N(µ,σ 2 ) = 1 2πσ 2 e (x µ ) 2 2σ 2

14 The Normal (Gaussian) Distribution p( x) = 2 1 πσ 2 e 1 2 x µ σ 2 mean variance µ = 1, σ = µ = 0, σ = Bryan Pardo, Machine Learning: EECS 349 Fall 2009

15 Central Limit Theorem Let {X 1, X 2,, X n } be a random sample of size n i.e. a set of IID discrete random variables from some distribution D with expected value µ and variance. Define the sample average as x = 1 n K = 1 n n X n 1 For large n, the distribution of is approximated by the normal distribution. Important: The distribution for the sample average approaches normality regardless of the shape of the distribution D governing our random samples X i. x x σ 2

16 Why the previous slides matter Classification is like a coin flip, you re either right or wrong. If classification is independent, then the number of correct classifications K is governed by a Binomial distribution If the Binomial distribution is approximated by the Normal distribution we can use what we know about the Normal distribution. The Normal distribution lets us estimate how close the TRUE error is to the SAMPLE error.

17 How many samples do I need before my sample s distribution is approximately normal? More is always better. The more samples you have, the closer it gets to a normal distribution. Rule of thumb: have at least 30 IID trials. (let s look)

18 The binomial distribution as n grows Source:

19 Running a statistical test 1. Pick a parameter to estimate 2. Choose an estimator 3. Determine the probability distribution governing the estimator 4. Find the interval such that N% of the probability mass falls in that interval 5. The parameter has a N% chance of falling in that interval.

20 Confidence Intervals: Estimating a value 1. Pick a parameter to estimate error D (h) 2. Choose an estimator error s (h) 3. Determine the probability distribution governing the estimator error s (h) governed by Binomial, approximated by Normal when n > Find the interval such that N% of the probability mass falls in that interval Use your favorite statistics software, or look up z N values.

21 How many samples do I need to give me good confidence intervals (assuming we already have a normal distribution)? The standard deviation of the sample mean is related to the standard deviation of the population s and the size of the sample, n by the following: SD X = σ / n Practical result: to decrease uncertainty in a mean estimate by a factor of n requires n 2 observations.

22 Setting 95% confidence interval size Recall that SD X = σ / For a normal distribution, 95% of the mass is within 2 standard deviations of the mean. For estimating a sample mean, an approximate 95% confidence interval has the form (x 2σ / n, x + 2σ / n) So, the 95% confidence interval width is n W = 4σ / n

23 N-fold cross validation Spilt data into N groups. Train on N-1 groups. Validate on the Nth. Rotate, repeat.

24 N-fold cross validation Spilt data into N groups. Train on N-1 groups. Validate on the Nth. Rotate, repeat.

25 N-fold cross validation Spilt data into N groups. Train on N-1 groups. Validate on the Nth. Rotate, repeat.

26 Student s t-test Facts One of the most commonly used statistical tests Assumes normally distributed data Different variants for different questions. one sample t-test: Is x the mean of a normally distributed population? independent samples t-test: Are the means of two normally distributed populations equal? paired samples t-test: Is 0 the mean difference between paired responses measured on the same data? Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007

27 Student s t-test Fact(oid)s The t-test was devised by William Gosset in 1908 It was used to monitor the quality of Guinness Stout (beer). Gosset published the t-statistic under the name student because Guinness considered it a trade secret

28 one sample t-test Abstract question: Is a known population mean m different from the mean of a sample population? Example: µ We know =0.3 is the error rate ID3 has on categorizing a given data set. I trained 30 neural nets to categorize the same data set and the mean error rate was Are neural nets better on this data set? Or was that a fluke? I d use a one-sample t-test to find out. x = 0.2

29 one sample t-test Null Hypothesis: There is no significant difference between the sample mean and the population mean Neural nets perform no better than ID3 on this data. Alternate Hypothesis: There is a significant difference between the sample mean and the population mean. Neural nets DO perform better than ID3 on this data.

30 Paired samples t-test Abstract question: Is 0 the mean difference between paired responses measured on the same data? Example Does eating ice cream make you heavier? Take 1000 people. Weigh each of them. Feed each one an ice cream cone. Weigh each of them again. A paired-samples t-test is appropriate ( Why? )

31 Paired samples t-test Null Hypothesis: There is no significant difference between the two sample means Ice cream does not make you heavier Alternate Hypothesis: There is a significant difference between the two sample means Ice cream makes you heavier. Or it makes you lighter. We didn t actually check which way the difference goes.

32 independent samples t-test Abstract question: Are the means of two normally distributed populations equal? Example Is C4.5 better than ID3 on identifying bad movies from a database of 1000 labeled movies? I do the following 30 times: Train C4.5 on of 500 randomly-selected examples Test on the other 500 I then repeat that for ID3 An independent samples t-test is appropriate ( Why? )

33 Common t-test pitfalls Data is not normally distributed (can t use a t-test) Not enough sample points (degrees of freedom) Using a paired-samples t-test on data where the samples aren t paired (use independent samples t- test, instead) Using a Student s independent samples t-test when the variances of the two sets are different (use Welch s t-test in this case)

34 Comparing populations: Which test? Are the means of two populations equal? What assumption does each test make? Name of test Student s T-Test: Paired samples Student s T-Test: Unpaired samples Samples IID Gaussian Paired X X X X X X X Both pops have same variance Welch s T-test X X Wilcoxon signed-rank test Mann Whitney U test X X X Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2014

35 Conclusions The error measure should captures what you really want to know.not what is easy to measure. Your measure may have variance/bias/noise. Therefore Results are more meaningful when a statistical significance test is done. Many tests depend on the data being normally distributed By taking the sample average of a large set of IID trials, you can ensure normal-like data The t-test is a good, easy test to use if you know when to use it and how

36 Appendix: Precision vs Recall Classifiers are often evaluated with an eye towards their being search engines. (e.g. labeling documents as either relevant or not to a search query). In this case people often use the following measures: precision p = recall r = F measure tp tp + fp tp tp + fn F = 2 p r p + r Machine s Classification True True Classification True True positive (tp) False False negative (fn) False False positive (fp) True negative (tn)

37 Appendix: ROC The Receiver Operating Characteristic (AKA ROC Curve) curve shows the tradeoff as you adjust parameters of your system. best 0 True Positive rate 1 worst 0 False Positive rate 1

38 Appendix: Confusion Matrix Lets us see which things the classifier is mixing up. Helps direct improvement. Correct Classification Machine s Classification Dog Coyote Cactus Road Runner Dog Coyote Cactus Road Runner

### Inferential Statistics

Inferential Statistics Sampling and the normal distribution Z-scores Confidence levels and intervals Hypothesis testing Commonly used statistical methods Inferential Statistics Descriptive statistics are

### Statistics Review PSY379

Statistics Review PSY379 Basic concepts Measurement scales Populations vs. samples Continuous vs. discrete variable Independent vs. dependent variable Descriptive vs. inferential stats Common analyses

### 4. Introduction to Statistics

Statistics for Engineers 4-1 4. Introduction to Statistics Descriptive Statistics Types of data A variate or random variable is a quantity or attribute whose value may vary from one unit of investigation

### MONT 107N Understanding Randomness Solutions For Final Examination May 11, 2010

MONT 07N Understanding Randomness Solutions For Final Examination May, 00 Short Answer (a) (0) How are the EV and SE for the sum of n draws with replacement from a box computed? Solution: The EV is n times

### Statistical Significance and Bivariate Tests

Statistical Significance and Bivariate Tests BUS 735: Business Decision Making and Research 1 1.1 Goals Goals Specific goals: Re-familiarize ourselves with basic statistics ideas: sampling distributions,

### Confidence intervals, t tests, P values

Confidence intervals, t tests, P values Joe Felsenstein Department of Genome Sciences and Department of Biology Confidence intervals, t tests, P values p.1/31 Normality Everybody believes in the normal

### Statistical Inference and t-tests

1 Statistical Inference and t-tests Objectives Evaluate the difference between a sample mean and a target value using a one-sample t-test. Evaluate the difference between a sample mean and a target value

### Statistiek I. t-tests. John Nerbonne. CLCG, Rijksuniversiteit Groningen. John Nerbonne 1/35

Statistiek I t-tests John Nerbonne CLCG, Rijksuniversiteit Groningen http://wwwletrugnl/nerbonne/teach/statistiek-i/ John Nerbonne 1/35 t-tests To test an average or pair of averages when σ is known, we

### Outline. Definitions Descriptive vs. Inferential Statistics The t-test - One-sample t-test

The t-test Outline Definitions Descriptive vs. Inferential Statistics The t-test - One-sample t-test - Dependent (related) groups t-test - Independent (unrelated) groups t-test Comparing means Correlation

### Evaluation & Validation: Credibility: Evaluating what has been learned

Evaluation & Validation: Credibility: Evaluating what has been learned How predictive is a learned model? How can we evaluate a model Test the model Statistical tests Considerations in evaluating a Model

### How to Conduct a Hypothesis Test

How to Conduct a Hypothesis Test The idea of hypothesis testing is relatively straightforward. In various studies we observe certain events. We must ask, is the event due to chance alone, or is there some

### Statistical Inference

Statistical Inference Idea: Estimate parameters of the population distribution using data. How: Use the sampling distribution of sample statistics and methods based on what would happen if we used this

### Data Analysis. Lecture Empirical Model Building and Methods (Empirische Modellbildung und Methoden) SS Analysis of Experiments - Introduction

Data Analysis Lecture Empirical Model Building and Methods (Empirische Modellbildung und Methoden) Prof. Dr. Dr. h.c. Dieter Rombach Dr. Andreas Jedlitschka SS 2014 Analysis of Experiments - Introduction

### Standard Deviation Calculator

CSS.com Chapter 35 Standard Deviation Calculator Introduction The is a tool to calculate the standard deviation from the data, the standard error, the range, percentiles, the COV, confidence limits, or

### Nonparametric Statistics

1 14.1 Using the Binomial Table Nonparametric Statistics In this chapter, we will survey several methods of inference from Nonparametric Statistics. These methods will introduce us to several new tables

### Study Guide for the Final Exam

Study Guide for the Final Exam When studying, remember that the computational portion of the exam will only involve new material (covered after the second midterm), that material from Exam 1 will make

### Joint Probability Distributions and Random Samples (Devore Chapter Five)

Joint Probability Distributions and Random Samples (Devore Chapter Five) 1016-345-01 Probability and Statistics for Engineers Winter 2010-2011 Contents 1 Joint Probability Distributions 1 1.1 Two Discrete

### Research Variables. Measurement. Scales of Measurement. Chapter 4: Data & the Nature of Measurement

Chapter 4: Data & the Nature of Graziano, Raulin. Research Methods, a Process of Inquiry Presented by Dustin Adams Research Variables Variable Any characteristic that can take more than one form or value.

### Fairfield Public Schools

Mathematics Fairfield Public Schools AP Statistics AP Statistics BOE Approved 04/08/2014 1 AP STATISTICS Critical Areas of Focus AP Statistics is a rigorous course that offers advanced students an opportunity

### Chapter 7 Part 2. Hypothesis testing Power

Chapter 7 Part 2 Hypothesis testing Power November 6, 2008 All of the normal curves in this handout are sampling distributions Goal: To understand the process of hypothesis testing and the relationship

### Hypothesis Testing or How to Decide to Decide Edpsy 580

Hypothesis Testing or How to Decide to Decide Edpsy 580 Carolyn J. Anderson Department of Educational Psychology University of Illinois at Urbana-Champaign Hypothesis Testing or How to Decide to Decide

### Nonparametric Test Procedures

Nonparametric Test Procedures 1 Introduction to Nonparametrics Nonparametric tests do not require that samples come from populations with normal distributions or any other specific distribution. Hence

### QUANTITATIVE METHODS BIOLOGY FINAL HONOUR SCHOOL NON-PARAMETRIC TESTS

QUANTITATIVE METHODS BIOLOGY FINAL HONOUR SCHOOL NON-PARAMETRIC TESTS This booklet contains lecture notes for the nonparametric work in the QM course. This booklet may be online at http://users.ox.ac.uk/~grafen/qmnotes/index.html.

### Power and Sample Size Determination

Power and Sample Size Determination Bret Hanlon and Bret Larget Department of Statistics University of Wisconsin Madison November 3 8, 2011 Power 1 / 31 Experimental Design To this point in the semester,

### Statistics I for QBIC. Contents and Objectives. Chapters 1 7. Revised: August 2013

Statistics I for QBIC Text Book: Biostatistics, 10 th edition, by Daniel & Cross Contents and Objectives Chapters 1 7 Revised: August 2013 Chapter 1: Nature of Statistics (sections 1.1-1.6) Objectives

### Statistics 641 - EXAM II - 1999 through 2003

Statistics 641 - EXAM II - 1999 through 2003 December 1, 1999 I. (40 points ) Place the letter of the best answer in the blank to the left of each question. (1) In testing H 0 : µ 5 vs H 1 : µ > 5, the

### Module 5 Hypotheses Tests: Comparing Two Groups

Module 5 Hypotheses Tests: Comparing Two Groups Objective: In medical research, we often compare the outcomes between two groups of patients, namely exposed and unexposed groups. At the completion of this

### Summary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4)

Summary of Formulas and Concepts Descriptive Statistics (Ch. 1-4) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume

### Null Hypothesis Significance Testing Signifcance Level, Power, t-tests Spring 2014 Jeremy Orloff and Jonathan Bloom

Null Hypothesis Significance Testing Signifcance Level, Power, t-tests 18.05 Spring 2014 Jeremy Orloff and Jonathan Bloom Simple and composite hypotheses Simple hypothesis: the sampling distribution is

### Psyc 250 Statistics & Experimental Design. Single & Paired Samples t-tests

Psyc 250 Statistics & Experimental Design Single & Paired Samples t-tests Part 1 Data Entry For any statistical analysis with any computer program, it is always important that data are entered correctly

### Hypothesis Testing & Data Analysis. Statistics. Descriptive Statistics. What is the difference between descriptive and inferential statistics?

2 Hypothesis Testing & Data Analysis 5 What is the difference between descriptive and inferential statistics? Statistics 8 Tools to help us understand our data. Makes a complicated mess simple to understand.

### Two-sample hypothesis testing, I 9.07 3/09/2004

Two-sample hypothesis testing, I 9.07 3/09/2004 But first, from last time More on the tradeoff between Type I and Type II errors The null and the alternative: Sampling distribution of the mean, m, given

### Introduction to. Hypothesis Testing CHAPTER LEARNING OBJECTIVES. 1 Identify the four steps of hypothesis testing.

Introduction to Hypothesis Testing CHAPTER 8 LEARNING OBJECTIVES After reading this chapter, you should be able to: 1 Identify the four steps of hypothesis testing. 2 Define null hypothesis, alternative

### Tutorial 5: Hypothesis Testing

Tutorial 5: Hypothesis Testing Rob Nicholls nicholls@mrc-lmb.cam.ac.uk MRC LMB Statistics Course 2014 Contents 1 Introduction................................ 1 2 Testing distributional assumptions....................

### Descriptive Statistics

Descriptive Statistics Primer Descriptive statistics Central tendency Variation Relative position Relationships Calculating descriptive statistics Descriptive Statistics Purpose to describe or summarize

### Introduction to Hypothesis Testing. Hypothesis Testing. Step 1: State the Hypotheses

Introduction to Hypothesis Testing 1 Hypothesis Testing A hypothesis test is a statistical procedure that uses sample data to evaluate a hypothesis about a population Hypothesis is stated in terms of the

### Lesson 1: Comparison of Population Means Part c: Comparison of Two- Means

Lesson : Comparison of Population Means Part c: Comparison of Two- Means Welcome to lesson c. This third lesson of lesson will discuss hypothesis testing for two independent means. Steps in Hypothesis

### Comparing Means in Two Populations

Comparing Means in Two Populations Overview The previous section discussed hypothesis testing when sampling from a single population (either a single mean or two means from the same population). Now we

### Two-sample inference: Continuous data

Two-sample inference: Continuous data Patrick Breheny April 5 Patrick Breheny STA 580: Biostatistics I 1/32 Introduction Our next two lectures will deal with two-sample inference for continuous data As

### Principles of Hypothesis Testing for Public Health

Principles of Hypothesis Testing for Public Health Laura Lee Johnson, Ph.D. Statistician National Center for Complementary and Alternative Medicine johnslau@mail.nih.gov Fall 2011 Answers to Questions

### Introduction to Hypothesis Testing. Point estimation and confidence intervals are useful statistical inference procedures.

Introduction to Hypothesis Testing Point estimation and confidence intervals are useful statistical inference procedures. Another type of inference is used frequently used concerns tests of hypotheses.

### 4: Probability. What is probability? Random variables (RVs)

4: Probability b binomial µ expected value [parameter] n number of trials [parameter] N normal p probability of success [parameter] pdf probability density function pmf probability mass function RV random

### Nonparametric tests, Bootstrapping

Nonparametric tests, Bootstrapping http://www.isrec.isb-sib.ch/~darlene/embnet/ Hypothesis testing review 2 competing theories regarding a population parameter: NULL hypothesis H ( straw man ) ALTERNATIVEhypothesis

### PROBABILITIES AND PROBABILITY DISTRIBUTIONS

Published in "Random Walks in Biology", 1983, Princeton University Press PROBABILITIES AND PROBABILITY DISTRIBUTIONS Howard C. Berg Table of Contents PROBABILITIES PROBABILITY DISTRIBUTIONS THE BINOMIAL

### Inferences About Differences Between Means Edpsy 580

Inferences About Differences Between Means Edpsy 580 Carolyn J. Anderson Department of Educational Psychology University of Illinois at Urbana-Champaign Inferences About Differences Between Means Slide

### Sampling Central Limit Theorem Proportions. Outline. 1 Sampling. 2 Central Limit Theorem. 3 Proportions

Outline 1 Sampling 2 Central Limit Theorem 3 Proportions Outline 1 Sampling 2 Central Limit Theorem 3 Proportions Populations and samples When we use statistics, we are trying to find out information about

### Simple Regression Theory II 2010 Samuel L. Baker

SIMPLE REGRESSION THEORY II 1 Simple Regression Theory II 2010 Samuel L. Baker Assessing how good the regression equation is likely to be Assignment 1A gets into drawing inferences about how close the

### Quantitative Biology Lecture 5 (Hypothesis Testing)

15 th Oct 2015 Quantitative Biology Lecture 5 (Hypothesis Testing) Gurinder Singh Mickey Atwal Center for Quantitative Biology Summary Classification Errors Statistical significance T-tests Q-values (Traditional)

### Statistiek (WISB361)

Statistiek (WISB361) Final exam June 29, 2015 Schrijf uw naam op elk in te leveren vel. Schrijf ook uw studentnummer op blad 1. The maximum number of points is 100. Points distribution: 23 20 20 20 17

### Exact Nonparametric Tests for Comparing Means - A Personal Summary

Exact Nonparametric Tests for Comparing Means - A Personal Summary Karl H. Schlag European University Institute 1 December 14, 2006 1 Economics Department, European University Institute. Via della Piazzuola

### Sampling Distribution of a Normal Variable

Ismor Fischer, 5/9/01 5.-1 5. Formal Statement and Examples Comments: Sampling Distribution of a Normal Variable Given a random variable. Suppose that the population distribution of is known to be normal,

### 9-3.4 Likelihood ratio test. Neyman-Pearson lemma

9-3.4 Likelihood ratio test Neyman-Pearson lemma 9-1 Hypothesis Testing 9-1.1 Statistical Hypotheses Statistical hypothesis testing and confidence interval estimation of parameters are the fundamental

### SCHOOL OF HEALTH AND HUMAN SCIENCES DON T FORGET TO RECODE YOUR MISSING VALUES

SCHOOL OF HEALTH AND HUMAN SCIENCES Using SPSS Topics addressed today: 1. Differences between groups 2. Graphing Use the s4data.sav file for the first part of this session. DON T FORGET TO RECODE YOUR

### Data Mining Practical Machine Learning Tools and Techniques

Counting the cost Data Mining Practical Machine Learning Tools and Techniques Slides for Section 5.7 In practice, different types of classification errors often incur different costs Examples: Loan decisions

### Basics of Statistical Machine Learning

CS761 Spring 2013 Advanced Machine Learning Basics of Statistical Machine Learning Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu Modern machine learning is rooted in statistics. You will find many familiar

### Discrete Mathematics and Probability Theory Fall 2009 Satish Rao,David Tse Note 11

CS 70 Discrete Mathematics and Probability Theory Fall 2009 Satish Rao,David Tse Note Conditional Probability A pharmaceutical company is marketing a new test for a certain medical condition. According

### Module 9: Nonparametric Tests. The Applied Research Center

Module 9: Nonparametric Tests The Applied Research Center Module 9 Overview } Nonparametric Tests } Parametric vs. Nonparametric Tests } Restrictions of Nonparametric Tests } One-Sample Chi-Square Test

### Multiple Hypothesis Testing: The F-test

Multiple Hypothesis Testing: The F-test Matt Blackwell December 3, 2008 1 A bit of review When moving into the matrix version of linear regression, it is easy to lose sight of the big picture and get lost

### Hypothesis testing S2

Basic medical statistics for clinical and experimental research Hypothesis testing S2 Katarzyna Jóźwiak k.jozwiak@nki.nl 2nd November 2015 1/43 Introduction Point estimation: use a sample statistic to

### Master s Theory Exam Spring 2006

Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem

### Measuring the Power of a Test

Textbook Reference: Chapter 9.5 Measuring the Power of a Test An economic problem motivates the statement of a null and alternative hypothesis. For a numeric data set, a decision rule can lead to the rejection

### Statistiek I. Proportions aka Sign Tests. John Nerbonne. CLCG, Rijksuniversiteit Groningen. http://www.let.rug.nl/nerbonne/teach/statistiek-i/

Statistiek I Proportions aka Sign Tests John Nerbonne CLCG, Rijksuniversiteit Groningen http://www.let.rug.nl/nerbonne/teach/statistiek-i/ John Nerbonne 1/34 Proportions aka Sign Test The relative frequency

### CHAPTER 14 NONPARAMETRIC TESTS

CHAPTER 14 NONPARAMETRIC TESTS Everything that we have done up until now in statistics has relied heavily on one major fact: that our data is normally distributed. We have been able to make inferences

### Let s explore SAS Proc T-Test

Let s explore SAS Proc T-Test Ana Yankovsky Research Statistical Analyst Screening Programs, AHS Ana.Yankovsky@albertahealthservices.ca Goals of the presentation: 1. Look at the structure of Proc TTEST;

### Stat 411/511 THE RANDOMIZATION TEST. Charlotte Wickham. stat511.cwick.co.nz. Oct 16 2015

Stat 411/511 THE RANDOMIZATION TEST Oct 16 2015 Charlotte Wickham stat511.cwick.co.nz Today Review randomization model Conduct randomization test What about CIs? Using a t-distribution as an approximation

### Sample Size and Power in Clinical Trials

Sample Size and Power in Clinical Trials Version 1.0 May 011 1. Power of a Test. Factors affecting Power 3. Required Sample Size RELATED ISSUES 1. Effect Size. Test Statistics 3. Variation 4. Significance

### Normal distribution Approximating binomial distribution by normal 2.10 Central Limit Theorem

1.1.2 Normal distribution 1.1.3 Approimating binomial distribution by normal 2.1 Central Limit Theorem Prof. Tesler Math 283 October 22, 214 Prof. Tesler 1.1.2-3, 2.1 Normal distribution Math 283 / October

### Module 7: Hypothesis Testing I Statistics (OA3102)

Module 7: Hypothesis Testing I Statistics (OA3102) Professor Ron Fricker Naval Postgraduate School Monterey, California Reading assignment: WM&S chapter 10.1-10.5 Revision: 2-12 1 Goals for this Module

### Paired vs. 2 sample comparisons. Comparing means. Paired comparisons allow us to account for a lot of extraneous variation.

Comparing means! Tests with one categorical and one numerical variable Paired vs. sample comparisons! Goal: to compare the mean of a numerical variable for different groups. Paired comparisons allow us

### Projects Involving Statistics (& SPSS)

Projects Involving Statistics (& SPSS) Academic Skills Advice Starting a project which involves using statistics can feel confusing as there seems to be many different things you can do (charts, graphs,

### Statistics 100 Binomial and Normal Random Variables

Statistics 100 Binomial and Normal Random Variables Three different random variables with common characteristics: 1. Flip a fair coin 10 times. Let X = number of heads out of 10 flips. 2. Poll a random

### NONPARAMETRIC STATISTICS 1. depend on assumptions about the underlying distribution of the data (or on the Central Limit Theorem)

NONPARAMETRIC STATISTICS 1 PREVIOUSLY parametric statistics in estimation and hypothesis testing... construction of confidence intervals computing of p-values classical significance testing depend on assumptions

### Single sample hypothesis testing, II 9.07 3/02/2004

Single sample hypothesis testing, II 9.07 3/02/2004 Outline Very brief review One-tailed vs. two-tailed tests Small sample testing Significance & multiple tests II: Data snooping What do our results mean?

### Nonparametric Two-Sample Tests. Nonparametric Tests. Sign Test

Nonparametric Two-Sample Tests Sign test Mann-Whitney U-test (a.k.a. Wilcoxon two-sample test) Kolmogorov-Smirnov Test Wilcoxon Signed-Rank Test Tukey-Duckworth Test 1 Nonparametric Tests Recall, nonparametric

### Statistics courses often teach the two-sample t-test, linear regression, and analysis of variance

2 Making Connections: The Two-Sample t-test, Regression, and ANOVA In theory, there s no difference between theory and practice. In practice, there is. Yogi Berra 1 Statistics courses often teach the two-sample

### Inferential Statistics. Probability. From Samples to Populations. Katie Rommel-Esham Education 504

Inferential Statistics Katie Rommel-Esham Education 504 Probability Probability is the scientific way of stating the degree of confidence we have in predicting something Tossing coins and rolling dice

### Statistical tests for SPSS

Statistical tests for SPSS Paolo Coletti A.Y. 2010/11 Free University of Bolzano Bozen Premise This book is a very quick, rough and fast description of statistical tests and their usage. It is explicitly

### Hypothesis Testing COMP 245 STATISTICS. Dr N A Heard. 1 Hypothesis Testing 2 1.1 Introduction... 2 1.2 Error Rates and Power of a Test...

Hypothesis Testing COMP 45 STATISTICS Dr N A Heard Contents 1 Hypothesis Testing 1.1 Introduction........................................ 1. Error Rates and Power of a Test.............................

### Introduction to Analysis of Variance (ANOVA) Limitations of the t-test

Introduction to Analysis of Variance (ANOVA) The Structural Model, The Summary Table, and the One- Way ANOVA Limitations of the t-test Although the t-test is commonly used, it has limitations Can only

### A crash course in probability and Naïve Bayes classification

Probability theory A crash course in probability and Naïve Bayes classification Chapter 9 Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s

### Come scegliere un test statistico

Come scegliere un test statistico Estratto dal Capitolo 37 of Intuitive Biostatistics (ISBN 0-19-508607-4) by Harvey Motulsky. Copyright 1995 by Oxfd University Press Inc. (disponibile in Iinternet) Table

### Histogram. Graphs, and measures of central tendency and spread. Alternative: density (or relative frequency ) plot /13/2004

Graphs, and measures of central tendency and spread 9.07 9/13/004 Histogram If discrete or categorical, bars don t touch. If continuous, can touch, should if there are lots of bins. Sum of bin heights

### MTH 140 Statistics Videos

MTH 140 Statistics Videos Chapter 1 Picturing Distributions with Graphs Individuals and Variables Categorical Variables: Pie Charts and Bar Graphs Categorical Variables: Pie Charts and Bar Graphs Quantitative

### A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution

A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution PSYC 943 (930): Fundamentals of Multivariate Modeling Lecture 4: September

2. DATA AND EXERCISES (Geos2911 students please read page 8) 2.1 Data set The data set available to you is an Excel spreadsheet file called cyclones.xls. The file consists of 3 sheets. Only the third is

### CA200 Quantitative Analysis for Business Decisions. File name: CA200_Section_04A_StatisticsIntroduction

CA200 Quantitative Analysis for Business Decisions File name: CA200_Section_04A_StatisticsIntroduction Table of Contents 4. Introduction to Statistics... 1 4.1 Overview... 3 4.2 Discrete or continuous

### Multivariate Normal Distribution

Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #4-7/21/2011 Slide 1 of 41 Last Time Matrices and vectors Eigenvalues

### THE FIRST SET OF EXAMPLES USE SUMMARY DATA... EXAMPLE 7.2, PAGE 227 DESCRIBES A PROBLEM AND A HYPOTHESIS TEST IS PERFORMED IN EXAMPLE 7.

THERE ARE TWO WAYS TO DO HYPOTHESIS TESTING WITH STATCRUNCH: WITH SUMMARY DATA (AS IN EXAMPLE 7.17, PAGE 236, IN ROSNER); WITH THE ORIGINAL DATA (AS IN EXAMPLE 8.5, PAGE 301 IN ROSNER THAT USES DATA FROM

### CONTENTS OF DAY 2. II. Why Random Sampling is Important 9 A myth, an urban legend, and the real reason NOTES FOR SUMMER STATISTICS INSTITUTE COURSE

1 2 CONTENTS OF DAY 2 I. More Precise Definition of Simple Random Sample 3 Connection with independent random variables 3 Problems with small populations 8 II. Why Random Sampling is Important 9 A myth,

### Statistical Foundations: Measures of Location and Central Tendency and Summation and Expectation

Statistical Foundations: and Central Tendency and and Lecture 4 September 5, 2006 Psychology 790 Lecture #4-9/05/2006 Slide 1 of 26 Today s Lecture Today s Lecture Where this Fits central tendency/location

### Hypothesis Testing. Bluman Chapter 8

CHAPTER 8 Learning Objectives C H A P T E R E I G H T Hypothesis Testing 1 Outline 8-1 Steps in Traditional Method 8-2 z Test for a Mean 8-3 t Test for a Mean 8-4 z Test for a Proportion 8-5 2 Test for

### Estimation and Confidence Intervals

Estimation and Confidence Intervals Fall 2001 Professor Paul Glasserman B6014: Managerial Statistics 403 Uris Hall Properties of Point Estimates 1 We have already encountered two point estimators: th e

### HYPOTHESIS TESTING WITH SPSS:

HYPOTHESIS TESTING WITH SPSS: A NON-STATISTICIAN S GUIDE & TUTORIAL by Dr. Jim Mirabella SPSS 14.0 screenshots reprinted with permission from SPSS Inc. Published June 2006 Copyright Dr. Jim Mirabella CHAPTER

### Testing Research and Statistical Hypotheses

Testing Research and Statistical Hypotheses Introduction In the last lab we analyzed metric artifact attributes such as thickness or width/thickness ratio. Those were continuous variables, which as you

### P (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )

Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =

### Overview. Evaluation Connectionist and Statistical Language Processing. Test and Validation Set. Training and Test Set

Overview Evaluation Connectionist and Statistical Language Processing Frank Keller keller@coli.uni-sb.de Computerlinguistik Universität des Saarlandes training set, validation set, test set holdout, stratification

### L10: Probability, statistics, and estimation theory

L10: Probability, statistics, and estimation theory Review of probability theory Bayes theorem Statistics and the Normal distribution Least Squares Error estimation Maximum Likelihood estimation Bayesian

### Lesson Lesson Outline Outline

Lesson 15 Linear Regression Lesson 15 Outline Review correlation analysis Dependent and Independent variables Least Squares Regression line Calculating l the slope Calculating the Intercept Residuals and