Herd Behavior and Underdogs in the NFL



Similar documents
A Test for Inherent Characteristic Bias in Betting Markets ABSTRACT. Keywords: Betting, Market, NFL, Efficiency, Bias, Home, Underdog

Fair Bets and Profitability in College Football Gambling

THE DETERMINANTS OF SCORING IN NFL GAMES AND BEATING THE SPREAD

Forecasting Accuracy and Line Changes in the NFL and College Football Betting Markets

The NCAA Basketball Betting Market: Tests of the Balanced Book and Levitt Hypotheses

Beating the Book: Are There Patterns in NFL Betting Lines?

Pick Me a Winner An Examination of the Accuracy of the Point-Spread in Predicting the Winner of an NFL Game

Testing Efficiency in the Major League of Baseball Sports Betting Market.

POINT SPREAD SHADING AND BEHAVIORAL BIASES IN NBA BETTING MARKETS. by Brad R. Humphreys *

Volume 30, Issue 4. Market Efficiency and the NHL totals betting market: Is there an under bias?

The Determinants of Scoring in NFL Games and Beating the Over/Under Line. C. Barry Pfitzner*, Steven D. Lang*, and Tracy D.

Basketball Market Efficiency and the Big Dog Bias. Ladd Kochman* and Randy Goodwin*

The degree of inefficiency in the. football betting market. Statistical tests

Testing the Efficiency of the NFL Point Spread Betting Market

Chapter 13 Gambling and the NFL

Does NFL Spread Betting Obey the E cient Market Hypothesis?

DOES SPORTSBOOK.COM SET POINTSPREADS TO MAXIMIZE PROFITS? TESTS OF THE LEVITT MODEL OF SPORTSBOOK BEHAVIOR

EXAMINING NCAA/NFL MARKET EFFICIENCY

Using Past Performance to Predict NFL Outcomes: A Chartist Approach

How To Bet On An Nfl Football Game With A Machine Learning Program

A Contrarian Approach to the Sports Betting Marketplace

Risk, Return, and Gambling Market Efficiency. William H. Dare Oklahoma State University September 5, 2006

NFL Betting Market: Using Adjusted Statistics to Test Market Efficiency and Build a Betting Model

Home Bias in the NFL Pointspread Market. Matt Cundith Department of Economics California State University, Sacramento

The Performance of Betting Lines for Predicting the Outcome of NFL Games

Does the Hot Hand Drive the Market? Evidence from Betting Markets

THE EFFICIENT MARKET HYPOTHESIS AND GAMBLING ON NATIONAL FOOTBALL LEAGUE GAMES YOON TAE SUNG THESIS

Prices, Point Spreads and Profits: Evidence from the National Football League

An Analysis of Sportsbook Behavior and How to Profit. Chris Ludwiczak. Advisor: Dr. John Clark

Do Gamblers Correctly Price Momentum in NBA Betting Markets?

UZH Business Working Paper Series (ISSN )

Efficiency of football betting markets: the economic significance of trading strategies

Testing the Efficiency of Sports Betting Markets: An Examination of National Football League and English Premier League Betting

International Statistical Institute, 56th Session, 2007: Phil Everson

NBER WORKING PAPER SERIES HOW DO MARKETS FUNCTION? AN EMPIRICAL ANALYSIS OF GAMBLING ON THE NATIONAL FOOTBALL LEAGUE. Steven D.

Sports Forecasting. H.O. Stekler. RPF Working Paper No August 13, 2007

VALIDATING A DIVISION I-A COLLEGE FOOTBALL SEASON SIMULATION SYSTEM. Rick L. Wilson

Testing the Efficiency of Sports Betting Markets

Predicting Margin of Victory in NFL Games: Machine Learning vs. the Las Vegas Line

Understanding Price Movements in Point Spread Betting Markets: Evidence from NCAA Basketball

Live Betting Extra Rules

The Efficiency of Australian Football Betting Markets

Testing Market Efficiency in a Fixed Odds Betting Market

SPORTS FORECASTING. There have been an enormous number of studies involving various aspects of sports. We

Predicting the NFL Using Twitter

Sam Schaefer April 2011

Sin City. In poker, the facility to buy additional chips in tournaments. Total payout liability of a casino during any one game.

EFFICIENCY IN BETTING MARKETS: EVIDENCE FROM ENGLISH FOOTBALL

Sentiment Bias in National Basketball Association. Betting

During the course of our research on NBA basketball, we found out a couple of interesting principles.

Rating Systems for Fixed Odds Football Match Prediction

The Secret To Making Consistent. Profits Betting On Sports

Does bettor sentiment affect bookmaker pricing?

Behavioural Biases in the European Football Betting Market

The Fibonacci Strategy Revisited: Can You Really Make Money by Betting on Soccer Draws?

THE ULTIMATE FOOTBALL BETTING GUIDE

Keywords: behavioral finance, anchoring, affect, market efficiency, sports wagering

Football Bets Explained

Journal of Quantitative Analysis in Sports

THE FAVOURITE-LONGSHOT BIAS AND MARKET EFFICIENCY IN UK FOOTBALL BETTING

Early Season Inefficiencies in the NFL Sports Betting Market. A Thesis. Presented to. The Honors Tutorial College. Ohio University

Applied Economics Publication details, including instructions for authors and subscription information:

Market Efficiency and Behavioral Finance. Chapter 12

Modelling football match results and testing the efficiency of the betting market

WHY ARE GAMBLING MARKETS ORGANISED SO DIFFERENTLY FROM FINANCIAL MARKETS?*

On the effect of taxation in the online sports betting market

Social Casino Postmortems. Derrick Morton CEO, FlowPlay

Betting Terms Explained

How Efficient is the European Football Betting Market? Evidence from Arbitrage and Trading Strategies

Efficiency of sports betting markets

LIVE BETTING ULTRA RULES

The Market for English Premier League (EPL) Odds

A Statistical Test of Association Football Betting Market Efficiency

The Independence Referendum: Predicting the Outcome 1. David N.F. Bell

ONLINE SPORTS GAMBLING: A LOOK INTO THE EFFICIENCY OF BOOKMAKERS ODDS AS FORECASTS IN THE CASE OF ENGLISH PREMIER LEAGUE

Earnings Announcement and Abnormal Return of S&P 500 Companies. Luke Qiu Washington University in St. Louis Economics Department Honors Thesis

Probability and Expected Value

CHAPTER 11: THE EFFICIENT MARKET HYPOTHESIS

Betting Markets and Social Media

Sport Hedge Millionaire s Guide to a growing portfolio. Sports Hedge

Information and e ciency: an empirical study of a xed odds betting market

Beating the NCAA Football Point Spread

RULES AND REGULATIONS OF FIXED ODDS BETTING GAMES

BETTING MARKET EFFICIENCY AT PREMIERE RACETRACKS

Market efficiency in greyhound racing: empirical evidence of absence of favorite-longshot bias

Point Shaving: Corruption in NCAA Basketball

ON ARBITRAGE AND MARKET EFFICIENCY: AN EXAMINATION OF NFL WAGERING. Mark Burkey*

Point Shaving in NCAA Basketball: Corrupt Behavior or Statistical Artifact?

Numerical Algorithms for Predicting Sports Results

The Effects of Atmospheric Conditions on Pitchers

MARKET REACTION TO PUBLIC INFORMATION: THE ATYPICAL CASE OF THE BOSTON CELTICS

Late Money and Betting Market Efficiency: Evidence from Australia

Game Theory and Poker

Transcription:

Herd Behavior and Underdogs in the NFL Sean Wever 1 David Aadland February 2010 Abstract. Previous research has failed to draw any clear conclusions about the efficiency of the billion-dollar gambling industry for National Football League (NFL) games. We build on the previous research and expose a new market inefficiency, which is consistent with the welldocumented notion of herd behavior in behavioral finance. A differential strategy of betting on home and visitor underdogs with large closing lines can produce statistically significant positive returns. Keywords. Market efficiency; Herd behavior; Sports betting; NFL JEL Codes. G14, C25 1 Sean Wever is a graduate student and David Aadland is an associate professor in the Department of Economics and Finance, University of Wyoming, Laramie WY 82071. 1

We add to an established literature that tests whether NFL betting markets are efficient. Financial economists have a long tradition of testing the efficiency of financial markets (Fama, 1970). Standard economic theory predicts that without frictions or informational failures, markets will be efficient and eliminate all arbitrage opportunities. NFL betting markets and financial markets are similar in several ways: large sums of money are involved, investments are inherently risky, and each market is designed to be efficient so abnormally high returns are not available. 2 The existence of profitable betting strategies on NFL games raises doubts about the efficiency of markets in general. Although a few small inefficiencies have been discovered in the NFL betting market, they are generally not robust and a consensus about market efficiency has yet to be reached. Studies such as Zuber et al. (1985), Gandar et al. (1988), Golec and Tamarkin (1992), Gray and Gray (1997), Borghesi (2007) and Sapra (2008) find evidence of inefficiencies in the NFL betting market. Conversely, Sauer et al. (1988), Dare and MacDonald (1996), and Dare and Holland (2004) find little-to-no evidence of market inefficiency. We propose a betting strategy that uncovers surprisingly large market inefficiencies in the NFL, implying that the betting line is systematically biased. The strategy is simple. We regress the bet outcome on a dummy variable for home favorites, home underdogs, and their interaction with the closing line. The estimates imply a strategy of only betting on certain underdogs home teams that are large underdogs and visiting teams that are even larger underdogs. Betting on these select games over the past nine NFL seasons generates out-of-sample winning percentages that are statistically greater than 52.38%, the rate necessary to generate economic 2 In betting markets such as the NFL, the odds maker has the responsibility of setting the price or the line or point spread that produces fair bets. The most popular type of bet for the NFL is a straight bet, which requires the bettor to choose a winner against the point spread. 2

profits. 3 In fact, the winning percentages from our preferred betting strategies are in the neighborhood of 60%. Levitt (2004) and Simmons et al. (2009) have also suggested possible biases against underdogs in the NFL betting market. A common explanation for the apparent irrational behavior in financial markets is that of herd behavior (Banerjee, 1992 and Lux, 1995). Rather than basing investment decisions on rational forecasts of future returns, individuals may instead base their decisions on the actions or words of others. We hypothesize that herd behavior is responsible for the bias toward favorites in the NFL betting market. The elite teams in the NFL receive excess national attention by pundits, sports analysts, and the general press. In response, bettors may tend to overvalue these elite teams, particularly when they are matched up against subpar teams, and place too many bets in their favor. This type of bandwagon effect when applied to the NFL betting market could alternatively be interpreted as an underestimate of the parity in the league. The 2006 season provides an example of how favorites are overvalued and how NFL bettors may display herd behavior. One of the elite teams in the league in 2006 was the Indianapolis Colts. The Colts started the year 10-1 and eventually went on to win the Super Bowl, receiving significant national press and TV coverage of their games. Indianapolis played six games in which they were at least a seven-point visiting favorite or ten-point home favorite against Houston (twice), the New York Jets, Tennessee (twice), and Buffalo. As expected, the combined winning percentage of their four opponents was below 0.500. Indianapolis only covered the spread in one of the six games week two against Houston. Of course, not all such cases are this 3 A bet on either side typically requires $110 with the chance of winning $100. This is known as the 11-for-10 rule or vigorish and is the cost of doing business with the casino. The 11-for-10 rule requires a winning percentage of 52.38% to break even. This break-even percentage can be found by setting the expected value of the bet equal to zero: 10 11 1 0 and solving for. 3

stark but it does highlight the recent tendency of bettors to underestimate the ability of subpar NFL teams to beat the spread. I. Methodology In this section, we describe the methods used to test for market efficiency. Previous models (e.g., Gray and Gray, 1997) have identified two possible sources of inefficiency in NFL betting markets: bias towards favorites and visiting teams. Following Borghesi (2007), we specify a probit model that incorporates these biases, as well as the magnitude of the closing line:,,,,,,,,,,,, (1) where a positive (negative) value for the latent dependent variable indicates that the home team will win the bet; and are dummy variables for home team being the favorite or underdog; is the closing line; is an i.i.d. standard normal error term with cumulative distribution function Φ; and the triplet,, indexes team 1,, ; season 1,, ; and game 1,,. 4 The closing line is calculated as visiting team minus the home team so that negative numbers denote that the home team is favored. 5 The efficient market hypothesis (EMH) implies that the closing line will incorporate all available information regarding the outcome of the bet. The EMH implies that EMH: 0. (2) 4 For our sample period, ranges from 28 to 32 teams; represents 24 seasons; and captures 16 regular season games plus up to three playoff games. 5 This specification, which essentially estimates separate models for home underdogs and home favorites, addresses the biases highlighted by Dare and McDonald (1994). 4

The first two restrictions state that, after accounting for the closing line, home favorites and home underdogs do not systematically win bets. The third and fourth restrictions state that the magnitude of the closing line is not correlated with the outcome of the bet. After imposing the EMH, equation (1) simplifies to,,,, or alternatively,,,,, 0 0.5, (3) where,, is the probability that the home team covers the spread. The EMH therefore implies that all betting strategies will produce an expected winning percentage of 50%. The predicted probability of the home team covering the spread ( ) can be used to develop betting strategies. If exceeds the critical probability, then a bet is placed on the home team. If instead, 1, then a bet is placed on the visiting team. If neither condition is satisfied, no bet is placed. The proportion of winning bets is then compared to 52.38% to test for economic inefficiency. If less than 50 bets are placed, the binomial distribution is used to perform the statistical test and calculate the P values. If 50 or more bets are placed, the normal approximation to the binomial distribution is used. II. Data The data were obtained from www.nfldata.com and consist of all games from 1985 through 2008, a total of 5,976 games. The closing lines are taken from www.vegasinsider.com and measured 12 hours prior to the beginning of each game. We removed all games played on a neutral site where there is no home or visitor designation, all games where neither team is 5

favored, all games that finished in a push, and all games in week 17. 6 All overtime games are determined by the final score after the overtime period. The final data set includes 5,397 games. Table I shows all relevant variables with their definitions and descriptive statistics. The home team covers the point spread in approximately half the games (49.8% of games) and is favored in the majority of games (68.8% of games). When the home (visiting) team is favored it is by an average of 4.82 (1.44) points, but they only win by an average of 4.26 (1.28) points. This is suggestive of a possible bias against underdogs in the NFL, which is explored in more detail below. III. Market Efficiency Results In this section, we describe the results of our market efficiency tests. First, we use classical testing methods for the EMH in (2). Using the language from the finance literature, this is a weak test of market efficiency (Fama, 1970). In the second or strong test of the EMH, we develop an out-of-sample betting strategy in an attempt to identify positive economic returns. Table II shows the probit estimates from model (1) during the period 1985 to 1999. The table shows the coefficient estimates and the marginal effects (i.e., the partial derivative in,, with respect to the closing line for home favorites and home underdogs). The coefficients for home favorite (HF) and the interaction between HF and the closing line are significant. The positive coefficient on the dummy variable ( 0.0828) indicates that, for a small closing line, when the home team is favored they are more likely to win the bet than when they are underdogs. This contrasts to Gray and Gray (1997) who find a small home-underdog bias for an 6 Week 17 is eliminated because teams that have secured a spot in the playoffs will often rest their key players to avoid injury, making it more difficult for odds makers to set efficient lines. 6

earlier sample period. The primary difference between our model and previous research is the interaction of home favorites and home underdogs with the closing line, which allows us to expose the differential underdog effect associated with large closing lines. The positive coefficient on the HF-closing line interaction ( 0.0156) indicates that an increase in the magnitude of the closing line (a more negative number for home favorites), increases the chance that the visiting team will win the bet. Finally, the likelihood ratio statistic is 8.46 (P value = 0.076) leading to a rejection of the hypothesis that the NFL betting market is efficient at the 10% significance level. For the strong test of market efficiency, we use equation (1) and the probit model to calculate the probability that the home team will cover the spread ( ), compare them to various critical probabilities ( ), count the proportion of bets won, and test whether the proportion exceeds the 52.38% threshold for economic efficiency. Table III shows the out-of-sample prediction results for the NFL seasons 2000-2008. When the critical probability ( ) is 0.5, a bet is placed on all 2,138 games. This betting strategy produces a winning percentage of approximately 50% (49.91% to be exact). Obviously, this strategy would lose money and does not show any market inefficiency. However, as the critical probability increases, bets are placed on fewer games and the winning percentages increase. 7 When the critical probability increases to 0.53, the number of games bet falls sharply to approximately two games per week with a winning percentage equal to 58%. At this level, we find evidence of economic inefficiency at the 5% significance level. 8 The winning percentages 7 We note that the tests of efficiency at different critical probabilities are not independent of one another. 8 The NFL introduced a salary cap in 1994. To test for a possible structural break, we also estimated the probit model separately for the 1985-1993 and 1994-1999 sample periods. The out-of-sample prediction results were 7

at higher critical probabilities are also above 52.38%. However, because of the small samples, they are not all statistically greater than the break-even percentage. To gain further understanding of the betting strategies implied by the estimates in Table II, consider solving for the critical closing line from the two following equations: :,, 0.0828 0.0156,, 1 4 :,, 0.0659 0.0016,,. 5 Using the standard normal density function, the closing lines that solve (4) and (5) when, for example, 0.53 are 10.5 and 6.5, respectively. This implies that the bet should be made exclusively on underdogs bet on the visiting team when 10.5 and the home team when 6.5. Higher critical probabilities imply increasing more restrictive underdog betting strategies. The lower panel of Table III shows the year-by-year prediction results for the betting strategy of p c = 0.53. Between 2000 and 2008, the winning percentages are greater than 52.38% for all but 2005 and greater than 52.38% for all but 2000 and 2005. The winning percentages are statistically greater than the 52.38% for 2001, 2002 and 2003 even considering the small number of games bet each year. This indicates that the inefficiencies are not driven by a single year, but appear to be fairly robust through the decade. One possible criticism of the previous analysis is that the results from Table III may not be robust over a smaller number of games. The out-of-sample results could be driven by a similar using the model with and without a structural break in 1994. To conserve on space, we only present the results without the structural break. 8

smaller set of games and therefore the inefficiencies may not be a general phenomenon or continue into future years. To investigate this possibility, we repeatedly randomize the games during the 2000-2008 period and bet on one-third of the randomized games (equivalent of three NFL seasons). Figure I shows the distribution of winning percentages over the subset of games. The majority of the distribution is clearly to the right of the 52.38% break-even level with 93.5% of the subset of games resulting in positive economic returns. Overall, the out-of-sample predictions from 2000-2008 indicate that a differential strategy of betting on underdogs with large closing lines will produce winning percentages in the neighborhood of 60% and statistically significant positive returns. 9 IV. Conclusions Previous research has failed to draw any clear conclusions about the efficiency of the billiondollar gambling industry for National Football League (NFL) games. Our results show that a differential strategy of betting on large underdogs can produce statistically significant profitable returns. 10 The evidence from our study suggests that the recent NFL betting market has underpriced large underdogs and bettors have failed to recognize the amount of parity in the NFL. This inefficiency is consistent with a certain amount of herd behavior toward highly publicized elite teams. 9 The winning percentages for home and visitor underdogs in Table III are similar. We omit the results to conserve on space, but they are available by request. 10 The 2009 NFL season is still ongoing. However, through week 18, the differential underdog strategy at p c = 0.53 has won 32 bets out of 57 games, for a 56.14% winning percentage. 9

References Banerjee A.V. 1992. "A simple model of herd behavior." Quarterly Journal of Economics 108(3), 797-817. Borghesi, R. 2007. "The home team weather advantage and biases in the NFL betting market." Journal of Economics and Business 59(4), 340-354. Dare, W.H. and A.S. Holland. 2004. "Efficiency in the NFL betting market: modifying and consolidating research methods." Applied Economics 36(1), 9-15. Dare, W.H. and S.S. MacDonald. 1996. "A generalized model for testing the home and favorite team advantage in point spread markets." Journal of Financial Economics 40(2), 295-318. Fama, E. 1970. "Efficient capital markets: a review of theory and empirical work". Journal of Finance 25, 383 417. Gandar J., R. Zuber, T. O'Brien and B. Russo. 1988. "Testing rationality in the point spread betting market." Journal of Finance 43(4), 995-1008. Golec, J. and M. Tamarkin. 1991. "The degree of inefficiency in the football betting market -- statistical tests." Journal of Financial Economics 30(2), 311-323. Gray, P.K. and S.F. Gray. 1997. "Testing market efficiency: Evidence from the NFL sports betting market." Journal of Finance 52(4), 1725-1737. Levitt, S.D. 2004. "Why are gambling markets organised so differently from financial markets?" Economic Journal 114(495), 223-246. Lux, T. 1995. "Herd behaviour, bubbles and crashes." Economic Journal 105(431), 881-896. Sapra, S.G. 2008. "Evidence of Betting Market Intraseason Efficiency and Interseason overreaction to unexpected NFL performance 1988-2006." Journal of Sports Economics 9(5), 488-503. Sauer, R.D., V. Brajer, S.P. Ferris, and M.W. Marr. 1988. "Hold your bets -- another look at the efficiency of the gambling market for national-football-league games." Journal of Political Economy 96(1), 206-213. Simmons, J.P., L.D. Nelson, J. Galak, and S. Frederick. 2009. "Are crowds wise when predicting against point spreads? It depends on who you ask." unpublished manuscript. Zuber, R.A., J.M. Gandar, and B.D. Bowers. 1985. "Beating the spread -- testing the efficiency of the gambling market for national-football-league games." Journal of Political Economy 93(4), 800-806. 10

Table I. Definitions and Descriptive Statistics Variables Definition Mean Min Max Win Bet ( ) =1 if the home team covers the spread; 0 otherwise. 0.498 0 1 Home Favorite =1 if the home team is the favorite; 0 otherwise. 0.688 0 1 Home Underdog =1 if the home team is the underdog; 0 otherwise. 0.312 0 1 HF Closing Line Home favorite closing line. -4.817-24 0 HU Closing Line Home underdog closing line. 1.443 0 23 HF Realized Spread (Away Score Home Score) when home team is favored. -4.255-55 44 HU Realized Spread (Away Score Home Score) when visitor is favored. 1.275-43 51 Table I gives the definitions and descriptive statistics for the variables used throughout the paper. The team of record is always the home team. Negative (positive) numbers for closing lines and realized point spreads denote that the home team is the favorite (underdog) or winning (losing) team. The sample period 1985-2008 covers 5,397 games. 11

Table II. Probit Estimates Variable Coefficient Standard Marginal P Value Error Effect Home Favorite (HF) 0.0828** 0.0389 0.0167 -- Home Underdog (HU) 0.0659 0.0622 0.1445 -- HF Closing Line 0.0156*** 0.0055 0.0023 0.0062 HU Closing Line 0.0016 0.0134 0.4541 0.0006 LR Statistic for EMH Degrees of Freedom P Value 8.4624* 4 0.0760 Table II shows the probit estimates for the 1985-1999 NFL seasons, a total of 3,259 observations. The marginal effects are calculated at the mean closing lines for HF and HU. (*), (**), and (***) indicate significance at the 10, 5 and 1% levels. 12

Table III. Out-of-Sample Prediction Results Out-of-Sample Betting for 2000-2008 Based on Probit Model for 1985-1999 Games Bet Games Bet Critical Winning P Value P Value Bets Won per Week Probability Percentage (WP) (WP>0.5) (WP>0.5238) 2138 13.97 0.50 1067 49.91% 0.5345 0.9890 1746 11.41 0.51 883 50.57% 0.4809 0.9740 1118 7.30 0.52 569 50.89% 0.2749 0.8400 328 2.14 0.53 191 58.23% 0.0014 0.0169 100 0.65 0.54 59 59.00% 0.0359 0.0925 Out-of-Sample Betting Performance by Year for a Sample Strategy (p > p c = 0.53) Year Games Bet Critical Winning P Value P Value Bets Won Probability Percentage (WP) (WP>0.5) (WP>0.5238) 2000 50 0.53 26 52.00% 0.3886 0.5215 2001 35 0.53 23 65.71% 0.0448 0.0783 2002 28 0.53 20 71.43% 0.0178 0.0322 2003 27 0.53 20 74.07% 0.0096 0.0180 2004 28 0.53 15 53.57% 0.4253 0.5262 2005 35 0.53 15 42.86% 0.8447 0.9028 2006 31 0.53 20 64.52% 0.0748 0.1199 2007 48 0.53 27 56.25% 0.2354 0.3483 2008 46 0.53 25 54.35% 0.3294 0.4534 Table III displays the out-of-sample prediction results for the 2000 through 2008 NFL seasons. The coefficients from the probit estimates (1985-1999) are used to form probabilities and bet on NFL games from week 1 through week 20, omitting week 17. If then we bet on the home team. If 1 then we bet on the visiting team. Winning percentages that are statistically greater than 50% show statistical inefficiency. Winning percentages that are statistically greater than 52.38% show economic inefficiency. When the games bet are less than or equal to 50, we use the binomial distribution to calculate the p values. For the other cases, we use the normal approximation. 13