Rate-Based Daily Arrival Process Models with Application to Call Centers

Transcription

1 Submtted to Operatons Research manuscrpt (Please, provde the manuscrpt number!) Authors are encouraged to submt new papers to INFORMS journals by means of a style fle template, whch ncludes the journal ttle. However, use of a template does not certfy that the paper has been accepted for publcaton n the named journal. INFORMS journal templates are for the exclusve purpose of submttng to an INFORMS journal and should not be used to dstrbute the papers n prnt or onlne or to submt the papers to another publcaton. Rate-Based Daly Arrval Process Models wth Applcaton to Call Centers Bors N. Oreshkn, Nazm Regnard, Perre L Ecuyer Département d Informatque et de Recherche Opératonnelle, Pavllon Asenstadt, Unversté de Montréal, C.P. 6128, Succ. Centre-Vlle, Montréal, Québec, Canada H3C 3J7, oreshkn@ro.umontreal.ca, lecuyer@ro.umontreal.ca We propose, develop, and compare new stochastc models for the daly arrval rate n a call center. Followng standard practce, the day s dvded n tme perods of equal length (e.g., 15 or 30 mnutes), the arrval rate s assumed random but constant n tme n each perod, and the arrvals are from a Posson process, condtonal on the rate. The random rate for each perod s taken as a determnstc base rate (or expected rate) multpled by a random busyness factor havng mean 1. Models n whch the busyness factors are ndependent across perods, or n whch a common busyness factor apples to all perods, have been studed prevously. But they are not suffcently realstc. We examne alternatve models for whch the busyness factors have some form of dependence across perods. Maxmum lkelhood parameter estmaton for these models s not easy, manly because the arrval rates themselves are never observed. We develop specalzed technques to perform ths estmaton. We compare the goodness-of-ft of these models on arrval data from three call centers, both n-sample and out-of-sample. Our models can represent arrvals n many other types of systems as well. Estmatng a model for the vector of counts (the number of arrvals n each perod) s generally easer than for the vector of rates, because the counts can be observed, but a model for the rates s often more convenent and natural, e.g., for smulaton. We examne and provde nsght on the relatonshp between these two types of modelng. In partcular, we gve explct formulas for the relatonshp between the correlaton between rates and that between counts n two gven perods, and for the varance and dsperson ndex n a gven perod. These formulas mply that for a gven correlaton between the rates, the correlaton between the counts s much smaller n low traffc than n hgh traffc. Key words : arrval process; arrval rate; doubly stochastc Posson process; nput modelng; copula; correlaton; call center 1. Introducton The randomness of customer arrvals s a prme source of uncertanty n servce systems such as restaurants, retal stores, emergency servces, and call centers, to name a few. In those systems, customers (or demands) arrve accordng to stochastc processes whose ntensty (or rate) vares 1

2 2 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) wth tme n a stochastc way, often nfluenced by external events that are not always predctable, and are generally dffcult to model n a realstc way. Ths modelng s nevertheless essental to study the performance of these systems and manage them effectvely. In ths paper, we are concerned wth modelng the arrval process n a call center for one day of operaton. Call centers (or contact centers) are a key component of many organzatons. They employ several mllon people n North Amerca alone, and much of ther operatng costs s to pay the agents who answer the calls (Gans et al. 2003, Akşn et al. 2007, Koole 2013). To optmze the staffng and work schedules of these agents, good models are requred to forecast the call arrval volumes (the demand) and also to smulate the detaled operatons of the call centers. Most large call centers are ndeed complcated stochastc systems whose realstc models can only be handled va stochastc dscrete-event smulaton (Mehrotra 1997, Avramds and L Ecuyer 2005, Bust and L Ecuyer 2005, 2012, Ibrahm et al. 2015a,b). Our dscusson here targets call centers, but our models could apply to many other settngs, such as customer arrvals n retal stores, basket arrvals to cashers n grocery stores, emergency arrvals n healthcare servces, demands for ambulances, for taxs, for pzza delveres, demands for a specfc product n a store or onlne, party arrvals n restaurants, and many more. We leave t to the readers to test how well our proposed models can ft data sets from these other areas. Call arrvals can usually be assumed ndependent over a short tme scale, because they are ntated by ndvduals who make decsons (approxmately) ndependently n the short term. For a gven expected number of arrvals wthn a selected mnute, say, the calls typcally arrve (approxmately) ndependently of each other. It s then natural and qute standard to model arrvals by a Posson process, whch s equvalent to assume that the arrvals occur one by one, ndependently of each other, condtonal on the arrval rate. There are many systems where ths modelng choce makes sense, at least to a good approxmaton, ncludng call centers. It s supported mathematcally by the Posson superposton theorem and s ubqutous n all the work on the modelng of call centers and other smlar servce systems. For detaled justfcatons and some examples, we refer the reader to Whtt (2001), Brown et al. (2005), Koole (2013), Km and Whtt (2014b), Ibrahm et al. (2015b) and the references gven there. There are a few stuatons were call center arrvals can hardly be seen as Posson, e.g., when dozens of people call the polce or ambulance at almost the same tme for the same accdent, or f many people call to order an tem mmedately after seeng a tv commercal for that tem (Soyer and Tarmclar 2008). These arrvals can be modeled by a Posson process whose arrval rate has large narrow peaks once n a whle, when such events occur. We do not model these specal types of bursts n ths paper. Emprcal evdence shows that f arrvals are from a Posson process, the arrval rate must change wth tme and also be stochastc. Such evdence s gven later n ths paper and also n Tanr and

3 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) 3 Booth (1999), Deslaurers (2003), Avramds et al. (2004), Brown et al. (2005), Steckley et al. (2009), Channouf and L Ecuyer (2012), Ibrahm et al. (2015b) and references theren. If the arrval rate s taken as a determnstc functon of tme, the Posson process model mples that the varance and the mean of the number of arrvals n any gven tme perod are equal. Ths dsagrees wth what s observed n call center data (and for many other systems): the varance of daly arrval counts s typcally larger than the mean, and often much larger. Ths s because the arrval rate changes and depends on several factors that are too hard to predct. A smple way of defnng a stochastc arrval rate over a gven tme perod s to assume that a determnstc base rate over that perod s multpled by a sngle random factor wth mean one; see Whtt (1999b) and Avramds et al. (2004). When the tme perod s one day, the base rate as a functon of tme s called the daly profle and the random factor s the busyness factor for the day. Wth a sngle factor for the day, however, the arrval rates over any two dsjont perods of the day are perfectly correlated, whch s more correlaton than what s mpled by observed data. At the other extreme, Jongbloed and Koole (2001) proposed a model n whch the day s dvded nto perods, each perod has ts own random busyness factor, and these factors are ndependent. Call center data strongly dsagrees wth ths ndependence assumpton: the arrval rates over dsjont perods are typcally postvely correlated. The explanaton s that factors that affect the arrval rates (e.g., weather condtons, etc.) typcally span over several perods of the day, so larger call volumes n the mornng are often assocated wth larger call volumes n the afternoon, for example. Neglectng ths dependence leads to an underestmaton of the queue buld up process and of watng tmes. Our am s to develop more realstc arrval rate models that are between these two extremes and for whch the means, varances, and correlatons of arrval counts better match those observed n data. In the models we consder, the arrvals are Posson wth a stochastc rate. Under such models, the varance of the arrval counts n any gven perod cannot be smaller than the mean;.e., underdsperson s not possble. We have never observed underdsperson, or negatve correlatons between perods, n call center arrval data. The daly profle can be taken n prncple as any fxed functon of tme. Although a contnuous functon may appear more realstc, the most popular choce by far s a pecewse-constant functon, for whch the day s dvded nto tme perods of equal length (usually 30 or 15 mnutes) and the arrval rate s assumed constant over each perod (Gans et al. 2003, Avramds et al. 2004, Brown et al. 2005, Akşn et al. 2007, Channouf and L Ecuyer 2012, Koole 2013, Km and Whtt 2014a, Ibrahm et al. 2015b). There are many reasons for ths. For one, most call center managers determne ther requred staffng by usng approxmatons va Erlang formulas. For each tme perod, they would compute how many agents they need to acheve a gven target performance measure (e.g., 80% of the calls answered wthn 20 seconds, or less than 3% abandonment, etc.) by assumng a

4 4 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) steady-state model over that perod and usng the Erlang formula to approxmate the performance. Ths can be done even f the arrval rate over the perod s random wth a known dstrbuton. The random arrval rates do not have to be ndependent across perods, there can be global performance targets for the day, and also multple call types (Gans et al. 2003, Harrson and Zeev 2005, Atlason et al. 2008, Ce zk and L Ecuyer 2008, Avramds et al. 2009a, Gurvch et al. 2010, Koole 2013). If one wshes to estmate the performance by a more detaled smulaton nstead of queung formulas, t s easer and faster to smulate a Posson process wth pecewse-constant rate than one whose rate changes contnuously. For the former t suffces to generate ndependent exponental nterarrval tmes n each perod, and eventually reschedule the next arrval tme when the arrval rate changes at a perod boundary (Bust and L Ecuyer 2005, Ibrahm et al. 2012). A thrd justfcaton for usng pecewse-constant rates s that exact call arrval tmes are rarely avalable n call center data. Typcally, the avalable data s n the form of number of arrvals (the arrval counts) n each tme perod, and the call by call arrval process model must be constructed based on ths data only. Moreover, agents can typcally be added or removed only at perod boundares. It s of course possble to construct models n whch the arrval rate changes contnuously. For example, Km and Whtt (2014a) consder a pecewse-lnear approxmaton and compare wth a pecewse-constant rate. Channouf (2008) developed a methodology that uses smoothng splnes to model the arrval rate functon, together wth a sngle random busyness factor for the day. The process s smulated va a thnnng technque. Hs methodology can estmate the parameters from aggregated data (arrval counts per perod) and s mplemented n the ContactCenters software (Bust and L Ecuyer 2005, 2012). In numercal experments wth real data from three dfferent call centers, Channouf (2008) observed a small dfference n smulated performance measures (a small mprovement) when swtchng from a pecewse-constant rate to a splne rate, both wth a sngle random busyness factor for the day. Note that f the (smooth) splne base rate s multpled by busyness factors that dffer across perods, the resultng rate functon wll no longer be contnuous. One could thnk of a model n whch some base rate s multpled by dfferent random busyness factors n dfferent perods and the smoothng splne s ftted to the resultng random rates afterward, or models n whch both the base rate and the random busyness factor are contnuous functons of tme, probably parameterzed, and whose parameters would be ftted to call-by-call arrval tmes. These types of models are beyond our scope; we leave them for future work. Assumng a pecewse-constant rate s reasonable f the arrval rate does not change too rapdly and the tme perods are taken small enough so the arrval rate s approxmately constant n each perod. Km and Whtt (2014b,a) have studed ths ssue and developed tests for the Posson process assumpton (wth ether determnstc or random rates). When the ndvdual arrval tmes are avalable, ther tests can be used to select an approprate perod length n whch the rate

5 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) 5 can be taken as approxmately constant. They also provde practcal gudelnes for selectng ths length and dscuss varous other related ssues. In ther tests wth real-lfe data, they found that the Posson assumpton was qute reasonable wth 30-mnute perods for call center data, and wth 60-mnute perods for arrvals to a hosptal emergency system. The choce of perod length may also depend on how much data s avalable to estmate the rate for each perod: f the perods are taken too short, the estmates can be too nosy and we can end up wth an overfttng problem. The methodology developed n ths paper works for an arbtrary perod length. We focus on modelng the call center arrval process over one day of operaton. Ths s useful for smulatng the call center one day at a tme, more than a few days before the days that are smulated, so the dependence between the currently avalable data and the days that are smulated can be seen as neglgble. Ths type of settng occurs when managers construct work schedules one or more weeks n advance. The predctve power of known tme-seres forecastng models n ths scenaro s weak, because the target day s too far ahead (Avramds et al. 2004, Ibrahm and L Ecuyer 2013). These models can be useful not only for smulaton, but also to obtan dstrbutonal forecasts that can be used n other algorthms or formulas. For the day-to-day management of real-lfe call centers, one would also need to model the dependence across successve days, the seasonaltes at weekly and yearly levels, specal-day effects, and n many cases the dependence between dfferent call types (Brown et al. 2005, Jaoua et al. 2013, Ibrahm et al. 2015b). Ths s beyond the scope of the present paper, but could be done n combnaton wth the models proposed here. Shen (2010) and Ibrahm et al. (2015b) gve overvews of exstng models, whch typcally have a regresson or tme seres flavor. For example, Wenberg et al. (2007) used a Bayesan approach to sample from the forecast dstrbutons based on a lnear regresson data model, whle Ibrahm et al. (2012), Aldor-Noman et al. (2009), Brown et al. (2005) used dfferent varants of lnear regresson models to produce pont forecasts of daly call volumes. One common characterstc of these papers s that they all use dfferent varants of the root-unroot varance stablzng transformaton proposed by Brown et al. (2001), whch approxmates the square root of the sum of 1/4 and a Posson random varable wth large mean, mnus the square root of the mean, by a normal random varable wth mean 0 and varance 1/4. However, ths approxmaton s often questonable because () the expected count n a gven tme perod s not always large and () the arrval counts are typcally not Posson, but have over-dsperson compared wth the Posson dstrbuton (the varance s larger than the mean). Most of these works focus on the pont forecastng of call volumes,.e., estmatng the expectaton, to plug t eventually nto an Erlang formula to determne the requred number of agents, rather than dstrbutonal forecasts. Models for the dstrbuton of the vector of arrval counts over a gven day, wth the day dvded nto equal-length perods, have also been proposed. Avramds et al. (2004) ntroduced two such

6 6 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) models. In the frst one, the vector of counts has a negatve multnomal dstrbuton whose parameters have a (multvarate) Drchlet dstrbuton. In the second model, the total number of arrvals durng the day has an arbtrary dstrbuton (they take the gamma dstrbuton n ther experments) and the vector of proportons of arrvals n each perod has an ndependent Drchlet dstrbuton. The coordnates of the resultng vector of counts must then be rounded to obtan nteger counts. These models are more flexble and mprove the matchng of correlatons compared wth the model wth a sngle busyness factor that multples the daly profle, but the correlatons stll reman too strong. Channouf and L Ecuyer (2012) llustrate ths wth real call center data and propose a general multvarate dstrbuton model for the vector of counts n whch the margnal dstrbutons are specfed ndvdually to match the dstrbuton of counts n each perod, and the dependence between the counts s modeled separately va a Gaussan copula, that matches (approxmately) the parwse rank correlatons. They also have versons n whch the correlaton matrx of the copula s parameterzed, to reduce the number of model parameters. Ther model provdes a much better match to data even after accountng for the fact that t has more parameters (va the Akake nformaton crteron). These authors model the vector of arrval counts, whereas n the present work we want to model the vector of arrval rates. Wth a dstrbutonal model for the counts, assumng a pecewse constant rate, one can smulate the arrvals by frst generatng the number of arrvals (the count) n each perod, and then generatng the arrval tmes unformly and ndependently over the gven tme perod. Ths s consstent wth the assumpton that arrvals are from a Posson process wth unknown (perhaps random) constant rate over the perod. But ths s much less convenent and effcent than generatng the arrvals one by one drectly from the constant rate when ths rate s known (Ibrahm et al. 2012). The former requres generatng, storng, and sortng all arrval tmes n the perod before dong the dscrete-event smulaton, whereas wth the latter we only need to store the next arrval tme. Our preference s therefore for dstrbutonal models of the rates. In a smulaton, we can then frst generate the (random) arrval rates for all perods of the day and then run the smulaton wth those rates. Another mportant motvaton for the rate-based models s that many staffng models need the dstrbuton of the rate as one of ther nputs (Gans et al. 2003). In ths paper, the day s dvded nto perods of equal length and the arrvals are assumed to be from a Posson process wth constant rate n each perod. All our models are developed for the settng n whch data are aggregated n the form of arrval counts per perod. For recommendatons on how to select the perod length when detaled call by call arrval tmes are avalable, we refer the reader to Km and Whtt (2014b,a). We start from two smple models mentoned prevously, n whch a determnstc daly profle for the rate s multpled ether by a sngle busyness factor for the day, or by ndependent busyness factors, one for each perod. Our frst dea s to combne

7 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) 7 these two models: we take one random local busyness factor for each perod plus a global one for the day. Ths provdes more freedom to control the correlatons. As n Jongbloed and Koole (2001) and Avramds et al. (2004), our busyness factors have a gamma dstrbuton. Then we generalze ths to a model n whch the global gamma factor s rased to a dfferent power (and normalzed) n each tme perod, whch gves further flexblty n matchng the correlatons. Fnally, we propose a model n whch the multvarate gamma random vector of busyness factors, rather than the vector of counts as n Channouf and L Ecuyer (2012), s determned by a Gaussan (normal) copula. Ths gves even more flexblty to match both the correlatons and the varance wthn each perod, at the expense of havng many more parameters to estmate. To reduce (and control) the number of parameters, we consder varants of ths model for whch the correlaton matrces are restrcted to classes havng a specal structure. We compare the goodness of ft of these dfferent models on real data sets. For the models of Jongbloed and Koole (2001), Avramds et al. (2004), and Channouf and L Ecuyer (2012), parameter estmaton was relatvely easy, va maxmum lkelhood and correlaton matchng. For our new models, t s much harder, manly because we do not observe the arrval rates themselves, but only the counts, whch gve only ndrect nformaton on the rates. An mportant part of our contrbuton s to develop vable methods to estmate the parameters for all the proposed models, va maxmum lkelhood. The rest of the paper s organzed as follows. In Secton 2, we defne our general settng wth busyness factors and pecewse constant arrval rates, and we examne some of ts propertes. In Secton 3 we ntroduce a two-level busyness factor model, whch combnes a sngle busyness factor for the day and a busyness factor for each perod. Secton 4 generalzes ths model by ncludng perod-wse exponentaton of the daly busyness factors. In Secton 5, we model the dependence structure n the vector of stochastc arrval rates va a normal copula. Estmaton methods for these models are developed n the Onlne Supplement. They consttute an mportant part of our contrbuton. Secton 6 reports the results of our experments. Addtonal results and plots are gven n the Onlne Supplement. All sectons whose numbers start by a letter from A to E are n the Onlne Supplement. 2. General Settng, Notaton, and Relatonshps Between Rates and Counts We consder one day of operaton of a call center. The openng hours are dvded nto p tme perods of equal length. For example, f the center receves calls from 8:00 to 18:00 and the perods are 30 mnutes long, we have p = 20. Let X = (X 1,..., X p ) be the vector of arrval counts n those p perods. We assume that the arrvals are from a Posson process wth a random rate Λ j, constant over perod j. To smplfy the notaton, the tme unt for ths rate s assumed to be one perod,.e., the rate

8 8 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) s expressed n (expected) number of arrvals per perod. The vector Λ = (Λ 1,..., Λ p ) can have an arbtrary multvarate dstrbuton over [0, ) p. Takng ts mean λ = (λ 1,..., λ p ) as a scalng factor (or base rate), we shall assume that Λ j = B j λ j where B j s a non-negatve random varable wth E[B j ] = 1 for each j. We call B j the busyness factor for perod j and we denote B = (B 1,..., B p ). Condtonal on Λ, the X j s are ndependent and each X j has a Posson dstrbuton wth mean Λ j. To summarze, we have Λ j = B j λ j and X j Posson(Λ j ), (1) where Posson(λ) denotes the Posson dstrbuton wth mean λ. By takng the logarthm on each sde of the equaton n (1), we obtan a lnear model wth mxed effects. However, ths model s non-standard because the rates Λ j are hdden and can be nferred only ndrectly through the counts X j, and ts parameters are often hard to estmate for ths reason. Note that ths settng does not capture trends or changes that mght occur over tme frames longer than a day. The mean and covarance matrx of the counts X can be expressed n terms of those of the rates Λ as follows. For each j, we have E[X j ] = E[Λ j ] = λ j and Var(X j ) = E[Var[X j B j ]] + Var[E[X j B j ]] = E[B j λ j ] + Var(B j λ j ) = λ j (1 + λ j Var(B j )). (2) The coeffcent of dsperson, or dsperson ndex (DI), defned as the rato of varance to the mean (Cox and Lews 1966), s then DI(X j ) = Var(X j) λ j = 1 + λ j Var(B j ) 1. (3) We use the DI rather than the coeffcent of varaton to measure the relatve varablty because t better ndcates the overdsperson wth respect to the Posson dstrbuton. Eq. (3) shows n partcular that the varance can never be smaller than the mean under a Posson process model. When Var(B j ) = 0, X j has a Posson dstrbuton wth mean λ j, so DI(X j ) = 1. More nterestngly, for a fxed value of Var(B j ), DI(X j ) ncreases lnearly wth λ j, and DI(X j ) 1 when λ j 0. Ths means that under ths general model n whch the rate λ j s multpled by a sngle factor over the tme nterval, the arrval process behaves pretty much lke a Posson process over very short tme ntervals, for whch λ j s small, and the overdsperson ncreases wth the length of the tme nterval, because λ j s then larger. In partcular, f the base rate s multpled by a sngle factor over the entre day, the count for the entre day under ths model wll typcally have a much larger DI than that for one hour or one half-hour. We also ntroduce a standardzed verson of the DI (SDI), defned as SDI(X j ) = DI(X j) 1 λ j, (4)

9 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) 9 whch under our model s equal to Var(B j ), so t measures the varablty of the rate ndependently of λ j. Note that f we merge two perods j and k havng a common busyness factor B = B j = B k, the SDI of the merged perods s the same as that of the orgnal ones: SDI(X j + X k ) = SDI(X j ) = SDI(X k ) = Var(B), and ths apples as well to more than two perods. Thus, lookng at how the SDI behaves when we merge perods permts one to test the dependence between ther busyness factors. In partcular, for the model wth a sngle common busyness factor B = B 1 = = B p, the SDI always remans the same when we merge perods. In general, we have SDI(X j + X k ) = λ2 j Var(B j ) + λ 2 k Var(B k ) + 2λ j λ k Cov(B j, B k ) (λ j + λ k ) 2 (λ2 j + λ 2 k + 2λ j λ k ) max(var(b j ), Var(B k )) (λ j + λ k ) 2 = max(var(b j ), Var(B k )) = max(sdi(x j ), SDI(X k )), (5) wth equalty holdng when Corr(B j, B k ) = 1. Ths generalzes to more than two perods. On the other hand, one can have SDI(X j + X k ) < mn(var(b j ), Var(B k )) = mn(sdi(x j ), SDI(X k )), so the SDI of merged perods can be smaller than the smallest SDI of those perods. For example, f Var(B j ) = Var(B k ) > 0 and Corr(B j, B k ) < 1, then SDI(X j + X k ) = Var(B j)[λ 2 j + λ 2 k + 2λ j λ k Corr(B j, B k )] (λ j + λ k ) 2 < Var(B j ) = mn(sdi(x j ), SDI(X k )). For j k, we also have Cov(X j, X k ) = E[(B j λ j )(B k λ k )] λ j λ k = λ j λ k Cov(B j, B k ) (6) and Corr(X j, X k ) = = Cov(B j, B k ) [(Var(B j ) + 1/λ j )(Var(B k ) + 1/λ k )] 1/2 Corr(B j, B k ). (7) 1/2 [((1 + 1/(Var(B j )λ j ))(1 + 1/(Var(B k )λ k ))] For a fxed dstrbuton of B = (B 1,..., B p ), we have Corr(X j, X k ) 0 when λ j 0 or λ k 0, whereas Corr(X j, X k ) Corr(B j, B k ) when both λ j and λ k. That s, the process behaves lke a Posson process over short tme perods (the correlaton between the counts over short dsjont perods s near zero), whle the correlaton s hgher over larger dsjont tme perods. For fxed values of λ j and λ k, Corr(X j, X k ) s close to Corr(B j, B k ) when B j and B k have large varance and s smaller otherwse. Three specal cases of ths model, studed earler, are defned as follows. We wll use them for comparson wth our new models.

10 10 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) The frst (smplest) case s the degenerate case where B j = 1 for all j, so Var(B j ) = 0 and Corr(X j, X k ) = 0. It gves an ordnary nonhomogenous Posson arrval process wth pecewse constant rate, as used n Brown et al. (2005) for example. We wll refer to ths case smply as Posson. It s well known that ths type of model s unacceptable because t typcally underestmates the varablty of the counts n a sgnfcant way; see, e.g., (Jongbloed and Koole 2001, Avramds et al. 2004, Steckley et al. 2009, Shen 2010). In the second specal case, ntroduced by Whtt (1999a), one takes B j = B for all j, where E[B] = 1, so the base rate s multpled by the same random busyness factor for the entre day. Then the SDI over any unon of perods s Var(B). Avramds et al. (2004) further studed ths model n the case where B has a gamma dstrbuton wth Var(B) = 1/γ, whch can take any value n the range (0, ), and Corr(B j, B k ) = 1. In ths case, the vector X has a negatve multnomal dstrbuton whose parameters are easy to estmate from the counts. We call ths model Possongamma-sngle-factor, or smply PGsngle. Two mportant drawbacks of ths model are that () t tends to overestmate the postve correlaton between the counts n dfferent perods and () t does not ft the varance equally well for all the perods of the day (see Avramds et al and Channouf and L Ecuyer 2012). These problems are due to the fact that there s a sngle busyness factor common to all perods of the day, and (gven the λ j s) a sngle parameter γ to be chosen that determnes all the varances and correlatons. The thrd smplfed settng, from Jongbloed and Koole (2001), uses ndependent busyness factors B j for the dfferent perods of the day. Ths gves Corr(X j, X k ) = 0. These authors use the gamma dstrbuton for the B j and show that the parameters are easy to estmate by maxmum lkelhood. We refer to ths model as Posson-gamma-ndependent, or PGndep. It has the mportant lmtaton of neglectng the dependence (often strong) between counts across dfferent perods. In the remander, we propose new nstances of the general arrval process model outlned earler, whch allow more flexblty for matchng the varances wthn perods and correlatons between perods. Frst, we combne the PGsngle and PGndep models nto a two-level busyness factor model that ncludes both a daly busyness factor and a busyness factor per perod. We then further extend ths model by ntroducng an exponentaton of the daly busyness factor n every perod. These new models are more flexble than the prevous ones and they contan only a few extra parameters compared to the PGndep model. Then we propose a normal copula model for B whch s even more flexble to ft the varances and correlatons of the counts, at the expense of havng more parameters. 3. Two-Level Busyness Factor Model We consder the followng two-level arrval process model, named PG2, based on the multplcatve combnaton of ndependent perod busyness factors B j and the busyness factor for the day, B.

11 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) 11 Let Gamma(a, b) denote a gamma dstrbuton wth mean a/b and varance a/b 2. We assume that B, B 1,..., B p are ndependent wth B Gamma(β, β) and Bj Gamma(α j, α j ) for each j, (8) for some postve parameters β, α 1,..., α p, and we take B j = B j B (9) as the busyness factor of perod j. Ths combnaton permts one to better control the correlaton between the B j s, n comparson wth the prevous specal cases where t was ether 0 or 1. Smple formulas are avalable for the moments n ths model: Var(B j ) = (1 + β + α j) βα j and Cov(B j, B k ) = 1 β. Then, Var(X j ) and Cov(X j, X k ) are easly obtaned from (2) and (6). Table 1 summarzes some statstcal propertes of ths model and compares wth the smpler specal cases. We see how the addtonal terms for ths PG2 model provde more flexblty to match the varances and correlatons. Snce we have smple formulas for the moments, moment-matchng estmators (MMEs) for ths model are easy to compute (see Secton A.1). However, n experments where we generated data sets by smulaton from the model wth known parameters, and then estmated these parameters by moment matchng, we found that these MMEs often returned values far off the correct ones, whch ndcates a lack of accuracy and robustness. Table 1 Some statstcal propertes (moments) for the PGsngle, PGndep, and PG2 models. Model E[X j ] Var(X j ) Corr(B j, B k ) Corr(X j, X k ) PGsngle λ j λ j + λ 2 j/β 1 [(1 + β/λ j )(1 + β/λ k )] 1/2 PGndep λ j λ j + λ 2 j/α j 0 0 PG2 λ j λ j + (1+β+α j)λ 2 j βα j [(1 + 1+β α j )(1 + 1+β α k )] 1/2 [(1 + β λ j + 1+β α j )(1 + β λ k + 1+β α k )] 1/2 Maxmum lkelhood estmators (MLEs) are generally more robust and accurate. However, whle these estmators were readly avalable for the prevous specal cases, here they are much harder to compute. More specfcally, the avalable expresson for the densty of X j, whch appears n the lkelhood functon for each j, nvolves an ntegral wth respect to the realzaton of the vector B of unobserved daly busyness factors (see (20) n Secton A.2) and we do not know how to evaluate ths ntegral n closed form. We opted to develop parameter estmaton methods for ths model based on Monte Carlo estmaton of the log-lkelhood functon. The stochastc optmzaton algorthm that we use to approxmate the MLEs for the PG2 model s descrbed and studed n Secton A.2.

12 12 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) When comparng the MLEs wth the MMEs usng smulaton experments as descrbed earler, usng mean square error n the parameter estmaton, for varous sets of parameter values, MLEs were always clear wnners. In ths PG2 model, the parameters α 1,..., α p can be specfed ndependently from each other, wthout any functonal relatonshp between them. Alternatvely, one may mpose as an addtonal constrant that α j as a functon of j belongs to some class of smooth functons, e.g., a cubc splne. Ths can provde a more realstc model n the stuaton where α j does not vary wth j too wldly on a gven day. Forcng α j to obey a splne as a functon of j s a way to reduce the overfttng of the model. We wll examne the smoothng splne varant of the model named PG2sp n our case studes. The addtonal ngredents requred to compute the MLE for the PG2sp model are descrbed n Secton A Extended Two-Level Busyness Factor Model The PG2 model of Secton 3 s more flexble than PGndep and PGsngle, but on close examnaton we see that n comparson wth PGndep, t has only one addtonal parameter β, so the addtonal flexblty n matchng all the varances and correlatons s stll lmted. Ths s because the busyness factor B for the day affects all the perods n exactly the same way. To remove ths restrcton, and to add flexblty n matchng the correlatons, here we rase the factor B to some power p j n each perod j, where the exponents p j s may dffer across perods, and we normalze so that the expectaton of B p j remans equal to 1 n each perod. The exponent permts one to modulate the mpact of B dfferently across perods. For example, t could cover a stuaton where the busyness factor for the day affects the arrval rates much more strongly n the mddle of the day than n the evenng. Accordng to our data, such types of stuatons do occur. Ths yelds the followng model, whch we call PG2pow: B j = B j Bp j /γ(p j ) (10) where γ(p j ) = E[ B p j] = β p jγ(p j + β)/γ(β) s the approprate normalzaton constant. Ths model remans n the class of gamma-posson processes as the dstrbuton of Bp j belongs to the class of generalzed gamma. For any fxed value of β > 0, when p j 0 we have Var( B p j) 0 and B p j becomes degenerate at 1, whle when p j, Var(B p j). Therefore, the mpact of the daly busyness factor B on perod j can be made arbtrarly small by decreasng p j, eventually completely decorrelatng ths perod j from the rest of the day, and arbtrarly large by ncreasng p j. The varances and covarances of the counts are gven by [ ] (1 + Var(X j ) = λ j + λ 2 αj )Γ(β)Γ(2p j + β) j 1, (11) α j Γ(p j + β) [ 2 ] Γ(β)Γ(pj + p k + β) Cov(X j, X k ) = λ j λ k Γ(p j + β)γ(p k + β) 1. (12)

13 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) 13 We see that f ether p j = 0 or p k = 0, then Cov(X j, X k ) = Corr(X j, X k ) = 0. Thus, n ths extended model, the correlatons and the varances can be further dsentangled compared to the PG2 model. Parameter estmaton for ths model can be performed usng technques smlar to those used for the PG2 model. Further detals are gven n Secton B. 5. A normal copula for the vector of rates The most general way of modelng the dstrbuton of B = (B 1,..., B p ) s to select an arbtrary margnal dstrbuton for each B j, and to model the dependence by a copula (Nelsen 1999). Here we propose a model that captures the dependence by a normal copula. The advantages of usng a normal copula are that t can match (approxmately) all the correlatons between the X j s, the parameters are not too hard to estmate even when the dmenson p s large, and t s not dffcult to generate the vector B from ths model. The resultng model has much more flexblty to match the varances and correlatons than the prevous ones, at the expense of havng many more parameters. Ths type of model based on a normal copula s also known as a NORTA (NORmal To Anythng) model (see, e.g., Hörmann et al. 2004, Avramds et al. 2009b). We call t PGnorta. Channouf and L Ecuyer (2012) also proposed a normal copula model, but t was to model drectly the vector X of arrval counts nstead of the vector B as we do here. Estmatng the copula parameters s more dffcult n our case because B s unobserved. The margnal dstrbutons of each B j can be arbtrary over [0, ). In our development and experments, we use a gamma dstrbuton wth mean 1, as n the prevous models Normal copula wth arbtrary correlaton matrx and gamma margnals We assume that each B j has a Gamma(α j, α j ) dstrbuton, wth cumulatve dstrbuton functon (cdf) G j. Each α j s s estmated ndvdually by MLE as n the PGndep model. The dependence between the B j s s modeled by a normal copula, defned as follows. Recall that a copula s a multvarate dstrbuton whose margnals are all unform over the nterval (0, 1). A normal copula n p dmensons s a specal type of copula that can be specfed by selectng an arbtrary (vald) p p correlaton matrx R Z. To generate a random vector U from that copula, we generate Z = (Z 1,..., Z p ) from the multnormal dstrbuton wth mean zero and covarance matrx R Z, then we put U = (U 1,..., U p ) = (Φ(Z 1 )),..., Φ(Z p )) where Φ s the standard normal cdf. Then to generate B from ths copula, we smply put B j = G 1 j (U j ) def = nf{x R : G j (x) U j } for all j. The choce of R Z defnes mplctly the covarance matrx of B, whch n turn determnes the covarance matrx of X. We want to choose R Z to match the emprcal correlatons for X observed n the data. As generally recommended because t s more robust (Hörmann et al. 2004, Avramds et al. 2009b), we want to match the Spearman (or rank) correlatons between the X j s. Wth ths approach,

14 14 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) the modelng of margnal dstrbutons and correlatons s hghly decoupled, snce the correlatons are estmated separately from the margnals. The varances and correlatons of the X j s can be matched very closely, as wth the model of Channouf and L Ecuyer (2012). Our results wth real data wll confrm ths. For each j, let F j be the cdf of X j and σ 2 F j = Var(F j (X j )). The Spearman correlaton between X j and X k s r X j,k = E{F j(x j )F k (X k )} E{F j (X j )}E{F k (X k )} σ Fj σ Fk. Here, F j (X j ) s not a unform random varable over (0, 1), because X j s a dscrete random varable. After the parameters α j have been estmated by MLE, we have estmates ˆF j of the margnal dstrbutons F j (obtaned by replacng each α j by ts estmate n the functon F j ), and the Spearman correlaton for the model that uses these estmates s r X j,k = E{ F j (X j ) F k (X k )} E{ ˆF j (X j )}E{ ˆF k (X k )} σ Fj σ Fk, (13) Let r X j,k be the (emprcal) Spearman correlaton coeffcent n the data. For each par (j, k), we want to fnd ρ Z j,k for whch r X j,k = r X j,k. (14) Ths s a root fndng problem n whch the left sde, gven n (13), contans an expectaton defned as a double ntegral nsde a double sum: we must ntegrate wth respect to the jont dstrbuton of (B j, B k ), then sum wth respect to the (condtonal) Posson dstrbutons of X j and X k. Ths s more complcated than n Channouf and L Ecuyer (2012), who could use the root-fndng methodology of Avramds et al. (2009b), whch does not apply to our case. To approxmate the root, we use a stochastc approxmaton (SA) root-fndng method n whch we estmate the multvarate expectaton by Monte Carlo for each value of ρ Z j,k that s consdered. The algorthm s gven n Secton C. For a recent coverage of stochastc root fndng methods, see Pasupathy and Km (2011). As s typcally the case when a large correlaton matrx s estmated from data, the matrx whose entres are the ρ Z j,k just obtaned may not be a vald (postve defnte) correlaton matrx. In our experments, these matrces were always ether postve defnte or only slghtly negatve defnte. Those that were not postve defnte were modfed slghtly nto postve defnte ones by applyng a small perturbaton usng the heurstc of Davenport and Iman (1982), whch fnds a vald correlaton matrx whch s as close as possble to the matrx wth entres ρ Z j,k. Ths method was also used successfully by Channouf and L Ecuyer (2012).

15 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) Parametrc models for the correlaton matrx In the PGnorta model presented so far, there are p(p 1)/2 correlatons to specfy n R Z. Ths can be too many when p s large, and may open the door to overfttng. To prevent ths, we can parameterze the matrx R Z by a small number of parameters, then estmate those parameters. Here we consder two such parametrc models, namely PGnortaAR1, where entres of the correlaton matrx R Z are assumed to obey an AR(1) process, ρ Z j,k = ρ j k, and PGnortaARM, where the AR(1) process s extended to ρ Z j,k = aρ j k + c for j k, and ρ Z j,j = 1. Channouf and L Ecuyer (2012) have used successfully smlar parameterzatons n ther model for X and ponted our ther usefulness for the stuaton where the correlaton between the counts may drop sharply between lag 0 and lag 1, then decays slowly as a functon of the lag j k, and does not approach 0. We have observed ths type of behavor n some of our data sets. It can happen when some factor has a strong mpact over the entre day. The parameters of these two new models are estmated by matchng the models to the correlaton matrx estmated by the algorthm of Secton C, usng least-squares fttng. 6. Case Studes In ths secton we report the results of fttng the dfferent models dscussed prevously to real data sets obtaned from three call centers located n Canada. The frst one s a 24-hour emergency call center, the second one s the commercal call center consdered n Ibrahm and L Ecuyer (2013) and the thrd one s the call center of the Quebec electrcty provder, Hydro-Québec. In all cases, our data comprses only a subset of the call types. We dstngush the dfferent days of the week, but assume otherwse that the arrval rates have the same dstrbutons across successve weeks. We wll see that notwthstandng the dfferent nature of these call centers and the fact that the data they generate have dfferent statstcal patterns, the results of the ft are qualtatvely consstent across dfferent datasets. We compare the statstcal performance of the followng nne models defned earler: Posson, PGndep, PGsngle, PG2, PG2sp, PG2pow, PGnorta, PGnortaAR1, and PGnortaARM An Emergency Call Center The emergency call center operates 24 hours a day for 7 days a week. We had access to the callby-call data (exact arrval tme of each call) over 616 successve days. The center receves calls categorzed n several dozen types. For the results reported here, we selected a subset of those types for whch the daly patterns were smlar and we consder the aggregated arrval process for those types. These results are representatve of a larger set of statstcal analyzes performed over dfferent

16 16 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) Mean Count Perod Fgure 1 Mean count per perod for the emergency center. subsets and over ndvdual call types havng suffcently large volume. Confdentalty agreements prevent us from provdng further detals. The days are dvded nto p = 48 half-hour perods. At frst, our days started and ended at mdnght, but we found n the data that there s often sgnfcant postve correlaton between the call volume n the evenng (say between 8 p.m. and mdnght) and the call volume durng the followng nght (say between mdnght and 5 a.m.), whch could not be captured well by our models when startng the days at mdnght. Ths type of dependence across successve days was n fact causng estmaton artfacts such as spurous humps n the correlaton curves between past and future arrval volumes durng the day, due to the correlatons between parts of dfferent days n the tranng dataset and not by the wthn-day effects. Ths problem was resolved by startng the days at 5 a.m. nstead of mdnght. Around 5 a.m., the traffc s usually very low and there s very lttle correlaton between the arrval volumes before and after 5 a.m. Holdays are an excepton to ths rule: the call volumes are usually larger durng the nght before a holday, and smaller n the mornng of the holday. For ths reason, n a preprocessng phase before fttng our models, we removed the data correspondng to specal days (Quebec statutory holdays), for whch the arrval volumes and patterns dffer sgnfcantly from the ordnary days. The arrval process for those days would have to be modeled separately. Some descrptve statstcs. Prelmnary analyss of data revealed that Frday, Saturday and Sunday have partcular statstcal patterns dfferent from the rest of the dataset. On the other hand, statstcal characterstcs of weekdays from Monday to Thursday are very smlar. The results reported here are for the data from Monday to Thursday regrouped n a sngle dataset. Ths data represents normal weekdays of the call center. Fgure 1 shows the average number of calls receved per half-hour perod. To see how the DI and SDI behave when we aggregate perods, we defne Y j,d = X j + + X j+d 1

17 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) 17 DI(Yj,d) mn aggregaton 1 hour aggregaton 2 hour aggregaton 4 hour aggregaton SDI(Yj,d) mn aggregaton 1 hour aggregaton 2 hour aggregaton 4 hour aggregaton Perod Perod Fgure 2 DI(Y j,d ) (left panel) and SDI(Y j,d ) (rght panel) as a functon of j, for d = 1, 2, 4, 8, for the emergency center. for j 1 and d p j + 1. Ths represents the count for an aggregaton of d successve perods startng at perod j. Fg. 2 shows DI(Y j,d ) and SDI(Y j,d ) as a functon of j, for d = 1, 2, 4, 8 (.e., tme slots of 30 mnutes to 4 hours). When we ncrease d for a fxed j, we fnd that DI(Y j,d ) ncreases as expected from (3), and that SDI(Y j,d ) remans pretty stable, whch suggests that the B j s are strongly correlated (we are close to a sngle busyness factor model; see (5)) at least wthn 4-hour tme slots. The SDI also depends much on j, whch shows that there s much more overdsperson (for the selected call types) n early mornng, the evenng, and durng the nght, than n the mddle of the day. We also see n the fgure that the curve for d = 8 s hgher than the other three for j between about 30 and 37. Ths may appear to contradct (5). The explanaton s that SDI(Y j,8 ) s for a tme slot of 8 perods that extends on the rght of perod j, and the maxmum of each of the other three curves over ths tme slot s ndeed larger than SDI(Y j,8 ). Fg. 18 n Secton E provdes a pcture of all the correlatons between pars of perods, and also between aggregated perods n blocks of 1, 2, and 4 hours. The correlatons are qute small n ths case compared wth other (typcal) call centers. They are larger n the evenng than n the rest of the day. How the models ft the data. Fg. 3 compares the DI obtaned for the sx models wth the sample DI calculated from the data (also gven as the lower curve n Fgure 2(a)), n each 30-mnute perod j. For the latter, we also provde a 95% confdence nterval for the DI (ndcated by the sold lnes) calculated usng bootstrap from a kernel densty estmator (KDE) of the data sample, wth a Gaussan (normal) kernel and a bandwdth chosen so that the varance assocated wth the estmated densty s equal to the sample varance n the data; see Secton D for the detals. The Posson and PGsngle models are far from matchng the emprcal DI; they both gve an SDI that s always too low. For PGsngle, ths s due to the fact that a sngle parameter s avalable,

18 18 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) Dsperson Index Posson PGsngle PGndep PG2 PG2pow PGnorta Data Perod Fgure 3 Comparson of DI(X j) as a functon of j for dfferent models and for the data, for the emergency call center. namely the varance of the sngle busyness factor, whch s the SDI common to all perods n the model. Ths parameter also affects the correlaton between the counts across perods. If t was set hgher, so that the average SDIs would match the average SDI n the data for example, then these correlatons would be much too large compared wth those n the data. The MLEs make a compomse. The PG2 model mproves the DI, but the values dffer from those n the data and wander around the boundares of the confdence nterval. The PG2pow and PGnorta provde a much better ft. Fg. 4 shows the correlaton between the demand Y 1,j n the frst j perods and the demand Y j+1,p j n the remanng p j perods, as a functon of j, for the data and for the models where ths correlaton s nonzero. The sold lnes ndcate 95% confdence ntervals calculated from the data usng the same KDE bootstrap methodology as n the prevous plot, but wth a KDE based on a two-dmensonal Gaussan kernel (see Secton D). We see that only PGnorta has suffcent flexblty to ft the correlatons. The smplfed model PGnortaARM also does well, wth much fewer parameters, whle PGnortaAR1 sgnfcantly underestmates the correlatons. PGsngle and PG2 are sgnfcantly off. PG2pow does better and captures the shape of the correlaton curve, but t does not ft the correlatons accurately. To test the qualty of ft of both the dsperson and correlaton n a combned way, we compare some quantles of the dstrbuton of the partal demand over the two-hour nterval (four perods) startng at j, Y j,4, for the emprcal dstrbuton of the data and the dstrbutons mpled by the models. For each model, we computed a predcton nterval (PI) (Gesser 1993) whose boundares

19 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) Corr(Y1,j, Yj+1,p j) PGsngle PG2 PG2pow PGnorta PGnortaAR1 PGnortaARM Data Perod Fgure 4 Comparson of the sample coeffcent of correlaton of past and future demand for the models wth correlaton, for the emergency call center % Partal volume cover Posson PGsngle PGndep PG2 PG2pow PGnorta PGnortaAR1 PGnortaARM Perod Fgure 5 Comparson of the emprcal coverage probablty of a 95% PI of the partal demand Y j,4, for dfferent models, for the emergency call center. are the and quantles of the mpled dstrbuton, and then computed the emprcal coverage probablty of ths PI, defned as the fracton of observatons of Y j,4 n the data that fall n the nterval. Ths coverage should be close to A coverage smaller than 0.95 ndcates that the model underestmates the dsperson, and vce-versa. Larger postve correlatons across perods, or larger varances of the counts wthn perods, tend to ncrease the varance of Y j,4, so f a model fals

20 20 Artcle submtted to Operatons Research; manuscrpt no. (Please, provde the manuscrpt number!) to properly capture any of these two effects, the coverage should devate from 0.95, sgnallng an ncorrect dstrbuton for Y j,4 n the model, whch n turn would lead to an ncorrect dstrbuton of the watng tme dstrbuton and of performance measure estmates when we smulate the model. Fg. 5 shows the emprcal coverage probablty for each model. We see that PG2pow and PGnorta best capture the dstrbuton of the partal demand, and PGnorta s the best performer. For the other four models, the PI coverage s too small n ntervals 30 to 45, whch correspond to the counts n perods 30 to 48. These are the perods wth the largest DIs (see Fg. 3). Recall that a larger DI s assocated wth a stronger departure from the Posson dstrbuton and larger correlatons between the X j s for gven correlatons between the B j s, as we have seen earler. As expected, PGndep performs poorly here even though t models well the DI for one perod at a tme (see Fg. 3), because t totally neglects the correlaton across perods. PGsngle fals because t largely underestmates the DI n ths regon. PG2 provdes only a small mprovement. The PG2pow model, even though t does not exactly capture the correlatons of the partal demand (see Fg. 4), s reasonably close to the 95% percent target n all segments. Ths suggests that ths model, whose number of parameters remans small, could be a reasonable choce n some stuatons such as the one llustrated here. An out-of-sample goodness-of-ft analyss. In what we have seen so far, PGnorta appears to be the best performer overall. It s the only model that matches both the varances and correlatons n the data, and provdes and quantles of the partal demand Y j,4 that match those of the data, n all cases. On the other hand, ths model has many more parameters (the entre correlaton matrx R Z ) than the other ones, so t could perhaps overft the data. For a farer comparson, we now examne how the models perform for out-of-sample dstrbutonal forecasts, usng a leave-one-out technque as follows. For each, we remove day from the data set, re-estmate the model wthout that day, and compute a PI [L,j,d, U,j,d ] wth nteger bounds, n whch Y j,d wll fall wth probablty P,j,d 1 α under ths model, where 1 α s a fxed number selected n advance. We do ths for each, j, and selected values of d. The PI boundares are ntegers because Y j,d s always an nteger, and for ths reason t s generally not possble to select the bounds so that the probablty of fallng n the nterval s exactly equal to 1 α. The dfference can be sgnfcant when Y j,d has a small mean. Then we compute the proporton of days for whch the realzaton Y,j,d of Y j,d for the removed day falls n the nterval, and we compare ths proporton wth our best estmate of P,j,d, usng a sum of squares crteron, as we now explan. To estmate the quantles L,j,d and U,j,d, we smulate n = 10 5 replcates of the vector of counts (X,1,..., X,p ) for day from the model estmated wth day removed, compute Y,j,d for each (j, d) of nterest and each replcate, say y,j,d,k for replcate k, then we compute the ntegers L,j,d