Setting Accuracy Targets for. Short-Term Judgemental Sales Forecasting

Transcription

1 Seing Accuracy Targes for Shor-Term Judgemenal Sales Forecasing Derek W. Bunn London Business School Sussex Place, Regen s Park London NW1 4SA, UK Tel: +44 (0) Fax: +44(0) dbunn@london.edu and James W. Taylor Saïd Business School, Universiy of Oxford 59 George Sree Oxford OX1 2BE, UK Tel: +44 (0) Fax: +44 (0) james.aylor@sbs.ox.ac.uk Inernaional Journal of Forecasing, 2001, Vol. 17, pp

2 Seing Accuracy Targes for Shor-Term Judgemenal Sales Forecasing Absrac Tradiionally, he qualiy of a forecasing model is judged by how i compares, in erms of accuracy, o alernaive models. However, by providing a relaive measure, no indicaion is given as o how much scope here migh be for improvemens beyond he benchmark model. When judgemenal mehods are used alongside simple forecasing models, he scope for such improvemens is considerable and difficul o benchmark. Derivaion of arges for forecasing qualiy is hus no sraighforward. The approach aken in his paper is o consider forecas error as consising of irreducible error due o inrinsic unpredicable uncerainy, and error due o less han perfec modelling, esimaion and forecasing. As he inrinsic uncerainy presens a bound on forecas accuracy, our derivaion of an accuracy arge is based on he measuremen of his irreducible uncerainy. The moivaion and daa for his case-sudy was aken from he shorerm sales forecasing process of a major, inernaional high-echnology manufacurer. Keywords: Accuracy arges; Judgemenal forecasing 1

3 1. Inroducion This research was moivaed by an organisaion ha wished o implemen a qualiy iniiaive hroughou is producion and invenory managemen. Like many companies, his organisaion sough o encourage a qualiy culure in heir operaions by he seing of arges for a number of key measurable aciviies (Juran and Gryna, 1993). However, one imporan aciviy ha presens special problems for such qualiy arge-seing is shor-erm sales forecasing. This paper addresses his problem, using he company as a case sudy. Benchmarking agains indusry leaders, and op performing companies in similar funcional areas in oher indusries, is worhwhile for arge-seing in many insances of oal qualiy managemen (Hradesky, 1995). However, cross-company comparisons have no generally been relevan, nor feasible, in he area of seing forecasing qualiy goals. Company specific and company sensiive marke issues ofen preclude his. Furhermore, when we look a he research lieraure on forecasing, i is eviden ha he focus is more upon models han processes, and ha he qualiy of a forecasing model ends o be judged by how i compares, in erms of accuracy, o a reasonable alernaive saisical model. However, he value of such a comparison clearly depends on he qualiy of he benchmark model. Moreover, as research has shown ha he fi of a model o hisorical daa is no always a good guide o he pos-sample accuracy of he model (Makridakis, 1986; Pan and Sarbuck, 1990), forecasers have been advised o judge accuracy based on pos-sample predicion error. Thus, we have seen many published sudies deriving he pos-sample forecas errors from a variey of saisical models (such as he M-Compeiion, Makridakis e al., 1982). However, o he exen ha he process of mos business forecasing in pracice involves considerable wellinformed judgemenal adjusmens o simple ime series mehods, or may indeed be mosly judgemenal, his research is herefore quie limied for he ask of qualiy arge-seing. Indeed, i is clear ha in circumsances where judgemenal inpus are of proven value in forecasing, he usefulness of saisical model benchmarking is, a bes, o provide lower qualiy 2

4 bounds on performance. This, herefore, sill leaves open he issue of assessing upper bounds which are heoreically feasible, bu srongly challenging and can hereby provide a viable moivaion for managerial forecasers. To address his, we presen a mehodological framework which considers he forecas error associaed wih a predicion o consis of wo componens: he irreducible error due o he inrinsic unpredicable uncerainy in he variable, and he error due o less han perfec modelling and esimaion. The inrinsic uncerainy clearly presens a bound on he accuracy of he forecasing process. Hence, our derivaion of an upper bound is based on he esimaion of his irreducible componen of uncerainy in he daa. The analogy is wih he sudy of physical sysems, where observaion noise can be seen as an upper limi o he accuracy of sysemaic measuremens. This concep has been exended in forecasing research. For example, Bunn and Seigal (1983) found ha here was an upper bound on he accuracy of minue-by-minue elecriciy load forecass due o load measuremen problems and used his as a basis for assessing he performance of various shor-erm predicors. Compared o a measure of ex pos accuracy, which evaluaes he forecas agains he acual ou-urn, he proposed qualiy arge is clearly more reasonable, bu is sill an idealised upper bound on performance. Measures of acual ex pos accuracy are, of course, essenial for monioring and, as we have observed, simple model based comparisons can provide a reasonable lower bound on performance. I would seem reasonable, herefore, o evaluae he usefulness of qualiy arge bounds where he upper ones are based upon esimaes of irreducible uncerainy, and he lower ones are derived from a simple ime series model (e.g. random walk). We applied his approach o he qualiy iniiaives of our collaboraing company. This company operaes worldwide in he fas changing, high-echnology secor, selling a range of personal compuers direcly o consumers, eiher by elephone or inerne. They hold no invenories of finished goods, jus componen pars, and assemble o order. The producs have quie shor life cycles, wih sales very dependen upon pricing and adverising. Their forecass 3

5 are mosly judgemenal esimaes, using sales force knowledge plus marke informaion on produc innovaions and promoions, agains a background of daily monioring of underlying sales rends per produc line. In his respec he case sudy described here is ypical of a more general class of consumer forecasing problems where here is high frequency daa (eg EPOS, inerne or phone), and a necessarily subsanial judgemenal componen. The hypohesis of his sudy is ha, in he spiri of TQM, a forecas qualiy arge, implemened wih regular monhly feedback, will moivae and monior improved forecasing hroughou he company and ha such a arge could consis of wo bounds. The upper bound could be an esimaor which forecass wih error due only o inrinsic uncerainy, whils he lower bound could be a naïve saisical mehod based upon a random walk. I is imporan o undersand ha his paper is no concerned wih he problem of esimaing predicion inervals. A predicion inerval conveys he inerval wihin which an acual ou-urn is likely o fall wih a given probabiliy, such as 95%. A qualiy arge is a measure ha one mus ry o aain. Since forecas qualiy is assessed by accuracy measures, such as MSE, his paper aims o provide a mehodology for deriving a value for he measure which would serve as a qualiy arge. In secion 2, we consider he lieraure on forecas accuracy measures, in order o provide an appropriae meric for hese bounds. We presen he company s own meric and hen, in secion 3, we presen a framework for deriving bounds on forecas accuracy. Secion 4 discusses he limiaions of an analyical approach o assessing he accuracy bounds, whils secion 5 presens an alernaive approach, which uses Mone Carlo simulaion. Secion 6 repors he applicaion of he simulaion procedure o he company s daa and he final secion offers some concluding commens. 4

6 2. Evaluaing Forecas Accuracy 2.1. Forecas Accuracy Measures In a survey of praciioners and academicians, Carbone and Armsrong (1982) found ha he mos preferred measure of forecas accuracy is he Mean Square Error (MSE). Chafield (1992) wries ha, for a single series, i is perfecly reasonable o fi a model by leas squares and evaluae forecass from differen models by he MSE. However, he MSE has been broadly criicised for use in comparing forecasing mehods across series, as i can be disasrous o average MSE from differen series since i is scale-dependen. Because of his, and is poor proecion o ouliers, Armsrong and Collopy (1992) recommend agains using he MSE. They found ha for selecion among forecasing mehods, he MdRAE, GMRAE and MdAPE are o be preferred o he MSE. The MdRAE is he median value of he relaive absolue error (RAE). This is calculaed for a given ime series by dividing he absolue forecas error, a a given horizon, for a proposed model by he corresponding error for he random walk. The GMRAE is he geomeric average of he RAE values. The MdAPE is he median absolue percenage error which has he advanage of having a closer relaionship o decision making. In his conex, Fildes (1992) argues ha forecas comparisons, based on a populaion of ime series, should no rely on a single ime origin for each series bu, insead, he error measure should be averaged across ime in some way, as well as across differen series. Of he alernaives considered, he found ha only he geomeric roo mean squared error (GRMSE) is well-behaved and has a sraighforward inerpreaion. Makridakis and Hibon (1995) disagree, arguing ha he GRMSE has very lile inuiive meaning as i involves squared erms, producs and geomeric roos. Their sudy evaluaes various accuracy measures using wo saisical and wo user-oriened crieria. They conclude ha selecing amongs hem depends upon he siuaion involved and he needs of decision or policy makers. They noe ha such a choice canno be made wihou radeoffs. Clearly his issue is sill a conroversial area in forecasing research. 5

7 2.2. The Company s Accuracy Measure Reurning o he conex of our case sudy and he company involved, afer much deliberaion concerning ease of inerpreaion and monioring, he forecas accuracy measure adoped by he company was ( f / x ) 100 if f < x and (1) ( x / f ) 100 if f > x where x is he acual value and f is he forecas. We have ermed his he similariy percenage (SP). If a summary measure is required for forecas accuracy for several periods, he mean or median of he SP values can be used. The median would be more consisen wih he recen forecas evaluaion lieraure as i would be more robus o ouliers. The SP seemed o fi he company s requiremens for a qualiy measure since i varies beween zero and 100%, wih a higher percenage reflecing more accurae forecasing. However, i is no one of he merics usually discussed in he lieraure and his is probably because of is ambiguous inerpreaion. For example, an SP of 80% can resul from wo possibiliies: eiher f is greaer han x and he forecas error is 20% of f, or f is less han x and he forecas error is 20% of x. Neverheless, his seemed o make sense o he company in erms of he relaive coss of over-socking versus los sales. An advanage of he SP is ha, since i is in percenage erms and no scale dependen, i can be averaged across boh ime periods and ime series. The measure seems o have one advanage over he absolue percenage error (APE), which is he mos widely used percenage measure. Makridakis (1993) noes ha he APE has he problem ha equal errors when x is larger han f give smaller percenage errors han when x is smaller han f. (For example, when x is 150 and f is 100, he APE is 33.33%, however, when x is 100 and f is 150, he APE is 50%.) This difference can creae serious problems when he value of x is small (close o zero) and f is large, as he size of he APE can become exremely large making he comparisons among 6

8 horizons and among series difficul. This ype of asymmery is no a problem, however, for he similariy percenage (SP) and, furhermore, he SP can never be greaer han 100% (for posiive values of x and f ). Ineresingly, he modified MAPE, proposed by Makridakis (1993) o overcome his ype of asymmery, has recenly been shown by Goodwin and Lawon (1999) o be far from symmeric in oher respecs. This may also be he case for he SP. The main conribuion of our work is no affeced by he choice of accuracy measure. Thus, since his paper is essenially a case sudy, and he company had already esablished he pracice of working wih he SP measure, he accuracy arges described below, which we developed and were implemened by he company, were mos appropriaely expressed in hese erms. 3. Framework for Deriving Accuracy Targes 3.1. The Componens of Forecas Error Having chosen an accuracy meric as a measure of forecas qualiy, we now address he problem of deriving arge values for he meric. The way ha we approach he problem is o esimae he limis on he accuracy of he company s sales forecass. Our mehodological framework considers he forecas error associaed wih a predicion as consising of wo componens: he irreducible error, e, due o he inrinsic unpredicable uncerainy in he variable, and he error, ε, due o less han perfec esimaion (i.e. forecasing). We can hen wrie he forecas error as x - f = e + ε Improvemen in forecasing will reduce ε bu i canno reduce he inheren uncerainy, e. Consequenly, an ideal predicor will have ε = 0 and i is clear ha e implies a bound on forecas accuracy. Suppose we are able o esimae he variance, σ 2, of e, and we assume ha e is normally disribued abou zero. We can hen say ha, wih 5% probabiliy, e will fall ouside he inerval [-1.96σ,1.96σ ]. Since ε = 0 for a perfec predicor, we can say ha he forecas 7

9 error for a perfec predicor would be expeced o fall wihin hese bounds 95% of he ime. This could hen serve as a arge for forecas accuracy. I is imporan o noe ha hese 95% limis are confidence limis for a perfec predicor and are, herefore, no sandard predicion inerval limis. In secion 4, we adap his idea for esimaing arges o obain bounds for he accuracy meric employed. Firs, however, we address he problem of assessing he variance, σ 2, of he inrinsic uncerainy, e, as his is fundamenal o he approach Esimaing he Inrinsic Uncerainy The problem, as posed by he company, is o esablish saisical arge bounds for calendar monh sales forecas accuracy measures based upon some measure of he inrinsic randomness in he daa. Thus, we wish o esimae he irreducible componen of uncerainy in he monhly values. The concepual framework ha we presen for esimaion is new and relies upon he availabiliy of daa wih frequency ha is higher han monhly daa, for example, daily or weekly observaions. The general proposiion is ha measuremens aken a higher frequency han ha posulaed for he underlying srucural process can provide a means of esimaing inrinsic randomness. The analogy is wih esimaing noise in physical sysems, from repeaed measuremens wih he same conrol variables. In our case sudy, he srucural process proposed for he daa by he company is ha of monhly shifs in demand which need o be forecased, bu weekly or daily variaions wihin he monh which exhibi inrinsic randomness abou ha monhly mean. Clearly, more complex srucural models could be developed impuing wihin monh rends, bu he daa series invesigaed here did no seem o warran he exra parameerisaion, a leas for a firs analysis. Produc life-cycles were shor and he monhly mean shif seemed o be an appropriae srucural assumpion. Thus, if average sales per week is correcly forecas, here would sill be saisical error in he monhly oal due o random week-by-week variaions abou his average. This variaion 8

10 could be used o esimae he inrinsic uncerainy in he monhly oals, and his esimae could hen be used o provide saisical bounds for monhly accuracy. The key underlying sabiliy assumpion is ha each week s value is varying independenly abou he same weekly average wihin a calendar monh, alhough we would expec small average changes from one calendar monh o anoher. The variance, σ 2, of he inrinsic uncerainy, e, in he calendar monh oal, x, is hen calculaed from he four weekly values, x 1, x 2, x 3 and x 4, hus σ 2 = var( x 1 + x 2 + x 3 + x 4 ) = 4σ w 2 (2) where σ 2 w is he variance in a weekly sales value. This variance, σ 2 w, is esimaed as he variance of he four values x 1, x 2, x 3 and x 4. Expression (2) relies on he assumpion ha he four weekly sales values are no correlaed. This was indeed rue for our company s daa which implies ha ime series forecasing mehods will be ineffecive. I is herefore no surprising ha he company uses judgemenal mehods o produce sales forecass for hese shor life-cycle producs. Alhough his framework is mos suiable for judgemenal forecasing, i is worh noing ha i can be adaped o he siuaion where sable or susained paerns do exis in he daa. In such cases, he underlying paern should be esimaed and hen he weekly randomness from i used o esimae inrinsic uncerainy in he monhly oals. Whils only weekly daa was available for he individual producs, daily daa was available for he oal sales. We calculaed he inrinsic variances for he oal monhly sales using weekly daa and compared he resuls o hose using daily daa. For monhly oals, he inrinsic variances compued from weekly and daily daa seemed sufficienly close o jusify calculaion of he monhly inrinsic variances for he individual producs using he weekly daa. We also invesigaed he significance of iner-correlaions beween sales of producs on a weekly basis, o see he exen o which saisical dependencies needed o be aken ino accoun when looking a averages of he accuracy measure across producs. Our analysis indicaed ha he 9

11 iner-correlaions could be omied from he analysis as hey did no appear o have a major effec. The idea of using higher frequency daa o esimae he variance in lower frequency daa is no enirely original. For example, i has been used in financial applicaions o evaluae volailiy forecass (e.g. Day and Lewis, 1992). As volailiy is unobservable, i is no obvious wha o use as he acual wih which o compare he forecas. Suppose we wish o evaluae forecass of weekly volailiy. If daily observaions of he reurns are available, a proxy for he acual weekly-reurn variance is creaed by compuing he daily variance from he daily reurns, and hen muliplying his by he number of rading days in he week. 4. Analyical Approach o Assessing Accuracy Bounds for SP We can work owards a bound on accuracy by considering he limi on he forecasing performance of an ideal predicor. The forecass, p, of an ideal predicor have ε = 0, so ha x - p = e If we assume ha e is normally disribued, we can say ha wih 5% probabiliy, e will fall ouside he inerval [-1.96σ,1.96σ ]. Using his, and recalling he definiion of he similariy percenage (SP) given in expression (1), we can make he following probabiliy saemens for he forecas of he ideal predicor: Since p > x +1.96σ wih probabiliy of 2.5%, hen wih probabiliy of 2.5%: x < p and x p < x x σ Since p < x -1.96σ wih probabiliy of 2.5%, hen wih probabiliy of 2.5%: x > p and p x < x 1.96σ x We can hus say ha, wih probabiliy less han 5% and greaer han 2.5%: 10

12 x x 1.96σ x 1.96σ SP of he Perfec Predicor < min, 100 = x + σ x x (3) The value of SP for he ideal predicor will hus be greaer han he expression in (3) beween 95 and 97.5% of he ime. If he value of SP for our forecas is found o be greaer han he expression in (3), hen we could conclude ha i is no significanly less accurae han he ideal predicor. The expression in (3) could hen be used as a forecas accuracy arge. However, his analyical approach o assessing bounds for he accuracy measure is unsaisfacory for wo reasons. Firsly, we may require an upper bound corresponding o a paricular percenage. However, he analysis is only able o supply he imprecise probabiliy saemen for he SP of he perfec predicor wih probabiliy less han 5% and greaer han 2.5%. The second reason concerns he company s addiional requiremen ha accuracy arges are also derived for he weighed average (WSP ) of he similariy percenage for monh, averaged across differen producs. WSP = i w i SP i where w i = i fi f i f i is he company s forecas for sales of produc i in monh, and SP i is he similariy percenage for f i. We require a 5% bound for WSP for a paricular monh. Unforunaely, i is impossible o form probabiliy saemens, as in he previous secion, for a weighed average of he accuracy meric. Consequenly, i is impossible o form a probabiliy saemen for WSP i. Wih he complexiy of he problem limiing he usefulness and feasibiliy of heoreical analysis, recourse o simulaion provides a pracical alernaive. 5. Simulaion Approach o Assessing Accuracy Targes for SP and WSP Upper bounds may be derived for he accuracy measure by simulaing he acual sales, x i, for produc i in monh. We proceed by considering he observed acual as being a random variable consising of a non-sochasic expecaion componen, E(x i ), plus an inrinsic error 11

13 erm, e i. Having esimaed he sandard deviaion, σ i, of e i, we are hen in a posiion o simulae values of x i as x i = E(x i ) + e i where e i is a value derived by Mone Carlo sampling from a normal disribuion wih zero mean and sandard deviaion σ i. We use he observed acual as E(x i ). The ideal predicor, p i, is hen modelled as a perfec predicor of E(x i ) so ha he only error is due o he inrinsic uncerainy. p i = E(x i ) For each simulaed value of x i, we record he value of he similariy percenage, SP i : ( p i / x i )% if p i < x i and ( x i / p i )% if p i > x i We also record he weighed similariy percenage for he ideal predicor using he simulaed acuals for a paricular monh, WSP = i w i SP i where w i = i pi p i By repeaedly sampling from he disribuion for e i, we produce a disribuion for SP i and for WSP. The 5h perceniles of he resulan probabiliy disribuions can hen be inerpreed as respecive upper bounds for he similariy percenage and weighed similariy percenage of he company s sales forecass. They have been compued a he 95% confidence level, in he sense ha, wih ideal forecasing of he monhly means, inrinsic variaion would imply ha one s forecas meric would be less han he bounds only 5% of he ime. The bound is herefore he limi of he accepance region corresponding o he one-sided hypohesis es wih null ha he forecas is a leas as accurae as he perfec predicor. I is worh reieraing ha he simulaion mehodology described in his secion aims o generae he disribuion for he similariy accuracy measure for a perfec predicor. We hen use his probabiliy disribuion o provide qualiy arges for he similariy accuracy measure for he 12

14 company s forecass. Of course, hese forecass have no been used o consruc he accuracy arges which, herefore, remain fixed over ime. I is hoped ha hese arges will moivae improvemen in he company s forecas accuracy over ime. Alhough he main aim of his paper is o derive upper bounds for forecasing accuracy, which can be inerpreed as qualiy arges, i seems useful o also consider a general approach o he derivaion of lower bounds for accuracy. The random walk is ofen aken as a benchmark agains which o es he success of oher forecasing mehods. Raher han simply calculaing he accuracy merics (SP and WSP) for random walk forecasing for each monh, we used he simulaed acuals o build up a disribuion for he accuracy merics for he random walk. The company s forecass are made wih a lead ime of hree monhs. In view of his, we used he simulaed acual for monh -3 as he random walk forecas for produc i in period. By recording he SP for each simulaed acual and he WSP for each monh, we generaed a disribuion for he SP and WSP for random walk forecasing for each produc in each monh. A sensible probabiliy saemen regarding a lower bound would be ha he forecas should no be significanly worse han he random walk. In oher words, he accuracy meric for he company s forecas should no be significanly lower han he accuracy measure for he random walk. The lower bound for accuracy would hen be he 5h percenile of he probabiliy disribuion of he simulaed accuracy measure for he random walk. 6. Case Sudy Resuls We had daa for 11 monhs. We used 1,000 ieraions in he Mone Carlo simulaions. In oher words, we produced a housand simulaed acuals from which we calculaed he SP and WSP for he ideal predicor and for he random walk. The 5h perceniles of he resulan disribuions were hen used as upper and lower bounds, respecively, on accuracy. The upper bound can serve as a arge for forecas qualiy. 13

15 The company groups all of is producs ino one of hree differen caegories. The weighed accuracy merics are calculaed separaely for each caegory. The resuls of he simulaions for he WSP for caegory 1 are given in Table 1. Figure 1 shows he disribuion generaed for he WSP of perfec predicor forecasing for he producs in caegory 1 in monh 1. I is easy o confirm ha he 5h percenile is approximaely 91%, as repored in Table 1 for monh 1. The negaive skewness of he disribuion was ypical for all he WSP disribuions. Briefly, he resuls indicaed ha he upper bound or arge value for he weighed similariy percenage (WSP) varied beween abou 80 and 90%. The company s forecass in his daa se generally had a WSP which was abou 30% lower. A noiceable difference beween he recorded accuracy and he arge is o be expeced as he arges are ulimae and are no realisically achievable. Raher more surprising are he values of he company s WSP relaive o he lower bounds. The resuls indicaed ha in more han half he cases he company s WSP is below he corresponding lower bound and hereby appeared o be no beer han a random walk model. However, in resricing he daa se o sable monhs, and requiring a hree monh lead ime, he sample size was small, already being judgemenally filered. Neverheless, since performance on his daa se was closer o he boom han he op of he arge bands, i did cause he company o hink abou more sysemaic ways of using he subjecive sales and markeing informaion. Overall, he main resul of his sudy was o sugges ha a arge of 85% accuracy in he WSPs should be used for performance monioring in he company and ha a qualiy iniiaive should susain a seady improvemen owards ha goal. Indeed in he six monhs ha followed he seing of a qualiy arge, he company did manage o move is average monhly WSPs o above 80%. Feedback on acual accuracy had been given o forecasers for several years, bu i seems ha being associaed wih a arge provided he exra moivaion for qualiy improvemen. Furhermore, i does seem ha qualiy improvemen is essenially in he forecasing of demand, raher han in he sales eams managing heir ou-urns. The forecass were esimaed cenrally, 14

16 whereas he sales were almos compleely made hrough elephone call cenres during he period of his sudy. Thus, given he shor-lead imes involved, he lack of personal conac beween he sales eam a he call cenres and mos of he cusomers and he fac ha hey had no discreion on discouning lis prices, here would appear o have been relaively lile scope for he sales eam o have influenced he aainmen of he forecass TABLE FIGURE Concluding Commens Based upon he assumpions of inrinsic randomness, we have presened a simulaion framework for deriving an upper and lower bound for he weighed accuracy measure. In he acual case sudy, he approach assumed ha here is unforecasable week-by-week variaion wihin each monh, bu ha he average, from which hese weeks are saisical oucomes, is predicable. The upper bound was compued a he 95% confidence level, in he sense ha, wih ideal forecasing of he monhly means, he forecas meric should achieve his bound 95% of he ime. I is, of course, unreasonable o assume ha, 3 monhs ou, produc ransiion effecs, media influences, logisics and oher evens can be anicipaed perfecly, and so we have o see he upper bound as an ideal arge agains which o moivae qualiy improvemens in forecasing; i should no be seen as a benchmark ha should be regularly aainable. Juran (1988) discusses how i should no be assumed ha qualiy goals can always be me. Neverheless, hey do provide a basis for moivaion. In his case, hey succeeded where simple feedback had failed, and qualiy improvemens in forecasing were achieved relaive o he arge. The qualiy arge bounds are, herefore, ulimae and are no realisically achievable every monh. In order o posiion forecasing performance wihin he range of aainable 15

17 accuracies, we calculaed lower bounds. The random walk model is ofen aken as a benchmark agains which o es he success of oher forecasing mehods. I essenially assumes ha fuure changes from he curren level are unpredicable and ha he bes forecas is he laes value. The lower bound was specified so ha if he accuracy measure falls below ha value, one can conclude wih 95% confidence ha he forecas has been ouperformed by he random walk. As we saw in his example, hisorical performance close o he lower bound can also insigae greaer effors a forecas improvemens. Finally, i should be observed ha his is a "bounding" exercise, looking a saisical variaion from wo exremes. The upper assuming ha he daa is predicable 3 monhs ou excep for some shor-erm, inrinsic week-by-week variaion. The lower assuming ha changes from he laes value are no a all predicable. I is no, herefore, a benchmarking exercise in erms of seeing wha sor of accuracy oher commonly used echniques, such as exponenial smoohing, could give. Nor does i address how a srucured process of forecasing qualiy improvemen migh be implemened. Neverheless, he saisical bounds derived here did provide a framework wihin which o moivae improvemens in forecasing. The principle is generalisable o more complex srucural models, such as rending and life-cycle models, and may well be he mos appropriae way o se qualiy arges for forecass which involve subsanial judgemenal inpus. Acknowledgemens We would like o acknowledge he helpful commens of an associae edior and wo anonymous referees. 16

18 References Armsrong, J. S. & Collopy, F. (1992). Error Measures for Generalising Abou Forecasing Mehods: Empirical Comparisons, Inernaional Journal of Forecasing 8, Bunn, D. W. & Seigal, J. P. (1983). Forecasing he Effecs of Television Programming upon Elecriciy Loads, Journal of he Operaional Research Sociey 34, Carbone, R. & Armsrong, J. S. (1982). Evaluaion of Exrapolaive Forecasing Mehods: Resuls of a Survey of Academicians and Praciioners, Journal of Forecasing 1, Chafield, C. (1992). A Commenary on Error Measures, Inernaional Journal of Forecasing 8, Day, T. E. & Lewis, C. M. (1992). Sock Marke Volailiy and he Informaional Conen of Sock Index Opions, Journal of Economerics 52, Fildes, R. (1992). The Evaluaion of Exrapolaive Forecasing Mehods, Inernaional Journal of Forecasing 8, Goodwin, P. & Lawon, R. (1999). On he asymmery of he symmeric MAPE, Inernaional Journal of Forecasing 15, Hradesky, J. (1995). Toal Qualiy Managemen Handbook, McGraw-Hill, New York, Juran, J. M.. (1988). Juran on Planning for Qualiy, The Free Press, New York, ch. 8. Juran, J. M. & Gryna, F. M. (1993). Qualiy Planning and Analysis - From Produc Developmen Through Use, 3rd Ed, McGraw-Hill, USA, ch. 8. Makridakis, S. (1986). The Ar and Science of Forecasing: An Assessmen and Fuure Direcions, Inernaional Journal of Forecasing 2, Makridakis, S. (1993). Accuracy measures: heoreical and pracical concerns. Inernaional Journal of Forecasing 9, Makridakis, S. & Hibon, M. (1995). Evaluaing Accuracy (or Error) Measures, Working Paper, INSEAD, Fonainebleau, France. Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newon, J., Parzen, E., & Winkler, R. (1982). The Accuracy of Exrapolaion (Time Series) Mehods: Resuls of a Forecasing Compeiion, Journal of Forecasing 1, Pan, P. N. & Sarbuck, W. H. (1990). Innocens in he Fores: Forecasing and Research Mehods, Journal of Managemen 16,

19 Monhs Average WSP Lower Bound Company s WSP WSP Upper Bound Table 1: Bounds for he WSP for produc caegory 1 18

20 Probabiliy WSP Figure 1: Disribuion for he WSP of he ideal predicor for he producs in caegory 1 in monh 1 19