REVISTA INVESTIGACION OPERACIONAL VOL. 31, No.2, , 2010

Transcription

1 REVISTA INVESTIGACION OPERACIONAL VOL. 3, No., 59-70, 00 AN ALGORITHM TO OBTAIN AN OPTIMAL STRATEGY FOR THE MARKOV DECISION PROCESSES, WITH PROBABILITY DISTRIBUTION FOR THE PLANNING HORIZON. Gouliois E. Joh Deparme of saisics ad isurace sciece Uiversiy of Pireas, 80 Karaoli a Dimiriou Sree, 8534 Piraeus, Greece ABSTRACT I his paper we formulae Markov Decisio Processes wih Radom Horizo. We show he opimaliy equaio for his problem, however here may o exis opimal saioary sraegies. For he MDP (Markov Decisio Process, wih probabiliy disribuio for he plaig horizo wih ifiie suppor, we show Turpike Plaig Horizo Theorem. We develop a algorihm obaiig a opimal firs sage decisio. We give some umerical examples. MSC: 90C40, 90B50, 90C39, 90C5 KEY WORDS: MDPs, opimizaio, probabiliies ad decisio makig, operaio research RESUMEN E ese rabajo formulamos u Proceso de Decisió Markoviao co Horizoe Aleaorio. Desarrollamos la ecuació de opimalidad para ese problema, si embargo puede o exisir esraegias opimales esacioarias. Para el MDP (Proceso de Decisió Markoviao, co disribució de probabilidad para horizoe de plaeamieo co sopore ifiio, demosramos el Teorema de Horizoe de Plaeamieo de Turpike. Desarrollamos u algorimo para obeer ua decisió de primera eapa opimal. Damos alguos ejemplos uméricos.. INTRODUCTION A muliperiod opimizaio problem is ofe modeled as a ifiie horizo problem whe is horizo is log sufficiely. We do o ecessarily kow he horizo of he problem i advace sice we ca o predic he fuure precisely. For example, we imagie vaguely ha a drasic chage of a projec may occur some day, ad we oly believe whe is chage will occur uder a cerai probabiliy disribuio. Thus i is o appropriae ha we simply model he problem as a ifiie or a fixed fiie horizo case. If he plaig horizo chages may cause a remarkable chage of opimal sraegy, ad he oal reward may differ much. Hece, i is ecessary o make a decisio cosiderig he probabiliy of he ime a which he projec will ed. We formulae hese problems usig MDPs i which probabiliy disribuios for he plaig horizo are give i advace, ha is, MDP wih Radom Horizo. I his paper we will cosider o homogeeous MDPs. I is kow oe ypical example wih a geomerically disribued plaig horizo, which is equivale o a ordiary discoued MDP (Ross [9]. We ca oice ha he discou rae represes a evaluaio of uceraiy expeced o be happeed i he fuure. Numerous researches have bee made for he ype of MDP which has variable discou raes (Whie [], Puerma [8], Sodik []. I he MDP wih radom horizo here may o exis a opimal saioary sraegy. Whe he suppor of he probabiliy disribuio for he plaig horizo is fiie, we ca easily ge a opimal sraegy by solvig he correspodig opimaliy equaio. Whe he suppor of he probabiliy disribuio for he plaig horizo is ifiie, i is difficul o solve he problem. So we adop a rollig horizo sraegy o obai a opimal sraegy, ha is, firs we obai he Turpike Plaig horizo for MDP ad solve he problem uder is horizo. Shapiro [] shows he exisece of he Turpike Plaig Horizo for he homogeeous discoued MDP. 59

2 This paper is relaed o he researches of Bea ad Smih []. They reas deermiisic decisio problems. I addiio Hopp, Bea ad Smih [7] cosiders he codiio for he exisece of a opimal sraegy for he o homogeeous o discoued MDP uder a weak ergodiciy assumpio. I Secio, he model is described i deail ad some assumpios are provided. We derive he opimaliy equaio for he MDP wih radom horizo. I his secio he opimal problem is formulaed as MDP. I Secio 3 we describe he srucure ad aure of a opimal sraegy i he case ha he suppor of he probabiliy disribuio for he plaig horizo is ifiie. I secio 4 we give a algorihm for solvig he problem based o he aure derived i secio 3. I secio 5 some coclusios are provided.. MODEL DESCRIPTION AND ASSUMPTIONS. Le (Ω,F,P deoe he uderlyig probabiliy space. Le T { 0,,, } K be he se of oegaive iegers. We cosider a discree ime o homogeeous Markov Decisio Model wih (i Couable sae space S, (ii measurable acio space A edowed wih σ-field, A coaiig all oe-poi subses of A. (iii ses of acio A( s available a s S, where A( s is a eleme of A, (iv rasiio probabiliies { p ( j, i α } a sae, T,where p ( j, i α is oegaive ad measurable i a, ad for each i S, a A( s, p ( j, i a, T, j S (v ses of reward fucios { (, } r i a a sage, T r i a is measurable i a, c i a whe he projec ed a sage, T, where c i a is measurable i a. The salvage cos is he icurred cos o sop he projec ad may deped o he sae ad acio a ha sage. (vi ses of salvage cos fucios { (, } he fucio (,, where he fucio (, Assumpio..For each sage, reward fucios ad salvage cos fucios are assumed o be bouded, ha is, r s, a R c s, a C <+ ( <+, ( Le a fucio a : S A, T ( a, T (,,, be a decisio fucio wih a ( s A( s. The sequece is called a sraegy. Le Δ deoe he se of all sraegies. We also use he oaio 0 K o represe firs decisios i. I his model, we also se, f wih which he projec ed a sage, T (vii a probabiliy disribuio Also we cosider a absorbig sae The we add ex hree o he above (iii, (iv ad (v, (iii A( s { a' }, (iv for ay j S, all T, (, 0 (v for all T r s, a o, (. s ' represeig he ed sae of projec ad le S' S { s' }. p j s a, ( p s' s, a, 60

3 Le H ( S A S be he space of hisories up o he sage T { }, where A A { a }. If a sraegy Δ ad iiial sae s are specified, rasiio probabiliies are deermied compleely. Accordigly a probabiliy measure P s is iduced. Le x deoe he sae of process a sae, M deoe he radom plaig horizo wih disribuio ϕ, A deoe he acio ake a sae, he he expeced oal reward for he -horizo problem is give by ( ( r X, A whe X S, < M R( X, A c X, A whe X S, M 0 whe X S. (. Cosiderig he -horizo problem, for a fixed, whe he process sars wih a iiial sae s uder a sraegy, he expeced oal reward for his problem is give by (,, (, V s E R X A, (. where s s 0 E is he correspodig expecaio operaor. We oice ha he expeced reward V ( s,, depeds oly o he firs decisios i each. Now, we ca describe he -horizo ad M-horizo opimal decisio problem. The -horizo problem is defied as sup V( s,, E R ( X, A for each s. (.3 0 A sraegy ( Δ is called a opimal sraegy for he -horizo problem if for each s S, V s,, sup V s,,. ( ( ( For he radom M-horizo problem, we se (, (, V s E R X A. (.4 s 0 I should be also oed ha a opimal sraegy for -horizo problem depeds oly o he firs decisios i each. Similarly a sraegy Δ is called a opimal sraegy for he radom M-horizo problem if for each s S V s, sup V s,., ( ( Le ε be a arbirary oegaive cosa. The a sraegy ( for he -horizo problem if for each s S, V s, (, V s,, ε ( ε ( ( is called ε - opimal sraegy. (.5 Now we cosider he opimaliy equaio for he MDP wih radom horizo. Le b be a probabiliy which he projec is sill coiuig a sage ( + uder codiio ha i has coiued uil sage, ha is, 6

4 κ κ b φ. (.6 φ κ κ Whe he process is i sae i ad acio a is used a sage, he expeced reward we ge is d i, a b r i, a + b c i, a. (.7 ( ( ( ( Now, le v ( i deoe he maximal value which we ca ge afer he sage whe he process is i sae i, a sage. Therefore we ca ge he opimaliy equaio as follows, v ( max (, (, i d i a + b p j ia v ( j a A +. (.8 j S Whe he suppor of he probabiliy disribuio for he plaig horizo is fiie, we ca easily obai he soluio of he problem as i a ordiary fiie horizo, by applyig he backward iducio mehod o he opimaliy equaio (.8 wih seig v ( i max c ( i, a, for all i S, (.9 a A( i where is a maximal value of he suppor of { φ, T}. 3. OPTIMAL STRATEGIES WHEN THE SUPPORT OF THE PROBABILITY DISTRIBUTION FOR THE PLANNING HORIZON IS INFINITE. I his secio we discuss he MDPs wih radom horizo which have he ifiie suppor of he probabiliy disribuio for he plaig horizo. We discuss he problem based o he idea ha if he opimal sraegies for he fiie horizo problem approach a paricular sraegy for he ifiie suppor problem, we will cosider ha sraegy as he opimal oe. Works of Hopp, Bea ad Smih [7], Bes ad Sehi [3] are based o his idea, oo. We ow defie a meric opology o he se of all sraegies Δ. The meric ρ below is he same oe which Bea ad Smih [] uses ρ (, σ (,, where σ (, ( ( ( ( ' a x a x x S ' 0 a x a x x S The ρ meric has he propery ha ay wo sraegies ha agree i he firs M policies, for ay M, are cosidered closer ha ay wo sraegies ha do o. Now we defie Defiiio 3..A sraegy % Δ is periodic forecas horizo (PFH opimal if for some subsequece i he ρ meric as m. of he iegers { M m}, ( % m M m Proposiio 3..(, ρ Δ is a meric space. If ρ( ', < ε < for all log ε α α. See Bea ad Smih [] 6

5 Assumpio 3..We assume ha he sraegy Space, Δ, is compac i meric space geeraed by ρ. This assumpio precludes he possibiliy of a sequece of feasible sraegies covergig o a ifeasible sraegy. For furher discussio of a relaed problem, see Bea ad Smih []. ' ' Le Δ { a} for all T. We defie he discree opology p (, a a σ ( a, a o hem. The he heorem below holds. Theorem 3.. Δ is compac if ad oly if T, Δ are fiie ses. If Δ is compac, he each cylider subse of Δ is compac. See [3] Theorem 3.3.A periodic forecas horizo opimal sraegy exiss for he ohomogeeous Markov decisio process. { m } Compacess of Δ implies ha he sequece ( M m has a coverge subsequece. The limi of such a sequece is PFH opimal by defiiio (3.. Whe S ad A are fiie ses, a compacess of Δ is esured. From he defiiio of V( s, a, we have he followig proposiio. Proposiio 3.3.Whe he expecaio of he plaig horizo is fiie, he oal expeced reward is fiie. Sice he expecaio of he plaig horizo, E( M, is fiie, ϕ <+. (3. The, Δ, N V ( s,, M Es R( X, A 0 E s r ( X, A M P[ M ] E s c ( X, A M P[ M ] > > + κ κ. κ < max { RC, } ϕ max { RC, } ( ϕ Thus from (3., ( V s,, M <+. Assumpio 3.4. The expecaio of he plaig horizo is fiie. Now we ca discuss he exisece of opimal sraegy for he MDP wih radom horizo ad ifiie suppor. Lemma 3.5. V ( s, is coiuous i Δ. 63

6 For ay 0 ε >, here exiss Λ, such ha max { RC, } ε <. b κ Λ+ κ Therefore we ge a v such ha Λ log v. The for ay ' M M ' V ( s, V ( s, ' Es R ( X, A Es R ( X, A ρ, ' < v, Δ such ha ( E s r ( X, A M P[ M ] E s c ( X, A M P[ M ] > > + Μ+ Μ+ ' ' s, s, M+ M+ max RC, b κ < ε. M+ κ ( [ ] ( [ ] E r X A M > P M > E c X A M P M { } Le ow ( { } Noe ha sice ( Δ be a opimal sraegy for -horizo problem ad Δ be a se of cluser pois of all he sequeces ( Δ α Δ, T V s,, is coiuous i ad Δ is compac, Δ is a oempy se., ha is, a se of PFH opimal sraegies ad le { }. Theorem 3.6 (exisece. Uder assumpios., 3. ad 3.4 here exiss a PFH opimal sraegy for he MDP wih radom horizo. Sice Δ is compac, V ( s, is uiformly coiuous o Δ. Thus here exiss a sraegy such ha V( s, max V ( s, wih radom horizo Δ. Δ. Therefore here exiss a PFH opimal sraegy for he MDP Lemma 3.7. lim max (,, max (, Le max {, } V s V s K RC b κ, he we have + κ ( ( ( + K as, lim max V( s,, max V ( s, max V s, K max V s,, max V s, K Thus sice 0 Le Δ deoes a se of all opimal sraegies Δ for he MDP wih radom horizo. Lemma 3.8. Le Δ Δ { ( } Δ. From he defiiio here exiss a sequece of sraegies m( j lim ( m( j. j Thus ( ( ( ( ( lim V s, m j, m j V s,. j j T such ha 64

7 , Sice from lemma (3.7 V ( s, max V( s, Δ. There may o ecessarily exiss a saioary deermiisic sraegy or saioary radomized sraegy for he MDP wih radom horizo. There may o exis eve a ε - opimal radomized aioary sraegy. We show a example. Example. Cosider he homogeeous model wih S {, } ad A { ab, } p( s, a p( s, b, for s,. r(, a, r( b r( a, (,, 0 c( s, x 0, for, s, x ab,. r, b.. Le We deoe he probabiliy disribuio for he plaig horizo { ϕ } as follows, β ( whe 0 ( ( whe ( β ( β β ( whe φ β β ha is, he geomeric disribuio of which parameer chages o β from β a sage. I his model, i is clear ha acio b is opimal a sae. Thus here are wo cadidaes for opimal deermiisic saioary sraegy as follows, ' : keep your sae (use acio a sae ad acio b a sae, '' :move o sae ad keep i (use oly acio b. We shall examie a opimal radomized saioary sraegy for his model. A radomized saioary sraegy is defied as a a a τ (, ( Whe, his model is equivale o he MDP wih discou rae β. Therefore he expeced v s, s,, is he uique soluio of he sysem of he followig liear equaios, reward (, { } { } (, ( β α( (, ( α (, (, ( β τ. (, + ( τ ( + (, v + v + v v v v Solvig he equaios, we have,. (a (, β (, β ( α( β + τ ( β a a( τ( β + ( a( τ( β aτ ( β + ( τ a( τ( β v v Similarly, 65

8 { } { } (, ( β α( (, ( α (, (, ( β τ (, + ( τ ( + (, v v + v v v v so ha v, π β α + v, + α v, { } ( ( ( ( ( ( 0 a( β a ( β ( α( τ( β ( ( + + ( a a v ( ( a( ( a ( v ( + β + τ, + β + τ, Sice from (a we have ϑv (, ϑv (, 0, 0, v(, v(,, ϑτ ϑτ ad we obai ϑv 0 (, 0. ϑτ Therefore i is see formally ha acio b is opimal a sae. 3 Now fix β, β so ha he expeced rewards of deermiisic saioary sraegies u ', 5 5 are v0 (, ' K, v0 (, '' K, ad he expeced reward of radomized saioary sraegy π is 0 ( 5a a v0 (,, 55 3a ( 5 0 which is maximized a a. The expeced reward associaed wih his 3 v 0 (, ( α Give iiial sae, he radomized saioary sraegy saioary sraegies. associaed wih. a is u '' a is he bes amog all Defie he sraegy as follows, ' if 0 '' if The expeced reward of his sraegy is v0 (, From he fac meioed above, i is see ha for ε < here does o exis a ε - opimal radomized saioary sraegy. We coiue ow wih showig ha a heorem similar o Turpike Plaig Horizo Theorem which Shapiro []shows for he homogeeous discoued MDP holds for his MDP wih radom horizo. Because, here may o ecessarily exiss a saioary deermiisic sraegy or saioary radomized sraegy for MDP wih radom horizo, a opimal sraegy we wish o kow may be o saioary, so i s difficul o ge i direcly. 66

9 We iroduce he followig wo oaios, F { : Δ } : a se of opimal decisios a he firs sae for he radom M- horizo problem, ad F :a se of opimal decisios a he firs sae for he - horizo problem. { : } ( ( Theorem 3.9 (Turpike Plaig Horizo Theorem. There exiss some L such ha for ay L F F., ( Assume as he corary ha here does o exis such a umber L. The here exiss a ieger M such ha he firs decisio of some opimal sraegy for he M horizo problem is o coaied i F,ad here exiss a ieger M ( > similarly, so we obai a sequece of sraegies M { ( M i } such ha ( M i F { ( m( M i } such ha is limi is Δ.Thus for sufficie large ( i (, p ( m( M < ε, so ( ( i m Mi. Therefore for all i. Sice Δ is compac, here exiss a subsequece 67 m M, F. O he oher had, from defiiio Δ %, ad from he lemma 3.8 Δ. Thus F, which is a coradicio. From he above heorem we ca make a firs opimal decisio by solvig he sufficie large - horizo problem. I should be oed ha here exiss a opimal rollig sraegy. 4. ALGORITHM FOR FINDING AN OPTIMAL FIRST DECISION Alhough he Turpike Plaig Theorem i he above secio saes he exisece of he urpike horizo, he heorem shows o way for fidig i. Hece i his secio we ivesigae a algorihm for fidig a opimal firs decisio or ε opimal firs decisio. If we ca fid a opimal firs decisio, ex we pay aeio o he secod sage, ha is, we cosider he secod sage as he firs sage, ad he apply he same algorihm o i. By meas of coiuig his procedure a hird, fourh, sage, we ca fid a sequece of opimal decisios oe by oe, ha is, a opimal rollig sraegy. Above procedures correspods o ideifyig he PFH opimal sraegy gradually, ha is, makig he eighborhood of PFH-opimal sraegy small. Le Δ ˆ deoes a se of sraegies such ha is firs decisio is o icluded i F(, ha is ˆ Δ { Δ : F( }. ˆ Theorem 4.. For ay Δ, if saisfies he followig codiio (, ' ( max (,, (,, > max {, }, (4. V s V s R C b κ is o opimal for he problem wih ifiie suppor. Le ' Δ be a sraegy saisfyig a codiio (. Se max {, } + κ K RC b κ, he from he codiio (, ( ( + κ ' max V s,, V s,, > K (4. ad

10 ( ( max V s,, K max V s,. (4.3 Therefore from (4. ad (4.3 ( ( max V s, V s, ', > K. Thus ' Δ is o opimal. Remark 4.. If F( is sigleo ad a codiio of heorem 4. holds for ay ( is a opimal firs decisio. ˆ Δ, F From he above heorem we ca fid a firs decisio which is o opimal ad he remove i. I cosequece we propose a algorihm which decreases he umber of decisios possible o be opimal by ieraig he above check. The followig algorihm fids eiher a opimal firs decisio or a ε opimal decisio. Algorihm 4.3 Sep. Se. u max R, C b κ. Sep.Le { } κ Sep 3. a A + a, compue ξ max V( s,, V ( s,, a, where, α α. If ξ > ad a F is sigleo, Sop. Is decisio is a opimal firs oe. Sep 4. If ε, Sop. Is decisio is a ε-opimal firs oe. Sep 5. +, ad go o Sep. Remark 4.4. From he heorem 3.6 he above algorihm sops i a fiie umber of seps. Remark 4.5. If Δ is sigleo, he above algorihm ca fid a opimal firs decisio i a fiie umber of seps. I is discussed by Bes ad Sehi [3] ha Δ is o rarely sigleo. As a umerical example, we cosider a followig iveory problem. A iem has a lifeime disribuio a accou of is lifecycle or appearace of a ew iem. We cosider ha his disribuio correspods o he radom horizo previously saed. We deoe is disribuio by { ϕ } projec ed, all remaiig iems may be se back a a salvage cos per ui. Whe he Here we assume ha oe-period demad, η, follows i.i.d. Poisso disribuio. Le a deoes he amou of order. So he amou of sock saisfies a followig relaio, s s + a η, (4.4 where he iiial sock, s 0, is eve. We assume ha S s S, ha is, a upper boud ad a lower boud of he sock is give. The cos we cosider are followig, k( a : he order cos i he period whe a iems are ordered, ( c s : he holdig cos i he period whe s 0, he backloggig cos i he period whe s < 0, ( r x : he icome i he period whe Accordigly he problem is o maximize he oal expeced reward: 68 x iems are sold, where x max{ mi { η, s },0}.

11 a Maximize Es { ( ( ( } 0 r X κ Α c S. (4.5 Now we assume ha he daa are as follows, s 0 5, S 5, S 0. The expeced value of he demads i oe-period is 7. 0x ( x α ( α 0 s ( s 0 r ( x k ( a c ( s. (4.6 0 ( x < 0 0 ( α < 0 4s ( s< 0 The le he salvage cos per ui be 7. Table. Opimal Firs Decisios ad Turpike Plaig Horizos CV Mea (.5,.75 (0.886,3. (0.589, (.5,3.49 (.8,4.9 (.34,4.66 (0.99,5.07 (0.55, (3.88,6. (3.00,7.00 (.3,7.69 (.68,8.3 (.09,8.9 (0.58, (6.3,3.9 (4.90,5. (3.76,6. (.65,7.3 (.57,8.4 (0.53, (8.58,.4 (6.88,3. (5.4,4.8 (3.64,6.4 (.07,7.9 (0.509, (.,8.9 (8.86,3. (6.73,33.3 (4.64,35.4 (.56,37.4 (0.506, (6.0,44.0 (.9,47. (9.73,50.3 (6.63,53.4 (3.56,56.4 (0.504, (6.0,74.0 (0.8,79. (5.7,84.3 (0.6,89.4 (5.56,94.4 (0.503, 99.5 (upper opimal firs decisio (middle Turpike Plaig Horizo (lower (λ, λ We examie how he probabiliy disribuio for he plaig horizo cause he chage of he firs opimal decisios. We use he followig composie disribuio of Poisso disribuios, P[ N ] 0.5Pλ [ N ] + 0.5P [ N ] λ, (4.7 which eables us o arrage various combiaios of values of he mea ad coefficie of variaio of he disribuio by chagig λ ad λ. We calculae he opimal decisios for amou of orders a he firs sage ad he Turpike plaig horizos for he cases i which meas are,3,5,0,5,0,30,50, ad coefficies of variaio are 0.5,0.6,0.7,0.8,0.9,.0. The resuls of calculaios are show i Table. From Table.we ca see wo edecies i his iveory problem, oe is ha qualiy of order a he firs sage icreases as he mea horizo icreases, ad he oher is ha i decreases as he coefficie of variaio icreases. The umerical resul shows he ieresig behaviour ha whe he coefficie of variaio is.0, he firs opimal decisios are always 5. I his umerical example, whe he coefficie of variaio is.0, λ becomes very small for each emas, which suggess he probabiliy ha he projec will ed soo is fairly large. Thus he firs decisio for amou of order is expeced o become small. From heses resuls he opimal firs decisios are cosidered o deped o he shape of he probabiliy disribuio for he plaig horizo much. 5. CONCLUSIONS The purpose of his paper is o aalyze a opimal sraegy for he MDP wih radom horizo, ad purpose he algorihm o obai i umerically by Turpike Plaig Horizo approach. For he processes here may o exis opimal saioary sraegies, so we evaluae rollig sraegies, derived by 69

12 usig he resul of Turpike Horizo Theorem. We develop a algorihm obaiig a opimal firs sage decisio, ad some umerical experimes. As a resul of umerical experimes, we ake ha he opimal firs decisios deped o he shape of he probabiliy disribuio for he plaig horizo. REFERENCES RECEIVED OCTOBER 008 REVISED NOVEMBER 009 [] ALDEN,J.M. ad SMITH R.(99: Rollig horizo procedures i ohomogeeous Markov decisio processes. Operaios Research, 40, [] BEAN, J.,ad SMITH R.(984: Codiios for he exisece of plaig horizos. Mah. of Operaios Research, 9, [3] BES C. ad SETH, S. (984: Coceps of forecas ad decisio horizos : applicaios o dyamic sochasic opimizaio problems. Mah., of Operaios Research, 3, [4] GOULIONIS E.J., (004: Periodic policies for parially observable Markov decisio processes, Workig paper No. 30, 5-8, Uiversiy of Piraeus 004. [5] GOULIONIS, E.J. (006: Α replaceme policy uder Markovia deerioraio. Mahemaical ispecio, 63, [6] GOULIONIS E.J. (005: P.O.M.D.Ps wih uiformly disribued sigal processes. Spoydai, 55, [7] HOPP,W.J,, BEAN J.C ad SMITH, R (987: A ew opimaliy crierio for ohomogeeous Markov decisio processes. Operaios Research, 35, [8] PUTERMAN, M.L (994: Discree Sochasic Dyamic Programmig. Joh Wiley ad Sos, New York. [9] ROSS,S.M. (984: Iroducio o Sochasic Dyamic Programmig. Academic Press, New York. [0] SETHI S. ad BHASKARAN,S. (985: Codiios for he Exisece of Decisio Horizos for Discoued Problems i a Sochasic Evirome. Operaios Research Leers, 4, [] SHAPIRO,J.F. (968: Turpike plaig horizos for a Markovia decisio model. Maageme Sci., 4, [] SONDIK.J.E. (978: Οpimal corol of parially observable Markov decisio processes over ifiie horizo. Qperaios Research, 6, [3] WHITE,D.J. (987: Ifiie horizo Markov decisio processes wih ukow variables discou facors. Euro. J. of Qperaios Research,.8,