Recrung Supplers for Reverse Producon Sysems: n MDP Heurscs Approch Wuhch Wonghsnekorn, Mhew J. Relff 2, Jne C. Ammons Georg Insue of Technology School of Indusrl nd Sysems Engneerng Aln, GA 30332-0205 2 Georg Insue of Technology School of Chemcl nd Bomoleculr Engneerng Aln, GA 30332-000 Correspondng Auhor: Wuhch Wonghsnekorn School of Indusrl nd Sysems Engneerng Georg Insue of Technology Aln, GA 30332 Eml: wwongh@sye.gech.edu Phone: 404-894-7792 Absrc In order o cheve sble nd susnble sysems for recyclng pos-consumer goods, frequenly s necessry o concenre he flows from mny collecon pons of supplers o mee he volume requremens for he recycler. The collecon nework mus be grown over me o mxmze he collecon volume whle keepng coss s low s possble. Ths pper ddresses complex nd nerconneced se of decsons h gude he nvesmen n recrung effor. Posed s sochsc dynmc progrmmng problem, he recrumen model cpures he decsons for he processor who s responsble for recrung merl sources o he nework. A key feure of he model s he behvor of he collecor, whose wllngness o jon he nework s modeled s Mrkov process. An exc mehod nd wo heurscs re developed o solve hs problem, hen her performnce s compred n solvng prcclly szed problems. Key words Recrung, Reverse Producon Sysem, MDP Heurscs. Inroducon The pper ddresses on he concenron of pos-consumer merl flows hrough he nercon beween supplers (gens) nd processor (recruer). In forwrd supply chns, one processor negoes wh mny supplers n order o cqure he requred resources for producon o mee he demnd. The processor hs he bly o conrol he moun of merl from he supplers. The mn uncerny s usully he demnd n he mrke. In reverse supply chns, he reurn flows from he consumers re sgnfcn uncerny. The reverse sysem consss of he cves such s collecon, clenng, dsssembly, esng, nd sorng, sorge, nd recovery operons. Ths pper focuses on he collecon segmen where he recyclng processor emps o rereve used merls from vrous sources under uncerny.
In order o buld successful collecon nework, we consder he problem of recrung supplers. In hs cse supplers re ypclly locl ggregors of merl who work wh consumers or very loclzed merl hndlers, such s supermrkes, o produce regulr supply of merl. Creful plnnng of he supply nework o suppor he recyclng pln cn be crcl fcor n he success or flure of he producon operons. The nercon n he recrumen process s beween busness-o-busness (B2B) enes rher hn busness-ocusomer (B2C). Ths mkes he process more complced becuse boh pres hve power o negoe. In reverse supply chn, plnnng he collecon nework o supply he cpl nensve processng pln cn be crucl fcor n he success or flure of recyclng operons. For exmple, n he ps decde, wo mjor crpe recyclng compnes (Evergreen Nylon Recyclng nd PA2000) suffered mjor fnncl problems h led o he closure of he recyclng plns. One explnon s he hgh cos of he supply merls resulng from wdely dspersed collecon pons. Hence, he objecve s o provde he collecon cpcy low enough cos o mke vble. However, he processor s fced wh sgnfcn chllenge. Processors re ypclly no fmlr wh he wse busness; he collecon of rsh s no core compeency of her orgnzon, nor do hey hve exsng wse hulng conrcs h hey cn explo o ge he merl. Ofen, hey do no hve vercl negron o he rel secor nd hence do no conrol he pon of conc wh he cusomers. Ths leds o he need o recru lyer of supplers o he sysem. In he cse of recycled crpe hs mgh be he relers who sell crpe, s hey re he ones o whom used crpe my be reurned by he nsllers. Accordng o he d from Crpe Amerc Recovery Effor s Annul Repor n 2004, 4,000 mllon pounds of used crpe s dscrded o he lndfll n yer 2003 whle less hn 00 mllon pounds ws recycled. Ths number s growng rpdly s rw merl coss rse. However, he crpe ndusry hs hd lmed success wh lrge scle recyclng of nylon crpes despe he fnncl poenl. Pos consumer crpe scrp s prced pproxmely $0.06 pound for ruck lod quny (40,000 pounds or more) whle nylon 6 (fer processng nylon-6 scrp) s prced $.59 2 for ruck lod quny. One soluon o hs recrumen problem s o subconrc he responsbly of recrung supplers o locl regonl collecor. The collecor s hen lloced budge wh whch o recru he relers o he nework, whch could nclude fnncl ncenves. Alernvely, he processor my decde o perform he recrumen nd collecon self, nd he budge would mos lkely reflec he moun of me nd personnel resources h re devoed o prculr regon. Recrumen models n he lerure focus on employmen recrumen, humn resource mngemen, nd physologcl models n medcl reserch (Drmon 2003, Treven 2006, Hwkns 992, Georgou nd Tsns (2002). Mehlmnn (980) use recrumen concep for long-erm mnpower plnnng problem. Coughln nd Gryson (998) exmne he problem where he ndvdul dsrbuors ply wo key roles n nework mrkeng orgnzons (e.g. Amwy, Mry Ky nd NuSkn): hey sell produc, nd hey recru new dsrbuors. They develop model of nework mrkeng orgnzon nework growh h shows how compenson nd oher nework chrcerscs ffec growh nd profbly of he dsrbuor. In her conex, one dsrbuor recrus ohers by soclly nercng wh hem n one form or noher. They represened hs process by dpng dffuson model formulon o he recrumen process (Bss 969). Ths model llows for nework growh Cnd's Wse Recyclng Mrkeplce (2006) 2 IDES The Plscs Web (2006) 2
v boh nheren rcon (he nnovon effec) nd he spred of word-of-mouh (he mon effec). They nroduce recrumen funcon whch ncludes nnovon nd mon erms. Ths pper explores he noon of he recrumen of he collecors, nsed of he dsrbuors. Furhermore, he recrumen process s represened n more complex form, no jus closed-form funcon. The recrumen model whch s posed s sochsc dynmc progrmmng problem hs some smlres o he Resless Bnds Problem, frs nroduced by Whle (988). The connuous-me verson of he problem wh me-verge rewrd creron ws developed n dynmc progrmmng frmework. He hen nroduced relxed verson of he problem, whch cn be solved opmlly n polynoml me. More lerure revew n hs re s dscussed he end of secon 3. The pper s orgnzed s follows. In secon 2, we more formlly defne he problem. The modelng of he problem s dscussed n secon 3. In secon 4, we presen hree mehods o solve he problem. In secon 5, we descrbe he compuonl expermens h llusre effcency of our lgorhms. Fnlly, n secon 6, we develop some conclusons nd des for fuure reserch. 2. Problem Defnon nd Modelng We refer o he problem of recrung supplers n reverse producon sysem s he recrumen problem. Recrumen s negong process h nvolves wo pres: processor nd suppler. We denoe suppler s n gen hroughou he pper nd processor s recruer. The recruer cnno rereve merl from he gen unless boh pres gree o hrough he recrumen process. There re gens o consder nd ech gen owns Resource B h he recruer wshes o collec. Typclly, hs s no very lrge number s here re mny relers bu only frcon hs suffcen volume of busness o jusfy recrumen. The objecve s o recru he gens o grow he recrumen nework by usng he lmed recrung budge effcenly o mxmze he expeced collecon volume he end of plnnng horzon. The recrumen problem s solved perodclly over perods. Fgure depcs he growh of recrumen nework over me. = 0 = Recrued Agen Unrecrued Agen Recruer = T- Fgure : Growng Recrumen Nework We ssume h here s only one recruer nd does no compee wh oher recruers for he resource. The recruer s gven Resource A, whch cn be used for he gens 3
recrumen process. Ths resource ypclly cn be nerpreed s money or dscoun h cn be used s n ncenve o recru he gens. A se of gens hs heerogeney n: ) The quny of Resource B h hey genere, b) Ther geogrphcl locon, c) Ther nl wllngness o sell/gve he recruer he resource bsed on some predefned fcors, nd d) Ther predsposon owrds becomng recrued o he nework. In order o cheve he objecve, he recruer mkes recrumen budge llocon decson n ech decson perod. Ths decson ffecs he decsons of he gens. I s ssumed h he wllngness se of ech gen s upded o he recruer n every perod. Also, he ol spendng budge n ll perods mus no exceed he ol recrung budge lm provded o he recruer he begnnng of plnnng perod. A he begnnng of he perod, fer n gen receves s llocon of budge (Resource A) from he recruer, decdes wheher o gve/sell s resource, Resource B, o he recruer. Is decson s bsed on s wllngness o pr wh he resource. In he cse where he gen does no conrc o provde he resource o he recruer, he gen s overll wllngness se cn chnge by beng nfluenced by he ncenves receves from he recruer. Fgure 2 summrzes how he decson of he recruer s reled o he decson of he gens n ech perod. Srng Budge Recruer Recruer s Decson Agen Agen 2 Agen 3 Agen Agen Jon Nework Agen 2 Agen 3 Jon Nework Agen s Decson Agen Fgure 2: Decsons of Recruer nd Agen n One Perod A key elemen of our recrumen frmework s model of n gen s wllngness o prcpe. Ths model cn be s smple s one-vrble funcon of he gven ncenve. However, s more lkely h gens hve more sophsced se relve o her wllngness o prcpe. A Mrkov model for he gen, whch we cll he Agen s Resource Wllngness Model (ARW), s developed n order o cpure more sophsced srucure of he gen behvor nd ye ren resonble represenonl nd compuonl smplcy. Agen s Resource Wllngness Model (ARW) The sgnfcn componens of hs model re he wllngness se nd he rnson probbles. We model ech gen s resource wllngness s Mrkov chn wh recrued se h s bsorbng. Ths mens h he recrued gen never leves he collecon nework nd he recruer s no confroned wh n gen reenon ssue. The model lso consss of oher ses h represen dsnce from recrumen bsed on he 4
probbly of rechng recrumen nd connecon o oher ses. Ech gen hs s own Mrkov model nd s ssumed he recruer knows he se of he gen n ech me perod. Consder gen n hs model. Wllngness Se Defnon (Agen s se) s { R, L, M, H}. Ths descrbes wh se gen s n me perod. There re four possble ses for ech gen:. Recrued (R) - The gen grees o gve he Resource B o he recruer. 2. Low (L) - The gen s no recrued by he recruer. Also, he gen s n se where wll be very chllengng o recru. 3. Medum (M) - The gen s no recrued by he recruer. The gen hs no bs gns he recrumen. 4. Hgh (H) - The gen s no recrued by he recruer. Also, he gen s n se h mkes recrumen esy. We ssume h when n gen s recrued, resdes n he R wllngness se, n bsorbng se. The ses L, M, nd H represen dsnce from recrumen bsed on he probbly of rechng he recrumen se nd connecon o oher ses. In oher words, f he gen s no recrued, resdes n eher he L, M, or H se. Fgure 3 shows symbolc represenon of he ses nd possble rnsons. R Probbly = L M H Fgure 3: Agen s Se Dgrm We denoe he moun of Resource B h gen cn genere beween ech decson epoch (ech perod) s g, whch s ssumed o be sngle vlue, lhough would no be dffcul o generlze o rndom vrble. We lso ssume h he recruer cn collec he full moun of vlble Resource B from every recrued reler n ech perod. In ddon, we denoe s he moun of budge (Resource A) h gen receves from he recruer n perod. Gven he con, he gen rnss o he nex se wh followng rnson probbles. Trnson Probbles The probbly of gen movng o se s, from se s by con s denoed by p( s, s, ) or Pr ( ) s s,. There re wo ypes of rnson probbles o consder. 5
The probbly of recrumen s he probbly of movng o se R from he L M, or H ses ( PrLR ( ), PrMR ( ), PrHR ( ) ). The dffculy of recrumen depends on hree fcors: he se of he gen, he budge llocon or con, nd he gen s recrumen budge hreshold,. The recrumen budge hreshold hs he sme uns s he Resource A budge llocon. In generl, represens mnmum vlue requred o recru he gen. In he cse of crpe relers, here my be some correlon beween he sze of he reler nd he budge hreshold requred becuse he ncenves offered my drecly scle wh he moun of used crpe vlble for pck-up. The hreshold my be nerpreed s subsdzng he cos of he reler s dsposl fee. In secon 5, when we provde he d for numercl sudy, we ssume h f he gen cn provde sgnfcn moun of Resource B, lso demnds lrge moun of llocon of Resource A from he recruer. Thus, he recrumen budge hreshold depends on he moun of Resource B collecon volume vlble from gen. A hgher g mples hgher. Ths mens h s more expensve o recru gens who hve hgher Resource B generon res. In order o cpure hese hree fcors ogeher, we pply sgmod funcon (Seggern 993) o clcule he probbly of recrumen. In ddon, we defne he recrumen wllngness fcor, s, bsed on he se of he gen such h H M L 0. The sgmod funcon o clcule probbly of recrumen of gen me s: Prs ( ) R. s ( ) e () Usng he probbly of recrumen funcon n (), we cn vry he vlue of he recrumen wllngness fcor so h ech se hs dfferen recrumen probbles s shown n Fgure 4. We se H 2, M, nd L 0.5, wh =0 for ll ses. Probbly of Recrumen H M L 0.5 0-9 -6-3 0 3 6 9 Fgure 4: Recrumen Probbly for Dfferen Recrumen Wllngness Ses The probbly of (unrecrued) se rnson cn be specfed ccordng o how redly prculr gen s moved mong he L, M, nd H ses f s no recrued. The probbly of se rnsons cn be se up such h s esy o move o M nd H from L. Ths mkes he gen eser o recru. On he oher hnd, he probbly of se rnson cn be se up such h s more dffcul o move o M nd H from L. Ths mkes he gen more dffcul o recru. Fgure 5 dsplys he overll rnson probbles of n gen. 6
R Pr ( ) RR Pr ( LR ) PrHR ( ) Pr ( ) MR Pr ( ) LM Pr ( ) MH L Pr ( ) LL M Pr ( ) MM Pr ( ) ML PrHM ( ) Pr ( ) HL Pr ( ) LH H Pr ( ) HH Fgure 5: Trnson Probbles for n Agen Usng he Agen s Resource Wllngness (ARW) Model, he decson for one perod h s shown n Fgure 2 cn be modfed s shown n Fgure 6. The ARW model provdes beer represenon of ech gen s prcpon sus for he recruer. Srng Budge Recruer Recruer s Decson Agen H Agen 2 L Agen 3 M Agen M R M R Agen s Decson M Fgure 6: Decsons of Recruer nd Agens wh he ARW Gven he problem defnon nd he generl frmework of he recrumen model, we now nroduce sochsc dynmc progrmmng formulon of he recrumen problem for he recruer n he nex secon. 3. Problem Formulon: Sochsc Dynmc Progrmmng Formulon Ths secon develops sochsc dynmc progrmmng model for he recrumen problem h cplzes on he Mrkov propery n he Agen s Resource Wllngness model. The formulon of hs model consss of he defnon of decson epochs, se spce, cons, rnson probbles, nd rewrds. A soluon for hs model provdes he opml recrung polcy o he regonl collecor. In hs formulon, we ssume h precse 7
nformon for he prmeer vlues s vlble. The number of gens s, he mxmum mx recrung Resource A budge s B nd he ol number of plnnng perods s T. The decson epochs nd se spces re defned s follows. Decson Epochs Se Spce {0,,..., T } Sr Y {, w, w2,..., w, } m B for ll Y, where he wllngness se of reler decson epoch s w { L, M, H, R } nd he srng recrumen budge he begnnng of perod s represened by In hs model, we defne he con se s follows. Sr B. Acon Ses Al { l, 2l,..., } ml, where he moun of resource A lloced o gen from con se ndex l me perod s represened by l such h l Sr Sr B nd 0 l B for l,..., A. A he Sr mx frs perod, B0 B. The sze of he con se depends on nd In hs model, we defne he se rnson rules s follows. Se Trnson Rules Sr B. () Inl Se There s more hn one possble nl gen se dependng on he nl vlue Sr of w. One exmple nl se s Y0 {0, w0, w20,..., w 0, B0 } {0, L,..., L,0} m where ll gens begn n he L wllngness se nd he srng recrumen budge s 0 uns of Resource A. (b) Se Trnson Probbles These probbles depend on he ARW model for ech gen. We ssume h ech gen s wllngness se chnges ndependenly, so he se rnson probbly s he mulplcon of he probbly of he wllngness se rnson for ech gen gven he specfc Resource A llocon provded by he recruer. If he curren se s Y nd con A l s ken n perod, he probbly rnson of movng o se Y or P ( Y Y, Al ) cn be represened n he followng form, P ( Y Y, Al ) Sr P (, w, w,... w, B ) ( ) 2( ) ( ) (, w, w,... w, B ),(,,..., ) Sr 2 l 2l l, Pr ( ) Pr ( ) Pr ( ), w, w( ) l w2, w2( ) 2 l w, w ( ) l Pr ( ), (2) w, w ( ) l 8
where Sr Sr Sr l B nd B B l wllngness se w ( ) s Pr w w ( ) l. Here, he probbly of movng o, ( ) f he prevous wllngness se s w nd llocon l s ken for gen. These probbles cn be clculed from he ARW model descrbed n secon 2. Rewrds In order o compue he rewrds, we ssume h he wllngness se of gen me hs s own vlue, V w. Le V R be he moun of Resource B h n gen cn provde o he recruer. Becuse he rewrd should represen he ncremen n collecon volume, he vlues for he non-recrued ses V L, V M, nd V H re se o zero. However, he rewrd should be defned such h here s n ncenve o move o hgher wllngness se. Hence, he vlue of V L, V M, nd V H re ssgned smll vlue such h VL VM VH VR. For exmple, V L = 0., V M = 0.2, nd V H = 0.3. Le r Y, Al, Y denoe he vlue me of he rewrd receved when he se of he sysem decson epoch s Y, con A l s ken, nd he sysem occupes se Y decson epoch. Ths vlue represens he ol ncremen n he collecve vlue of ll of he gens se chnges. If he recruer moves mny gens o se R, cn obn hgh rewrd from he cumulve collecon volume for he recrued gens. Ths vlue cn be obned by: r Y, Al, Y Vw V ( ) w. (3) The regonl recruer s expeced rewrd of se Y nd for con by compung: r Y, A P( Y Y, Al ) r Y, Al, Y l, Y A l cn be evlued Prw ( ), w ( ) l Vw V ( ) w. (4) Y Gven he descrpon of he rewrd funcon, he objecve funcon of hs model cn be defned s follows. Objecve Funcon The objecve s o mxmze he expeced collecon volume usng he specfed Resource A budge. In oher words, under fxed budge, he recruer wns o move subse of gens o se R over he horzon such h he recrued gens yeld he mxmum expeced ol Resource B collecon volume n he fnl perod. Snce he overll purpose s o mxmze he ol Resource B collecon volume he end of he me horzon nd he merl volume s no quny h chnges wh me, here s no convenonl dscoun fcor nvolved. Le = ( d0, d,..., dt ) represen he polcy for every me perod. Hence, = * * * ( d, d,..., d ) denoes he opml polcy n ech me perod. Defne he expeced ol 0 T rewrd obned decson epoch,,..., T by usng polcy o be u ( Y ) srng se Y n decson epoch s: wh 9
Le T u ( Y ) EY r (, ) Y Al. (5) u * ( Y ) denoe he mxmum expeced ol rewrd obned decson epochs,,.., T wh srng se Y n decson epoch. Then he opmly equon for he recrumen problem s: * *! *! u ( Y ) u ( Y ) mx r ( Y, Al ) P ( Y Y, Al ) u ( Y ), (6) Al! Y!!! A ( Y ) rg mx r ( Y, A ) P ( Y Y, A ) u ( Y ). (7)!! * * l l Y A l The opml con n ses Y epoch s denoed by A * ( Y ). In oher words, he mxmum expeced ol rewrd perod, u * ( Y ), s he relzon from ll possble cons of he mmede rewrd nd expeced fuure rewrd from prculr con. * Essenlly, he objecve of he recrumen problem s o fnd u0( Y 0). We hve no found smlr formulon n he lerure. The formulon h ppers closes s he resless bnd problem. Bersms nd Nno-Mor (2000) ddresses he resless bnd problem s follows. Consder ol of N projecs, nmed n N {, 2,..., N}. Projec ncn be n one of fne number of ses ' I '. A 0,,2,..., excly M N projecs mus be chosen o work on or se cve. If projec n, n se ' n, s cve, n cve rewrd R s receved, nd s se rnson chnge follows from n cve rnson ' n probbly mrx no se j ' n wh probbly ' worked on, pssve rewrd 0 ' n P n ' n jn n. On he oher hnd, f projec s no R s erned, nd s se rnson chnge follows from n pssve rnson probbly mrx no se j ' n wh probbly ' P 0 ' n jn. Rewrds re medscouned by specfc dscoun fcor. The problem s objecve s o fnd schedulng polcy h mxmzes he ol expeced dscouned rewrd over n nfne horzon. Our recrumen problem formulon s generlzon of he resless bnds o nclude () more possble cons for ech seleced bnd oher hn jus selecon or no, nd (2) n overll budge consrn. Frs, he recrumen problem s number of cons for ech gen or projec cn be greer hn he wo n he resless bnd problem (cve nd pssve). Second, he resless bnd problem hs one lnkng consrn whch s excly M projecs mus be worked on, or se cve. In conrs, we cnno fx how mny gens re worked on n ech me perod. Insed, we hve he budge lnkng consrn whch lms he con se n ler perods. In ddon, he resless bnds do no hve nernl se, wheres our "gens" do hve nernl se (R,L,M,H) whch hey remember from one perod o he nex. Ths ler generlzon of ner-perod couplng s sgnfcn becuse he couplng beween me perods s herefore no jus v he budge consrns bu lso hrough he se equons. Furhermore, our recrumen problem hs some smlres o he wekly coupled dynmc progrm (WC-DP) nroduced by Adelmn nd Mersereu (2004). The problem descrpon of WC-DP s s follows. There re I subproblems h re ech Mrkov decson problems on dsjon se spces. Correspondng o subproblem, defne he followng: 0
- Se spce S, ssumed fne - Conrol spce A ( s ), dependng on he curren se s nd ssumed fne for ll s S. - Mrkov rnson probbles p ( s s, ) for ll A nd s, s S. Here we ndce h he curren se by s nd he nex se n me s s. Condonl on he locl se s nd con, rnsons re ssumed ndependen of oher subproblems. - Expeced rewrd r( s, ) ccrung n se when conrol s dmnsered. The overll WC-DP problem by Adelmn nd Mersereu (2004) s collecon of subproblems of hs form solved smulneously subjec o N lnkng consrns of he form I N N D ( s, ) b where b R nd D :{( s, ) : s S, A }" R. The recrumen problem my be reduced no WC-DP. Insed of D(s,), he recrumen problem wll hve D() or only budge or con mers. Also, we wll only hve one lnkng consrn whch s summed over he me horzon s well s he ndex of he con. Hence, b hs only dmenson. Ths consrn mkes he problem more complced becuse couples cross me s well s cross gens. Adelmn nd Mersereu (2004) proposed he LP-Bsed relxon lgorhm o solve WC-DP wh column generon echnque. Solvng hs problem wll be me-consumng f he number of cons s lrge. The compuonl exmple of her pper s performed over dervves of he resless bnd problem where he cons re eher pssve or cve only. Our problem hs much lrger con spce. Gven he sochsc dynmc progrmmng formulon of he recrumen problem, he exc mehod o solve hs problem s developed n he nex secon. 4. Soluon Approch In hs secon, hree soluon pproches re dscussed. They re he Dynmc Progrmmng Algorhm, he Q-lernng Bsed Heursc nd he Rollng IP wh DP Heursc. The frs pproch s n exc mehod whle he oher wo pproches re heursc mehods proposed o solve relsclly szed problems. 4. Dynmc Progrmmng Algorhm In hs subsecon, n exc lgorhm o solve for he opml polcy of he recrumen problem s proposed. The lgorhm kes dvnge of he opmly equon developed n secon 3. Becuse he sochsc recrumen problem s fne horzon problem, cn be modeled s sochsc ph problem where he number of phs s exponenlly lrge. In ddon, he recrumen problem s rewrd flls under he ol rewrd problem. For fne perod sochsc ph problem wh ol rewrd, one could use vlue eron bsed scheme o solve he problem. Hence, for smll szed problem, bckwrd nducon or dynmc progrmmng (DP) provdes n effcen mehod o solve he recrumen problem. The procedure of he DP lgorhm s shown s follows.
Bckwrd Inducon (DP) Algorhm Procedure Sep Se T nd Sep 2 u * ( Y ) r ( Y ) 0 for ll possble ses n. T T T T Subsue for nd compue u * ( Y ) for ech Y from *! *! u ( Y ) mx r ( Y, Al ) P ( Y Y, Al ) u ( Y ), Al! Y! Se m!, ( ) ( ) * u ( Y ) mx Prw w ( l ) Vw Vw Al! Y # $! * % Prw ( ), ( ) ( ) w l u Y &. Y! ' (!! A ( Y ) rg mx r ( Y, A ) P ( Y Y, A ) u ( Y ).!! * * l l Y A l (8) (9) Sep 3 If 0, sop. Oherwse reurn o sep 2. Usng heorem 4.5. from Puermn (994), cn be shown h he opml vlue for ll decsons epochs s u * ( Y ) nd correspondng o he opml con (polcy) n ll ses Y epoch s opml con A * ( Y ). For smll szed problems, he DP lgorhm redly provdes n opml polcy for decson mkng bsed on he ses nd he me perod. I enbles he recruer o fnd whch gens o recru nd how much of Resource A o lloce o ech gen for ech perod. However, hs lgorhm suffers from he curse of dmensonly s descrbed n Bellmn (957). Ths mens h compuonl effors grow exponenlly wh he number of se vrbles or wh he problem sze. For lrge-scle problems, he DP lgorhm s dffcul o solve n resonble me becuse hs o exmne every possble con n ech se n order o fnd he opml soluon, even hough mny ses would no be reched by he opml polcy. In he nex secon, we nroduce wo heurscs s wy o solve he lrge-scle recrumen problem n resonble me. 4.2 Q-Lernng Bsed Heursc Ths secon develops heursc bsed on he Q-Lernng mehod o obn soluon polcy for he recrumen problem. Ths heursc provdes n lernve wy o solve he lrge-scle recrumen problem whn resonble effor. Q-lernng (Wkns 989) s n exenson o rdonl dynmc progrmmng or vlue eron. Q-Lernng s one of he mehods of renforcemen lernng (RL) or smulon-bsed opmzon conceps. Accordng o Keblng (996), RL s he problem where solver mus lern how o cheve he bes con v rl-nd-error wh nercon n dynmc envronmen. Snce compuonl effor s prmry concern for soluon of he recrumen problem, we dp he Q-Lernng pproch dscussed by Gosv (2003). Frs, we nroduce he Q- vlue, Q( se, con ) or Q( Y, ), h corresponds o vlue of ech se-con pr. 2
The sep-by-sep procedure of he Q-Lernng Bsed Heursc (QBH) procedure s shown s follows. Q-Lernng Bsed Heursc (QBH) Procedure Sep 0 Se he eron number o 0. Selec vlue for ) where 0) nd nlze he eron lm. Sep Inlze me perod,, o 0 nd srng se o Y. Ths represens he nl budge nd nl wllngness se of ech gen. Sep 2 Genere n con usng n con selecon heursc, descrbed below. Sep 3 Smule con o rereve he nex perod con, Y. Le r ( Y,, Y ) be he mmede rewrd erned n he rnson o se Y from se Y under he nfluence of con. Sep 4 Upde Q( Y, ) usng he followng equon: Q( Y, ) * ( ) Q( Y, ) ) [ r ( Y,, Y ) mx Q( Y, b)], 0), (0) b A( Y ) where A( Y ) represen ll possble cons n se Y nd f Q( Y, b) hs no vlue, se s nl vlue s se o 0. Sep 5 If T, ncrese by nd go o sep 2. Else, ncrese he eron number by. If he eron number exceeds he lm, go o sep 6. Else, ncrese by nd go o sep 2. Sep 6 For ech Y, selec A ( Y ) rg mx Q( Y, b). () b A( Y ) The lernng re s represened by ) n (0). Is vlue weghs how much he prevous vlue of Q( Y, ) nd he evluon of mmede rewrd wh fuure rewrd should ffec he new vlue of Q( Y, ). The Q-vlue s predcon of he sum of he renforcemen one receves when performng he ssoced con nd he followng gven polcy. To upde he predcon Q( Y, ), one mus perform he ssoced con, cusng rnson o he nex se Y, nd reurnng sclr renforcemen r ( Y,, Y ). Then one only needs o fnd he mxmum Q-vlue n he new se, mx Q( Y, b), o hve ll necessry b A( Y ) nformon for revsng he predcon (Q-Vlue) ssoced wh he con jus performed. Q-lernng does no requre one o clcule he rnson probbles o successor ses. The reson s h sngle smple or successor se for gven con s n unbsed esme of he expeced vlue of he successor se. The con selecon heurscs n sep 2 of he QBH procedure re descrbed s follows. 3
Acon Selecon Heurscs Budge llocon for ech gen represens he con n sep 2 of he QBH. I s very mporn o selec n con wsely s hs s he exploron pr of he RL. Three heurscs re nroduced s follows. In he Q-Lernng QBH procedure, one of he heurscs s rndomly seleced durng ech execuon of con selecon. Heursc : Rndom Allocon In hs heursc, he remnng budge s lloced o rndom se of gens rndom moun level. Heursc 2: Hgh Wllngness Se Agen Frs Ths heursc focuses on llocng he remnng budge o hose gens who hve hgher chnce of recrumen success. Ths my no be he bes wy o obn he hghes pyoff becuse he gens wh hgh wllngness se my genere smller moun of Resource B collecon volume compred o gens wh low wllngness se who genere hgher moun of collecon volume. Heursc 3: Hgh Collecon Volume Agen Frs Ths heursc focuses on llocng he remnng budge mong hose gens who genere hgher moun of Resource B collecon volume. Ths my no be he bes wy o obn he hghes pyoff becuse gens wh hgher collecon volume my be very hrd o recru. In oher words, recrung mny wllng smll gens my resul n hgher moun of ol collecon volume. The QBH uses he con selecon heurscs o explore he con nd se spces. The exploon pples (0) o upde he Q-vlue for se-con pr. Accordng o Gosv (2003), he Q-Lernng mehod gves ner-opml soluon when he mxmum number of erons s lrge enough. In order o perform lrge number of erons n resonble compuon me, he compuonl complexy of he lgorhm should be nlyzed. In sep 4, he number of seps requred o upde Q( Y, ) n (0) requres frs serch for he nl vlue of Q( Y, ) nd second he mxmzon of Q( Y, b) for every vlue of b A( Y ). The vlue look-up for Q( Y, ) s performed n O ( + ) seps, where + s he sze of ypclly lrge Q-ble. A Q-ble s look-up ble h sores he vlue of Q( se, con ) for every encounered se-con pr. Ths sep kes O ( + ), number of cons ses Y, whch s ypclly lrge. In summry, every compuon of (0) n sep 4 of he Q-Lernng Bsed Heursc requres: Tme Complexy of (0) = O( + ) A. (2) Two modfcons re nroduced o speed up hs sep. The frs s o se he lernng re ) equl o nd he second s o sore he Q-vlues usng hsh ble. Ech of hese modfcons s descrbed n he followng prgrphs. Lernng Re Equl o One Wh ) =, equon (0) becomes Q( Y, ) * r ( Y,, Y ) mx Q( Y, b). (3) b A( Y ) 4
Insed of sorng Q( Y, b) for every vlue of b A( Y ) nd serchng for he mxmum of mx Q( Y, b) n every eron, s much smpler o sore he mxmum of Q( Y, b) no b A( Y ) Qmx ( Y, bmx ). Under hs modfcon, he upde of Q( Y, ) becomes: Qmx ( Y, mx ) mx[ Qmx ( Y, mx ), r ( Y,, Y ) Qmx ( Y, bmx )]. (4) Bsclly, rerevng mx Q( Y, b) cn be done n complexy of O ( + ) by lookng up b A( Y ) Qmx ( Y, bmx ). The upde of Qmx ( Y, mx ) s performed f he new vlue of r ( Y,, Y ) Qmx ( Y, bmx ) s hgher hn he prevous vlue of Qmx ( Y, mx ). In hs sep, he bes con mx s lso upded ccordngly. In ddon o hs modfcon, he Hsh Tble d srucure s ppled o he QBH mehod. I s descrbed s follows. Hsh Tble The QBH mehod requres lrge Q-ble n order o rereve Q-vlues of correspondng ses nd cons. Compuonlly, s me-consumng o rereve he seleced Q-vlue usng rdonl rry for he d-srucure. As beer lerve, Hsh Tbles (Knuh 973) re used s Q-vlue d srucure o mprove he look-up me. The Q-vlue cn be rereved n complexy of O () n he verge cse nd bes cses. The wors cse serch me s O ( + ) ; however, he probbly of hs hppenng s vnshngly smll. Ths d srucure echnque does no hve n mpc on he soluon quly of he QBH mehod. The procedure s he sme. The only chnge s he rerevl me of he Q-vlue of ny secon pr n (0). Employng he hsh ble d srucure for Q-vlues nd fxng he lernng re ) o one, he compuonl complexy of (0) n sep 4 of he QBH s reduced from O( + ) A o O () n he verge nd bes cses. In he wors cse, s O ( + ). Ths mprovemen reduces compuonl requremens for exploons. Compleely gnorng he prevous vlue of Q-vlue by seng he lernng re ) o one my ffec he resulng quly of he heursc soluon, bu he compuonl effor s sgnfcnly reduced o fcle overll problem soluon. 4.3 A Rollng IP wh DP Heursc As n lernve o he QBH procedure, heursc employng neger progrmmng (IP) hs been explored. The heursc s bsed on n observon of he DP lgorhm descrbed n secon 4.. An opml recrumen polcy for n ndvdul gen cn be found usng he DP lgorhm becuse he number of ses nd cons s smll. Ths chrcersc my be exploed o solve he overll recrumen problem. The mn concep of hs heursc s o shrnk mul-perod problem so s o hnk of one-perod problem. Frs, he opml polcy for ech ndvdul gen s solved for T perods usng he DP lgorhm. Then, ll of hese ndvdul gen soluons re used o fnd he bes combnon of budge llocons mong gens. The resulng soluon s mplemened for he frs me perod where he seleced gens receve her gven frs perod budge llocons. Nex, he opml polcy for ech ndvdul gen s solved for he remnng T perods. Then he procedure repes self unl he fnl perod s solved. The exmple of n gen for whom recrumen budge s lloced n he frs nd second perod s shown n Fgure 7. 5
Opml Polcy for 2 perods If lloced, spend budge n he frs perod Opml Polcy for perod If lloced, spend budge n he second Fgure 7: The Rollng Horzon Concep A key sep n hs pproch s n opmzon problem h selecs he bes combnon of ndvdul polces o use o mxmze he collecve recrued gens collecon volume, subjec o he overll recrumen budge consrn. Ths pproch s subopml becuse does no ke dvnge of he bly o observe nd respond o recrumen durng he polcy execuon. To mprove performnce, rollng horzon mplemenon s ppled. The remnng unspen funds lloced o hose gens who hve been recrued nd unspen funds lloced o relers no recrued re dded bck o he vlble recrumen budge moun nd hen he opmzon problem s resolved wh he upded nformon. The reson why hs pproch s subopml s becuse does no ke n con bsed on he nformon bou how he relers respond o he expendures. I lloces budge o be spen for he enre perod on he reler nd only recvely relloces money from mong relers who re recrued erly on n he process. mx The sochsc recrumen funcon s denoed by SR(, B, ) s funcon h reurns he soluon from solvng he recrumen problem wh he DP lgorhm (developed n mx secon 4.) for reler for perods gven he srng ol budge B. The soluon yelds he opml budge llocon polcy n ech perod nd he expeced collecon volume from reler over perods. Nex, he opmzon problem h selecs he bes combnon of ndvdul polces mong ll relers s formuled. In order o lm he problem sze, we dscreze he budge prmeer n he formulon. The ndex, prmeers, nd vrbles re defned s: Index: Index of gens ( =, 2,, m ) j Index of budge levels ( j =, 2,, J ) Prmeers: mx B Mxmum srng ol budge over ol T perods sr B Mxmum srng budge perod b Budge llocon he collecor choose o spend on he reler, whch s he vlue j h of j enry n B ( b,..., bj,..., bj ) v Mxmum expeced ncremen of cpcy volume h cn be colleced from j reler f budge moun b j s lloced o h reler. The vlue of v j cn be obned from solvng SR(, bj, ) usng he DP pproch. 6
Decson Vrbles: f budge moun b j s lloced o reler x j = 0 oherwse The neger progrmmng problem denoed he Rollng IP for perod cn be formuled s: Rollng IP ( RP ) for Perod Mxmze Subjec o: x v j xj (5) j j j j b x B sr j j (6) (7) x {0,}, j. (8) j The objecve funcon (5) s he sum of collecon volume. Consrns (6) perm only one budge moun o be lloced o reler. Consrn (7) resrcs he overll spendng budge o be less hn he budge lm. Consrns (8) force x vrbles s bnry vrbles. The procedure for he Rollng IP wh DP mehod s dscussed nex by combnng he Rollng IP formulon ogeher wh he rollng horzon concep. j Rollng IP wh DP Heursc (RIDH) Soluon Procedure Sep 0 Se nd sr B0 mx B. Solve for v j from SR(, bj, T ) s defned erler n hs secon for ll gens nd budge level j usng he DP pproch developed n secon 4.. The nl se of SR(, bj, T ) s [0, nl wllngness se of reler, b j ]. Sep Formule he rollng IP ( RP ) model nd solve for x j. Sep 2 For he relers for where recrung budge hs been lloced, smule he con n perod only. If T, obn he ol ncremen n collecon volume from perod o perod T nd ex. Oherwse, go o Sep 3. Sep 3 Se. Upde he vlue of v j from for ll, j. Noe h here s no need o resolve MDP for ech reler. Obn he v j by chngng he srng nl se o [+, new wllngness se, remnng budge]. For exmple, f he nl se s [0,M,30], budge moun 0 s ppled o hs perod, he nex perod sus chnge o H, nd he overll remnng budge s 0, hen v j cn be looked up 7
from se [,H,0]. Upde remnng budge prevous perod). sr sr sr B, ( B B cul budge spen n he Go o Sep These seps cn be summrzed by he flow chr shown n Fgure 8. Se 0 Obn he vlue for ll v j Formule nd solve IP ( RP ) Smule nd upde he cul se of ech reler. No T Se Yes Obn fnl resul Upde remnng budge nd v j Fgure 8: Procedure for he RIDH Soluon Approch 5. Compuonl Resuls In hs secon, he lernve soluon pproches re ppled o smll nd lrge exmples. Snce he DP lgorhm cn fnd n opml soluon for smll exmple n resonble compuon me, s soluon cn be used s benchmrk gns he soluons obned by he RIDH, nd QBH. For he lrge exmple, he compuonl requremens re prohbve for he DP lgorhm. Thus, only he resuls from he wo heurscs re compred. All he compuon expermens re solved usng Wndows 2000-bsed Penum 4.80 GHz personl compuer wh 640MB of RAM wh CPLEX verson 8.0 (www.log.com) for he opmzon sofwre. 5. Smll Exmple For our smll exmple, he recrumen wllngness fcors for he wllngness se of ech reler re defned s H 2, M, nd L 0.5. Equon () s used o compue he probbles of recrumen, ( PrLR ( ), PrMR ( ), PrHR ( ) ), for he gven wllngness se nd budge llocon ( ). If recrumen does no occur, hen here s sll he chnce he se of reler wll chnge. Ths lso depends on he recrumen budge hreshold of he 8
reler. When he gven budge llocon fls o recru he reler, wo cses re consdered. Cse A: If -, he rnson probbles re depced n Fgure 9. 0.9 0 0.9 L M H 0 0. 0. 0. 0.9 0 Fgure 9: Trnson Probbles for Cse A Cse B: If 0, he rnson probbles re depced n Fgure 0. 0 0.6 0.4 L 0.3 0.4 0.3 0.6 M Fgure 0: Trnson Probbles for Cse B Wh hese sengs, we genere four es cses h hve dfferen relers nl wllngness ses s shown n Tble. Tble 2 shows he moun of collecon cpcy nd recrumen budge hreshold of ech reler. The lernve budge lmon sengs re spced 0 uns pr 0, 20,, 00 nd he budge llocon sengs re smlrly spced. The number of me perods s chosen o be hree. From he collecve reler collecon cpces, he mxmum sysem collecon cpcy n ll hese four cses s 220 pounds, over gven me perod. 0.4 Tble : Smll Exmple D Cse Inl Wllngness Se LLLLL 2 MMMMM 3 HHHHH 4 MHHMH 0 H 9
Tble 2: Smll Exmple D Reler Collecon Cpcy (lb.) Recrumen Budge Threshold 0 5 2 30 5 3 70 49 4 20 20 5 90 8 Tbles 3,4,5, nd 6, dsply he soluon verge collecon cpcy, compuon me, nd opmly gp for soluon pproches DP, RIDH, nd QBH for dfferen mxmum budge sengs of cses,2,3, nd 4 respecvely. The verge collecon cpcy s compued from he resuls obned by pplyng he polcy resulng from he dfferen soluon mehods for 00 replcons. The opmly gp llusres he soluon quly found by he RIDH nd QBH mehods compred o he opml soluon obned by he DP lgorhm. For he DP lgorhm, he compuon me s obned by exmnng every possble se nd con n every perod nd selecng he polcy h yelds he mxmum verge collecon cpcy. For he QBH soluon pproch, he mxmum number of eron s se o 00,000. Tble 3: Cse Soluon: Cpcy Collecon, Soluon Tme nd Opmly Gp Mxmum Budge Soluon Approches DP RIDH QBH Collecon Soluon Collecon Soluon Collecon Soluon Cpcy Tme Cpcy Tme Opmly Cpcy Tme (lb.) (sec.) (lb.) (sec.) Gp (%) (lb.) (sec.) Opmly Gp (%) 0 2.4 6 2.4 5 0.0.6 6 6.4 20 27.6 8 28.5 7 0.0 2.5 6 22. 30 46.3 568 4.4 9 0.5 40.4 7 2.7 40 76.8 2,086 80.4 4 0.0 8.0 8 89.5 50 90. 8,282 9.0 9 0.0 27.9 9 69.0 60 06.7 6,229 03.0 23 3.4 75.9 9 28.8 70-86,400 3 4.7 32-92.8 0-80 - 86,400 38. 4-92. - 90-86,400 53.8 59-93.0-00 - 86,400 65.2 65-95.6 - The resuls for Cse show h he soluon verges for he collecon cpcy obned by he RIDH mehod re close o he opml soluon for every mxmum budge seng. The lrges opmly gp s 0.5%. For mxmum budge sengs of 20, 40, nd 50, he verge soluon for collecon cpcy found by he RIDH pproch hppens o be slghly hgher hn he vlue found by he DP pproch becuse of he rndom numercl evluon found by smulng 00 replcons. For hs suon, he opmly gp s se o zero. The compuon me requremens for he RIDH pproch re much smller hn hose for he DP mehod. The QBH mehod requres he les moun of soluon me for every 3 Algorhm ws sopped when he compuon me requremen reched 86,400 seconds or one dy. 20
mxmum budge seng, bu he opmly gp s lrger hn h found by he RIDH pproch. In fc, s soluon s worse hn soluon obned by he RIDH pproch for every mxmum budge seng. For mxmum budge sengs of 80 o 00, he DP mehod cnno obn opml polcy whn he soppng me lm of one dy of compuonl effor. Tble 4: Cse 2 Soluon: Cpcy Collecon, Soluon Tme nd Opmly Gp Mxmum Budge Soluon Approches DP RIDH QBH Collecon Soluon Collecon Soluon Collecon Soluon Cpcy Tme Cpcy Tme Opmly Cpcy Tme (lb.) (sec.) (lb.) (sec.) Gp (%) (lb.) (sec.) Opmly Gp (%) 0 2.2 2.6 4 0.0.5 6 87.7 20 47.2 8 44.4 6 5.9 5.6 6 88. 30 84.8 40 82.6 8 2.5 6.3 6 80.7 40 00.6 38 99..4 56.7 7 43.6 50 3.4 393 30.7 5 0.5 52. 8 60.3 60 58.9 962 55.5 6 2. 45.4 8 7.4 70 75.3 2,097 65.4 23 5.6 82.3 9 53.0 80 87.2 4,49 80. 26 3.7 84.7 9 54.7 90 20.5 8,47 97.5 38.9 7.9 9 4.4 00 20.0 5,309 20.4 4 4. 8.3 0 3.6 The overll resuls for Cse 2 follow he sme rends s Cse. For he sme mxmum budge seng, he verge soluon s collecon cpcy n hs cse s hgher hn n Cse becuse he relers n Cse 2 sr n more fvorble ses hn ones n Cse. Furhermore, he compuon me requremens re less n hs cse becuse he probbly of he reler movng bck o se L s smll. Tble 5: Cse 3 Soluon: Cpcy Collecon, Soluon Tme nd Opmly Gp Mxmum Budge Soluon Approches DP RIDH QBH Collecon Soluon Collecon Soluon Collecon Soluon Cpcy Tme Cpcy Tme Opmly Cpcy Tme (lb.) (sec.) (lb.) (sec.) Gp (%) (lb.) (sec.) Opmly Gp (%) 0 44. 46.9 5 0.0 5.4 6 87.7 20 90.0 90.0 3 0.0 54.9 6 39.0 30 37.7 2 39.7 5 0.0 63. 6 54. 40 7. 5 66.9 6 2.4 84.8 7 50.4 50 99. 89.4 6 4.8 00.8 7 49.3 60 2.5 27 20.0 0 0.7 94.8 8 7.9 70 220.0 59 220.0 9 0.0 205.3 8 6.6 80 220.0 6 220.0 6 0.0 220.0 8 0.0 90 220.0 25 220.0 22 0.0 220.0 9 0.0 00 220.0 377 220.0 24 0.0 220.0 9 0.0 2
For he soluon of Cse 3, he overll resuls follow he sme rends s n Cses nd 2. For he sme budge lm, he verge soluon s collecon cpcy n hs cse s hgher hn ones n Cses nd 2 becuse he relers n Cse 3 sr wh hghes fvorble ses compred o he ones n Cses nd 2. Furhermore, he compuon me requremens re less n hs cse. When he mxmum budge equls 70, every reler cn be recrued no he sysem. I s neresng o see h he QBH pproch performs lmos s well s he DP lgorhm when he budge lm s equl or greer hn 60. Tble 6: Cse 4 Soluon: Cpcy Collecon, Soluon Tme nd Opmly Gp Mxmum Budge Soluon Approches DP RIDH QBH Collecon Soluon Collecon Soluon Collecon Soluon Cpcy Tme Cpcy Tme Opmly Cpcy Tme (lb.) (sec.) (lb.) (sec.) Gp (%) (lb.) (sec.) Opmly Gp (%) 0 46.2 47.5 4 0.0 30.0 6 35.0 20 90.0 84.8 4 5.7 50.0 6 44.4 30 38.3 4 39.2 5 0.0 60.0 7 56.6 40 77.7 5 73.6 6 2.3 64.6 7 63.6 50 78.2 44 92.5 8 0.0 77.8 7 56.3 60 99. 03 99. 0 0.0 90. 8 54.7 70 22.5 226 23.4 2 0.0.0 8 47.7 80 29.9 468 29.8 9 0. 27.3 8 42. 90 220.0 863 220.0 24 0.0 68.4 9 23.4 00 220.0,65 220.0 26 0.0 50.5 9 3.5 For Cse 4, he quly of he soluon obned by he RIDH pproch s lmos s good s he soluon obned by he DP lgorhm for every mxmum budge seng. The QBH pproch does no provde very good soluons for hs cse. The resuls from he smll exmple show h he RIDH soluon pproch performs lmos s well s he opml DP procedure n ll cses, wh much lower compuonl effor. The QBH mehod solves he smll exmple recrumen problem wh he les compuonl effor, bu yelds he wors verge collecon cpcy soluons compred o he DP nd RIDH mehods. Nex, we es hese procedures on lrger problem. 5.2 Lrge Exmple In hs secon, we consruc hree cses o exmne for he lrge exmple. Cse 5 consss of dfferen number of relers (5,0,5,20) wh lrge collecon cpces nd ll relers srng n wllngness se L. The soluon resuls re shown n Tble 7. Cse 6 consss of dfferen number of relers (5,0,5,20) wh smll nd md-sze collecon cpces nd ll relers srng n wllngness se M. The soluon resuls re shown n Tble 8. Cse 7 consss of dfferen number of relers (5,0,5,20) wh lrge collecon cpces nd ll relers srng n wllngness se H. The soluon resuls re shown n Tble 9. 22
Tble 7: Resuls for Cse 5 Soluon Approches RIDH QBH Number of Mxmum Collecon Bes Collecon Soluon Tme Collecon Bes Collecon Soluon Tme Relers Budge Cpcy (lb.) Cpcy(lb.) (sec.) Cpcy(lb.) Cpcy (lb.) (sec.) 5 50 66.3 90 9 6. 90 6 00 46.4 230 6 2.6 50 7 50 242 320 3 4.6 220 9 200 325.8 390 52 87.7 240 0 0 50 76.5 90 2 8.5 90 00 62 230 25 85.4 40 6 50 25.3 340 56 8 70 26 200 347.5 430 4 93.2 230 32 5 50 74.7 90 6.6 80 7 00 52.5 230 33 93.8 40 30 50 26.7 340 82 86.7 70 54 200 362.8 450 65 96.8 230 65 20 50 74.7 90 6 2.8 80 23 00 47.5 230 4 93. 40 45 50 263. 340 24 4. 20 74 200 359 450 9 92.4 230 85 Tble 8: Resuls for Cse 6 Soluon Approches RIDH QBH Number of Mxmum Collecon Bes Collecon Soluon Tme Collecon Bes Collecon Soluon Tme Relers Budge Cpcy (lb.) Cpcy(lb.) (sec.) Cpcy(lb.) Cpcy (lb.) (sec.) 5 50 98.2 0 8 79.8 0 8 00 44.6 50 9 4.6 50 9 50 50 50 9 50 50 8 200 50 50 3 50 50 9 0 50 7.8 30 9 94.6 20 3 00 207 230 7 34.8 80 20 50 274.3 300 33 20.8 250 26 200 302.7 30 62 27.4 30 27 5 50 22.2 40 2 73.6 20 2 00 235.9 260 23 46 80 35 50 323.6 350 49 223.8 260 5 200 399. 430 96 298 340 56 20 50 27.2 40 2 03.5 0 30 00 260. 280 27 48 90 5 50 360. 390 63 22.2 270 66 200 45.7 490 4 327.6 330 75 23
Tble 9: Resuls for Cse 7 Soluon Approches RIDH QBH Number of Mxmum Collecon Bes Collecon Soluon Tme Collecon Bes Collecon Soluon Tme Relers Budge Cpcy (lb.) Cpcy(lb.) (sec.) Cpcy(lb.) Cpcy (lb.) (sec.) 5 50 28.5 30 6 55.6 320 6 00 387.9 390 6 37.5 390 7 50 390 390 5 38.6 390 9 200 390 390 2 390 390 9 0 50 23.7 300 8 56.6 330 00 462.7 550 349.3 530 3 50 66.3 790 2 528.5 560 6 200 788.6 790 33 600.4 720 6 5 50 24.5 350 0 22.2 250 5 00 465.9 550 4 42.3 500 8 50 688.7 80 27 464.6 570 23 200 879.7 970 5 736.4 790 26 20 50 248.5 350 0 88. 320 20 00 473.4 600 8 425.6 480 25 50 695.3 860 36 586.8 680 37 200 908.6 040 62 667.3 760 43 The resuls from Cses 5, 6, nd 7 show h he RIDH pproch ouperforms he QBH mehod n every cse. The bes collecon cpcy represens he lrges collecon cpcy h he soluon mehod hs found so fr nd hs se s rge o cheve. Even hough he QBH pproch requres less compuonl effor o obn he resuln polcy, s verge soluon collecon cpcy s domned by he one obned by he RIDH pproch, whch lso provdes hgher bes collecon cpces hn he one obned by he QBH mehod. 6. Conclusons Ths pper s he frs o employ recrumen concep for reverse supply chn pplcons. We model he behvor of relers who hve dfferen udes owrds prcpng n recyclng cves s Mrkov process descrbng rnsons h chrcerze her movemen owrds jonng he nework. Usng hs mechnsm, he recrumen problem s formuled s sochsc dynmc progrmmng problem. Ths pper provdes n exc soluon mehod (DP lgorhm) for smll problems nd wo heurscs, QBH nd RIDH, for lrger problems. The QBH pproch s bsed on smulon-bsed opmzon echnque o vod compung he lrge rnson probbly mrx. The RIDH mehod ulzes he benef of rollng horzon feure nd IP cpbles n order o cpure he recrumen decsons over me. I uses heursc decomposon of he problem bsed on he polces h would be opml for ech reler n he bsence of ny ohers. Numercl sudy demonsres h he RIDH pproch provdes he verge soluon collecon cpces lmos s good s he ones obned by exc he DP pproch when smll exmple s consdered. Furhermore, he compuonl resuls lso show h he RIDH mehod cn solve lrge recrumen problems quckly wh good soluon quly. The bly o solve n cul sze recrumen problem cn enbles us exmne mulple recrumen problems sregclly. In ddon, he recrumen model cn ppled o mny 24
ypes of recrumen problems such s recrung supermrke sores for he collecon of plsc boles n he plsc recyclng ndusry nd recrung he mjor elecronc sores such s Bes Buy nd Crcu Cy for used elecroncs equpmen n he elecronc scrp recyclng ndusry. If hese decsons were lef up o ndvdul sore mngers nd no cenrlly mnded. Resuls from hs pper rse new quesons nd severl poenl drecons of fuure reserch. Fuure exensons cn be envsoned n boh he modelng nd soluon mehodology res. Currenly we hve developed ccl collecon model under he ssumpon h once he gen s recrued o he nework, lwys sys n he nework. However, n cul suons, somemes collecon gen my op o leve he nework. Fuure work ncludes exendng he recrumen model o nclude reenon nd defecon consderons. An mporn subsk s o defne he crer h deermne he reenon nd defecon cons of he gen fer s recrued. The ddonl complexy wll mpc he cpbly of he curren pproch o solve lrge scle collecon recrumen problems. Anoher drecon of fuure work ncludes explorng he collecon logscs where he generon re of collecon merl s sochsc mong he gens. Wh hs uncerny, he problem of roung fxed number of fne cpcy rucks o collec he merl from he collecon gens s more dffcul. In ddon, n he rel pplcon, he recrumen process nd reler reenon my depend no only on he connecon beween he relers nd he recruer, bu lso on he ousde mrke, compeor. For exmple, currenly compnes n Chn re showng neres n used crpe from U.S. sources o brng o Chn for recyclng. Hence, here cn be compeon for he desred source. Relers boh n nd ou of he collecon nework my op o gve he source o compeor collecors who provde beer ncenve. Ths lso ffecs he recrumen llocon pln for he crpe recycler n he U.S. Addng compeon feure from he gme heory perspecve o he recrumen model cn complce he model frmework bu provdes deeper undersndng of how he enes mgh c n he rel suon. References Adelmn, D. nd Mersereu, A.J. (2004), Relxons of Wekly Coupled Sochsc Dynmc Progrms, Workng Pper, Grdue School of Busness, The Unversy of Chcgo Bss, F. (969), A New Produc Growh Model for Consumer Durbles, Mngemen Scence, 5(), 25-227. Bersms, D. nd Nno-Mor, J. (2000), Resless Bnds, Lner Progrmmng Relxons, nd Prml-Dul Index Heursc, Operons Reserch, 48(), 80-90. Bellmn, R.E. (957), Dynmc Progrmmng, Prnceon Unversy Press, Prnceon, New Jersey, USA. Cnd's Wse Recyclng Mrkeplce (2006), hp://www.recyclexchnge.com, vewed 03/6/2006. Crpe Amerc Recovery Effor (2004), Crpe Amerc Recovery Effor s Annul Repor n 2004, hp://www.crperecovery.org/nnul_repor/04_care-nnul-rp.pdf, vewed 03/6/2006. Coughln, A.T., Gryson, K. (998), Nework Mrkeng Orgnzons: Compenson Plns, Rel Nework Growh, nd Profbly, Inernonl Journl of Reserch n Mrkeng, 5, 40-426. 25
Drmon, R. Y. (2003), Conrollng Sles Force Turnover Coss hough Opml Recrung nd Trnng Polces, Europen Journl of Operon Reserch, 54, 29-303. Gosv, A. (2003), Smulon-bsed Opmzon: Prmerc Opmzon Technques nd Renforcemen Lernng, Kluwer Acdemc Publshers. Georgou, A.C., Tsns, N. (2002), Modelng Recrung Trnng n Mhemcl Humn Resource Plnnng, Appled Sochsc Models n Busness nd Indusry, 8, 53-74. Hwkns, D.A. (992), An Acvon-recrumen Scheme for Use n Muscle Modelng, Journl of Bomechncs, 25, 467-476. IDES The Plscs Web (2006),hp://www.des.com/resnprce/resnprcngrepor.sp, vewed 03/6/2006. Kelblng, L.P., Lmn, M.L., Moore, A.W. (996), Renforcemen Lernng: A Survey, Journl of Arfcl Inellgence Reserch, 4, 237-285. Knuh, D. (973), The Ar of Compuer Progrmmng, Volume 3: Sorng nd Serchng, Addson Wesley. Mehlmnn, A. (980), An pproch o opml recrumen nd rnson sreges for mnpower sysems usng dynmc progrmmng, Journl of he Operonl Reserch Socey, 3, 009-05. Puermn, M.L. (994), Mrkov Decson Processes, Wley Inerscence, New York, USA. Seggern, V.D. (993), CRC Sndrd Curves nd Surfces. Boc Ron, FL: CRC Press, p. 24, 993. Treven, S. (2006), Humn Resources Mngemen n he Globl Envronmen, Journl of Amercn Acdemy of Busness, 8, 20-25. Wkns, C.J. (989), Lernng from Delyed Rewrds, Ph.D. Dsseron, Kngs College, Cmbrdge, Englnd, My 989. Whle, P. (988), Resless Bnds: Acvy Allocon n Chngng World, J. Gn, ed. A Celebron of Appled Probbly, Journl of Appled Probbly, 25A, 287-298. 26