Strategic, Tactical and Real Time Planning of Locomotives at Norfolk Southern Using Approximate Dynamic Programming

Transcription

1 Sraegic, Tacical and Real Time Planning of Locomoives a Norfolk Souhern Using Approximae Dynamic Programming Warren B. Powell powell@princeon.edu (609) Belgacem Bouzaiene Ayari belgacem@princeon.edu Deparmen of Operaions Research and Financial Engineering Sherrerd Hall Princeon Universiy Princeon, NJ Coleman Lawrence coleman.lawrence@nscorp.com Clark Cheng clark.cheng@nscorp.com Sourav Das sourav.das@nscorp.com Ricardo Fiorillo ricardo.fiorillo@nscorp.com Norfolk Souhern Corporaion 1200 Peachree S. NE, Box Alana GA March 9, 2012

2 Absrac Locomoive planning has been a popular applicaion of classical opimizaion models for decades, bu wih very few success sories. There are a hos of complex rules governing how locomoives should be used. In addiion, i is necessary o simulaneously manage locomoive invenories by balancing he need for holding power agains he need for power a oher yards. A he same ime, we have o plan he need o reurn foreign power, and move power o mainenance faciliies for scheduled FRA appoinmens. An addiional complicaion arises as a resul of he high level of uncerainy in ransi imes and delays due o yard processing, and as a resul we may have o plan addiional invenories in order o move oubound rains on ime despie inbound delays. We describe a novel modeling and algorihmic sraegy known as approximae dynamic programming, which can also be described as a form of opimizing simulaor which uses feedback learning o plan locomoive movemens in a way ha closely mimics how humans plan real world operaions. This sraegy can be used for sraegic and acical planning, and can also be adaped o real ime operaions. We describe he sraegy, and summarize experiences a Norfolk Souhern wih a sraegic planning sysem.

3 Page 1 Locomoive planning is one of he mos complex operaional problems in freigh ransporaion. Planners have o ake ino consideraion a hos of operaional characerisics ha describe a locomoive o bes uilize he flee o mee he service requiremens of he rains. Locomoive flees can represen billions of dollars in invesmens, and as a resul railroads have every incenive o manage his invesmen as efficienly as possible. The complexiy of he problem has pu i well pas he capabiliies of even oday s advanced opimizaion solvers. Compleely overlooked in hese models are he imporan sources of uncerainy such as ransi ime delays, he dynamics of scheduling commodiies such as coal and grain, and he ever presen problem of equipmen failures and mainenance. Railroads face hree classes of planning problems when managing railroads: Sraegic planning The major quesion here is flee size and mix, bu oher quesions can include undersanding he impac of improvemens in ransi ime reliabiliy, he effec of changes in rain plans and changes in inerchange policies wih oher railroads. Tacical planning and operaional forecasing Here he horizon is 1 7 days, and he quesion is wheher here will be significan shorages of power ha migh require addiional reposiioning, shor erm modificaions in leasing decisions and perhaps he decision o reain locomoives ha belong o oher railroads (known as foreign power ) and incur addiional per diem charges. Real ime planning The decision here is he assignmen of specific locomoives o specific rains ha are deparing over he immediae horizon (ypically he nex 12 hours). A large railroad may have billions invesed in heir flee of locomoives. Too many locomoives mean ha hundreds of millions are invesed in equipmen ha is no yielding a reurn. However, a failure o mainain a sufficienly large flee ranslaes o delayed rains and service failures ha can seriously impac revenue. Despie he scale of his invesmen, i is no unusual for a large railroad o plan heir locomoive flees using a simple linear regression ha relaes operaing saisics such as forecased onnage and operaing speeds o he amoun of power required o run he railroad. Given he size of he invesmen, railroads have ried for decades o ap he power of opimizaion ools o manage heir flees more effecively. Booler (1980) presens a very early aemp a using linear programming o solve a scheduling model. Chih (1986) and Chih e al. (1990) describe an implemenaion of an ineger programming model for locomoive scheduling a he Burlingon Norhern Sane Fe Railroad (see also Forbes (1990)). This early work sruggled wih he limiaion of ineger programming algorihms, despie using highly simplified models of locomoive operaions. A number of advances have been made in he design of specialized algorihms o solve ineger programming formulaions locomoive scheduling. Ziarai (1999) presens a new branch and cuc algorihm. Ahuja e al (2005) describe a heurisic based on very large scale neighborhoods o find near opimal schedules for locomoives which considers consis breakups and he desire for weekly paerns in he flows of locomoives. Vaidyanahan e al. (2008a) provides a deailed model of he locomoive rouing problem capuring a number of operaional consrains wih an adapaion of heir large neighborhood search sraegy (see also Vaidyanahan e al. (2008b) for addiional experimenal work). Cordeau e a. (2000, 2001) describe he use of Benders decomposiion for he simulaneous assignmen of cars and 1

4 Page 2 locomoives. A he same ime, i is imporan o recognize major advances o general purpose ineger programming solvers such as Cplex and Gurobi ha have occurred since Below, we repor on experimens wih Cplex 12 running on a mulihreaded machine. This paper describes a muliyear effor o develop a family of locomoive planning models for Norfolk Souhern Railroad. The resul is he Princeon Locomoive And Shop MAnagemen sysem (PLASMA), which can be used for sraegic planning, shor erm operaional forecasing, and real ime operaions. PLASMA has been imbedded in a larger informaion sysem developed a Norfolk Souhern called he Locomoive Assignmen and Rouing Sysem (LARS). As of his wriing, he sysem has been used for sraegic flee sizing for several years and has become an inegral par of he company s nework and resource planning processes. The operaional forecasing sysem has been undergoing exensive user accepance esing while Norfolk Souhern has been upgrading is informaion sysems o improve he accuracy of some of he daa. We hen provide an indicaion of how he sysem can be implemened as a real ime, ineracive sysem. Locomoive operaions Locomoives are described by a hos of aribues including horsepower and racive effor, he owning railroad, is mainenance saus (e.g. days unil he nex federally mandaed mainenance appoinmen), and equipmen deails such as communicaions gear (for coordinaing muliple locomoives). Locomoives ypically need o be bundled ogeher ino consiss of one o perhaps five locomoives which are needed o pull a paricular rain. The process of connecing locomoives is ime consuming, requiring he connecing and esing of cables ha allow he se of locomoives o work as a single uni. Trains ofen arrive from a neighboring railroad using locomoives owned by ha railroad (known as foreign power ). Normally foreign locomoives are reurned o he owning railroad, bu i is possible o use hese locomoives, bu only wihin negoiaed limis If he locomoive is reurned, his has o be handled hrough pre defined exchange poins. The rains hemselves also have numerous aribues. The number of locomoives needed o pull a rain depends on he weigh of he rain, he speed requiremens (merchandise rains need o move more quickly han coal and grain rains), and he seepes grade ha he rain has o navigae. Trains have differen service prioriies, and a scheduler has o consider if here are no enough locomoives o move all he rains on ime. Shop rouing (geing he locomoive o a mainenance shop) is one of he mos complex issues facing a locomoive manager. Locomoives need o have regular mainenance reviews on a periodic schedule (ypically every 90 days). If a locomoive does no make is shop appoinmen on ime, i has o be urned off and owed o he shop. Of course, on some rains i is possible o simply add a locomoive as an exra locomoive and move i o is shop appoinmen, bu i is beer o use he locomoive producively. A he same ime, he scheduler has o balance geing he locomoive o shop early (which coss produciviy) or risk ha i may arrive lae (for example, by missing a criical connecion). On op of his, here may be muliple shop locaions ha can service a locomoive. I is necessary o anicipae he number of locomoives ha are scheduled a each shop in order o balance he loads and mainain a seady flow of work. 2

5 Page 3 A separae issue ha is widely discussed bu rarely solved concerns he differen ypes of uncerainy ha plague locomoive operaions. These include: Transi ime delays These can be as long as six o 12 hours for he shorer movemens of an Easern railroad, o more han a day for he long movemens of he wesern railroads. Dynamic schedule changes Planners also have o deal wih he paern of scheduling addiional rains for commodiies such as coal and grain, wih as lile as one or wo days advance noice. Shop delays Mainenance managers will provide esimaes of when a locomoive will be ready o leave a shop, bu hese are jus esimaes and frequen calls o a mainenance faciliy are ofen needed o deermine if a locomoive will be ready for a paricular rain. Equipmen failures Locomoives may fail unexpecedly, and his represens an addiional source of uncerainy. The model presened in his paper is designed o handle uncerainy, bu producion applicaions of he model have ye o exploi his capabiliy. Figure 1 Illusraion of mulicommodiy nework flow problem over a ime space nework. Deerminisic opimizaion models The mos common sraegy for modeling locomoives as an opimizaion problem is o use he framework of mulicommodiy flows over a ime space nework, depiced in figure 1. Mahemaically, such a model can be wrien in a generic forma using 3

6 Page 4 min T K k k x cx ij ij 1 k1 (1) subjec o A x B x R (2) k k k k 1 1 Dx u (3) x 0 and ineger. (4) k ij k Here, x ij represens he flow of locomoives of ype k moving from yard i o yard j deparing a ime. Equaion (2) capures flow conservaion for locomoives of each ype. Equaion (3) capures he number of locomoives needed o move a rain (summing across differen ypes of locomoives). Equaion (4) requires ha he flow variables be ineger. This basic model has o be adjused o allow for he following: If a rain requires hree locomoives o mee speed and grade requiremens, we can sill use he rain o pull more locomoives if we need o reposiion power from one yard o anoher. One or more locomoives can be moved from one yard o anoher wihou pulling a rain, an operaion known as a ligh engine move. Ligh engine moves incur crew coss and as a resul need o be carefully managed. There is a cos for coupling locomoives ogeher ino a consis o pull a single rain, as well as a cos for decoupling he locomoives. This operaion also akes ime which has o be buil ino he dynamics. Locomoives ha need o be roued o shop have o be modeled. A paricularly difficul challenge is he modeling of rain delay. The mos common assumpion is o use a ime space nework as shown in figure 1, bu o hen replicae rain movemens over muliple ime periods. Since he same rain canno move a differen poins in ime, hey are linked by a bundle consrain ha ensures ha only one copy of he rain can move, as shown in figure 2. Figure 2 Time space represenaion of rains wih muliple deparure imes. 4

7 Page 5 Figure 3 Task graph represenaion of rains and locomoive arrivals. Our experience wih his sraegy was ha i inroduced discreizaion errors ha were unaccepable o he railroad. If we use a one hour ime sep, he nework simply explodes. However, if we use a slighly more manageable four hour ime sep (which sill increases he size of he problem dramaically), we encouner siuaions where a locomoive may arrive 20 minues oo lae o serve a rain, bu his hen forces us o impose a four hour delay. This was fel o be an unaccepable disorion, and i dramaically inflaed rain delay. We avoid his problem by using a classic ask graph formulaion, where a rain is modeled as a ask which is characerized by an iniial ime of availabiliy. The basic idea is illusraed in figure 3. Locomoives are modeled as arriving in coninuous ime. Dashed assignmen arcs can join a locomoive a any poin in ime o a specific rain. The deparure ime of he rain is governed by he laes locomoive ha is acually assigned o he rain, which allows us o model rain delays of any lengh. Deerminisic models are exremely hard o solve over long planning horizons, especially when modeling he abiliy o delay rains. Figure 4 shows he run imes as he horizon is exended for he Norfolk Souhern flee. Noe he excepionally fas execuion ime for he single day horizon. For his problem, locomoives are being assigned o a mos a single rain. As he horizon grows, we have o model he cascading of rain delays in he ask graph. Wih a horizon of as lile as four days, he run imes are already exceeding 50 hours. Figure 4 Execuion imes for increasing planning horizons for he full flee of locomoives. 5

8 Page 6 An approximae dynamic programming model and algorihm Approximae dynamic programming is a modeling and algorihmic sraegy ha decomposes decisions over ime (see Powell (2011) for an inroducion using he conceps and noaion in his paper; approximae dynamic programming is closely relaed o he field of reinforcemen learning, see Suon and Baro (1998)). I was originally developed o handle problems which involve uncerainy. Our own work, however, has focused on is abiliy o decompose large deerminisic problems, overcoming he dramaic increase in CPU imes documened in figure 4. In fac, all of he producion applicaions of ADP a Norfolk Souhern have been conduced using a deerminisic model. In his paper, we repor on some recen experimens using ADP o improve he robusness of he soluion in he presence of uncerainy in ransi imes. This paper will no aemp o presen a complee mahemaical model. Insead, we provide a skech of how approximae dynamic programming approaches he problem. The basic idea in an ADP approach is o se up a subproblem where we assign locomoives o rains over a relaively shor horizon (perhaps 4 6 hours). We could sar by maximizing he conribuion we earn now, ignoring he impac of decisions now on he fuure. In such a model, le S The sae of our sysem, including he saus of curren and inbound rains, and he oubound schedule of rains ha we know of a ime. x A vecor of assignmens of curren and possibly inbound locomoives o oubound rains over a specific horizon (such as 4 6 hours). CS (, x) Conribuion earned if we are in sae S and make decision x. This would include bonuses for moving rains (which reflec he prioriy of he rain), coss for forming or breaking consiss, and any coss ha are incurred implemening he decision. If we are willing o ignore he fuure, we would make decisions using a myopic policy defined by M X ( S ) argmax C( S, x ). xx This means we find he se of assignmens ha produces he highes conribuion now. Such a policy would never reposiion power for use in he fuure. Furhermore, we migh use power now on a low prioriy rain, ignoring he very high prioriy rain ha has o leave, say, 10 hours from now. A policy ha overcomes his limiaion looks like x 1 1 M X ( S ) arg max X C( S, x ) V ( S ) S (5) M where S 1 S ( S, x, W 1) is he sae a he nex ime period (ypically four hours from now) given ha we are in sae S righ now (his specifies he available locomoives and rains), we make decision x (his deermines which locomoives are assigned o each rain, and which locomoives are held), and where W 1 capures he random informaion such as rain delays, schedule changes and equipmen 6

9 Page 7 Figure 5 Illusraion of he single period decision problem using value funcion approximaions for locomoives in he fuure. problems ha were no known a ime. In a deerminisic implemenaion, he variable W 1 does no conain anyhing (no rains are delayed, no schedule changes are made, and no equipmen fails). The expecaion in equaion (5) generally canno be compued, bu we can ge around his (for sochasic problems) by using he concep of he pos decision sae variable. While his arises in differen forms, in his seing i is easies o hink of i as ha he sae ha we would land in if he random informaion W 1 is wha we expec i o be (say, he average ravel ime, or assuming ha here are no changes in he schedule and no equipmen failures). Le W, 1 be a forecas of W 1 given wha we know a ime. We can wrie he pos decision sae x M S S ( S, x, W, 1). x S using We can hen replace equaion (5) wih V x x X ( S) argmax xx C( S, x) V ( S ). (6) x x Finally, we will never be able o calculae he value funcion V ( S ) exacly, so we replace i wih an approximaion ha we wrie as V( S x ) V x ( S x ). The challenge hen is o design an approximaion V( S x ) ha is easy o esimae, allows equaion (6) o be solved fairly easily, and works well. 7

10 Page 8 Figure 6 Illusraion of he process of sepping forward hrough ime, solving sequences of small locomoive assignmen problems wih value funcion approximaions. Recall ha he sae variable consiss of locomoives and rains. We have found ha an approximaion sraegy ha works well ignores he value of rains in he fuure, and uses a piecewise linear funcion o approximae he value of he major classes of locomoive (low/high adhesion, low/high horsepower). Capuring he value of rains in he fuure conribues o beer decisions only when rains are delayed, which happens fairly infrequenly. This produces an opimizaion problem ha is depiced in figure 5. In he figure, we see he iniial assignmen of locomoives o rains. We also use a piecewise linear funcion (idenified as he rain reward funcion ) o capure he value of puing power on he rain. Each rain has a criical poin, below which he rain canno move. We hen model he fac ha here may be value in adding power above he criical poin, bu only up o a specified goal horsepower. We can coninue o add power because i is needed a a downsream yard, bu here is a cos o doing so (he posiive conribuion is reduced). Finally, we model he value of each ype of locomoive a he desinaion locaion using piecewise linear value funcions. This is done a wo levels of aggregaion: he value of each of he major ypes of locomoive a a locaion, and he value of he oal amoun of power a a yard. This helps he model learn boh he righ mix of power, as well as he righ amoun of power in aggregae. The policy in equaion (6) (and depiced in figure 5) is an ineger program, bu wih a very small horizon. As a resul, i can be solved in a few seconds. The value funcion approximaions will no be known in advance. These are learned adapively by compuing he marginal value of each ype of locomoive a each yard, a each ime period. There is by now a fairly exensive lieraure on hese mehods, and we refer he deermined reader o Powell (2011) (chaper 13). The basic idea, hen, is o sar wih a value n funcion V 1 () s which deermines our policy (basically, he decision funcion in equaion (6)). We sep hrough ime simulaing he policy, as depiced in figure 7, and use he informaion o produce an n updaed value funcion approximaion V () s. If we wan o capure uncerainy, we use Mone Carlo simulaion o sample rain delays, schedule changes and equipmen failures in he informaion variable W1, W2,..., W,... However, if we are using a deerminisic model, we jus simulae he average ransi ime, and ignore schedule changes and equipmen failures. This way, we can solve large deerminisic 8

11 Page 9 problems using a sequence of very small ineger programming problems. The price is ha we have o run hese simulaions ieraively o learn he value funcion Ieraions Figure 7 Growh in objecive funcion due o ieraive learning The improvemen in he objecive funcion as a resul of he adapive learning is shown in figure 7. Noe ha he jump in he objecive funcion afer 10 ieraions is due o an algorihmic sraegy where no rains are allowed o be delayed for he firs 10 ieraions, a sraegy ha helps o sabilize he value funcions. We found ha 50 ieraions gave good soluions. A major feaure of his sraegy, compleely separae from he abiliy o handle uncerainy, is ha we can model individual locomoives and rains a a very high level of deail. If we ry o simulaneously opimize he problem over a long horizon using a single deerminisic formulaion, i is essenial ha he problem be simplified, such as grouping locomoives ino a small number of classes (known as commodiies ) and discreizing ime fairly coarsely. Our sraegy makes i possible o handle he differen aribues ha are required for a ruly realisic model. For example, we can capure ha a paricular locomoive needs o ge o shop. During he simulaions where we sweep forward in ime, we can calculae he ime ha would be required o ge o each shop locaion, and use his when we solve he problem in equaion (5). This adapive learning sraegy can be used for sraegic, acical and real ime planning. The bigges challenge, however, was calibraing he model so ha we could be confiden ha i was accuraely capuring real world operaions. Model calibraion and validaion The model wen hrough several years of careful calibraion agains hisorical performance. This required he painsaking examinaion of deailed assignmens, along wih he comparison of high level performance merics. The process involved he ieraive idenificaion and correcion of daa errors, as well as enhancemens in he model and, from ime o ime, improvemens in he basic algorihm. For example, i was hrough his process ha we idenified he need o use wo layers of aggregaion in he value funcion approximaions as shown in figure 5. The mos common daa problems arose in he iniial locaion of locomoives, and he represenaion of he rain schedule and onnage requiremens. 9

12 Page 10 Examples of modeling problems included changes required in he handling of foreign power and he rules for consis formaion. In addiion o he careful examinaion of individual assignmens, Norfolk Souhern focused on rain delay as he mos imporan meric of overall performance. Maching rain delay a a sysem level is an Figure 8 Snapsho of Piloview, showing assignmens of individual locomoives o rains. exremely difficul arge because i requires ha he model mach locomoive produciviy almos perfecly. For example, i is imporan ha we accuraely capure he coss and ime required for breaking up locomoive consiss. If we ignore his componen, we would over represen he abiliy o use power o move rains, which in urn would underesimae rain delay. In he early sages of he calibraion process, he model would produce delays ha were an order of magniude larger han hisory, largely as a resul of daa errors ha had locomoives hopelessly ou of posiion. I is no possible o mach hisorical performance simply by uning parameers wihin he model. I was essenial ha he deailed assignmens pass he examinaion of experienced schedulers. This process was simplified by a powerful diagnosic ool called Piloview (figure 8) ha we developed for complex resource allocaion problems such as his. We proceeded by creaing a curve from esimaes of oal rain delay as a funcion of he flee size. Afer finally geing he model o closely mach hisorical performance, we repeaed he exercise wih an enirely new daase. The resul is he curve shown in figure 9, which shows a very close mach beween he curve and he hisorical delay a he curren effecive flee size. Also noe ha he relaionship beween flee size and rain delay is smooh and predicable. Achieving his behavior wih a model ha capures his level of deail is acually quie hard, as i requires ha we have he abiliy o model rain delays coninuously. Oher forms of model validaion involved esing he sensiiviy of he model o key inpu parameers. One such es evaluaed he effec of increasing he consis breakup cos o deermine is impac on boh he number of consiss being broken and overall soluion qualiy. Figure 10 shows he effec of 10

13 Page 11 increasing he consis breakup cos, using he rae of consis breakups wih a cos of zero as a baseline. Figure 9 Simulaed rain delay versus flee size, compared o hisorical performance. The char demonsraes ha increasing he consis breakup cos produces a seady decline in he number of broken consiss. We noe ha his was achieved wih no discernible reducion in he model objecive funcion which capures reposiioning coss and penalies for delayed rains. The model also has he abiliy o balance loads across shop locaions when rouing power o shop. Shop rouing is a paricularly sophisicaed feaure of he model. I uses adapive learning o esimae he ime required o ge a locomoive o each shop (given all downsream evens), and o esimae he backlog a each shop. We can hen inroduce a penaly o reduce hese backlogs. Figure 11 shows he oal backlog across all he shop locaions as a funcion of he ime wihin he simulaion. The model can do lile o reduce backlogs early in he simulaion, which are largely a resul of iniial condiions, bu wih a higher penaly, he backlogs are reduced as he simulaion progresses (noe ha his behavior is learned over he course of abou 50 ieraions). These feaures make i possible o une he model so ha i can achieve realisic behaviors. For example, i is imporan ha he model handle consis breakups and shop rouing in a realisic way if i is going o be used for flee sizing. Ignoring hese imporan operaional issues would allow he model o achieve levels of uilizaion higher han wha could be achieved in he field, a common problem wih he 11 Figure 10 Effec of he consis breakup penaly on he number of broken consiss relaive o a base case.

14 Page 12 use of opimizaion models. In addiion, hese feaures mean ha i is possible o perform sraegic planning sudies ha are realisic o railroad operaions. A his poin, we concluded ha he model was calibraed, and responded in a smooh and consisen way o changes in he inpu parameers. Figure 11 Toal shop backlog over he course of a simulaion, wih and wihou a congesion penaly. Sraegic planning The mos imporan sraegic planning quesion a Norfolk Souhern involved esimaing he appropriae flee size and mix given a projeced rain schedule. NS had used a simple regression model o esimae flee size, bu managemen came o feel ha inefficiencies were baked ino his model. The developmen of PLASMA was moivaed by he desire o have an engineering soluion ha could adap in a realisic way o assumpions abou rain schedules, flee size and mix, and nework performance. When he model is used for sraegic planning, all locomoives sar in a super source node. We do no have o specify where locomoives are iniially, and we do no even have o specify he flee mix, alhough we are allowed o do so. The model hen figures ou where o firs posiion each locomoive a he beginning of he planning horizon. Afer his decision is made, he adapive learning logic assigns power o rains over a planning period (ypically a monh). The use of an opimizaion based modeling sraegy means ha he model simulaes a well rained group of locomoive planners. Figure 12 illusraes how he model is used o esimae flee size. The model is firs used o creae a rain delay curve (oal delay as a funcion of flee size) for he curren year. Then, a projeced rain schedule is creaed for some period in he fuure, afer which he model is used o creae a new delay curve. If we would like o mainain he same level of rain delay, we can simply pick off he required number of locomoives. If we do no consrain he model o a fixed proporion of differen locomoive ypes, he model will also specify he flee mix. The model can also be used o perform differen ypes of policy sudies. Figure 13 illusraes an analysis of he effec of changes in average rain speed. Train delay curves were generaed for a base case, and hen for six scenarios where he average rain speed was varied. We noe ha he curves are quie consisen and well behaved, simplifying he ask of idenifying he correc flee size. 12

15 Page 13 Figure 12 Delay curves based on curren and forecased schedules, showing he flee size needed o mainain he same level of rain delay. While Norfolk Souhern has primarily used he model for flee sizing sudies, i can be used for oher quesions such as quanifying he effec of changes o he rain schedule, changes in inerchange poins o foreign railroads, and changes in he size and locaion of mainenance shops. Exensions We close wih a discussion of several significan ways in which he model can be exended. These include: managing uncerainy, using he model for shor erm operaional forecasing, and finally, using he model for real ime operaional planning. Managing uncerainy Locomoives have o be managed in he presence of a significan level of uncerainy. The mos imporan is he variabiliy in ransi imes due primarily o he need for lower prioriy rains o si on Figure 13 Train delay curves for differen variaions of average rain speed. 13

16 Page 14 sidings o allow higher prioriy rains o pass. Oher sources of uncerainy arise as a resul of yard delays, changes in he rain schedule (e.g. scheduling new coal and grain rains), and equipmen problems. Approximae dynamic programming makes i exremely easy o handle uncerainy. The algorihm requires ha we sep forward hrough ime repeaedly as we learn he value of locomoives in he fuure. If we wan o incorporae uncerainy, all we have o do is o sample from disribuions describing ransi imes, yard delays and equipmen failures. The effec can be quickly seen in he value funcions. Figure 14(a) illusraes wha a value funcion migh look like if rained on deerminisic daa. The value rises as a funcion of he number of locomoives, bu here is clearly a poin where addiional locomoives are jus no needed. Figure 14(b) illusraes wha he same value funcion migh look like if rained in he presence of uncerainy. Here, he value funcion coninues o rise, because siuaions migh arise where an inbound rain is delayed, and as a resul here is value in holding addiional locomoives. Figure 14(a) Value funcion rained on deerminisic daa. Figure 14(b) Value funcion rained on sochasic daa. When we use sochasically rained value funcions, here is value o holding addiional power in a yard. This reflecs he endency of erminal managers o hold ono a few exra locomoives, jus in case of problems. Of course, every yard manager would like o hold ono a few more locomoives, creaing he widely recognized problem where he railroad appears o need more locomoives han i has. The logic will creae he highes values (in he form of higher slopes) a he yards which ruly need he addiional power he mos. A he same ime, jus because here is value o holding more locomoives does no mean ha a yard will be allowed o keep addiional locomoives. Insead of simplisic rules o hold a paricular number of locomoives a specific yards, he value funcions provide a flexible policy ha adaps o he general availabiliy of locomoives. For example, each railroad experiences a ime every week when power invenories are a heir ighes. This is he poin where yards simply canno hold ono buffer invenories. By conras, a oher imes here are yards which really benefi from holding ono power o proec agains inbound delays or equipmen failures. Figure 15 shows rain delay over 150 simulaions of random ransi imes, using deerminisically and sochasically rained value funcions. Noe ha he sochasically rained value funcions no only produce smaller delays, bu also much more sable resuls. Given he effec of randomness on he value funcions (depiced in figure 14), we would expec ha he sochasically rained value funcions should 14

17 Page 15 Figure 15 Train delay over muliple simulaions using deerminisically and sochasically rained value funcion approximaions. be more inclined o hold power in invenory. Figure 16 shows ha his hypohesis is accurae. When aggregaed across he railroad, he sochasically rained VFAs show consisenly higher invenories, alhough he difference varies by ime reflecing, we believe, he changing abiliy of he nework o hold power. This is he firs repored soluion of a sochasic formulaion of he locomoive managemen problem. The mehods presened are based on a mahemaically rigorous formulaion, bu hey are also quie inuiive and pracical. Figure 16 Aggregae power invenories over he course of a simulaion using sochasically and deerminisically rained value funcion approximaions. Operaional forecasing An operaional forecasing model produces a plan over perhaps a five o seven day horizon. The model is used o idenify surpluses and deficis of power, and o anicipae locomoive reposiioning and ligh engine moves (moving power wihou a rain). Such a model requires ha we know where he locomoives are iniially. Thus, while he sraegic planning model has o figure ou where each 15

18 Page 16 locomoive should be a he beginning of he simulaion, he operaional forecasing model works from a live snapsho. The operaional forecasing model a NS runs in a producion seing. Afer each forward sweep (over, say, a seven day horizon), he model would refresh he locomoive snapsho, as well as capure any changes o he rain schedule. This process should repea iself approximaely once each minue (for a nework comparable o ha of Norfolk Souhern). In he process, he model is consanly refining he value funcion approximaions. The operaional forecasing model requires ha he rain schedule and locomoive snapsho be accurae (he sraegic planning model does no require a locomoive snapsho). This is no a small reques for a railroad. Norfolk Souhern has been exensively esing and validaing he operaional forecasing model, bu in he meanime he process has helped o idenify areas where daa reporing needs o be more accurae. This will be realized hrough upgrades o he informaion sysems and improved reporing procedures. A byproduc of his implemenaion has sparked a major revision of heir daa collecion and reporing process for locomoives. This is a familiar experience, where he process of implemening advanced decision suppor sysems has he effec of raising he bar on he qualiy of informaion sysems. Real ime planning The las and arguably mos ambiious use of opimizaion would be he real ime assignmen of locomoives o rains. Real ime assignmen models have been used for years for ruckload rucking wih remendous success, and several railroads use real ime opimizaion o assign cars o orders. PLASMA can easily be adaped o perform real ime operaional planning for locomoives. Assuming ha he operaional forecasing model is running in producion, we have access o he value funcion approximaions. A real ime operaional model requires ha insead of solving a sequence of decision problems (depiced in figure 5) over ime (as illusraed in figure 6), we only have o solve a single problem, reflecing wha is known now. Such a model can be solved from scrach in a maer of seconds, bu i can be solved even more quickly by holding he soluion live in memory. The challenge of any real ime model is ha human planners always have access o informaion ha will simply no be available in he compuer. For example, ofen he firs source of updaed informaion abou he saus of a locomoive comes from an inspecor alking on a cell phone o a planner. As a resul, regardless of he sophisicaion of a model or he qualiy of a daabase, here will always be insances when a human will simply disagree (and in some cases, correcly) wih he recommendaion of a model. This is no a problem if he planner is allowed o override he model, and if he model can hen be updaed exremely quickly (which is o say in a second or wo). The speed wih which he model needs o be updaed is no relaed o he rae a which updaes come in from he railroad. The main consrain is he speed wih which planners make decisions. Real ime operaional model pose addiional demands on he qualiy of daa, over and above wha is required for a acical model. However, we believe ha a fully ineracive model can be robus, adding value even in he presence of imperfec informaion. 16

19 Page 17 Conclusions Approximae dynamic programming offers a novel modeling and algorihmic sraegy ha combines he realism of simulaion wih he inelligence of opimizaion. Classical opimizaion models have offered he promise of beer decisions, bu he echnology has required he use of major simplifying assumpions. As a resul, he savings produced by such models are ofen a by produc of simplified models raher han inelligen decisions. PLASMA has been shown o produce high qualiy, accurae soluions o sraegic and acical planning problems a Norfolk Souhern. Furhermore, i has shown very promising resuls for operaional forecasing and has high poenial for real ime locomoive assignmens. I is he firs opimizaionbased model ha calibraes accuraely agains hisory, making i useful as a ool for flee sizing, one of he mos demanding sraegic planning problems. The echnology allows locomoives and rains o be modeled a an exremely high level of deail. Train delays can be modeled down o he minue. The model can simulaneously handle consis breakups and shop rouing, while also planning he empy reposiioning of power. In addiion, i can handle uncerainies in ransi imes, yard delays and equipmen failures in a simple and inuiive way. The enire mehodology is based on firs principles, and as a resul avoids he need for heurisic rules ha have o be reuned as he daa changes. Acknowledgemens: We would like o hank he following people who have made significan conribuions o his projec: Don Grabb, Ed Courney, Junxia Chang, Brian Wilker, and Jermaine Wilkinson. The early sages of his projec were managed by Ajih Wijerane (who was he firs o use he rain delay curves), and he firs auhor would like o hank he very early suppor of Roger Baugher who recognized he poenial of approximae dynamic programming for rail operaions. The research behind his work has been suppored over he years by he Air Force Office of Scienific Research and he Naional Science Foundaion. 17

20 Page 18 References Ahuja, R. K., Liu, J., Orlin, J. B., Sharma, D., & Shughar, L. A. (2005). Solving real life locomoivescheduling problems. Transporaion Science, 39, Booler, J. (1980). The soluion of a railway locomoive scheduling problem.journal of he Operaional Research Sociey, 31(10), Pergamon Press. Rerieved from hp:// Chih, K., Hornung, M., Rohenberg, M., & Kornhauser, A. (1990). Implemenaion of a real ime locomoive disribuion sysem. Compuer Applicaions in Railway Planning and Managemen (pp ). Compuaional Mechanics Publicaions, Souhampon, UK. Chih, K. C. K. (1986). A Real Time Dynamic Opimal Freigh Car Managemen Simulaion Model of he Muliple Railroad, Mulicommodiy Temporal Spaial Nework Flow Problem. Princeon Universiy. Cordeau, J. F., Soumis, F., &Desrosiers, J. (2000).A Benders decomposiion approach for he locomoive and car assignmen problem. Transporaion Science, 34, Cordeau, J. F., Soumis, F., &Desrosiers, J. (2001). Simulaneous assignmen of locomoives and cars o passenger rains. Operaions Research, 49, Forbes, M., Hol, J., & Was, A. (1991).Exac soluion of locomoive scheduling problems.journal of he Operaional Research Sociey, Powell, W.B. (2011) Approximae Dynamic Programming: Solving he curses of dimensionaliy, 2 nd ediion, John Wiley and Sons, Hoboken, NJ. Rouillon, S., Desaulniers, G., &Soumis, F. (2006).An exended branch and bound mehod for locomoive assignmen.transporaion Research Par B: Mehodological, 40(5), doi: /j.rb Suon, R. and A. Baro, (1998) Reinforcemen Learning, MIT Press, Cambridge, MA. Vaidyanahan, B., Ahuja, R. K., & Orlin, J. B. (2008a). The Locomoive Rouing Problem. Transporaion Science, 42(4), doi: /rsc Vaidyanahan, B., & Ahuja, R. K. (2008b). Real life locomoive planning : New formulaions and compuaional resuls. Transporaion Research B, 42, Ziarai, K., Soumis, F., Desrosiers, J., Gelinas, S., &Sainonge, A. (1997). Locomoive assignmen wih heerogeneous consiss a CN Norh America. European journal of operaional research, 97, Elsevier. Ziarai, K., Soumis, F., Desrosiers, J., & Solomon, M. (1999).A branch firs, cu second approach for locomoive assignmen.managemen Science, 45,