Optimal control of an emergency room triage and treatment process

Transcription

1 Submitted to Management Science manuscript MS- Authors are encouraged to submit new papers to INFORMS journals by means of a style file template, which includes the journal title. However, use of a template does not certify that the paper has been accepted for publication in the named journal. INFORMS journal templates are for the exclusive purpose of submitting to an INFORMS journal and should not be used to distribute the papers in print or online or to submit the papers to another publication. Optimal control of an emergency room triage and treatment process Authors names blinded for peer review) Patient care in many healthcare systems consists of two phases of service: assessment or triage) and treatment. It is sometimes the case that these phases are carried out by the same medical providers. We consider the question of how to prioritize the work by the medical providers to balance initial delays for care with the need to discharge patients in a timely fashion. To address this question, we present a multi-server two-stage tandem queueing model for a hospital emergency department ED) triage and treatment process. We assume that all patients first receive service i.e. triage) from the first station. After completing this service some patients leave the system for some other part of the ED. The remaining patients are served or await service from the second station where they may abandon before receiving treatment. We use a Markov decision process formulation and sample path arguments to determine the optimal dynamic policy. In particular, we show that there exists optimal control policies that do not idle servers when there is work available and do not split servers except to avoid idling. We then focus on the states that have more patients than there are medical service providers. We consider a single server model as an approximation for these states and provide conditions under which it is optimal to prioritize phase-one service triage) or phase-two service treatment). We show that in the average reward case, it s optimal to prioritize station 2 regardless of the reward structure, as long as the system is stable. Otherwise, prioritizing station 1 or using a threshold policy dominates. Hence, we introduce a new class of threshold policies as alternatives to priority rules. Using data from an actual hospital, we compare the performance of all of the aforementioned policies and several other potential service policies in a simulation study. Results show that for a wide range of parameter values, the threshold service disciplines perform well. Key words : ER triage and treatment process, optimal control, tandem queue, abandonments 1. Introduction In 2010, the number of emergency department ED) visits in the United States was already about million, and the annual increasing rate was 2 3% in average The National Ambulatory Medical Care Survey, NAMCS). This increasing demand has occurred while the number of ED beds has been decreasing. The result is overcrowded departments, long waiting times, overworked 1

2 2 Article submitted to Management Science; manuscript no. MSstaff and patient dissatisfaction. Many patients seeking care from an ED present with low-acuity conditions and do not require hospitalization. Yet they must be treated, and so divert resources from more critically ill patients. To address this problem, EDs are developing new models of care to handle these lower-acuity patients and thereby facilitating patient flow. Motivated by the Triage-Treat-and-Release TTR) program developed in 2010 by the Lutheran Medical Center LMC) in Brooklyn, New York, we model the triage and treatment process as a two-phase stochastic service system. Services are provided by multiple providers i.e., physicians or physician assistants) who handle both phases of the system. At LMC, patients arriving to the ED are automatically registered and then proceed to triage phase one service) which is conducted on a first-come-first-served basis. After triage, high severity patients who are likely to be admitted to the hospital are assigned to another part of the ED for testing and/or treatment while those who are low severity and low complexity await treatment, phase-two service), in the triage area. Patients awaiting triage rarely leave before being seen, but patients who are awaiting phase-two service have limited patience and may abandon the system before receiving final treatment. After receiving phase-two service, patients are discharged and leave the system. The TTR program is a new and creative way to help reduce long waiting times in the ED. Placing physicians or physician assistants) in triage allows for earlier patient contact with a physician and results in earlier advanced decision making. In addition, there have been several studies that suggest that physicians or physician assistants are more reliable in their assessment of patients during triage Soremekun et al. 2012, Burström et al. 2012). While motivated by the TTR program, our results have broader implications for related models. For example, the fast-track system suggested for use in the ED identifies patients at triage who can be quickly discharged after treatment and directs them to a separate area. The TTR program is a variant of the traditional fast-track in that the person performing triage is also the one doing the treatment. We contend that if a hospital is going to use a highly trained professional to perform triage, it may also be desirable that the same person is involved in treatment. The goal of this paper is to develop insights for how to allocate healthcare providers between triage and treatment. Many hospitals, including LMC, have multiple people performing triage. As a result, we model the TTR systems as a general multi-server model and treat the single-server model as a special case. We assume that rewards are received for completing each phase of service, while no rewards are received for patients who abandon the system. We use this model to develop service disciplines that maximize the total discounted expected reward or long-run average reward of the system and to compare these with alternative service policies. We note that the reward structure for this application does not necessarily correspond to the accrual of actual cash rewards by the hospital. In fact, hospitals rarely charge for triage alone but certainly charge for treatment. Moreover,

3 Article submitted to Management Science; manuscript no. MS- 3 the decision-making scenario whether to triage or treat) takes place in a relatively short time frame/horizon, and the seasonality and growth of emergency arrivals implies that it is sensible to re-evaluate specific policy parameters frequently over time. As a result, we contend and emphasize) the average reward case, but consider the discounted reward case as well. We formulate the multiple medical provider allocation problem as a continuous-time Markov decision process CTMDP). In contrast to many existing models, the transition rates of our CTMDP model are unbounded due to potential abandonments by patients waiting for or in service in phase-two. This complicates our analysis, since a technique known as uniformization used to transform the CTMDP into an equivalent discrete-time MDP, can no longer be used see for example Lippman 1975). It follows that an alternate approach for the solution and analysis of our model is needed. Although the model developed here is general, the results and several findings that relate directly to the LMCs TTR program and could be useful to other hospitals who may want to implement similar system.s We show that as long as prioritizing treatment over triage leads to a stable system, one should in the average case) prioritize treatment. This confirms the current practice at LMC. Even when implementing a policy that prioritizes station 2, we believe that decision-makers would divert resources to station 1 in the case that the triage queue gets long. When the policy that prioritizes station 2 does not yield a stable system, the policy is unrealistic to implement. We provide easy to compute sufficient conditions for this so that a hospital could identify this in advance. We suggest that in this case, the decision-maker could use the number of patients waiting for triage as an alternative criterion. Our study suggests that threshold policies are a viable option. In the following, we will explicitly enumerate the contributions of the paper, and emphasize the most important results and insights. To our best knowledge, this is the first paper to deal with a single set of flexible servers in a two-stage tandem queue with abandonments. The main technical contribution of the paper is to find the stationary optimal control policies. We first show that there exist optimal policies that are non-idling. We then show that if the number of patients at each station exceeds the number of servers, there exists an optimal policy that does not split the servers. When the number of patients is high, the service process is akin to that of a single server. In this case, we focus on the single-sever model, which acts a proxy for the LMC s TTR program. We show the existence of stationary optimal policies and a solution of the optimality equations for both the discounted reward and the average reward cases. We then use the optimality equations and sample path arguments to analyze the structure of the optimal policy in each case. We provide sufficient conditions for when it is optimal to prioritize phase 2 service i.e. treatment). In the discounted case, the sufficient condition is akin to the classic c-µ rule cf. Buyukkoc et al. 1985). Furthermore, while there are seldom abandonments before phase 1 treatment at the LMC,

4 4 Article submitted to Management Science; manuscript no. MSwe extend the single server version of our model to allow for said abandonments and verify the usefulness of our recommendations by simulation. From the managerial perspective, this paper is the first to model this TTR system and provide simple control policies. One of the most important insights is that in the average reward case, it s optimal to prioritize station 2 regardless of the reward structure, as long as the system is stable. This policy, which is the current practice in the LMC ED, benefits both the patient and provider since it means that once a patient receives phase-one treatment, s)he immediately proceeds to phase-two service. The other major contribution is to show that when the system isn t stable, prioritizing station 1 or using a threshold policy dominates. Finally, we present a new class of policies that we call K-level threshold policies as a compromise between the two priority service disciplines, and provide insights on the value K for the threshold policy. We then compare several service disciplines, including the two priority rules and the K-level threshold service disciplines, in a simulation study. We mention here that although the optimal control policies are developed under typical modeling assumptions, relaxing these assumptions is considered in the simulation study. The simulations verify the robustness of our recommended controls. The rest of this paper is organized as follows. Section 2 contains a summary of the literature relevant to our work. Section 3 describes the queueing dynamics in detail and provides a Markov decision process formulation of the problem under both the discounted expected reward and the long run average reward optimality criteria. Section 4 contains our main theoretical results. We show that it is enough to consider policies which do not idle the providers whenever there are patients waiting to be served and that if the number of patients at each station exceeds the number of servers, there exists an optimal policy that does not split the servers. We then focus on the single-server model as a reasonable approximation to the multi-server problem when the number of patients exceeds the number of providers. We give sufficient conditions for when to prioritize phase-two service under each reward criteria. In Section 5, we introduce a class of threshold policies as a compromise between the two priority service disciplines. We then present a simulation study. In Section 6 we consider the same problem, but allow abandonments from station 1. We provide sufficient conditions for the optimal control to prioritize station 1 or 2 in the single server case. Although, this model does not match closely the TTR system considered, the tradeoffs between abandonments and rewards makes analysis more challenging. The results may be of independent interest. A short numerical study confirms our previous results. We discuss in detail extensions of the previous insights. We conclude with a brief discussion in Section 7.

5 Article submitted to Management Science; manuscript no. MS Literature Review In this section we highlight those papers most relevant to our work from two aspects; models that apply queueing methods in ED patient flow and models that analyze and control tandem queues in the queueing literature at large ED patient flow Saghafian et al. 2014) discuss a complexity-based triage system, based on the number of visits that patients pay to the ED physician. Saghafian et al. 2012) analyze the advantage of streaming patients and compare this practice with pooling. Based on a given triage system, Huang et al. 2012) consider how to balance the work of ED physicians on new patients after triage and workin-progress WIP) patients, and prove that the Gcµ-rule is asymptotically optimal. Yom-Tov and Mandelbaum 2011) take time-dependent arrivals into consideration, and model the ED as singleclass time-varying queueing system with feedback namely the Erlang-R model) in the QED regime in support of ED staffing. Dobson et al. 2013) develop an overloaded queueing network to analyze the impact of interruption on ED throughput. Our work differs from the existing literature in two major ways. First, we consider the triage, treatment and release process through the ED, a model that has reduced delays at Lutheran Medical Center but that has not been studied previously. And second, we use a CTMDP to model the flow control problem Tandem queues From an analytical perspective, our model falls into the category of tandem queues served by a common set of servers. We divide these related papers into two categories. The first includes those that analyze the performance of such queues under a specified policy and the second consists of those that consider optimal service policies. Among those that analyze the performance of single server tandem queues, Nair 1971) considers one with Poisson arrivals and general service times under a non-zero switching rule for phase-one and a zero switching rule for phase two. A non-zero switching rule is one that the provider continues to serve in a phase until some specified number of consecutive services have been completed and then switches to the other phase, while a zero switching rule continues to serve until the phase is empty before switching to the other phase. Taube-Netto 1977) considers the same model but with a zero switching rule at each phase. Katayama 1980) analyzes the system under a zero switching rule at each phase but with non-zero switchover times. A K-limited policy has the provider visit a phase and continue serving that phase until either it is empty or K patients are served, whichever occurs first. A gated service policy is one in which, once the provider switches phases, s)he serves only patients who are in that phase at the time

6 6 Article submitted to Management Science; manuscript no. MSimmediately following the switch. In Katayama 1981) and Katayama 1983), Katayama extends his previous work to consider gated service and K-limited service disciplines, respectively. In both papers, intermediate finite waiting room is allowed. Katayama and Kobayashi 1995) analyze the sojourn time under general K-decrementing service policies; policies in which once the server visits phase one, s)he continues serving that phase until either this phase becomes empty or K patients, are served, whichever occurs first, and then serves at phase-two until it is empty. We mention two additional service disciplines, that we consider in our simulation study. The first are exhaustive service disciplines, where the provider visits phase 1 or 2) and serves patients in that phase until it becomes empty, and then switches to the other phase. The second are priority policies, in which provider serves patients in phase 1 or 2) according to a preemptive or non-preemptive priority rule. Nelson 1966) is the first to consider optimal service policies for a tandem queueing system over a finite horizon. Johri and Katehakis 1988) also consider the optimal service disciplines for queues in series. Customers arrive according to a Poisson process and the service times at each queue are independent and exponentially distributed. They show that the discipline that always assigns the provider to the non-empty queue closest to the exit stochastically minimizes the total number of customers in the system. Vanoyen and Teneketzis 1994) consider optimal service disciplines in a forest network of N queues without arrivals and where the service times at each queue are independent and generally distributed. The goal is to minimize the total expected discounted cost, which consists of linear holding costs, lump sum switching costs or set-up delays. They consider the two node model as a special case, which is similar to the holding cost version of our model without abandonments, and in which two nodes are connected probabilistically in tandem. That is to say, a job served at node 1 is routed to 2 with probability p and exits the system with probability 1 p. They define an optimal policy for the latter model and show that an optimal policy must be exhaustive if the holding cost in node 1 is larger than node 2. Iravani et al. 1997) consider optimal service disciplines in a two-stage tandem queue with Poisson arrivals and general service times. The goal is to minimize total long run average holding and switching costs. They show that the optimal policy in the second stage is greedy and that if the holding cost rate in the second stage is greater or equal to the rate in the first stage, then it is also exhaustive. They also give a condition under which a sequential service policy is optimal. Duenyas et al. 1998) consider optimal service disciplines in a system where there is a setup time for the provider to switch between different phases. The arrival process is a Poisson process and the service time at each phase is an independent and generally distributed random variable. Neither setups nor services can be preempted. They assume that the holding cost at each operation is non-decreasing in operation number and seek to minimize discounted and average cost. The main result is that the optimal policy when the

7 Article submitted to Management Science; manuscript no. MS- 7 provider is set up for phase one service is completely described by a monotone switching curve. More recently, Ahn et al. 2002) consider optimal service disciplines for two flexible service providers in a two-stage tandem queueing system assuming Poisson arrivals and exponential service times. They consider a collaborative scenario and the non-collaborative scenario. In the former, providers can collaborate to work on the same job, which is similar to the single provider case. They provide sufficient conditions under which it is optimal to allocate both providers to phase 1 or 2 in the collaborative case. They also show that the optimal rule for prioritizing phase-one service in the collaborative case also holds in the non-collaborative case. Wang and Wolff 2005) consider queues in tandem with a single service provider, in which the provider may allocate a fraction α of the service capacity to station 1 and 1 α to station 2 when both are busy. Using work conservation and FIFO, they show that the wait time in system increases with α on every sample path. For Poisson arrivals and an arbitrary joint distribution of service times of the same customer at each station, the average waiting time at each station is given for the special cases α = 0 and α = 1. In Andradóttir and Ayhan 2005) the authors consider policies that maximize throughput of a tandem queue with a finite intermediate buffer. To the best of our knowledge, our work is the first to consider optimal service policies for service providers in a two-phase stochastic service system in which patients can abandon before receiving service. In addition, in our simulation study, we introduce and analyze a new class of policies that we call K-level threshold policies. In a threshold policy with level K, the server continues to prioritize station 2, but moves to station 1 when the number of customers at station 1 reaches K. That is, the threshold policy has the server work at station 2 until either station 2 is empty or the number of patients at station 1 reaches K. In the first case, s)he continues to follow the prioritize station 2 policy. In the second case, the server works at 1 until it is empty and then returns to prioritizing station 2. Note that the policy that prioritizes station 2 spends the highest proportion of effort at station 2, while that which prioritizes station 1 spends the least. The prioritize station 1 2) policy is equivalent to the K-level threshold policy with K = 1 ). In between these two extremes are threshold policies with higher thresholds spending more time at station Model Description and Preliminaries We approximate the triage-treatment system with the following model. Customers arrive to the service system ED) according to a Poisson process of rate λ and immediately join the first queue. After receiving service at station 1 a reward of R 1 is accrued. Independent of the service time and arrival process, with probability p the customer joins the queue at station 2. With probability q := 1 p, the customer leaves the system forever. If the customer joins station 2 his/her random and hidden) patience time is generated. Assume that the service times at station i are exponential

8 8 Article submitted to Management Science; manuscript no. MSwith rate µ i > 0, i = 1, 2 and the abandonment time is exponential with rate β. If the customer does not complete service before the abandonment time ends, the customer leaves the system without receiving service at the second station. There are N 1 servers, each of which can be assigned to either station. After each event arrival, service completion or abandonment) the servers) view the number of customers at each station and decides where to serve next Model assumptions and justification It is important to justify the assumptions of the queueing model described above. There are three key components: the arrival process, the service times, and the abandonments. Arrival process It has been shown that a non-homogeneous Poisson process is a good approximation for time-varying arrivals in a health care system Shi et al. 2014, Xie et al. 2014). We can partition a workday into relatively short periods e.g. 2 4 hour periods) and treat the arrival process within each period as a homogeneous Poisson process. The results under the stationary assumption can be extended to the time-dependent model by varying the arrival rate. This methodology has been shown very useful in supporting emergency department staffing Green et al. 2006). Service times For our work at Lutheran, we collected data on actual service times throughout the ED to both calibrate the inputs for our model as well as to estimate the coefficients of variation. We found that for the triage stage, the mean time was 7.25 minutes and the std. deviation was 6.46 minutes, so that the coefficient of variation was close to one, making the exponential assumption very reasonable. For the treatment stage the mean was 13.2 and the std. deviation was about 7, raising some concerns about this assumption. Consequently, we conducted sensitivity analysis to determine if the exponential assumption would significantly affect suggested staffing levels to achieve service targets of, e.g. 90% of patients waiting less than 10 minutes and it did not. In addition, in our simulation study in this paper we use a phase-type distribution for the treatment area and note that it did not affect the optimal policies. Abandonments With regard to abandonments, our understanding based on work with several hospitals, including Lutheran, is that there are virtually no abandonments before triage. This is because waiting times for triage are generally far shorter than waiting times to be seen by a provider. For example, at LMC, the staffing goal for triage at Lutheran is that no more than 10% of patients wait more than 10 minutes. Their automated registration system, which records patient arrivals as soon as they walk through the door, substantiates that there are no abandonments between arrival and triage. Despite this, for completeness we consider abandonments from both stages in Section 6 which extends our model beyond the LMC TTR that originally motivated this study.

9 Article submitted to Management Science; manuscript no. MS MDP Formulation To model the two-phase decision problem we consider a Markov decision process MDP) formulation. Let {t n, n 1} denote the sequence of event times that includes arrivals, abandonments and potential service completions. Define the state space X := {i, j) i, j Z + }, where i j) represents the number of customers at station 1 2). The available actions in state x = i, j) are Ax) = {n 1, n 2 ) n 1, n 2 Z +, n 1 + n 2 N}, where n 1 n 2 ) represents the number of providers assigned to station 1 2). A policy prescribes how many providers should be allocated to stations 1 and 2 for all states for all time. The function rx, a) is the expected reward function since each state change does not represent the accrual of a reward), rk, l), n 1, n 2 )) = min{k, n 1 } R 1 λ + min{k, n 1 } + lβ + min{l, n 2 } R 2 λ + min{l, n 2 } + lβ. Under the α-discounted expected reward criterion the value of the policy f given that the system starts in state i, j) over the horizon of length t is given by Nt) vα,ti, f j) = E i,j) e αtn rxt n ), fxt n ))) ], n=0 where Nt) is the counting process that counts the number of decision epochs in the first t time units, and {Xs), s 0} is the continous-time Markov chain denoting the state of the system i.e. the number of patients at each queue) at time s. The infinite horizon α discounted reward of the policy is v f αi, j) := lim t v f α,ti, j) and the optimal reward is v α i, j) := max π Π v π αi, j), where Π is the set of all non-anticipating policies. Similarly, for the average reward case, the average reward of the policy f is ρ f i, j) := lim inf t v f 0,t i,j) t. 4. Dynamic Control Our first result states that there is an optimal policy that does not idle the provider whenever there are patients waiting. This is used to simplify the optimality equations that follow. Proposition 1. Under the α discounted reward finite or infinite horizon) or the average reward criterion, there exists a non-idling policy that is optimal. Proof. We prove the result using a sample path argument. Suppose we start two processes in the same state i, j), and on the same probability space so that all of the potential) events coincide. Moreover, each customer that enters the system has attached to it an abandonment time and a mark indicating whether or not it will leave the system after completing service at station 1 or continue on to station 2. Of course, both are unbeknownst to the provider at the time of the arrival.

10 10 Article submitted to Management Science; manuscript no. MS- Fix time t 0 and assume α > 0. Process 1 uses a policy π 1 that idles providers unnecessarily say instead of assigning them to station 1) until the first event and uses a stationary optimal policy thereafter. In what follows, we show how to construct a potentially sub-optimal) policy for Process 2, that we denote by π 2, that assigns the extra providers to station 1 until the first event and satisfies v π 2 α,ti, j) v π 1 α,ti, j). 1) Since π 2 is not required to be optimal and π 1 uses the optimal actions after the first event, the result follows. Suppose the first event is an arrival, a service completion seen by the two processes, or an abandonment. In all cases, the two processes transition to the same state. At this point let π 2 = π 1 so that the two processes couple. It follows that if an arrival, a service completion seen by the two processes, or an abandonment occurs first, then 1) holds with equality. Suppose now that the first event is a service completion at station 1 in Process 2 that is not seen by Process 1. In this case, Process 1 remains in state i, j) while Process 2 transitions to either state i 1, j + 1) or state i 1, j) depending on the mark of customer 1) and accrues a reward of R 1. From this point forward, if we reach time t, before another event occurs, 1) holds strictly since Process 2 has seen an extra reward. Otherwise, let policy π 2 choose exactly the same action that policy π 1 chooses at every subsequent decision epoch except in the case that it cannot assign the same number of providers to station 1 since it now has one less patient at station 1 than Process 1). In this case, it assigns the same number to station 2 as Process 1 and one less at station 1 than Process 1 idling the extra provider). Since each customer in queue for Process 1 is a replica of each customer in Process 2 including the one just served), it follows from the description of π 2 that Process 1 can see at most one service completion at station 1 that is not seen at station 2, at which point, the two processes couple. Discounting in this case implies that 1) holds strictly. Continuing in this fashion for all initial states yields the result for fixed t. Note that the results hold pathwise for any t. This implies that the result holds as t, and hence, for the infinite horizon discounted case. Moreover, the discounting is only used to show that 1) holds strictly. Setting α = 0 and repeating the proof yields the result for the average case. A typical method for obtaining the optimal value and control policy under either the discounted or average reward criterion is to use the dynamic programming optimality equations. In theory since the state space is countably infinite) this is still possible. On the other hand, it is also quite common to use the optimality equations to obtain the structure of an optimal control in hopes that said structure is simple enough to be easily implementable. We pursue this direction

11 Article submitted to Management Science; manuscript no. MS- 11 with the caveat that none of the typical methods for using the optimality equations successive approximations, action elimination) work directly since the transition rates are not bounded. This means that the problem is not uniformizable and no discrete-time equivalent exists. Instead, we rely on the optimality equations to simplify our search for optimal policies. Proposition 1 implies that we may restrict our attention to non-idling policies thereby simplifying the search for the optimal control policy. We next provide conditions under which the optimality equations have a solution. To simplify notation for a function h on X define the following mapping T hi, j) = max λhi + 1, j) + jβhi, j 1) + min{i, a} R1 + phi 1, j + 1) + qhi 1, j) ) ) + min{j, N a} R2 + hi, j 1). 2) a {0,1,,N} Theorem 1. The MDP has the following properties: 1. Fix α > 0. Under the α discounted expected reward criterion: a) The value function v α satisfies the discounted reward optimality equations DROE), αv α = T v α. 3) b) There exists a deterministic stationary optimal policy c) Any f satisfying the maximum in the DROE defines a stationary optimal policy. 2. Under the average cost criterion: a) There exists a constant g and function w on the state space such that g, w) satisfies the average reward optimality equations AROE), g1 = T w, where 1 is the vector of ones. b) There exists a deterministic stationary optimal policy. c) Any f satisfying the maximum in the AROE defines a stationary optimal policy. Proof. Since the rewards are bounded and there are only a few states that can be reached during transitions, Assumptions A of Guo and Zhu 2002) hold with w n = 1. The results for the discounted reward case now follow directly from Lemma 3 parts ii)-iv) of Guo and Zhu 2002). The next result shows that we need only consider policies that never divide the providers between queues whenever the number of patients at each queue exceeds the number of servers. Proposition 2. If the number of patients at each queue exceeds the number of servers, there exists a discounted reward optimal control policy that does not split the servers. Similarly in the average reward case.

12 12 Article submitted to Management Science; manuscript no. MS- Proof. We show the result in the discounted expected reward case. The average reward case is analogous. Assume that i, j N, and hence, min{i, a} = a and min{j, N a} = N a for any a in the action set Ai, j) = {0, 1,, N}. Let G α i, j, a) := λv α i + 1, j) + jβv α i, j 1) + a R1 + pv α i 1, j + 1) + qv α i 1, j) ) + N a) R2 + v α i, j 1) ), and note that G α i, j, a) is linear in a. Since αv α i, j) = max a {0,1,...,N} G α i, j, a)] and a linear function achieves its maximum and minimum) at the extreme points, it follows that it is optimal to set a equal to 0 or N, the two extreme points in {0, 1,, N}. Proposition 2 implies that we may restrict attention to policies that always allocate all workers to one station or the other in those states where i, j N. Similarly, Proposition 1 implies that we should keep all service providers busy in states such that i + j N. We are left without any guidance in states with i or j but not both) greater than N, nor how to choose the priorities when i, j N. We point out that solving the problem generally is quite difficult and we simply were unable to do so. Instead, we discuss under what conditions simple priority policies are optimal in the single server model see next section) and use it as a proxy for the multi-server model. In the numerical study that follows we then compare these heuristics to other reasonable policies. In addition, we note that the LMC s TTR program is a two-phase stochastic service system with the number of providers ranging from 1 to 5 depending on the schedule. That is to say that sometimes the single provider proxy is exact of course modulo the other modeling assumptions) Optimal Control for the Single Server Proxy In the single server case, the optimality equations tell us what structure of the value functions implies that simple optimal policies exist. We then prove that structure via sample path arguments. The non-idling result of Proposition 1 leads to the next result which says that ignoring one station completely is sub-optimal. Proposition 3. In the case of the infinite horizon discounted reward or average reward, if λ < and p > 0, then there is an optimal policy that moves the provider from one station to the other at least once. Proof. We need to show that there exists a policy that does as well or better) than either the policy that works solely at station 1 letting every potential abandonment occur) or the policy that works solely at station 2. Note that λ < implies that under the policy that works at station 1 only, the first station acts like a stable) M/M/1 queueing system. Since p > 0, it is well-known that the exit process of the first station those patients that enter the second station) is a Poisson process

13 Article submitted to Management Science; manuscript no. MS- 13 with rate λp. Moreover, the second station acts as an M/M/ queue with each abandonment acting as a provider. That is to say, the joint process taking both stations together) is an ergodic continuous-time Markov chain. In particular, any state of the form 0, j), for j 1 is eventually reached. The fact that there is an optimal non-idling policy now yields the result. Consider now the system that works only at station 2. In this case, station 2 acts as a linear) pure death process, which eventually empties station 2. At this time, the system is in some state i, 0). If i 1 the previous results on non-idling policies yield the result. If i = 0, after the first arrival the results apply. The next result, the main result of the section, provides conditions under which it is optimal to prioritize station 2 under the discounted and average reward criteria. Theorem 2. The following hold: 1. Under the α discounted reward criterion, if R 2 R 1, then it is optimal to serve at station 2 whenever station 2 is not empty. 2. Under the average reward criterion, if λ station 2 whenever station 2 is not empty. Proof. See Appendix β ) < 1 and β > 0, it is optimal to serve at A few remarks regarding Theorem 2 are in order. In the discounted reward case, we require the condition R 2 R 1 which is closely related to the classic result of the c µ rule. In the average case, to get sufficiency we do not need this inequality. Intuitively, if it is known that there have been several arrivals, in the average case it does not matter when they arrived. In fact, we could collect the reward upon arrival instead of after service completion at station 1). This is not true for customers moved to station 2 due to the potential abandonment. The way to get the most reward is by serving customers in station 2 as soon as possible; by prioritizing station 2. It should also be noted that the policy that prioritizes station 2 by no means ignores station 1. On the contrary, the set of recurrent states under this policy is {i, j) i 0, j {0, 1}} so that customers receive service at station 1, move to station 2 and leave the system; the provider serves a customer completely before starting on the next customer. This leads to several questions ) Do we need the assumption that λ < 1 to solve the average reward problem when β the system is clearly stabilizable with λ < 1 just prioritize station 1)? Is it solely the abandonments that make the c-µ result fail? Under what conditions is it optimal to prioritize station 1? The answer to the first question is no due to the fact that the policy that prioritizes station 2 is not stable without this assumption. We will study it further numerically. To answer the second question consider the following result.

14 14 Article submitted to Management Science; manuscript no. MS- Proposition 4. Suppose β = 0 and R 1 > R 2, then under either the α discounted reward or the average reward criterion, it is optimal to serve at station 1 whenever station 1 is not empty. Proof. The result is shown in the discounted reward case. The average case is similar. In this case the problem is uniformizable with uniformization rate Ψ := λ + + and discrete-time discount factor γ = Ψ+α. The optimal policies are unchanged between the continuous and discrete Ψ time models and the optimal values coincide up to a multiplicative constant. Without loss of generality assume that Ψ = 1. The discrete time DROE mapping for any function h on X is now Uh0, j) = γ λh1, j) h0, j)] + h0, j) ) + 1j 1) R 2 + γh0, j 1) h0, j))]} Uhi, 0) = γ λhi + 1, 0) hi, 0)) + hi, 0) ] + 1i 1) R 1 + γphi 1, 1) + 1 p)hi 1, 0) hi, 0))]. For i, j 1, Uhi, j) = λhi + 1, j) hi, j)] + max{ R 1 + phi 1, j + 1) + 1 p)hi 1, j) hi, j)], R 2 + hi, j 1) hi, j)]}. The discrete-time value function satisfies ṽ γ = Uṽ γ and can be computed via successive approximations ṽ n+1,γ = Uṽ n,γ with ṽ 0,γ = 0 and lim n ṽ n,γ = ṽ γ. An argument similar to that in Theorem 2 implies that in state i, j) it is optimal to assign the worker to station 1 in state i, j) if R 1 R 2 + pṽ γ i 1, j + 1) + qṽ γ i 1, j) ) ṽ γ i, j)] + ṽ γ i, j) ṽ γ i, j 1)] 0. 4) We thus prove 4) with ṽ γ replaced with ṽ n,γ via a sample path argument and induction. Trivially, 4) holds for ṽ 0,γ. Assume that the inequality 4) holds for ṽ n,γ and consider it for ṽ n+1,γ. Suppose we start five processes in the same probability space. Processes 1-5 begin at time n + 1 in states i 1, j + 1), i 1, j), i, j), i, j), and i, j 1), respectively. The inductive hypothesis implies that it is optimal to provide service at station 1 when there is work to do there. Processes 3 and 5 use optimal policies which we denote by π 3 and π 5, respectively. In what follows we show how to construct potentially sub-optimal) policies π 1, π 2, and π 4 for processes 1, 2, and 4, respectively, so that R 1 R 2 + pṽ π 1 n+1,γi 1, j + 1) + qṽ π 2 n+1,γi 1, j) ) ṽ π 3 n+1,γi, j)] + ṽ π 4 n+1,γi, j) ṽ π 5 n+1,γi, j 1)] 0. 5)

15 Article submitted to Management Science; manuscript no. MS- 15 Since policies π 1, π 2, and π 4 are potentially sub-optimal, 4) follows from 5). In any of the cases that i 2 each process serves at station 1. So, each process moves to the next state according to the first event with the relative position of each intact. Since there are now n steps left, the inductive hypothesis yields the result. It remains to consider the case where one or more of the processes cannot serve at station 1; when i = 1. Policies π 1 and π 2 have the provider work at station 2 and policies π 3 and π 5 have the provider work at station 1. Let π 4 have the provider work at station 1 also. If the first event is a service completion at station 1 in Processes 3-5, after which all processes follow an optimal control, then the remaining rewards in 5) are R 1 R 2 R 1 + pṽn,γ 0, j + 1) + qṽ n,γ 0, j) pṽ n,γ 0, j) qṽ n,γ 0, j 1) )]. 6) If the first event is a service completion at station 2 in Processes 1-2, again after which optimal controls are used, then the remaining rewards from 5) are R 1 R 2 + R 2 + pṽn,γ 0, j) + qṽ n,γ 0, j 1)) ṽ n,γ 1, j) ) + ṽn,γ 1, j) ṽ n,γ 1, j 1) )]. 7) Adding expressions 6) and 7) yields R 1 R 2 + pṽn,γ 0, j + 1) + qṽ n,γ 0, j) ṽ n,γ 1, j) ) + ṽn,γ 1, j) ṽ n,γ 1, j 1) )] 8) Note that expression 8) is implied by the left hand side of inequality 5) and the inductive hypothesis. The result follows. Of course the issue with the policy that prioritizes station 1 is that it is also the policy that spends the least time on average) at station Numerical Study In this section, we study numerically the control policies discussed thus far and compare them to other potential policies in practice. To do so, we recall that the Lutheran Medical Center LMC) located in Brooklyn is a 468-bed academic teaching hospital. For the LMC s emergency room, it is reported that the average service time for phase 1 triage) is approximately 7 minutes and that the average phase 2 treatment) time is approximately 13 minutes. As a result, we fix service times with rates = per hour and µ 7 2 = per hour. For the LMC the number of providers 13 ranges from 1 to 5. For this study, we fix the number of providers at the nominal level 3. We also

16 16 Article submitted to Management Science; manuscript no. MSassume that the triage time is exponential while the treatment time has an Erlang distribution with parameters 3, 3 ). This last assumption keeps the average service time 1. We assume that the system accrues a reward of 20 after a service completion at station 2 i.e. R 2 = 20), and that patients join the second station immediately after completing service at station 1 i.e. that p = 1). We compare policies for two scenarios. In the first scenario R 1 = 10 so that R 1 < R 2. In the second scenario, R 1 = 15 so that R 1 > R 2. Hospitals do not generally track the time that a patient spends waiting for treatment before leaving the system. As a result, we do not have direct estimates for β. We propose the following for estimating a range for β and then parameterize β over that range. Assuming that abandonment time strongly correlates with service time, let ˆθ = µˆβ, where ˆβ is our estimate of the abandonment rate. This is the average patience measured in units of average service time. We use ˆθ to estimate the abandonment rate. That is to say, vary ˆθ over a sensible range of values, and for each value of ˆθ in this range, determine the corresponding value of ˆβ. For this study, we assume that ˆθ 5, 30). This implies that the mean abandonment actually patience) time is from 5 to 30 times that of the mean service time i.e. about one to 6.5 hours). This yields that the range for the abandonment rate β is approximately 0.15, 0.92). To justify the range we have proposed for θ, assume there is a linear relationship between the fraction of abandonments, PAb), and average waiting time, EW ], see for example Mandelbaum and Zeltyn 2007) where it is shown that this relationship holds for the Erlang A model): PAb) = β EW ]. 9) The relation 9) between the average wait in queue, EW ], and the fraction of patients who left without being seen, PLWBS), provides a method of estimating β as follows ˆβ = PLWBS) EW ] = %LWBS Average wait. 10) Using the data in Table 1 of Batt and Terwiesch 2013), we use 10) to obtain estimates for the abandonment rate ˆβ see Table 1). The Emergency Severity Index ESI), which is used in triage, has 5 levels of severity with levels 1 and 2 signifying patients requiring prompt attention and in need of hospitalization. The TTR is designed for patients who will not need to be hospitalized and generally only includes those in levels 3-5. We have extended this to include level 2 to expand the range of consideration for our study. If we compute the corresponding ˆθ estimates we get a range for ˆθ that is approximately 5,20). We extend this to 5,30) which leads to.29,.92)) which fall in the range of 0.15, 0.92); a proper subset of the range for β we have proposed. In other words, we have proposed a range of values for θ that includes the range of values for ˆβ we have estimated

17 Article submitted to Management Science; manuscript no. MS- 17 using the data from Batt and Terwiesch 2013) and an estimation procedure based on the Erlang-A model Mandelbaum and Zeltyn 2007). We consider a slightly wider interval to account for the possibility of a smaller hospital than that considered in Batt and Terwiesch 2013). Let S,, β) := β and recall that Theorem 2 states that under the exponential service 1 times, single provider system assumptions, λ < S, implies that the provider should prioritize,β) station 2. The intuition as has already been alluded to) is that this policy minimizes the number 1 of abandonments. However, this policy is unstable when λ S,. The analogous condition,β) when there are N servers is λ < ) 1 N S,. The policy that prioritizes station 1 leads to a stable,β) system whenever λ N < so we would also like to study the region, 1 S,,β) λ N <. We consider the arrival rates {6, 7, 8, 9, 10, 11, 12}. Note that under the exponential assumption if λ 10, then the system under the policy that prioritizes station 2 is unstable. The policy that prioritizes station 2 has the potential disadvantage that in stationarity it leaves station 1 to work at station 2 after every station 1 service. In other words, it is the policy that spends the least amount of time working at station 1 on average) out of all of the policies considered here. Since triage is supposed to take place rather quickly to safeguard getting timely care to those who have truly emergent problems recall the description in Section 2), it is desirable to minimize the number of patients awaiting triage. As a result, using a threshold policy, for example, may be a sensible alternative to prioritizing station 2. In what follows, we use a discrete event simulation to compare the average reward of several policies for different values of λ and β, see Table 2 for a complete list of the parameter values). In each scenario for R 1, the simulation length is one year 24 hours per day) per replication with 30 replications. We compare the following policies, that are described in detail in Section 2: 1. Prioritize station 1 P1); 2. Prioritize station 2 P2); 3. Exhaustive policy E); 4. Threshold policy T) having a threshold values, which we denote by K, of 5, 10, 15, and 20. Two more points should be made about the simulation study. First, while the dynamic programming formulation allows patients in service to abandon, in reality this is unlikely so in the simulation we disallow customer abandonments during service. Second, medical providers generally do not stop treatment on one non-emergent patient to help another; so in the study preemption is disallowed Summary of Results from Numerical Study We have assumed throughout that maximizing the average reward is our primary concern, but that some providers may prefer to forego some average reward in favor of having the provider prioritize

18 18 Article submitted to Management Science; manuscript no. MSstation 1 more often. One reason for this is that minimizing the number of patients awaiting triage may be desirable given that triage is supposed to take place rather quickly to safeguard getting timely care to those who have truly emergent problems. Moreover, it seems that the model may have higher average rewards, but larger average total number in the system; and thus, longer wait times. This trade-off is also examined. The basic insights obtained from the numerical study follow with details in the following sections. It has already been shown in the exponential case that as long as the policy P2 yields a stable Markov process, the average reward is maximized. The numerical study confirms this finding, but shows that when the system is lightly loaded there may not be significant differences between the optimal policy and several others. When the system is highly loaded there may be significant loss of average reward when deviating from P2. Some of the loss can be recovered using threshold policies. When P2 does not yield a stable Markov chain, the average waiting times are unbounded making this policy impractical. However, the average reward of the other candidate policies are not significantly different. A provider might look to a policy with a low average number in the system. Depending on the priorities of the provider s)he may choose a threshold policy or the policy that prioritizes station 1. Finally, we would like to mention that while for fixed K, threshold policies could be implemented with the service provider having full knowledge of the current number of patients at station 1, they could also be implemented with the server only having a sense of the magnitude of the queue. In particular, whether or not they switch from working at station 2 to station 1 when the number of patients at station 1 is 15 or 16 does not significantly effect the average reward. We ran each of the simulations for threshold levels ±1 from those considered to verify this assertion. Since the average rewards are virtually indistinguishable we omitted the numbers for brevity Average Reward for λ N < 1 S,,β) Subcase I: R 1 = 10) Results are summarized in Table 3. In all instances, the policy that prioritizes station 1 P1), which corresponds to K = 1 performed the worst. In almost all instances, the policy that prioritizes station 2 P2), corresponding to K =, performed the best. In the instances that policy P2 did not perform the best, it was a less than 0.03% away from the optimal. The last two observations suggest that Theorem 2 may hold under different service distributions for phase-two service i.e. treatment). The exhaustive policy E) sits between the threshold policies with K = 1 and K = 5 except when λ = 9. In the latter case, the exhaustive policy and the threshold policy with K = 5 are comparable. For each value of β tested, lightly loaded systems with λ {6, 7}), are comparable; all of the policies are within 2% of the highest value. In general, for a given value of β, all policies tend to