An Application of Yield Management for Internet Service Providers

Transcription

1 An Application of Yield Management for Internet Service Providers Suresh K. Nair, 1 Ravi Bapna 2 1 Department of Operations and Information Management, School of Business Administration, U41-IM, University of Connecticut, Storrs, Connecticut College of Business Administration, 214 Hayden Hall, Northeastern University, Boston, Massachusetts Received March 1999; revised January 2001; accepted 31 January 2001 Abstract: In this paper we study strategies for better utilizing the network capacity of Internet Service Providers (ISPs) when they are faced with stochastic and dynamic arrivals and departures of customers attempting to log-on or log-off, respectively. We propose a method in which, depending on the number of modems available, and the arrival and departure rates of different classes of customers, a decision is made whether to accept or reject a log-on request. The problem is formulated as a continuous time Markov Decision Process for which optimal policies can be readily derived using techniques such as value iteration. This decision maximizes the discounted value to ISPs while improving service levels for higher class customers. The methodology is similar to yield management techniques successfully used in airlines, hotels, etc. However, there are sufficient differences, such as no predefined time horizon or reservations, that make this model interesting to pursue and challenging. This work was completed in collaboration with one of the largest ISPs in Connecticut. The problem is topical, and approaches such as those proposed here are sought by users. c 2001 John Wiley & Sons, Inc. Naval Research Logistics 48: , 2001 Keywords: yield management; internet service providers; continuous time MDP 1. INTRODUCTION AND MOTIVATION Internet Service Providers (ISPs), companies that are engaged in providing direct online access to the Internet to individuals and corporations, are increasingly challenged to keep up with competition from other ISPs and the so-called commercial services like America Online and Prodigy. These commercial services do provide access to the Internet, but their real draw is their proprietary content that reflects their tie-ups with particular merchandisers, television channels, and business associates. ISPs have to deal with rapid technological advancements and growing demand from an expanding customer base while at the same time maintaining a desirable customer service level. It is not uncommon to hear of frustrated users who have been bumped, or simply disconnected from the network arbitrarily, or of others who struggle to get in at peak hours typically lunch hours on business days, evenings, or on days of inclement weather. Recall Correspondence to: S.K. Nair c 2001 John Wiley & Sons, Inc.

2 Nair and Bapna: Yield Management for Internet Service Providers 349 the problems that America Online (AOL) faced due to capacity limitations. AOL had to spend about $350 million to add capacity to overcome these problems. In this paper we study optimal strategies for utilizing the network capacity of ISPs when they are faced with stochastic arrivals and departures of customers attempting to log-on or -off, respectively. The optimal policy maximizes discounted net profits over an infinite planing horizon for the ISPs while improving service levels for higher class customers. This research was conducted in collaboration with one of the largest ISPs in Connecticut. Our approach is based on the general yield management framework that has been successfully applied to a broad spectrum of applications in the service sector. Pioneering efforts of American Airlines [2] were successfully replicated by the other airlines and by companies like Hertz for car rentals (Carroll and Grimes [5]), the leading hotel chains for rooms, and United Artists for managing the sales of movie rights. American Airlines estimated a quantifiable benefit of $1.4 billion over the period and expects an annual revenue contribution of over $500 million to continue into the future (Smith, Leimkuhler, and Darrow [17]). Recently the term perishableasset revenue management or PARM has been coined (Weatherford and Bodily [20]) to cover the separate but related problems of yield management, overbooking, and pricing. We shall stick with the term yield management (YM) which in essence is an integrated demand-management, overbooking and capacity utilization system that focuses on improving a strategic objective, such as revenue or service level, by carefully setting differential treatments for various market segments and dynamically reallocating fixed but perishable capacity between segments. The perishability of the resource, that is, the existence of a date or time after which it is either unavailable or it ages at a significant cost distinguishes this class of problems from inventory control problems (Weatherford and Bodily [20]). Airline seats, theater seats, hotel rooms, fashionable clothing, and traffic on communication channels are a few examples of such perishable assets. Airlines, for instance, deny advance bookings to price-sensitive customers for peak travel periods because they anticipate enough demand from the higher paying customers. We are not aware of any work in yield management for ISPs. Based upon our discussions with our collaborators, whom we will call Deciles, Inc., the Internet services division of a regional telephone company and a recent entrant in the field of ISPs; and after a close examination of their operational topology, we identified the presence of three common characteristics as articulated by Weatherford and Bodily [20] with situations where yield management is currently practiced. Perishable assets. Although there exist many stages that a user dialing-in from his or her home or office needs to cross before reaching the Internet backbone, as exhibited in Figure 1, the bottleneck points were identified by the managers of Deciles as the ISP s modem racks. A modem rack is a set of 24 modems that is typically connected to a T1 type transmission channel that serves as a gateway for users trying to log-on. The number of modem racks a given access point has is proportional to the population of the area the access point covers. A modems to users ratio (MUR) such as 1:10 (all numbers are camouflaged to protect proprietary data) is planned, and this significantly influences service. Obviously, a 1:8 MUR will provide better service and a higher chance of access to the network than an MUR of 1:10. Access points are located across Connecticut such that any user could dial into the system using a local number. Thus the perishable asset here is the modem capacity available at any instant of time, at any access point, for users to log onto. This capacity perishes and is regenerated with time and is dependent on the rate of users logging on and logging off over time. A fixed number of units. Based on the 1:10 MUR, a total of 1200 modems or 50 modem racks are distributed across the state of Connecticut. This allows Deciles to service a maximum of 1200 customers at any one time on the network and cover a population of 12,000 ( ). Thus the

3 350 Naval Research Logistics, Vol. 48 (2001) Figure 1. Steps involved in connecting a user to the Internet. assumption is that 1 in 10 customers would log-in at any time. In case this assumption is violated, then the service quality would deteriorate and customers would either be denied access, bumped, or asked to retry. Thus the total number of modems in the system is the maximum capacity available. While the above-mentioned scenario best represents the current capacity constraints of ISPs, future trends in Internet access technology indicate the likelihood of packet-switching approaches that may not require a dedicated modem. In such systems the densities of information exchanges is more significant than the number of modems. We defer the analysis of such systems to future research. Deciles needs to manage its fixed perishable capacity in the face of uncertain demand. At a strategic level, the number of modems used, and its location around the state is another interesting optimization problem. This compares directly with the well-known news vendor problem where the vendor must decide how many papers to order in the face of uncertain demand. However, in that problem the order quantity may be varied without much expense and the decision is tactical in nature, whereas in the ISP situation the problem is strategic with major capital expenses involved. The possibility of segmenting price-sensitive customers. Unlike airlines, ISPs do not at present differentiate between their customers. Instead they offer a variety of services based on the number of access hours. Typically a base fee is charged which allows up to certain fixed number of Internet access hours. Additional access hours are charged on an hourly basis. Another approach adopted by providers is to provide unlimited access for higher fees. We do not consider the case where providers, such as Netzero.com, provide free dial-up service and rely solely on advertising for their revenues. Based on our discussion with the managers at Deciles, Inc., we proposed to segment their customers into two classes which we call Platinum and Gold. The basis for this segmentation is quality of service. Platinum customers would be guaranteed a higher quality of service which we shall define as the probability of getting access to the Internet when they

4 Nair and Bapna: Yield Management for Internet Service Providers 351 attempt to dial-in. For this they would pay higher fees compared to gold customers who would correspondingly receive a lower quality of service. To demonstrate how an ISP can operationalize such a segmentation is one of the goals of this research. Customer segmentation gives us the opportunity of introducing incentive mechanisms similar to those pursued by airlines. One can imagine free upgrades to higher classes conditioned upon certain numbers of logged hours like the airlines do with air miles. Also due to the recent telecommunications deregulation bill, competition has heated up for both the long distance and local service, forcing Deciles to think of innovative services to provide to customers. The remainder of the paper is organized as follows. Section 2 discusses the differences between YM for ISPs and traditional YM applications, and linkages of this work to the admission control in queueing literature. Section 3 presents the problem formulation and the continuous time Markov decision process model, and Section 4 discusses the solution methodology. Finally, Section 5 concludes by summarizing the research effort and discusses directions for future research. 2. LITERATURE REVIEW Differences between YM for ISPs and Traditional YM Applications As one hotel industry expert put it yield management is charging a different rate for the same service to a different individual (Nykiel [15]). In effect all yield management applications attempt to arrive at an optimal tradeoff between average price paid and capacity utilization. This is achieved by making decisions at two levels. First, at the tactical level, decisions have to made that determine: 1. Aggregate capacity. For the airlines it could be the fleet size and mix or, for hotels, the number of rooms. In our case it is the number of modems racks and their location. The location problem here is not different from the typical set covering or the maximal population covering models, on which extensive work has been done (Fisher and Kedia [7]). 2. Market segmentation policy. The key decisions here would be to arrive at the number of segments and the value of each segment. Care should be taken that arbitrage opportunities the possibility that customers belonging to the lowprice segments acquire goods or services expressly for resale to others in higher price segments do not exist. Airlines, for instance, have restrictions such as mandatory Saturday night stayover and making tickets nonrefundable in event of cancellations. It is here that yield management interacts with marketing and uses theories of quantity discounting or time of purchase to segment customers. 3. Price setting. Based upon the market segmentation decisions and using knowledge of demand patterns, decision-makers have to decide what price is appropriate for a particular segment. For ISPs, this may be a flat fee or a pay-as-you-go approach. In close relation to the tactical level decisions there are the operational level decisions. Here the primary concern is to make optimal day-to-day decisions that ensure the attainment of specified objectives. Hotels, for instance, have to decide whether or not to accept a customer s request for a reservation based on the number of available rooms belonging to that customer s segment and the expected mix of customers that the hotel expects in the remainder of the time horizon, which is usually 6 p.m. each day. Airlines make similar decisions on requests for seat reservations

5 352 Naval Research Logistics, Vol. 48 (2001) on particular flights. There are three issues that make YM in ISPs different from airline/hotel applications from a modeling standpoint: 1. The ISP problem is inherently continuous both in state and time. True capacity is modem-hours rather than number of modems, and customers draw upon this capacity in a continuous fashion. There is no natural cutoff time which could be used as a time horizon to solve the problem. Airlines use flight takeoff time and hotels use 6 p.m. For ISPs, customers log-on and -off all day long. 2. Service is determined by the time it takes to get on the network. Thus the request and the service happen simultaneously. This is not the case in airlines and hotels where the request is made at one time (making the reservation) and the capacity is used up at another (the flight taking off or the hotel room gets occupied). 3. For the above reason, overbooking is not an issue in YM for ISPs. We address the first problem by formulating our problem as an infinite horizon, homogeneous, continuous-time Markov decision process (CTMDP) for which optimal policies are readily derived using standard techniques such as value iteration. The second issue creates an operational problem. How does one give priority to platinum customers over gold customers without knowing beforehand which type of customer is making the call. Once the customer seizes the modem, she is on the network and the service is already initiated. Deciles wanted to avoid bumping customers once they were on the network. We worked around this problem by proposing to supplement the customer-authentication process that is activated every time a customer tries to log-on. Here our algorithm could be used to decide, based on the class of the customer (either Platinum or Gold), whether to let the customer log-on or not. This decision would require knowledge of the number of modems available at the time, the customer profile, and the expected mix of customers that may attempt to utilize the service in the near future. An alternative approach would be to utilize dynamic pricing that would, based on the system load, indicate to customers the spot premium they could pay to obtain a higher class of service. In this paper we assume that customer segmentation is undertaken a priori. The existing research has focused primarily on the operational level (Bitran and Mondschein [3] and Lee and Hersh [11]). Most of the yield management literature assumes simplified dynamic relationships about customer behavior during the planning horizon. Alstrup et al. [1] assume that all customers arrive sequentially during the target date while studying booking policies for a single flight leg with two types of customers. Ladany [9] makes a similar assumption in his formulation for managing reservations in the motel industry. Lee and Hersh [11] relax this assumption for the airline reservation problem and do not require any advance knowledge about the arrival pattern for the various booking classes. They generate critical values which represent the optimal policy for making accept/reject decisions provided that the systems parameters remain constant, an approach followed by us. Bodily and Weatherford [4] and Weatherford and Bodily [20] present a comprehensive taxonomy for general PARM problems. Table 1 uses their taxonomy to place our problem in comparison with the airline and hotel problems as formulated by Lee and Hersh [11] and Bitran and Mondschein [3], respectively. Lautenbacher and Stidham [10] describe the underlying Markov decision process in the singleleg airline yield management problem. They introduce the terms static and dynamic for approaches that allow customers of different classes to book concomitantly, and those that assume the demand for different fare classes arrive in some predetermined order, respectively. We model the ISP problem dynamically. Subramanian, Stidham, and Lautenbacher [19], present a dynamic programming approach to the airline yield management problem with overbooking, cancellations,

6 Nair and Bapna: Yield Management for Internet Service Providers 353 Table 1. Taxonomical comparison of select YM models, based on Bodily and Weatherford [4]. Elements Airlines Hotels ISP Resource Discrete Discrete Continuous Capacity Fixed Fixed Fixed Cutoff time Yes Yes No Prices Predetermined Predetermined Predetermined Willingness to pay Buildup Buildup Not applicable Discount price classes k k k Arrival pattern Stochastic Stochastic Stochastic Departure pattern No No Stochastic log-offs Show-up of discount reservation Certain Certain Stochastic Show-up of full price Certain Certain Stochastic Group reservations Yes Yes Not applicable Overbooking Yes Yes Not applicable Diversion No No Possible Displacement No Downgrading Not applicable Bumping procedure None None None Asset control mechanism Nested Nested Nested Decision rule Dynamic Dynamic Dynamic and no shows. Bitran and Mondschein [3] develop renting policies for hotels making no assumptions concerning the particular order between the arrival of different types of customers which is treated as a stochastic process. They propose heuristics for problems that involve multiple night stays based upon results obtained from the single-night case. This issue is particularly relevant for our problem, because, in contrast to business travelers who visit airport hotels and motels for single nights (Ladany [9]), virtually no user shall log-on to the Internet for just a single decision period. Typically people log onto the Internet to visit some web sites or browse through contents of newsgroups, both of which consume time. We have addressed this issue by not just considering the number of users attempting to log-on but also the number logging-off during a decision period. Observed distributions of these two phenomena are closely related as evident in Figure 2. One of the conditions in our model is that customers are not bumped once they have been connected, which is an undesirable business practice in the first place Linkages to the Admissions Control in Queueing Literature Another perspective of the YM for ISPs problem can be obtained from examining the vast literature on admission control in queueing systems. Stidham [18] reviews this branch of the literature for static and dynamic systems, in the case of single and multiple servers. This research stream (Naor [14], Yechiali [22], and Mendelson and Yechaili [13]) brings to light the effect of negative network externalities imposed by individually optimizing decision rules. Such rules neglect the impact of a given job on the performance of the system as a whole. These negative externalities translate into deteriorated performance for existing users as a result of admission of additional jobs and are typically remedied by congestion tolls. Similarly, in our setting, a shortsighted individually optimizing approach would accept any class of customer provided there was any available capacity and a nonnegative revenue contribution. This, however, would decrease the probability of higher-valued customers getting access in the successive epochs and would not lead to maximal long-term expected revenue, as shown in Section 4. In light of the above, we

7 354 Naval Research Logistics, Vol. 48 (2001) Figure 2. Log-in/off distribution by quarter hour for Deciles, Inc. borrow from this literature the overall approach of considering the impact of a log-on request on the long-term benefit from the system as a whole. In the analysis of many traditional queueing systems (Stidham [18]), jobs are buffered if no server is immediately available, and servers are assumed to have a certain processing rate. Both of these factors may ultimately influence important parameters like the average response time and the long-run average number of jobs in the system. Contrast this with the YM for ISPs case where the average response time and the average number of jobs in the system are not of concern and the jobs have random durations that are dependent on the web-browsing behavior of the consumers, who cannot be buffered. We assume that the ISP desires to maximize long-term expected discounted benefit given a modem capacity constraint and multiple customer segments. In queueing, other criteria have been used, such as the power criteria proposed by Kleinrock [8] which takes the ratio of the throughput and the average response time to achieve admission control. Xu and Shantikumar [21] introduce a dual system to address optimal admission control for a first-come first-served ordered-entry M/M/m queueing system that maximizes expected discounted profit. The dual approach constructs an isomorphic preemptive last-come, first-served ordered-entry queueing system that is subject to expulsion control. Their unique approach determines the solution from the behavior of individual customers and could be used in the YM for ISPs problem. Applying their approach to our setting with multiple service classes would provide an alternative to the conventional dynamic programming approach based on value iteration, and is an exciting direction for future research. Additional related streams of work can be found in providing admission control in ATM networks that allow for multiple service classes (e.g., audio, video, text) that could have different quality service requirements. Elwalid and Mitra [6] provide analytic approximations, using which they calculate the admissible set but fail to provide an economic objective function which is a must in our setting. In summary, there is a close association with the various aspects of the admission control literature with the YM problem for ISPs. By formulating our problem as a CTMDP, as done by

8 Nair and Bapna: Yield Management for Internet Service Providers 355 Puterman [16], and using value iteration to solve for the optimal control policy, we do draw upon the findings of this interesting research stream. 3. PROBLEM FORMULATION We consider the case where ISPs have segmented their markets into two classes, Platinum denoted by 1 and Gold denoted by 2. Let λ 1 and λ 2 represent the Poisson arrival rates of platinum and gold class customers, respectively. Also, assuming exponential stays let µ 1 and µ 2 represent the service rates of platinum and gold class customers, respectively. The problem is formulated as a queuing admission control model in the form of a continuoustime Markov decision process (CTMDP), a special case of the general semi-markov decision process. In such systems, intertransition times are exponentially distributed, and actions are chosen at every decision epoch. In what follows, we use notation and terminology consistent with Puterman [16]. Our results are an extension of his discussion of the single server single customer class queue admission problem to a multiserver situation with multiple customer classes. We assume that the ISP s objective is to maximize the expected total discounted reward where α>0 is the continuous time discount rate, α = ln(λ), where λ is the discrete time discount rate, λ =1/(1 + i), where i is the interest rate. Suppose the state is defined as i, j, b when there are i platinum and j gold customers in the system, and event b occurs. We denote b =1as a platinum arrival, and b = 1 as a platinum departure. Similarly, we denote b =2as a gold arrival and b = 2 as a gold departure. Suppose that there are a total of M modems available at the ISP; then clearly i + j M at all times. In this situation, the decision epochs are when a customer arrives or departs. When a customer arrives, the possible actions, a, would be to admit the customer, denoted by a =1, or to refuse service to the customer, denoted by a =0. When there are no arrivals, or when there is a departure, the only action possible would be to continue, also denoted as a =0. Let β ijba be the rate of transition out of state i, j, b given action a is taken. Let the transition probabilities be denoted by q(k i, j, b ), which is the probability of transition from state i, j, b to state k. In state i, j, 1, where i + j<m, that is, when a platinum customer arrives and there are still modems available; when the action is to admit, that is, a =1, the transition probabilities can be written as (i +1)µ 1 /β ij11, k = i, j, 1, jµ 2 /β ij11, k = i +1,j 1, 2, j 1, q(k i, j, 1 ) = (1) λ 1 /β ij11, k = i +1,j,1, λ 2 /β ij11, k = i +1,j,2, where the rate of transition out of the state, β ij11 =(i +1)µ 1 + jµ 2 + λ 1 + λ 2. These can be explained in the following manner: We use the convention of noting the state just after a departure and just before an arrival. In state i, j, 1, when the decision is to accept the platinum customer that arrives, a =1, the number of platinum customers in the system gets incremented to i +1. The next event could be a platinum departure, b = 1, a gold departure, b = 2, a platinum arrival, b =1, or a gold arrival, b =2. That is, when there is a platinum departure the system would transition to state i, j, 1 at a rate of (i +1)µ 1 since there are i +1customers in the system, when there is a gold departure, the system will transition to state i +1,j 1, 2 at a rate of jµ 2, when there is a platinum arrival, the system will transition to state i +1,j,1 at a rate of λ 1, and when there is a gold arrival, the system will transition to state i +1,j,2 at a rate

9 356 Naval Research Logistics, Vol. 48 (2001) of λ 2. Since any of these four events could occur next, the next decision epoch occurs at the rate of β ij11 =(i +1)µ 1 + jµ 2 + λ 1 + λ 2. Therefore, the transition probability of going from state i, j, 1 to state i, j, 1 is (i +1)µ 1 /β ij11, the transition probability to state i +1,j 1, 2 is jµ 2 /β ij11, the transition probability to state i+1,j,1 is λ 1 /β ij11, and the transition probability to state i +1,j,2 is λ 2 /β ij11. In state i, j, 2, where i + j<m, that is, when a gold customer arrives and there are modems available; when the action is to admit, that is, a =1, the transition probabilities can be written as iµ 1 /β ij21, k = i 1,j+1, 1, i 1, (j +1)µ 2 /β ij21, k = i, j, 2, q(k i, j, 2 ) = (2) λ 1 /β ij21, k = i, j +1, 1, λ 2 /β ij21, k = i, j +1, 2, where the rate of transition out of the state, β ij21 = iµ 1 +(j +1)µ 2 +λ 1 +λ 2. The explanation is similar to that given above noting that when the decision is to accept a gold customer, the number of gold customers in the system get incremented to j +1. In all states i, j, b, where i + j M and b { 2, 1, 1, 2}, when the action is to refuse service or continue, a =0, the transition probabilities can be written as iµ 1 /β ijb0, k = i 1,j, 1, i > 1, jµ 2 /β ijb0, k = i, j 1, 2, j > 1, q(k i, j, b ) = (3) λ 1 /β ijb0, k = i, j, 1 λ 2 /β ijb0, k = i, j, 2, where the rate of transition out of the state, β ijb0 = iµ 1 +jµ 2 +λ 1 +λ 2. Again the explanation is similar to the one above, noting that since the action is to refuse entry to the customer, the number of platinum and gold customers is not immediately incremented. Note that β ij11 = β (i+1)jb0, (4) β ij21 = β i(j+1)b0. (5) Suppose every arriving platinum customer contributes a revenue of K 1, and every arriving gold customer a revenue of K 2, as modeled in Puterman [16]. Let the system holding cost due to the use of network infrastructure be at a rate of c 1 (i) when there are i platinum customers in the system, and c 2 (j) the corresponding rate for gold customers when there are j gold customers in the system. We assume that c 1 (i) and c 2 (j) are nondecreasing and convex, K 1 K 2 and c 1 (i) c 2 (i) for all i. As a first approximation, the revenue could be estimated by the ratio of total membership fee and other revenue from a class of customers to the average number of log-ons in a month by that class of customer. Similarly, the holding cost rate could be estimated by apportioning the network and overhead costs based on usage by each class of customer. This approach to estimating costs would be akin to estimating ordering and holding costs in inventory models. The expected discounted reward r( i, j, b,a), between decision epochs, given the system is in state i, j, b and action a is taken, could be written as (from Puterman [16, ]) r( i, j, 1, 1) = K 1 [c 1(i +1)+c 2 (j)] α + β ij11, (6)

10 Nair and Bapna: Yield Management for Internet Service Providers 357 r( i, j, 2, 1) = K 2 [c 1(i)+c 2 (j + 1)] α + β ij21, (7) r( i, j, b, 0) = [c 1(i)+c 2 (j)] (8) α + β ijb0 for b { 2, 1, 1, 2}. We analyze the model by using uniformization (Lippman [12]). The idea of uniformization is to convert a continuous time model with state-dependent exponential transition rates (the β ijba stated above) to a model with a state-independent transition rate, C. The trick is to ensure that the probabilistic behavior of the uniformized model be same as that of the original model. This can be done by ensuring that the infinitesimal generator of the two models is identical. We pick a constant transition rate that would be larger than all possible transition rates β ijba. A look at equations for β ij11,β ij21, and β ijb0 will show that the maximum value that these could achieve would be C = M max(µ 1,µ 2 )+λ 1 + λ 2. Since the constant C is chosen to be at least as large as the largest rate seen in the original system, uniformization results in more frequent transitions than in the original system. One could think of these as fictitious transitions from a state to itself. Because of this, we need to adjust the transition probabilities and rewards under the uniformized system (designated with a tilde) as follows: { 1 [1 q(k i, j, b )]βijba /C, k = i, j, b, q(k i, j, b ) = q(k i, j, b )β ijba /C, k i, j, b, (9) r( i, j, b,a)=r( i, j, b,a) α + β ijba α + C. The optimality equation has the form V ( i, j, b ) = max a { r( i, j, b,a)+ C α + C } q(k i, j, b )V (k), and from Puterman [16, Theorem , p. 566], if a maximum is attained for each V ( ) above, then a stationary deterministic optimal policy exists, since the action space is finite and compact, and the rewards, transition rates, and transition probabilities are continuous in a for each state. We can also write these value functions in expanded form as (α+β ij11)k 1 [c 1(i+1)+c 2(j)] α+c + 1 α+c [(i +1)µ 1V ( i, j, 1 ) + jµ V i, j, 1 = max 2 V ( i +1,j 1, 2 )+λ 1 V ( i +1,j,1 ) + λ 2 V ( i +1,j,2 )+(C β ij11 )V ( i, j, 1 )] V ( i, j, 1 ), (α+β ij21)k 2 [c 1(i)+c 2(j+1)] α+c + 1 α+c [iµ 1V ( i 1,j+1, 1 ) +(j +1)µ V i, j, 2 = max 2 V ( i, j, 2 )+λ 1 V ( i, j +1, 1 ) + λ 2 V ( i, j +1, 2 )+(C β ij21 )V ( i, j, 2 )] V ( i, j, 1 ), k

11 358 Naval Research Logistics, Vol. 48 (2001) where V ( i, j, 1 ) = c 1(i)+c 2 (j) + 1 α + C α + C [iµ 1V ( i 1,j, 1 ) + jµ 2 V ( i, j 1, 2 )+λ 1 V ( i, j, 1 ) + λ 2 V ( i, j, 2 )+(C β ijb0 )V ( i, j, 1 )]. By taking the last term to the left-hand side and simplifying, we get V ( i, j, 1 ) = c 1(i)+c 2 (j) 1 + [iµ 1 V ( i 1,j, 1 ) α + β ijb0 α + β ijb0 +jµ 2 V ( i, j 1, 2 )+λ 1 V ( i, j, 1 )+λ 2 V ( i, j, 2 )] (10) and V ( i, j, 2 ) =V ( i, j, 1 ). From the above equations and from (4) and (5) we can also see that which simplifies to (α+β ij11)k 1 α+c + V ( i +1,j, 1 ) V i, j, 1 = max C α+c [V ( i +1,j, 1 ) V ( i, j, 1 )] V ( i, j, 1 ), { K1 + V ( i +1,j, 1 ) V i, j, 1 = max V ( i, j, 1 ). (11) Similarly, { K2 + V ( i, j +1, 1 ) V i, j, 2 = max V ( i, j, 1 ). (12) Let 4. SOLUTION METHODOLOGY ij1 = K 1 +[V ( i +1,j, 1 ) V ( i, j, 1 )], (13) ij2 = K 2 +[V ( i, j +1, 1 ) V ( i, j, 1 )], (14) which is the difference between the accept and reject rows in (11) and (12). It is clear then from (13) and (14) that, in state V i, j, 1, the optimal decision would be to accept the platinum customer if ij1 > 0, and in state V i, j, 2 the optimal decision would be to accept the gold customer, ij2 > 0. We will show next that ij1 and ij2 are nonincreasing in i and j. LEMMA 1: For all i, ij1 and ij2 are nonincreasing in both i and j.

12 Nair and Bapna: Yield Management for Internet Service Providers 359 PROOF: We prove these by induction using a policy iteration approach. Set V 0 =0; then from (10) we have V 1 ( i +1,j, 1 ) V 1 ( i, j, 1 ) = c 1(i +1) c 1 (i) α + β ijb0, V 1 ( i, j +1, 1 ) V 1 ( i, j, 1 ) = c 2(j +1) c 2 (j). α + β ijb0 Since the costs are increasing convex, and β ijb0 increases linearly in i and j, the above equations are nonincreasing in i and j. Suppose the result were true for V n 1 ( i +1,j, 1 ) V n 1 ( i, j, 1 ) and V n 1 ( i, j +1, 1 ) V n 1 ( i, j, 1 ). We can then show from (10) that V n ( i +1,j, 1 ) V n ( i, j, 1 ) is nonincreasing in i and j. The result follows by induction and the fact that V n ( ) converges monotonically to the optimal V ( ) (see Puterman [16], Section 6.11). We now present a control limit result for this model. THEOREM 1: If the optimal decision in state V ( i, j, 2 ) is to reject the gold customer, then the optimal decision in state V ( i, j, 2 ),j >j, is also to reject the gold customer. PROOF: The proof follows directly from Lemma 1 and the paragraph before Lemma 1. EXAMPLE: Suppose the lump sum rewards are K 1 =20and K 2 =10for platinum and gold customers respectively. Let the cost rates be c 1 (I) =1.3 i and c 2 (j) =1.8 j. Let the maximum number of modems, M = 9. In this case one could confirm that there would be 220 states i, j, b. Suppose the arrival rates for platinum and gold customers are λ 1 =0.05 and λ 2 =0.2, respectively, and their service rates are µ 1 =0.5 and µ 2 =0.25, respectively. Suppose the discrete time discount rate λ =0.9, then the continuous time discount rate, α = ln(λ) = The uniformization constant C = M max(µ 1,µ 2 )+λ 1 + λ 2 =4.75. Tables 2 and 3 gives values of ij1 and ij2 under various conditions, and Figure 3 shows these graphically. Figure 3 shows that when there are already three gold customers in the system, then no more gold customers are admitted. Therefore, gold customers may be rejected even if there are available modems in the system. For this example, this threshold value remains the same for varying values of number of platinum customers in the system, as can be seen from Figure 4 top. Figure 4 bottom shows that, as the number of gold customers increase, the incremental value from platinum customers also reduces, albeit by a very small amount. Table 2. Values of i,j,b for i = 1 and varying j. j i = 1, b = 1 i = 1, b =

13 360 Naval Research Logistics, Vol. 48 (2001) Table 3. Values of i,j,2 for varying i and j. j i = 1 i = 2 i = 3 i = 4 i j = 1 j = 2 j = 3 j = CONCLUSION AND EXTENSIONS This paper studies optimal policies for allocating modems capacity among segments of customers using a continuous time Markov decision process model formulation considering stochastic arrivals and departures of customers attempting to use the Internet. As with other yield management applications, a limited capacity plays a critical role in the determination of such optimal strategies. However, there are sufficient differences, such as no prespecified time horizon and no prebooking or reservations, to make this model interesting to pursue. Finally, it also seems to be a tool that is needed by ISPs. Real time implementation is made possible by relying on a database of critical values generated by the model, which serves as input to the customer authentication module which gets activated every time a customer dials up the ISP to get connected to the Internet. This database would have to be updated periodically when there is a significant change in the system or customer parameters. More research needs to be done on modeling customer segment specific, and time of day/week specific customer behavior of using the Internet. Another approach may be to assess a price for each service in real time, and give better service to customers who are willing to pay more at that instant in time. However, these will make the model more complicated. Figure 3. Incremental values, ij1 and ij2, of accepting platinum and gold customers, respectively, as a function of gold customers in the system, j.

14 Nair and Bapna: Yield Management for Internet Service Providers 361 Figure 4. Values of i,j,2 for varying number of gold and platinum customers. Future research also needs to be done to incorporate real-time learning capabilities into our model so that it can dynamically detect patterns in log-in/off behavior using neural networks. Such a tool would automatically alter the behavior of the system in response to environmental changes like lunch-hour traffic jams on the information superhighway and inclement weather. ACKNOWLEDGMENTS We thank Bob D., Dave C., and Tony D. of Deciles, Inc., for taking the time to explain the intricacies of the Internet to us beyond what we knew before, which was not much. We thank Professor Marty Puterman for his advice. We also express our gratitude to the Associate Editor and a Referee for a very careful reading of our manuscript and for making many constructive suggestions.

15 362 Naval Research Logistics, Vol. 48 (2001) REFERENCES [1] J. Alstrup, S. Boas, O. Madsen, and R. Vidal, Booking policy for flights with two types of passengers, Eur J Oper Res 27(3) (1986), [2] American Airlines Annual Report, The art of yield management, 1987, pp [3] G.R. Bitran and S.V. Mondschein, An application of yield management to the hotel industry considering multiple day stays, Oper Res 43(3) (1995), [4] S.E. Bodily and L.R. Weatherford, Perishable-asset revenue management: Generic and multiple-price yield management with diversion, Omega 23(2) (1995), [5] W.J. Carroll and R.C. Grimes, Evolutionary change in product management: Experiences in the car rental industry, Interfaces 25(5) (1995), [6] A.I. Elwalid and D. Mitra, Statistical multiplexing with loss priorities in rate-based congestion control of high-speed networks, IEEE Trans Commun 42 (1994), [7] M. Fisher and P. Kedia, Optimal solution of set covering/partitioning problems using dual heuristics, Manage Sci 36 (1990), , [8] L. Kleinrock, Power and deterministic rules of thumb for probabilistic problems in computer communications, Proc Int Conf Commun, June [9] S. Ladany, Dynamic operating rules for motel reservations, Decision Sci 7 (1976), [10] C.J. Lautenbacher and S. Stidham, The underlying Markov decision process in the single-leg airline yield management problem, Transport Sci 33(2) (1999), [11] T.C. Lee and M. Hersh, A model for dynamic airline seat inventory control with multiple seat bookings, Transport Sci 27(3) (1993), [12] S.A. Lippman, Applying a new device in the optimization of exponential queueing systems, Oper Res 23 (1975), [13] H. Mendelson and U. Yechiali, Controlling the GI/M/1 queue by conditional acceptance of customers, Eur J Oper Res 7 (1981), [14] P. Naor, On the regulation of queue size by levying tolls, Econometrica 37 (1969), [15] R. Nykiel, 1990 s: Decade of yield management, presented at 1989 Yield Management Multi-Industry Conference, Charlotte, NC, [16] M.L. Puterman, Markov decision processes: Discrete stochastic dynamic programming, Wiley, New York, [17] B.C. Smith, J.F. Leimkuhler, and R.M. Darrow, Yield management at American Airlines, Interfaces 22(1) (1992), [18] S. Stidham, Jr., Optimal control of admission to a queueing system, IEEE Trans Automat Control AC-30(8) (1985), [19] J. Subramanian, S. Stidham, and C.J. Lautenbacher, Airline yield management problem with overbooking, cancellations, and no shows, Transport Sci 33(2) (1999), [20] L.R. Weatherford and S.E. Bodily, A taxonomy and research overview of perishable-asset revenue management: Yield management, overbooking, and pricing, Oper Res 40(5) (1992), [21] S.H. Xu and J.G. Shanthikumar, Optimal expulsion control a dual approach to admission control, Oper Res 41(6) (1993), [22] U. Yechiali, On optimal balking rules and toll charges in a GI/M/1 queueing process, Oper Res 19 (1971),