No. of Pages 15, Model 5G ARTICLE IN PRESS. Contents lists available at ScienceDirect. Electronic Commerce Research and Applications

Transcription

1 Eectronic Commerce Research and Appications xxx (2008) xxx xxx 1 Contents ists avaiabe at ScienceDirect Eectronic Commerce Research and Appications journa homepage: 2 Forecasting market prices in a suppy chain game q 3 Christopher Kiekintved a, *, Jason Mier b, Patrick R. Jordan a, Lee F. Caender a, Michae P. Weman a 4 a Computer Science and Engineering, University of Michigan, 2260 Hayward Avenue, Ann Arbor, MI , USA 5 b Department of Mathematics, Stanford University, Buiding 380, Stanford, CA 94305, USA 6 artice info Artice history: 10 Received 17 October Received in revised form 4 November Accepted 5 November Avaiabe onine xxxx 14 Keywords: 15 Forecasting 16 Markets 17 Price prediction 18 Trading agent competition 19 Suppy chain management 20 Machine earning Introduction abstract 36 The quaity of economic decisions made now depend on market 37 conditions that wi prevai in the future. This is particuary true of 38 suppy chain environments, which evove dynamicay based on 39 compex interactions among mutipe payers across severa tiers. 40 We present and evauate methods of forecasting market conditions 41 deveoped to predict prices in the trading agent competition sup- 42 py chain management game (TAC/SCM). In TAC/SCM, automated 43 trading agents compete to maximize their profits in a simuated 44 suppy chain environment. They face many chaenges, incuding 45 strategic interactions and uncertainty about market conditions. 46 Though compex, this domain offers a reativey controed envi- 47 ronment amenabe to extensive experimentation. TAC aso attracts 48 a diverse poo of participants, which faciitates benchmarking and 49 evauation. This is particuary important for prediction tasks, 50 where the quaity of predictions depends in part on the actions 51 of opponents. Research competitions such as TAC offer the oppor- Predicting the uncertain and dynamic future of market conditions on the suppy chain, as refected in prices, is an essentia component of effective operationa decision-making. We present and evauate methods used by our agent, Deep Maize, to forecast market prices in the trading agent competition suppy chain management game (TAC/SCM). We empoy a variety of machine earning and representationa techniques to expoit as many types of information as possibe, integrating we-known methods in nove ways. We evauate these techniques through controed experiments as we as performance in both the main TAC/SCM tournament and suppementary Prediction Chaenge. Our prediction methods demonstrate strong performance in controed experiments and achieved the best overa score in the Prediction Chaenge. Ó 2008 Esevier B.V. A rights reserved. tunity to test predictions in environments where the opposing agents are designed and seected by others, mitigating an important source of potentia bias reative to using opponents of one s own design (Weman et a., 2007). Our agent for the SCM game, Deep Maize, reies on price predictions in both the upstream and downstream markets to make purchasing, production, and saes decisions. Fig. 1 shows an overview of the daiy decision process. Predictions about market conditions feed into an approximate optimization procedure that uses vaues to decompose the decision probem. The architecture is described in detai esewhere (Kiekintved et a., 2006). Since our predictions are integrated into a task-situated agent, we can evauate the benefits of improved predictions on overa performance in addition to prediction error metrics. We appy machine earning to forecast prices in two different markets with distinct negotiation mechanisms and information reveation rues. Like many rea markets, the avaiabe information on market conditions comes from a variety of heterogeneous sources. Integrating different earning techniques and representations to expoit a of the avaiabe information is a centra theme of our approach. We find that representing the current market conditions we is particuary important for effective earning. We begin with a discussion of reated work and an overview of the TAC/SCM game, highighting the benefits and drawbacks of the different forms of information avaiabe for making predictions. The first set of methods we describe are used for predicting prices in the downstream consumer market. Centra to a of these methods is a nearest-neighbors earning agorithm that uses historica q This is a revised and substantiay extended version of a paper presented at the Sixth internationa joint conference on autonomous agents and mutiagent systems (Kiekintved et a., 2007). * Corresponding author. Te.: E-mai addresses: ckiekint@umich.edu (C. Kiekintved), jmier@math.stanford. edu (J. Mier), prjordan@umich.edu (P.R. Jordan), aseep@umich.edu (L.F. Caender), weman@umich.edu (M.P. Weman). URLs: (C. Kiekintved), umich.edu/~prjordan (P.R. Jordan), (M.P. Weman) /$34.00 Ó 2008 Esevier B.V. A rights reserved.

2 2 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx Suppy Market Prediction Component Vaues Suppy Purchasing 80 game data to induce the reationship between market indica- 81 tors and prices. We test variations that empoy two different 82 representations for price distributions: an absoute representation, 83 and a reative representation that expoits oca price stabiity. 84 Finay, we deveop an onine earning method that procedure that 85 combines and shifts predictions using additiona observations 86 within a game instance. Our evauation of these methods incudes 87 both prediction error and suppy chain performance. We show sub- 88 stantia improvements over a baseine predictor, with onine adap- 89 tation exhibiting especiay strong performance improvements. 90 We proceed to discuss methods for making predictions in the 91 market for component suppies. There are two eements to our ap- 92 proach in this market. The first is a inear interpoation mode that 93 uses recent observations to make reativey accurate predictions 94 about the current spot market. The second is a earning mode that 95 regresses prices against market indicators. This earning mode can 96 be used to make absoute price predictions, or to predict residuas 97 for the inear interpoation mode. We show that combining the 98 inear interpoation mode for current market prices with residua 99 predictions improves overa performance, particuary for ong- 100 term predictions. 101 We cose with discussion of the performance of both prediction 102 agorithms based on the resuts of TAC/SCM tournaments and the 103 inaugura TAC/SCM Prediction Chaenge. Deep Maize has been a 104 perennia finaist in the primary competition, and paced first in 105 the Prediction Chaenge. Our forecasting methods demonstrate 106 strong performance on prediction error measures and contribute 107 to strong overa performance in the context of decision-making Reated work State Estimation Long-term Production Schedue Factory Schedue Customer Market Prediction PC Vaues Customer Saes Market Predictions High-eve Decisions Low-eve Decisions Fig. 1. Overview of Deep Maize s decision process on each TAC/SCM day. 109 Forecasting is studied across many discipines, and many ap- 110 proaches have been deveoped for making predictions based on 111 historica data. We do not attempt to survey a of the various 112 methods here, but refer the reader to one of many introductory 113 texts that describe exponentia smoothing, GARCH, ARIMA, spec- 114 tra anaysis, neura networks, Kaman fiters, and other popuar 115 methods (Brockwe and Davis, 1996; Chatfied, 2001; Hastie 116 et a., 2001). Pindyck and Rubinfed (1998) is a good introduction 117 to many forecasting methods used in the context of economic pre- 118 dictions and modeing. The proiferation of techniques is in part an 119 indication that the best method for any particuar probem may de- 120 pend on many factors, incuding the type and quantity of informa- 121 tion avaiabe and the properties of the domain. Many appications focus on comparing methods to identify the most effective approach for a particuar probem (for exampe, see the appication of forecasting to retai saes by Aon et a. or the appication to predict eectricity prices by Nogaes et a. (2002)). Our approach for price prediction focuses on combining predictions made using difference sources of information and methods. Various methods have been deveoped for combining predictions, often motivated by the probem of combining the opinions of experts (Bates and Granger, 1969; Cemen and Winker, 1999). Recenty, predictions markets have become a popuar method for aggregating information (Hanson, 2003). Our approach shares some features with these markets, incuding the use of ogarithmic scoring rues to evauate the quaity of predictions. Price forecasting is a very important eement of decision-making in a wide variety of market settings, incuding financia markets (Kaufman, 1998) and energy markets (Nogaes et a., 2002). The rise of internet auctions has ed to an increasing interest in computationa methods for price forecasting in auction environments (e.g., Ghani, 2005). Brooks et a. (2002) discusses many issues reated to earning price modes in the context of onine markets. Price prediction methods have aso been used as the basis for successfu bidding strategies in many specific types of auctions, incuding simutaneous ascending auctions (Osepayshvii et a., 2005). It is not surprising then that price prediction has payed an important roe in the design of agents for the trading agent competition since its inception. For the TAC trave game (Weman et a., 2007), a crucia factor in agent performance was the abiity to predict the cosing price of auctions for hote rooms. Participants expored many approaches for making these predictions, spanning statistica anaysis, machine earning, and economic modeing (Cheng et a., 2005; Stone et a., 2003; Weman et a., 2004). Post-tournament anaysis reveaed that the common thread among the most effective methods was not the particuar technique appied, but rather the information taken into account by the predictor (Weman et a., 2004); specificay, the best predictors accounted for the initia auction prices. Most agents for the TAC/SCM game aso empoy some form of price prediction, but vary significanty both in what quantities are predicted and the methods used for making predictions. Many agents ony estimate prices for the current simuation day, and do not attempt to forecast changes in prices over time (Benisch et a., 2004, 2006; Burke et a., 2006; He et a., 2006). One of the most successfu agents in the tournament, TacTex, has appied a variety of sophisticated approaches for price prediction. The specific methods have evoved in successive tournaments, but generay rey on a combination of statistica machine earning appied to historica data and onine adaptation (Pardoe and Stone, 2005, 2006, 2007a,b). The most recent version of TacTex specificay uses machine earning to predict both current and future prices. The Minne- TAC team has aso devoted significant effort to deveoping prediction methods for the TAC/SCM domain (Ketter, 2007; Ketter et a., 2007). Their approach is somewhat unique in that it focus on predicting changes in the economic regimes, which refect oversuppy, under-suppy, or other characteristic market conditions. Our use of market state is simiar in spirit to the idea of an economic regime, but we do not distinguish among a discrete set of possibe regimes. The first version of Deep Maize used an anaytic soution based on competitive equiibrium to predict market prices (Kiekintved et a., 2004). This approach became infeasibe when the game rues were modified due to the greater compexity of the suppier pricing mode. Another unique approach was used by RedAgent, the winner of the first TAC/SCM competition (Keer et a., 2004). This agent used an interna market to coordinate decisions and make predictions; to our knowedge no current agents use this approach

3 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx 3 CPU Motherboard Memory HardDisk Pinte IMD Basus Macrostar Mec Queenmax Watergate Mintor component RFQ suppier offer offer acceptance The TAC suppy chain game 189 Agents in TAC/SCM pay the roe of persona computer (PC) 190 manufacturers (Coins et a., 2007; Eriksson et a., 2006). 1 The six 191 automated agents compete to maximize their profits on a simuated 192 suppy chain, shown in Fig. 2. Agents assembe 16 different types of 193 PCs for sae, defined by compatibe combinations of CPU, mother- 194 board, hard disk, and memory components. Each day agents negoti- 195 ate with suppiers to purchase components, negotiate with 196 customers to se finished products, and create manufacturing and 197 shipping schedues for the next day. Each agent has an identica fac- 198 tory with imited capacity for production. Each game has 220 simu- 199 ated days, each of which takes 15 s of rea time The persona computer market 201 The 16 PC types are separated into three market segments rep- 202 resenting high-, mid-, and ow-grade PCs. The average demand e- 203 ve for each segment varies separatey according to a random wak 204 with a stochastic trend. Each day, the customer demand process 205 generates a set of RFQs (request-for-quotes) representing the 206 aggregate demand. Each customer request is vaid for a singe 207 day and specifies a PC type, due date, reserve price, and penaty 208 for ate deivery. The negotiations for these requests use a simuta- 209 neous seaed-bid auction mechanism. Agents submit price offers 210 for a subset of the RFQs. The owest bidder for each request is 211 awarded the order, and must deiver the product by the due date 212 or pay the penaty specified in the contract. Due to the configura- 213 tion of the market, there are strong interactions among auctions 214 both within and across days The component market Agent 1 Agent 2 Agent 3 Agent 4 Agent 5 Agent 6 PCRFQ Customer 216 To procure components to buid PCs, agents must negotiate 217 with eight suppiers whose behavior is defined in the game speci- 218 fication. Each suppier ses two distinct grades of a singe compo- 219 nent type. CPU components have a unique suppier, whereas a 220 other types are provided by two suppiers. Suppiers have a imited 221 daiy production capacity for each component they se. These 222 capacities vary according to a mean-reverting random wak, and 223 are not directy observabe by the manufacturers. 224 Manufacturers negotiate with suppiers via an RFQ mechanism. 225 Each day, a manufacturer may send up to five RFQs per component bid Fig. 2. TAC/SCM suppy chain configuration. to each suppier specifying type, due date, quantity, and reserve price. Any future day may be specified as the due date. The suppiers respond the foowing day with offers specifying a proposed quantity, price, and deivery date. 2 Suppiers may opt to accept a subset of the offers by responding with orders. Suppiers determine offer prices based on the ratio of their committed and avaiabe production capacity. This pricing function accounts for inventory, previous commitments, future capacity, and the demand represented in the current set of customer RFQs. In genera, component prices rise as demand rises and suppiers become more constrained. In some cases, the suppier may be capacity constrained and unabe to offer components at any price. Prices depend on both the requested due date and the day the request is made. Price predictions must be made for this entire space of possibiities to support important decisions about the optima time to make component purchases Observabe market data Agents may utiize severa compementary forms of evidence to make predictions about market prices. Each provides unique insights into current activity and how the market may evove. The first category of evidence comprises observations of the markets during a game instance. This information is the most timey and specific to the current situation, but is of reativey ow fideity. More detaied observations of market activity are avaiabe in game ogs after each game instance has competed. 3 Logs from previous rounds are pubicy avaiabe, and additiona data may be generated in private simuations. During a game, agents receive daiy observations from participating in the PC and component markets. In the PC market, agents observe the set of requests and whether or not each bid resuted in an order. They aso receive a daiy report for each PC type isting the maximum and minimum seing price from the previous day. The detais of individua market transactions and opponents bids are not reveaed. In the component market, agents receive price quotes for each request they send, some of which may be price probes. Neither the suppier s state (inventory, capacity, and commitments) nor other agents requests are reveaed. Every 20 simuation days, manufacturers receive a report containing aggregate market data for the preceding 20-day period. This data incudes the mean seing price and quantity sod for each PC type, and the mean seing price, quantity ordered, and quantity shipped for component type. The report aso incudes the mean production capacity for each suppy ine. The importance of this evidence is that it is specific to the current market conditions and current set of manufacturing agents. However, these observations say reativey itte about how market conditions are ikey to evove, especiay if simiar conditions have not been observed earier in the game. The historica ogs revea the entire transaction history in each market (incuding a bids, offers, and orders), as we as suppier capacities and other state attributes hidden during a game instance. This precise information is potentiay very usefu for predicting the evoution of market prices over time. However, conditions in the current game are unikey to be identica to conditions in historica games. A particuary difficut issue is that the poo of competing agents is constanty changing as teams update their agents, so historica data may not be representative of the current strategic environment. Seecting reevant historica data is an important chaenge. 1 We present an overview of the game rues here. The fu rue specifications for each competition, tournament resuts, and genera TAC information are avaiabe at 2 Suppiers may offer either a partia quantity or a ater deivery if they are unabe to meet the initia request due to capacity constraints. 3 A rue change in 2007 expicity disaows agents from coecting and utiizing these ogs during a tournament round, but ogs from previous rounds or simuations may sti be used

4 4 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx Tabe 1 Four categories of predictions made in the Prediction Chaenge. Prediction Current PC prices Future PC prices Current component prices Future component prices Description The Prediction Chaenge 285 The 2007 TAC event introduced two chaenge competitions to 286 compement the origina TAC/SCM scenario. In the TAC/SCM Pre- 287 diction Chaenge (Pardoe, 2007), agents compete to make the most 288 accurate price predictions in four categories: short- and ong-term, 289 for both component and PC markets (see Tabe 1). The chaenge 290 isoates the prediction component of the game by having partici- 291 pants make predictions on behaf of the same designated agent, 292 PAgent. Simuations incuding a PAgent are run before the cha- 293 enge to generate og fies. During the chaenge, the og fies are 294 used to simuate the experience of paying the game for the predic- 295 tors by providing the exact sequence of messages observed by PA- 296 gent during the game. The predictions made based on this 297 information by competitors in the chaenge do not affect the 298 behavior of the PAgent. This simuation framework aows a pre- 299 dictors to make an identica set of predictions, and aso aows pre- 300 dictions to be made many times for the same set of game instances. 301 The predictions are scored against actua prices using the root 302 mean squared (RMS) error, normaizing by component and PC base 303 prices. 304 Each round of the chaenge used three sets of 16 game in- 305 stances, with different agents competing with PAgent on the sup- 306 py chain. The agents comprising the competitive environment 307 were seected from those in the pubic agent repository. 4 Since 308 tests can be run independenty and repeated, the software frame- 309 work for the chaenge aso provides a usefu tookit for conducting 310 one s own prediction tests. We took advantage of this in deveoping 311 and testing new prediction methods for the component market in 312 preparation for the 2007 TAC/SCM tournament and Prediction 313 Chaenge Methods for PC market predictions Lowest price offered by manufacturers for each current RFQ Median owest price offered for each PC type in 20 days Offer price for each component RFQ sent on current day Offer price for each component RFQ sent in 20 days 315 Each day, Deep Maize estimates the underying demand state in 316 the customer market and creates predictions of the price curves it 317 faces. 5 For the current simuation day, the set of requests is known. 318 Estimating the price curve in this case is equivaent to estimating 319 the probabiity that a given bid on an RFQ wi resut in an order, 320 denoted PrðwinjbidÞ. To predict the price curve for future simua- 321 tion days, it is aso necessary to predict the set of request that wi 322 be generated. We combine predictions about the set of requests 323 and the distribution of seing prices for each request (i.e., 324 PrðwinjbidÞ) to estimate the effective demand curve for each future 325 day. The effective demand curve represents the expected margina 326 seing prices for each additiona PC sod. The agent uses these fore- 327 cast demand curves to make customer bidding decisions and create a projected manufacturing schedue. We begin by discussion of our methods for estimating a key eement of the game state, and proceed with a discussion of our methods for predicting customer market prices Customer demand projection The customer demand processes that determine the number of 333 RFQs to generate exert a great infuence on market prices in the 334 SCM game. There are three stochastic demand processes, corre- 335 sponding to the respective market segments. Each process is cap- 336 tured by two state variabes: a mean Q and trend s. The number 337 of RFQs generated in each market segment on day d is determined 338 by a draw from a poisson distribution with mean Q. The initia de- 339 mand eves for each segment are drawn uniformy from known 340 intervas. The mean demand evoves according to the daiy update 341 rue Q dþ1 ¼ minðq max ; maxðq min ; s d Q d ÞÞ: ð1þ 345 Both the minimum and maximum vaues of Q are given for each market segment in the specification. The trend s is initiay neutra, s 0 ¼ 1, and is updated by the stochastic perturbation U½ 0:01; 0:01Š: 350 s dþ1 ¼ maxð0:95; minð1=0:95; s d þ ÞÞ: ð2þ 352 ying demand process using a Bayesian mode, iustrated in Fig When the demand Q reaches a maximum or minimum vaue, the 353 trend resets to the neutra eve. 354 The state Q and s are hidden from the manufacturers, but can be 355 estimated based on the observed demand. We estimate the under Note that the updated demand, Q dþ1, is a deterministic function of 358 Q d and s d, as specified in Eq. (1). This deterministic reationship sim- 359 pifies the update of our beiefs about the demand state given a new 360 observation. Let PrðQ d ; s d Þ denote the probabiity distribution over 361 demand states given our observations through day d. We can incor- 362 porate a new observation Q b dþ1 as foows: PrðQ d ; s d jq b dþ1 Þ/PrðQ b dþ1 jq d ; s d ÞPrðQ d ; s d Þ: ð3þ 366 Eq. (3) expoits the fact that the observation at d þ 1 is conditionay independent of prior observations given the demand state ðq d ; s d Þ. We can eaborate the first term of the RHS using the demand update (1): PrðQ b dþ1 jq d ; s d Þ¼PoissonðminðQ max ; maxðq min ; s d Q d ÞÞÞ: 372 After cacuating the RHS of Eq. (3) for a possibe demand states, we normaize to recover the updated probabiity distribution. We can then project forward our beiefs over ðq d ; s d Þ using the demand and trend update Eqs. (1) and (2), to obtain a probabiity distribution over ðq dþ1 ; s dþ1 Þ. A sequence of updates by Eq. (3) represents an exact posterior distribution over demand state given a series of observations. To maintain a finite encoding of this joint distribution, we consider ony integer vaues for demand and divide the trend range into T discrete vaues, eveny spaced (Deep Maize uses T ¼ 9 for tournament pay). Given a distribution over demand states, we can project this distribution forward to estimate future demand Learning prices from historica data The TAC agent repository stores binary impementations of many agents from past TAC tournaments, reeased by their deveopers. 5 The methods described in this section were deveoped for the 2005 TAC/SCM tournament, and used with minor updates and new data sets in the2006 competition. The version used in the 2007 tournament and Prediction Chaenge were identica to the 2006 version (incuding the data sets). We use a k-nearest-neighbors (k-nn) agorithm to induce the reationship between indicators of market conditions (features) and ikey distributions of current and future prices. The idea is 6 We have reeased source code for this estimation method. It is avaiabe in the TAC agent repository

5 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx 5 τ d Q d τ d+1 Q d simpe: given a set of games, find situations that most cosey 390 resembe the current situation and predict based on the observed 391 outcomes. When prices are not changing much from day to day, 392 earning prices from historica data has itte additiona benefit 393 over simpy estimating the prices in the current game instance. 394 However, when prices are changing, historica data may hep the 395 agent to predict how the prices wi move based on features of 396 the current situation. 397 For each PC type and day, we define the state using a vector of 398 features F ¼ðf 0 ;...; f n Þ. 7 The difference between two states f 1 and 399 f 2 is defined by weighted Eucidean distance: vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ux n d ¼ t w i ðf 1 i f 2 i Þ 2 ; 401 i¼0 402 where w i is a weight associated with each feature. Additionay, we 403 bound the maximum distance for any individua feature, so d ¼1if 404 f 1 i f 2 i is greater than the bound. To predict the prices during a 405 game, the agent first computes the current state features. It then 406 finds the k state vectors in a database of historica game instances 407 that are most simiar to the current state, using d as the metric. 408 To ensure diversity in the data set, we seect at most one of the k 409 vectors from each historica game instance. Associated with each 410 state in the historica database are the observed price distributions 411 in that game instance. We average these for the k observations to 412 predict prices in the current game. Preiminary experiments varying 413 the vaue of k showed the best resuts for vaues between 15 and 20; 414 we use a vaue of 18 for a competitions and experimenta resuts 415 presented beow. 416 The potentia features we experimented with incuded the cur- 417 rent simuation day, statistics about current prices, and estimates 418 of underying suppy, demand, and inventory conditions. Many of 419 these were motivated by anaysis showing that reated measures 420 were strongy correated with customer market prices in previous 421 competitions (Kiekintved et a., 2005). We tested a range of possi- 422 be features, weights, and bounds in preiminary experiments, and 423 seected those which had the owest prediction error after exten- 424 sive but ad hoc manua exporation. A of the foowing resuts 425 use equa weights and bounds. For predicting the current days 426 prices, we seected the foowing features: average high and aver- 427 age ow seing price from the previous five days, estimated mean 428 customer demand (Q), the simuation day, and the inear trend of 429 seing prices from the previous ten days. For predicting future 430 prices, we used the same set of features with the addition of the 431 estimated customer demand trend (s) Data sets 433 We empoy two distinct data sets for making predictions, one 434 based on tournament games and one based on interna simua- 435 tions. The tournament data set contained games from earier 436 rounds in the tournament. As new rounds were payed, we added #Q Fig. 3. Bayesian network mode of demand evoution. the additiona games to the training set. During the 2005 tournament we automaticay added new games within a round to the data set; this was disaowed by the 2007 rue change mentioned above. The second data set consisted of games run using six copies of Deep Maize. We payed a sequence of roughy 100 games, initiay using just the tournament data set to make predictions. After each game, we repaced the odest og in the data set with the new one, so eventuay the data set consisted entirey of games payed by the six Deep Maize agents. The fina 45 games of this sequence were seected as the sef-pay data set. Ideay, as the data sets contain more data from this specific environment, the predictions wi come to refect the true market prices and agents behavior wi improve. In the imit, we can imagine agents converging to a competitive equiibrium, which may be coser to the environment of the fina round than games from ess competitive rounds eary in the tournament. As it happened, our anaysis indicates that predictions based on this data set were not very accurate Representing and estimating price distributions We experiment with two different representations for the distribution of PC prices. One uses absoute price eves, and the other is reative to recent price observations. The advantage of absoute prices is that they transate readiy across games and can easiy represent changes in the overa eve of prices. The reative distribution is centered on moving averages of the high and ow seing price reports for the previous five days. Between these two averages we use 60 uniformy distributed points, above the average of the highs and beow the average of the ows we use 20 points distributed uniformy. The strength of this representation is in its abiity to expoit the fact that prices tend to be stabe over short periods of time in the game (a few simuation days). It effectivey shifts observations from historica games into the reevant range of prices for the current game, which is often sma. When extracting data from historica games we adjust for the number of opponent agents by randomy seecting one agent and removing that agent s bid, if it exists. This accounts for the effects of the agent s own bid on the price; it is effectivey earning the price that woud be expected based on ony the competitors bids. We aso smooth the observed distributions by considering a requests in the range d 10;...; d þ 10. To compute the estimated price distribution we take the average of the distributions stored for the k-nearest-neighbors, as described above. For each price point j, we take the predicted probabiities for this point in each of the historica instances and average them. For reative predictions, these points represent pricing statistics reative to current prices; for absoute, they represent absoute prices. After averaging, we enforce monotonicity in the distribution. Starting from the ow end of the distribution, we take the maximum of the estimated probabiity for the price point and the estimated probabiity of the immediatey preceding price point. To forecast prices i days into the future, we project based on what happened i days into the future in each historica game. The agent requires predictions for a future days, so using this method requires that the historica instances have at east as many days remaining in the game as the current game does. We enforce the tighter constraint that the historica predictions must start on exacty the same simuation day. For reative price predictions, we aso account for changes in the overa price eves over time by modifying the pricing statistics. For each price statistic, we compute the average change over the intervening i days in the historica data. Pricing statistics for the future days are modified by adding the average change. The absoute distributions do not need 7 When initiay computing the vaues of features, we normaize the vaues for each feature by the standard deviation of the feature in the training set

6 6 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx 501 to account for this, since the price points are based on absoute 502 vaues Onine earning 504 We empoy an onine earning procedure that optimizes predic- 505 tions according to a ogarithmic scoring rue. 8 First, we produce 506 baseine predictions P tournament and P sef-pay based on the two data sets 507 described in Section We then construct adjusted predictions of the form 511 aðb P tournament þð1 bþp sef-pay Þþc; ð4þ 512 where b weights the two prediction sources based on their accuracy 513 in the current environment, and the coefficients a and c define an 514 affine transformation aowing us to correct for systematic errors. 515 We determine the vaues of a, b and c using a brute-force search 516 to find the set of vaues (from a discretized range) that minimizes 517 the scoring rue. The possibe transformations are scored using the 518 recent history of bids won and ost, a vauabe but otherwise un- 519 used source of information. Each possibe set of vaues is scored 520 on the measure P ogða bids bidþ where a bid is the magnitude of 521 the difference between the predicted probabiity of winning and 522 the resut of the bid, which is 1 if the bid resuted in a win and otherwise. To prevent very arge changes in the prediction we 524 oosey bound the possibe vaues of a, b and c. This optimization 525 is performed using the actua bids submitted by the agent in recent 526 days, and modifies the distribution ony within the range of prices 527 where bids are made. This typicay eaves the extremes of the dis- 528 tribution (very high or ow probabiity bids) unchanged, since 529 there is itte bidding activity at these extremes Evauation of PC market predictions 531 We evauate our forecasting techniques using both prediction 532 error and suppy chain performance. We evauate severa varia- 533 tions of the predictor that empoy subsets of the functionaity de- 534 scribed in Section 4 to assess the contributions of each. As a 535 baseine we use a heuristic predictor described by Pardoe and 536 Stone (2005). The baseine predictor uses ony information con- 537 tained in the current price eve, and reies competey on oca 538 price stabiity for predictive power. This is simiar to what our pre- 539 dictor might ook ike using the reative representation without 540 any historica data or onine transformations. This predictor pro- 541 duces surprisingy accurate predictions despite its simpicity. The 542 reason is that prices in the TAC SCM game tend to be quite stabe 543 for ong periods, especiay if the start and end game conditions 544 are excuded. 9 One of the primary reasons we use this as a baseine 545 is that we beieve it is refective of the defaut heuristic methods 546 used by many of the agents to make decisions (especiay in the eary 547 years of the competition). In particuar, it makes use of ony oca 548 information about prices in the current game, and does not attempt 549 to project changes in price eves over time. 550 The baseine method predicts the distribution of PC prices using 551 a weighted average of uniform densities between the ow and high 552 prices from the previous five days. 10 We project this into the future 553 by assuming that the distribution of prices does not change and adjusting for the predicted change in the number of RFQs generated, as given by the mode of customer demand described in Section 4.1. We present resuts for the baseine and four variations of our k- NN-based predictor. These variations use either the reative or absoute representation, and either appy onine affine transformations or not. A resuts use both the tournament and sef-pay data sets, which contain 95 and 45 game instances respectivey. Regardess of whether overa affine transformations are appied we use the onine scoring method to weight the predictions from these two sources. At the end of this section we briefy discuss the effects of varying the data sets. A error resuts are derived from the 14 cean finas games from the 2005 tournament Prediction error We discuss two different types of prediction error. The first is for predictions made about the probabiity of winning the current day s auctions, and the second is for predictions about the effective demand curve on future days. As we discuss the prediction error resuts, it is important to keep in mind that there is a substantia amount of uncertainty inherent in the domain mode; we do not expect any prediction method to achieve anywhere near perfect predictions. To demonstrate this and provide some caibration, we consider the process that generates the overa eve of customer demand in the SCM game. The eve of customer demand (number of requests generated each day) is one of the most important factors infuencing market prices. Reca that we use a Bayesian mode to track the mean customer demand and trend; this mode is the theoreticay optima predictor of the process. We ran tests comparing this predictor to a naive predictor that predicts that the mean demand on a future days wi remain exacty the demand observed today. The optima Bayesian mode improved on the naive mode, but the difference was reativey sma. The naive mode had an RMS error of approximatey 35, and the Bayesian mode achieved an RMS error of approximatey 30 a reduction on the order of 15%. Whereas this may seem a reativey sma improvement in prediction quaity, these sma improvements can transate into substantia differences in overa agent performance Current day prediction error Predicting the conditiona distribution PrðwinjbidÞ for the current day s RFQ set is an important specia case because these estimates are used for making bidding decisions. It is aso unique because it is the ony day for which we have the actua set of customer RFQs avaiabe; for future days, there is substantia additiona uncertainty since the quantities, PC types, due dates, penaties, and reserve prices are a unknown. We measure RMS error over a range of possibe prices (i.e., potentia bid vaues). The error may vary over the distribution, and this aows us to present a more detaied picture of the errors each predictor makes. The range of vaues we consider is reative to the recent high and ow seing prices in the game. We distributed 125 price points over the range from 10% beow the average ow seing price from the previous five simuation days, to 10% above the average of the high seing price over the same window. Bids towards the high and ow ends of this distribution are very rare, since they are either very high or ow reative to the market. Deep Maize s bids are centered around price point 62, with a arge majority faing between price points 50 and The ogarithmic scoring rue is stricty proper, in that the optima score is obtained ony for the true probabiity. Scoring according to rues that are not proper (incuding RMS error) can ead to biased estimates. 9 A inear regression on the average seing prices from one day to the next resuts in an R 2 vaue on the order of 0.99, so current prices are extremey predictive of prices in the near future. 10 We use weights of 0.3 for the most recent two days, 0.2 for the midde day, and 0.1 for the odest two days. 11 In two games (3718 and 4254) the University of Michigan network ost connectivity and this team s agent was unabe to connect to the game server for a arge part of the game. This distorts the resuts for uninteresting reasons, so we omit these games

7 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx 7 RMS Error Reative with onine earning Reative Absoute with onine earning Absoute Baseine Heuristic Optima Distribution 611 To compute the error measure, we process each RFQ separatey. 612 For each of the 125 price points, we measure the error as the dif- 613 ference between the predicted probabiity and the actua outcome. 614 The outcome is either 1 if the owest bid was higher than the price 615 point, or 0 if the owest bid was ower than the price point (corre- 616 sponding to whether a bid at that price woud have won or ost the 617 auction). A subtety of measuring error based on individua RFQs in 618 this way is that any probabiistic prediction that is not either 1 or wi, by definition, show error on this measure. To provide a bench- 620 mark for this effect, we aso pot the error for the optima distribu- 621 tion. This is a distribution derived from the actua prices for the PC 622 type and day being measured, but without RFQ-specific features. 623 Fig. 4 shows the resuts of this error measure computed for 624 each of the five candidate predictors, as we as the optima distri- 625 bution. We compute errors averaged over a games from the tournament fina round, restricted to predictions made for 627 the midde part of the game from simuation day 20 to 200. This 628 restriction is primariy because the baseine predictor is not de- 629 signed for border cases, such as when no recent price observa- 630 tions are avaiabe. The error of the baseine is artificiay high 631 in these cases, which woud distort the overa error resuts in fa- 632 vor of our methods. 633 Of the candidate predictors, the reative predictor with onine 634 earning has the owest overa prediction error. A four of our pre- 635 dictors have ower error than the baseine in the center part of the 636 distribution. The two absoute predictors have greater error when 637 making predictions for very high or very ow prices. Onine earn- 638 ing reduces prediction error across the entire distribution for both 639 the absoute and reative representations. The reative predictors 640 generay perform better than the absoute predictors. However, 641 the absoute predictor with onine earning has ower error than 642 the reative predictor without onine earning in the center of the 643 price distribution. This is where onine earning shoud have the 644 greatest impact, since more bids are paced in this region and it 645 has more data to make updates. The differences in error between 646 our predictors and the baseine are quite substantia. In the center 647 of the distribution, the difference between the baseine method 648 and the optima method is 0.3, and the difference between the 649 optima and the reative predictor with onine earning is 0.2, 650 an improvement of roughy 30%. 651 Conceptuay, errors towards the center of the distribution are 652 ikey to be much more important because bids are centered there. Current Day Prediction Error Reative Price Fig. 4. RMS prediction error for current day predictions in the TAC/SCM fina round games. The points on the horizonta axis represent a uniform distribution of points over a range defined by the recent high and ow seing prices in the market. However, is not cear how errors shoud be weighted, since bid densities may vary in different situations (and certainy for different agents). These predictions are aso used for making non-bidding decisions, which may use the information in different ways. Using suppy chain performance (as we do in Section 5.2 beow) is one way to resove this diemma, since we can test the predictors as they are used by a rea decision-making procedure. ven bid eve. Inverting this function yieds the desired prices Future forecasting error 660 A distinctive feature of Deep Maize is that it expicity forecasts 661 changes in future market conditions. For future days, the agent 662 transates its predictions into an effective demand curve for use 663 in production scheduing. This demand curve specifies the price 664 the agent woud need to bid to win a particuar quantity of PCs, 665 on average. These prices are given in an ordered ist, with the first 666 price representing the bid to win one PC, the second price the bid 667 to win two PCs, and so on. The predictor gives the probabiity of 668 winning a bid Prðwin;jbidÞ, which we combine with the predicted 669 overa demand to determine the expected quantity won for any gi We measure the error for this effective demand curve, which 672 combines errors from the price predictions and demand predic- 673 tions. For each simuation day, we compute the actua demand 674 curve by sorting the observed prices for each PC. We then measure 675 the RMS error between the prices in this ist and the predicted de- 676 mand curve, summing the errors for each PC in the ist up to a max- 677 imum of 200 vaues per day. 13 Fig. 5 compares the five predictors on 678 this aggregate measure of future forecasting error out to a prediction 679 horizon of 50 days. 680 The pattern of resuts is simiar to the resuts for the current 681 day. The two reative predictors score the best over a horizons, 682 foowed by the baseine and then the two absoute predictors. 683 The onine affine transformations are beneficia regardess of the 684 representation used. We note that this measure averages error 685 over the distribution (each data point represents a compression 686 of the fu distribution shown in Fig. 4). We present information 687 in this way partiay for easy visuaization, but aso because future Our functiona form aows simpe inversion of this function, but this inversion can aso be computed using a simpe binary search to find the necessary bid eve. 13 An agent typicay projects seing 100 or fewer PCs of any type on a given day, so this is roughy twice the number of vaues the agent actuay uses to make decisions

8 8 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx RMS Error Reative with onine earning Reative Absoute with onine earning Absoute Baseine Heuristic 689 panning may be more dependent on errors over the entire distri- 690 bution than bidding decisions. However, we shoud note that the 691 absoute predictor shows higher overa error on this measure pri- 692 mariy because of errors at the extreme parts of the distribution, as 693 in the short-term error resuts Suppy chain performance Future Prediction Error 695 Since our forecasting methods were specificay designed to 696 support decision-making in a fuy impemented and automated 697 agent, we have the opportunity to assess how forecasts impact 698 agent performance. This is particuary interesting because of the 699 compex interactions between prediction error and decisions. As 700 discussed above, it can be difficut to determine how to weight dif- 701 ferent types of prediction errors. Another issue with onine predic- 702 tions is that the decisions made affect the information avaiabe for 703 making future predictions. 704 We ran simuations using a profie of two MinneTAC-05 agents, 705 two TacTex-05 agents, one static version of Deep Maize, and one 706 modified version. MinneTAC-05 and TacTex-05 were finaists in that were among the first to reease agent binaries. We ran 708 four sets of games with a minimum of 35 sampes. The static ver- 709 sion of Deep Maize used the reative affine predictor. We present re- 710 suts as differences between the static and variabe versions of 711 Deep Maize. This pairing reduces variance but introduces bias to 712 the extent that varying the sixth agent affects the performance of 713 the static version. Another important caveat is that these resuts 714 are for a singe profie of strategies. More systematic approaches 715 to seecting test environments may be better justified on game- 716 theoretic grounds (Jordan et a., 2007), but we make do with a sin- 717 ge convenient abeit ad hoc profie here. 718 In addition to scores we give three measures of customer saes 719 performance: average seing price (ASP), seing efficiency, and 720 timing efficiency. Seing efficiency is the fraction of achieved 721 revenue to the maximum possibe revenue for identica daiy saes, 722 given perfect information about opponents bids and the option to 723 partiay fi orders. Timing efficiency measures how we the agent 724 distributed saes over time. Let the t-optima same day seing price 725 be the highest ASP the agent coud achieve by seing each PC at 726 most t days after it was actuay deivered and no earier than it 727 was actuay produced. PCs are abeed according to a FIFO queuing 728 poicy. Timing efficiency is the ratio of the t-optima same day se- 729 ing price to the seing efficiency. This factors out the effects of bid- 730 ding poicy and eaves the effects of deciding which days to se on. 731 The appea of these additiona metrics is that they aow us to com- 732 pare a specific eement of decision performance against an optima 733 vaue, though we must remain cognizant of the additiona con- 734 straints when interpreting the resuts Prediction Horizon (days) Fig. 5. RMS error between predicted and actua effective demand curves out to a horizon of 50 days, computed on the 2005 finas. Tabe 2 Performance measures in simuated games. The numbers represent the difference between two versions of Deep Maize using the reative affine predictor and the indicated predictor. Predictor Score ASP Seing efficiency 20-Day timing Reative, no affine 2.3 M Absoute, affine 0.3 M Absoute, no affine 1.2 M Baseine 9.1 M The simuation resuts are in Tabe 2. The most striking point is that a of our k-nn-based predictors comfortaby outperform the same agent using the baseine predictor. To caibrate, the difference between the top and bottom scores in the 2005 finas was 6.5 M, ess than the advantage of any of our predictors over the baseine. Ceary, effective prediction can have a sizabe impact on agent performance when the decision-making architecture is capabe of expoiting the information. The affine transformations again showed benefits for both representations. The absoute predictors scored better than the reative predictors, abeit by fairy sma margins. This is somewhat unexpected, given the error resuts. Earier resuts using a smaer data set for the predictors actuay showed a sight advantage for the reative predictor. We expect a arger data set to benefit the absoute representation disproportionatey because some of the features are based on current prices. With more data more simiar instances are avaiabe, and the absoute representation shoud predict more ike the reative representation. We conjecture that the reative representation may have greater advantages for smaer data sets, but have not verified this experimentay. Another ikey expanation for this discrepancy is that the absoute representation better refects the actua uncertainty present because it is abe to spread predictions over a broader range of prices. This may cause the agent to deay making purchasing decisions unti more information is avaiabe, potentiay improving performance. In genera, the differences between the agent variations on the saes metrics are reativey sma, but we note some interesting features. The timing measure correates quite we with the overa scores. The baseine predictor performs very poory on this measure as we as having a ow ASP. The seing efficiency numbers do not show a strong pattern, and the ASP numbers for the k-nn variations are aso ambiguous. This suggest that one of the more consistent benefits of better forecasts for Deep Maize is the usefuness of these forecasts in timing saes activity Tournament and sef-pay data We aso generated error scores for the four predictor variants using ony the tournament data set and ony the sef-pay data set (a tota of 12 different variations). We do not present fu data here, but can describe the pattern of resuts fairy easiy. Using ony the tournament data resuted in amost exacty equivaent performance to using the combined data sets. The resuts using ony the sef-pay data were amost aways worse than either the tournament ony or combined settings. One concusion we draw from this is that the process we used to generate the sef-pay data did not produce the most reevant experience for making predictions. Another is that the onine scoring procedure was effective at detecting the source of more saient information to use in the combined predictor. It does this by pacing a very high vaue on the variabe b in Eq. (4), weighting the tournament data set much more heaviy than the sef-pay data set. As the agent pays a game, we often observe vaues set to the maximum possibe vaue, within the range of aowabe vaues

9 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx 9 Price Reative to Base Price Current Day Methods for component market predictions 788 As for the PC market, Deep Maize uses price forecasts to deter- 789 mine how to procure components in the suppy market. Compo- 790 nent price forecasts take the form p ðs;cþ i;j ðqþ, denoting the predicted 791 offer price for a quantity q of component type c, requested from 792 suppier s on day i, with due date j. The price predictions for each 793 possibe pair of request date and due date form a surface, as de- 794 picted in Fig. 6. These prediction are made each day for each sup- 795 pier-component pair ðs; cþ. This contrasts with the PC market, 796 where we mode a price curve with a singe time parameter (the 797 request date) Each day, manufacturers receive offers from suppiers in re- 799 sponse to the RFQs sent the previous day. Information from these 800 offers may be used as input to price predictions. Note that it is gen- 801 eray not possibe to get a direct price quote for each future date, 802 due to the imitation of five RFQs per day for each suppier and 803 component combination. Thus, there is significant uncertainty 804 regarding even the current price curve. 805 We consider three casses of prediction methods for the compo- 806 nent market. The first creates an estimate of the current price curve 807 using the observed spot price quotes and inear interpoation. This 808 method does not attempt to predict changes in prices over time. 809 The second approach augments the first by using market indicators 810 to predict residua changes from the estimated price curve over 811 time. This method expoits oca price stabiity in the short term 812 by using the inear interpoation mode as a base, but sti aows 813 for modeing price variabiity over time. The fina method appies 814 regression to predict prices directy based on market indicators. 815 This method aows for the greatest fexibiity in predicting price 816 changes Linear interpoation price predictor 818 As noted in many accounts of TAC/SCM agents, market prices in 819 the game tend to exhibit oca price stabiity over short horizons. 820 The degree of voatiity may vary depending on the particuar 821 agents paying and random variations in the underying game 822 state. However, there are structura reasons to expect such stabi- 823 ity. One is that the underying suppier capacity and customer de- 824 mand eves change over time, but rarey exhibit drastic changes 825 over short time periods. In addition, the outstanding commitments 826 and inventory of suppiers and manufacturers tend to exert a 827 damping effect on component price changes Reative Due Date Fig. 6. Sampe price predictions p i;iþd ð1þ for a singe component where i is the current day and d is the reative due date Day Due Deep Maize estimates the current market state based on price quotes from recent days. We empoy inear interpoation to estimate prices for due dates with no recent observations. Fig. 7 shows an exampe price curve estimated using this method, which we term the inear interpoation price predictor (LIPP). Due to oca price stabiity, we expect LIPP to be a reasonaby good predictor for prices in the near future. Empirica resuts beow support this proposition Reative component price predictor In anayzing the performance of LIPP, we note a tendency for predictions over the entire price curve to systematicay overestimate or underestimate prices. There are severa pausibe reasons for this. Reca that suppiers current capacity varies according to a random wak during the game. Suppiers construct prices in part based on projecting their current production capacity into the future to determine their avaiabe capacity. Since avaiabe capacity is used in pricing for a request dates, fuctuations in capacity cause prices for a of these dates to rise or fa in unison. This pattern coud aso resut from manufacturing agents increasing or decreasing order quantities, but maintaining approximatey the same timing in their purchase schedues. This might resut, for instance, from rising or faing customer demand eves. LIPP does not take into account any information beyond recent price quotes that might hep to predict these sorts of price fuctuations. We introduce the reative component price predictor (RCPP) as a way of expoiting these additiona forms of information. RCPP predicts the residua error of LIPP by appying regression to technica attributes summarizing the observabe market state. A ist of the attributes we used is given in Tabe 3. The predictions made by RCPP are formed by cacuating the summation of the residua errors and LIPP predictions. The intuition for this approach is that the RCPP agorithm identifies game states that are ikey to resut in predictabe price shifts, eading to inaccurate LIPP estimates. The residuas modify the current price estimates to account for these shifts. This approach expoits the short-term accuracy of the inear price predictor, whie expoiting historica data and market indicators to adapt future forecasts. We note that there is a parae between this approach and the reative representation used in predicting prices in the PC market. Price Fig. 7. A sampe inear interpoation of recent price quotes Absoute component price predictor We have found empiricay that the due dates for PC requests have minor effects on the price, so we do not mode these directy. The fina cass of predictors we consider is the absoute component price predictor (ACPP). ACPP appies regression to directy earn prices from the technica indicators. Unike RCPP, this approach does not bootstrap the short-term accuracy of LIPP, so it

10 10 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx Tabe 3 Attributes considered by RCPP and their RReiefF vaues for current and future days. The first five attributes correspond to parameters of an individua RFQ and the remainder of market state. Attribute Current Future Description Day Current day Sent-day Day the RFQ wi be sent Due-day Day the RFQ wi be due Duration Due-day minus sent-day Quantity RFQ quantity Linear-price LIPP price prediction Nearest-price-point-diff Nearest known price point distance (days from due date) Days-since-data Days since due date had a price quote Market-capacity Last suppier capacity from market report Estimated-capacity Estimate of suppier capacity using market reports and trend Surpus Avaiabe capacity minus agent demand in recent period Aggregate-surpus Avaiabe capacity minus agent demand in tota Mean-demand Mean customer demand for component Mean-ow-demand Mean customer demand for ow-end components Mean-mid-demand Mean customer demand for mid-range components Mean-high-demand Mean customer demand for high-end components 5-Day-demand Customer demand for ast 5 days 20-Day-demand Customer demand for ast 20 days 40-Day-demand Customer demand for ast 40 days 873 is not reative to the estimated current prices. The potentia advan- 874 tage of this approach is that it offers greater fexibiity for modeing 875 price shifts, since it is not constrained by the LIPP prediction. If 876 there are prevaiing ong-term price trends not captured in the 877 RCPP residuas, ACPP may yied more accurate predictions at ong 878 horizons. We note that there is a parae between this approach 879 and the absoute representation used in predicting prices in the 880 PC market Evauation of component market predictions 882 We evauate the reative performance of the three casses of 883 predictors on the basis of prediction error using the Prediction 884 Chaenge framework. We present the resuts of experiments run 885 after the competition using the data set from the fina round of 886 the 2007 Prediction Chaenge. 15 The data consists of a tota of games containing the designated PAgent, divided into three sets of games payed against the same set of opponents. A opponents 889 were seected from the agent repository. The predictors are scored 890 using normaized RMS error on a required component price 891 predictions Setting expiration intervas for LIPP 893 We begin by exporing an important parameter of the LIPP 894 method, which we then fix for subsequent anaysis. Reca that 895 manufacturers are imited in the number of RFQs they may submit 896 each day. We can increase the number of observations avaiabe for 897 estimating the price curve by incuding RFQ responses from previ- 898 ous days, with the caveat that these quotes may represent stae 899 information. We represent this tradeoff using an expiration inter- 900 va that determines how many days of previous observations are 901 used to construct the LIPP estimate. Using a onger interva in- 902 creases the amount of data avaiabe, aong with the risk that the 903 data no onger accuratey refects the current situation. To deter- 904 mine the best setting of the expiration interva, we tested intervas 905 from one to five days. The resuts are shown in Fig The mean accuracy of the LIPP predictions decreases monoton- 907 icay as the size of the ength of the expiration interva is in- 908 creased. This resut impies that the day-to-day price voatiity in RMS Error EX1 EX2 EX3 EX4 EX5 Expiration Interva Fig. 8. RMS error for LIPP with various expiration intervas. TAC/SCM is high enough that the benefit of having data points for additiona due dates is outweighed by the reduced accuracy of the oder data points. It is possibe that adapting this expiration interva dynamicay during a game based on observed price voatiity or other factors coud yied more accurate predictions, but we have not yet fuy expored this possibiity Estimating attribute quaity for regression modes As a preiminary step to regression, we seek to evauate the quaity of various attributes that may be used to predict current and future component prices. Unike cassification probems, there are few measures of attribute quaity avaiabe from the current iterature. For exampe, information gain, j-measure, and v 2 cannot be used for estimating attribute quaity in a regression. Three measures that have been proposed for regression with numerica attributes are mean absoute and mean squared error (Breiman et a., 1984), and RReiefF (Robnik-Šikonja and Kononenko, 1997). Because the error measures assume independence among attributes, they are ess appropriate given strong interdependencies ike those introduced in our anaysis. In contrast, RReiefF has be shown empiricay to perform we in situations where there are strong interdependencies, such as parity probems. A short expanation of the RReiefF vaues is rather eusive (see reated work Robnik- Šikonja and Kononenko, 2003 for a thorough exposition), but we attempt here to give some intuition as to what the vaues represent. 15 Many of these anayses were aso performed before the Chaenge using other test sets; we repeat a of the experiments using this data set for consistency

11 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx RReiefF is an extension of the ReiefF agorithm to regression. In 935 cassification, ReiefF measures an attribute s quaity according to 936 how we a given attribute distinguishes instances that are simiar 937 in the attribute space. The vaue itsef is the approximated differ- 938 ence of two probabiities with respect to a given attribute vaue. 939 The first is the probabiity that a different vaue is observed given 940 near instances from different casses. The second is the probabiity 941 that a different vaue is observed given near instances from the 942 same cass. Because price predictions and, in genera, regressions 943 are continuous, we cannot cacuate these probabiities. RReiefF 944 attempts to sove this probem by introducing a probabiity distri- 945 bution which determines whether two instances are different. The 946 vaues cacuated by RReiefF ie in the range [ 1, 1] with random 947 attributes tending to be sighty negative. At the extremes, a vaue 948 of near one signifies that a different attribute vaue impies a differ- 949 ent regression vaue for nearby instances. A RReiefF vaue greater 950 than zero indicates that the first probabiity subsumes the second 951 and thus the attribute is ikey to discriminate nearby instances to 952 a usefu extent. We adopt this simpe threshod to determine 953 whether attributes are incuded in our regression modes. Our 954 experience is consistent with the empirica findings that RReiefF 955 attribute seection improves our regression modes over the de- 956 faut mode that incudes a avaiabe attributes. 957 A compete ist of a attributes considered for regression ana- 958 ysis as we as the RReiefF vaues for each attribute on both current 959 and future day predictions is given in Tabe 3. The attributes are a 960 mix of RFQ parameters and estimated market state. Note that RRe- 961 ieff vaues often differ significanty with respect to current and fu- 962 ture predictions. In particuar, the tabe suggests that current 963 predictions are best determined by the attribute subset quantity, 964 due-day, duration, and inear-price, a but the ast of which are 965 RFQ parameter attributes. This differs notaby from future predic- 966 tions, for which the majority of attributes are sti of threshod 967 quaity. Component prices vary oosey according to a few market 968 state parameters that are hidden from direct observation. Our 969 attribute set was chosen in an attempt to extract usefu observabe 970 parameters from the market in ieu of the hidden state. For current 971 predictions, it appears that the market state attributes are not very 972 usefu in regression. This is ikey expained by the observation that 973 the interna market state changes graduay and that usefu por- 974 tions of the interna state are aready expressed in the LIPP prices 975 from the prior day. Thus, current prices can best be determined 976 by attributes from the RFQ and LIPP prices, whereas incuding 977 the estimates of market state is not beneficia. This is not to say 978 that estimating interna market state is without predictive power 979 in a cases. As the RRefiefF vaues in the future coumn suggest, 980 the future market state is no onger refected in LIPP prices and 981 incuding these attributes is ikey beneficia Comparing regression methods for RCPP 983 There are many machine earning agorithms that coud be ap- 984 pied to earn the residua errors for RCPP. We tested a variety of 985 regression agorithms from the Weka machine earning package 986 (Witten and Frank, 2005) for this task, incuding inear regression, 987 k-nn, support vector machines, and various decision-tree regres- 988 sions. We generated training data from approximatey 50 game in- 989 stances simuated on a oca computing custer. Each game of 990 training data had one PAgent paying, aong with a coection of 991 other agents taken from the agent repository (the same set of po- 992 tentia opponents used for the Prediction Chaenge). Each game 993 contains approximatey 1800 sampe price predictions for each 994 CPU component and twice as many for non-cpu components. We 995 train separate predictors for each component type, so we have 996 approximatey 90,000 data points for CPU components and ,000 for non-cpu components. We take sampes from the view- point of the PAgent in the training games, so the market indicators are computed with respect to this agent s observations. After some preiminary experimentation, we seected additive regression 16 and reduced-error pruning trees (REP trees) as the most promising candidates. Many of the other agorithms had simiar resuts in terms of prediction error, but required much onger to train or generated very arge modes. For some of the agorithms, overfitting was a severe probem. We tested severa parameter settings for both additive regression and REP trees using the data set from the fina round of the Prediction Chaenge. Additive regression was tested both with defaut settings and with a shrinkage rate of 10%. 17 REP Trees were tested with maximum tree depths of four through seven. A variations earned residuas from LIPP predictions with an expiration interva of one (the best setting from the previous section). Fig. 9 shows the prediction error resuts for these methods on both current and future predictions. The different earning agorithms produced very simiar resuts; each woud have paced first in both component prediction categories. For the actua chaenge, we used the REP tree agorithm with maximum depth five to earn residuas for RCPP, since this agorithm was consistenty accurate for both current and future predictions Comparison of prediction methods In the previous sections, we expored variations of the LIPP and RCPP methods to identify good parameter settings for these agorithms. We now directy compare the performance of the three different casses of methods introduced in Section 6. We use the LIPP method with expiration interva one, the RCPP method earning residuas using REP trees of max depth five, and an ACPP predictor that uses additive regression to make price predictions directy from the technica indicators. 18 Prediction error resuts for a three agorithms on both current and future predictions are given in Fig. 10. A current comparisons in the figure are statisticay significant beyond the eve and a future comparisons in the figure are statisticay significant beyond the 0.05 eve using a standard t- test. For the current price predictions, the performance for LIPP and RCPP is very cose, with both outperforming the absoute predictor by a wide margin. This is strong evidence that the LIPP method effectivey summarizes the current price state, and that this is a very good predictor of prices in the near future due to oca price stabiity. Interestingy, none of the methods we tried for earning residuas were abe to improve LIPP predictions by a significant margin. This suggests that in the near term, most of the information about market conditions is effectivey summarized by the current prices. The benefits of earning prices from historica data using technica indicators is evident in the resuts for predicting future prices. Both ACPP and RCPP perform better than the LIPP method in this case, demonstrating the vaue of representing changes in prices over time. RCPP performs better than ACPP on these future predictions, indicating that the current prices sti have significant predictive vaue, even at a 20-day horizon. Overa, the RCPP method, which combines the price estimates from inear interpoation with earned residuas, offers the best of both words; it retains shortterm accuracy whie significanty improving ong-term forecasts. 16 Additive regression is a meta-earning agorithm wherein each decision stump is fit to the residua of the previous step every iteration. 17 The shrinkage rate considers how much of the trained decision stump is used each iteration. Normay, the shrinkage rate is set to 100%, however, a smaer shrinkage rate induces a smoothing effect and reduces overfitting. 18 Based on the resuts in the previous section, we woud expect a REP-tree version of ACPP to have very simiar performance

12 12 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx Current RMS Error We seected RCPP for use in both the Prediction Chaenge and the 1055 main tournament Suppy chain performance AddReg Shrink REP4 REP5 REP6 REP7 Predictor 1057 In this section, we anayze the effects of component prediction 1058 methods on aggregate agent performance, simiar to the anaysis 1059 performed on the customer prediction effects described in Section We ran simuations using a profie of TacTex-07-Finas, Phant Agent-07, Mertacor-05, MinneTAC-06, and two versions of Deep Maize One version of Deep Maize stayed static and was used as the base ine agent. In this case, the baseine agent was Deep-Maize-07-Finas, 1064 which used RCPP with LIPP5 for its component predictions. The 1065 second Deep Maize agent competing in the simuations used either 1066 LIPP1, ACPP, or RCPP with LIPP1 for its component predictions. To 1067 evauate each regression method, we ran three sets of games, 1068 invoving no ess than 35 games in each set. We report the differ ence in score between the variation of the baseine using the indi cated predictor and the baseine itsef, as we as the p-vaue of the 1071 corresponding paired t-test. We found a pairwise comparisons to 1072 be significanty different at a eve beyond The resuts are pre sented in Tabe We expect that the reative performance of each variation of 1075 Deep Maize wi correspond to the accuracy of the prediction meth od used by the agent. The resuts support this for current price pre dictions, but show ess sensitivity to the accuracy of ong-term 1078 predictions. The predictor which fared the best in the prior section, 1079 RCPP with LIPP1, was an $0.8 M improvement over the baseine 1080 agent, with the ony difference being an expiration interva of Future RMS Error AddReg Shrink REP4 REP5 REP6 REP7 Predictor Fig. 9. Prediction error resuts for various candidate earning agorithms on the data set from the fina round of the Prediction Chaenge. Future corresponds to a 20-day horizon. Current RMS Error ACPP RCPP LIPP Predictor Future RMS Error one versus five in LIPP. Averaged over a arge number of games, this is a substantia difference in a profie of competitive agents. Surprisingy, LIPP1 without RCPP was aso an improvement over the baseine agent. This is interesting because LIPP1 predicts future prices simpy using its current prediction with no adjustment. Thus, Deep Maize was abe to perform quite we without making an effort to adjust for future component prices. This effect is aso evident in the resut for the ACPP version. In the prior section, we noted that athough ACPP performed poory in current predictions, it fared we in future predictions when compared to LIPP. The poor performance in the current prices had drastic consequences for the overa agent score. The effect of the arge errors in current prices caused Deep Maize to procure far fewer components and thus caim ess PC market share, which exacerbated the reative score difference by opening up more profitabe market share to the other agents. A three tests support the proposition that prediction accuracy exerts a noticeabe effect on an agent s performance, especiay when evauating current price predictions ACPP RCPP LIPP Predictor Fig. 10. Prediction errors for the absoute (ACPP), reative (RCPP), and spot (LIPP) predictors on the Prediction Chaenge fina round data set. Future corresponds to a 20-day horizon. Tabe 4 Reative score with respect to the baseine agent in simuated games. The numbers represent the difference between two versions of Deep Maize, one using the indicated predictor and the other using the version payed in the 2007 tournament finas (Deep- Maize-07-Finas). In addition the p-vaue of each paired t-test is shown. Predictor Reative score p-vaue RCPP with LIPP M ACPP M <2.2e-16 LIPP M

13 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx TAC/SCM tournament and prediction chaenge 1101 The TAC/SCM tournament and Prediction Chaenge offer dis tinct opportunities for evauation of our prediction methods in 1103 comparison with aternatives outside our contro. The Prediction 1104 Chaenge faciitates direct comparisons with competing prediction 1105 approaches, isoating this aspect of the game. The genera tourna ment resuts aow us to evauate the genera competency of our 1107 agent reative to the other competitors. Predictions are an impor tant component of Deep Maize s decision-making, so the overa 1109 performance of the agent refects in part the quaity of these pre dictions. An important aspect of both the SCM tournament and 1111 the Prediction Chaenge is that each new game instance may pres ent nove strategic environments not be refected in any historica 1113 data. Thus, this exercise serves as an important indication of the 1114 robustness of our prediction resuts TAC/SCM tournaments 1116 Tournament performance is an important indicator of the over a competence of TAC/SCM agents. Each TAC/SCM tournament 1118 comprises a sequence of rounds: quaifying, seeding, quarter-finas, 1119 semi-finas and finas. The quaifying and seeding rounds span sev era days and are used primariy for deveopment and testing. The 1121 quarter-finas, semi-finas and finas are hed on successive days. In and 2006, 24 agents were seected to begin the quarter-finas, 1123 with haf eiminated in each round. In 2007, 18 agents started the 1124 quarter-finas, with six eiminated in each round. Resuts from the , 2006 and 2007 tournament semi-fina and fina rounds 1126 incuding Deep Maize are presented in Tabes Deep Maize has performed very we in each of the TAC/SCM 1128 tournaments to date. Our agent reached the fina round each year, 1129 pacing fourth in 2005, third in 2006 and third in Deep Maize 1130 aso demonstrated strong performance in the semi-fina rounds, 1131 pacing first in each heat. This record of overa performance pro vides evidence that Deep Maize is effective in a wide variety of com petitive environments. The version of the customer prediction 1134 modue presented here is very simiar to the modues that payed 1135 in a the tournaments. This aspect of the agent was ar gey unmodified between 2005 and 2007, though other eements of 1137 the agent underwent significant revisions In 2005 and 2006, Deep Maize used ony LIPP5 for component 1139 market predictions. RCPP was added to the predictions for the tournament (using a REP Tree with a maximum depth five), 1141 however we sti maintained an expiration interva of five for LIPP Unfortunatey, we did not discover the superior accuracy of ower 1143 expiration intervas unti shorty before the finas, such that we 1144 coud not make the modification and competey test the strategy The training data used for 2007 tournament pay was derived from 1146 approximatey 100 game instances, incuding games from the ate 1147 rounds of the 2006 tournament and additiona simuations using 1148 the best avaiabe agents from the agent repository TAC/SCM 2007 Prediction Chaenge 1150 The introduction of the Prediction Chaenge aows us to di recty compare our prediction approaches to other competitors, 1152 suppementing our own anaysis based on controed experiments The Prediction Chaenge was payed in two rounds, a semi-fina 1154 and a fina. The semi-fina round took pace during the same 1155 time-frame as the TAC/SCM seeding round, before the main tour nament. Like the quaifiers and seeding rounds in the main TAC/ Tabe 5 Scores and rankings from the rounds Deep Maize payed in during the 2005 SCM tournament. A scores are in miions of doars. The adjusted coumn eiminates two game instances (3718 and 4259) where the University of Michigan network ost connectivity during the game, distorting the resuts substantiay. Semi-Finas 2 Finas Finas (adjusted) Deep Maize 3.68 TacTex TacTex TacTex SouthamptonSCM 1.60 Deep Maize 2.06 MinneTAC 2.27 Mertacor 0.55 SouthamptonSCM 1.55 RationaSCM 2.28 Deep Maize 0.22 Mertacor 0.53 CMieux 2.33 MinneTAC 0.31 MinneTAC 0.33 PhantAgent 6.64 Maxon 1.99 Maxon 1.74 Tabe 6 Scores and rankings from the rounds Deep Maize payed in during the 2006 SCM tournament. A scores are in miions of doars. Semi-finas 2 SCM tournament, this round was meant as a preiminary test of the overa Prediction Chaenge for both the organizers and competitors. The fina round took pace the day before the main TAC/ SCM finas. In both rounds, every competitor was given eight hours to make predictions about the 48 games defining the round. Competitors were aowed to produce their predictions any time during this period, independenty of the other competitors. The resuts from the Chaenge semi-finas and finas are shown in Tabe 8. Deep Maize paced first overa in the finas, achieving the top score in both component prediction categories and the secondbest scores in both PC prediction categories. These resuts provide 19 More detais about the tournament structure and fu resuts are avaiabe at the TAC web site, Finas Deep Maize 6.45 TacTex Maxon 4.08 PhantAgent 4.15 Botticei 1.95 Deep Maize 3.58 CMieux 1.81 Maxon 1.75 SouthamptonSCM 1.63 Botticei 0.48 Mertacor 1.44 MinneTAC 2.70 Tabe 7 Scores and rankings from the rounds Deep Maize payed in during the 2007 SCM tournament. A scores are in miions of doars. Semi-Finas 1 Finas Deep Maize 9.76 PhantAgent 8.67 Tinhorn 6.94 TacTex 6.31 Maxon 5.63 DeepMaize 5.45 MasterBham 5.40 Maxon 1.79 CrocodieAgent 5.12 Tinhorn 1.34 Sorter CMieux 1.24 Tabe 8 TAC/SCM Prediction Chaenge 2007 Resuts (RMS error normaized by base price). Prediction Agent Semi-finas Finas Current PCs TacTex Deep Maize Botticei Kshitij Future PCs TacTex Deep Maize Botticei Kshitij Current components Deep Maize Botticei TacTex Kshitij Future components Deep Maize Botticei TacTex Kshitij

14 14 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx 1168 additiona support for the overa efficacy of our methods. Deep 1169 Maize used a variation of RCPP for the semi-fina round of the Pre diction Chaenge. Among other differences, this version used an 1171 expiration interva of five for the inear price interpoations. The fi na round version used an expiration interva of one, which argey 1173 account for the significant improvement in current component 1174 price predictions between the semi-fina and fina round. The data 1175 set for component price predictions was based on games specifi cay simuated for the Prediction Chaenge, as discussed in Section The customer price predictor used in the Chaenge was iden tica to the version used by Deep Maize in the 2006 TAC/SCM tour nament, incuding the data set Concusions 1181 We document and anayze successfu approaches for forecast ing prices in a chaenging market domain. The genera principe 1183 of our design is to support expoitation of a possibe sources of 1184 information. To this end, we integrate severa we-known ideas 1185 in an nove way, resuting in better overa predictions In the PC market, we evauate our ideas against a baseine pre dictor simiar to those used by many competing agents, appying 1188 both error measures and simuation performance as benchmarks We show that improved predictions can dramaticay improve 1190 agent performance in TAC/SCM. This is particuary interesting in 1191 ight of the substantia uncertainties inherent in the domain and 1192 chaenges of predicting the behavior of other agents The technique we present for appying transformations to pre dictions using onine scoring is particuary interesting, and shoud 1195 be appicabe in many domains. This method provides a way to 1196 perform principed combinations of predictions based on different 1197 sources of data, as we as a way to reduce systematic errors. It 1198 does these by making use of additiona information derived from 1199 onine performance. This method showed consistent benefits in 1200 our experiments across a range of conditions on both prediction er ror and simuation performance metrics We aso investigated the use of absoute and reative represen tations for predictions. It appears that each of these representa tions has advantages under certain conditions, particuary 1205 reated to how the information is used in decision-making In the component market we deveop methods to integrate vo atie spot price predictions with a ong-term predictor using muti pe sources of inter-market information. This reative predictor is 1209 more accurate than either the spot predictions or absoute price 1210 predictions from historica data in isoation Our prediction methods for both markets demonstrate strong 1212 overa performance, demonstrated in interna experiments and in 1213 our first-pace finish in the inaugura TAC/SCM Prediction Chaenge These predictions have aso contributed to the strong overa perfor mance of Deep Maize in the TAC/SCM tournament in recent years Acknowedgements 1217 Thanks to the participants and organizers of the TAC/SCM com petition for providing such a fruitfu research environment. We owe 1219 particuar gratitude to David Pardoe for designing and operating the 1220 Prediction Chaenge. Kevin O Maey contributed to the Deep Maize 1221 infrastructure. This work was supported in part by NSF grant IIS , and the STIET program under NSF IGERT grant References 1224 Bates JM, Granger CWJ. The combination of forecasts. Oper Res 1969;20(4): Benisch M, Greenwad A, Naroditskiy V, Tschantz M. A stochastic programming 1226 approach to scheduing in TAC SCM. In: Fifth ACM conference on eectronic 1227 commerce. New York, 2004; p Benisch M, Sardinha A, Andrews J, Sadeh N. CMieux: adaptive strategies for competitive suppy chain trading. In: Eighth internationa conference on eectronic commerce. Ann Arbor, 2006; p Breiman L, Friedman J, Stone CJ, Oshen R. Cassification and regression trees. Chapman & Ha/CRC; Brockwe PJ, Davis RA. Introduction to time series and forecasting. Springer- Verag; Brooks CH, Gazzae R, Das R, Kephart JO, MacKie-Mason JK, Durfee EH. Mode seection in an information economy: choosing what to earn. Comput Inte Burke DA, Brown KN, Hnich B, Tarim A. Learning market prices for a rea-time suppy chain management trading agent. In: AAMAS-06 workshop on trading agent design and anaysis. Hakodate; Chatfied C. Time-series forecasting. Chapman & Ha/CRC; Cheng S-F, Leung E, Lochner KM, O Maey K, Reeves DM, Weman MP. Waverine: a Warasian trading agent. Decis Support Syst 2005;39: Cemen RT, Winker RL. Combining probabiity distributions from experts in risk anaysis. Risk Ana 1999;19(2): Coins J, Arunachaam R, Sadeh N, Eriksson J, Finne N, Janson S. The suppy chain management game for the 2007 trading agent competition. In: Technica report CMU-ISRI Carnegie Meon University; Eriksson J, Finne N, Janson S. Evoution of a suppy chain management game for the trading agent competition. AI Commun 2006;9:1 12. Ghani R. Price prediction and insurance for onine auctions. In: Internationa conference on knowedge discovery in data mining. 2005; p Hanson R. Combinatoria information market design. Inform Syst Front 2003;5(1): Hastie T, Tibshirani R, Friedman J. Eements of statistica earning. Springer- Verag; He M, Rogers A, Luo X, Jennings NR. Designing a successfu trading agent for suppy chain management. In: Fifth internationa joint conference on autonomous agents and mutiagent systems. Hakodate, 2006; p Jordan PR, Kiekintved C, Weman MP. Empirica game-theoretic anaysis of the TAC suppy chain game. In: Sixth internationa joint conference on autonomous agents and mutiagent systems. 2007; p Kaufman PJ. Trading systems and methods. third ed. John Wiey and Sons; Keer PW, Duguay F-O, Precup D. RedAgent-2003: an autonomous market-based suppy-chain management agent. In: Third internationa conference on autonomous agents and mutiagent systems. 2004; p Ketter W. Identification and prediction of economic regimes to guide decision making in muti-agent marketpaces. PhD thesis, University of Minnesota; January Ketter W, Coins J, Gini ML, Gupta A, Schrater P. A predictive empirica mode for pricing and resource aocation decisions. In: Ninth internationa conference on eectronic commerce. Minneapois, 2007; p Kiekintved C, Weman MP, Singh S, Estee J, Vorobeychik Y, Soni V, Rudary M. Distributed feedback contro for decision making on suppy chains. In: Fourteenth internationa conference on automated panning and scheduing. Whister, BC, 2004; p Kiekintved C, Vorobeychik Y, Weman MP. An anaysis of the 2004 suppy chain management trading agent competition. In: IJCAI-05 workshop on trading agent design and anaysis. Edinburgh; Kiekintved C, Mier J, Jordan PR, Weman MP. Controing a suppy chain agent using vaue-based decomposition. In: Seventh ACM conference on eectronic commerce. Ann Arbor, 2006; p Kiekintved C, Mier J, Jordan PR, Weman MP. Forecasting market prices in a suppy chain game. In: Sixth internationa joint conference on autonomous agents and mutiagent systems. 2007; p Nogaes FJ, Contreras J, Conejo AJ, Espinoa R. Forecasting next-day eectricity prices by time series modes. IEEE Trans Power Syst 2002;17: Osepayshvii A, Weman MP, Reeves DM, MacKie-Mason JK. Sef-confirming price prediction for bidding in simutaneous ascending auctions. In: Twentyfirst conference on uncertainty in artificia inteigence. Edinburgh, 2005; p Pardoe D. The TAC SCM prediction chaenge. 2007; < edu/tactex/predictionchaenge/>. Pardoe D, Stone P. Bidding for customer orders in TAC SCM. In: Agent mediated eectronic commerce VI, ecture notes in artificia inteigence, vo Springer; Pardoe D, Stone P. TacTex-05: a champion suppy chain management agent. In: Twenty-first nationa conference on artificia inteigence. Boston, 2006; p Pardoe D, Stone P. Adapting price predictions in TAC SCM. In: AAMAS-07 workshop on agent mediated eectronic commerce IX. Honouu; 2007a. Pardoe D, Stone P. An autonomous agent for suppy chain management. In: Adomavicius G, Gupta A, editors. Handbooks in information systems series: business computing. Esevier; 2007b. Pindyck RS, Rubinfed DL. Econometric modes and econometric forecasts. 4th ed. Irwin/McGraw-Hi; Robnik-Šikonja M, Kononenko I. An adaptation of reief for attribute estimation in regression. In: Fourteenth internationa conference on machine earning. Nashvie, 1997; p Robnik-Šikonja M, Kononenko I. Theoretica and empirica anaysis of ReiefF and RReiefF. Mach Learn 2003;53:

15 C. Kiekintved et a. / Eectronic Commerce Research and Appications xxx (2008) xxx xxx Stone P, Schapire RE, Littman ML, Csirik JA, McAester D. Decision-theoretic bidding 1314 based on earned density modes in simutaneous, interacting auctions. J Artif 1315 Inte Res 2003;19: Weman MP, Reeves DM, Lochner KM, Vorobeychik Y. Price prediction in a trading 1317 agent competition. J Artif Inte Res 2004;21: Weman MP, Greenwad A, Stone P. Autonomous bidding agents: strategies and essons from the trading agent competition. MIT Press; Witten IH, Frank E. Data mining: practica machine earning toos and techniques with java impementations. Morgan Kaufmann;