RoxyBot-06: Stochastic Prediction and Optimization in TAC Travel

In this paper, we describe our autonomous bidding agent, RoxyBot, who emerged victorious in the travel division of the 2006 Trading Agent Competition in a photo finish. At a high level, the design of many successful trading agents can be summarized as follows: (i) price prediction: build a model of market prices; and (ii) optimization: solve for an approximately optimal set of bids, given this model. To predict, RoxyBot builds a stochastic model of market prices by simulating simultaneous ascending auctions. To optimize, RoxyBot relies on the sample average approximation method, a stochastic optimization technique.


Introduction
The annual Trading Agent Competition (TAC) challenges its entrants to design and build autonomous agents capable of effective trading in an online travel 1 shopping game.The first TAC, held in Boston in 2000, attracted 16 entrants from six countries in North America, Europe, and Asia.Excitement generated from this event led to refinement of the game rules, and continuation of regular tournaments with increasing levels of competition over the next six years.Year-by-year, entrants improved their designs, developing new ideas and building on previously successful techniques.Since TAC's inception, the lead author has entered successive modifications of her autonomous trading agent, RoxyBot.This paper reports on RoxyBot-06, the latest incarnation and the top scorer in the TAC-06 tournament.
The key feature captured by the TAC travel game is that goods are highly interdependent (e.g., flights and hotels must be coordinated), yet the markets for these goods operate independently.A second important feature of TAC is that agents trade via three different kinds of market mechanisms, each of which presents distinct challenges.Flights are traded in a posted-price environment, where a designated party sets a price that the other parties must "take or leave."Hotels are traded in simultaneous ascending auctions, like the FCC spectrum auctions.Entertainment tickets are traded in continuous double auctions, like the New York Stock Exchange.In grappling with all three mechanisms while constructing their agent strategies, participants are confronted by a number of interesting problems.
The success of an autonomous trading agent such as a TAC agent often hinges upon the solutions to two key problems: (i) price prediction, in which the agent builds a model of market prices; and (ii) optimization, in which the agent solves for an approximately optimal set of bids, given this model.For example, at the core of RoxyBot's 2000 architecture (Greenwald & Boyan, 2005) was a deterministic optimization problem, namely how to bid given price predictions in the form of point estimates.In spite of its effectiveness in the TAC-00 tournament, a weakness of the 2000 design was that RoxyBot could not explicitly reason about variance within prices.In the years since 2000, we recast the key challenges faced by TAC agents as several different stochastic bidding problems (see, for example, the paper by Greenwald & Boyan, 2004), whose solutions exploit price predictions in the form of distributions.In spite of our perseverance, RoxyBot fared unimpressively in tournament conditions year after year, until 2006.Half a decade in the laboratory spent searching for bidding heuristics that can exploit stochastic information at reasonable computational expense finally bore fruit, as RoxyBot emerged victorious in TAC-06.In a nutshell, the secret of RoxyBot-06's success is: (hotel) price prediction by simulating simultaneous ascending auctions, and optimization based on the sample average approximation method.Details of our approach are the subject of the present article.
Overview This paper is organized as follows.Starting in Section 2, we summarize the TAC market game.Next, in Section 3, we present a high-level view of RoxyBot's 2006 architecture.In Section 4, we describe RoxyBot's price prediction techniques for flights, hotels, and entertainment, in turn.Perhaps of greatest interest is our hotel price prediction method.Following Wellman et al. (2005), we predict hotel prices by computing approximate competitive equilibrium prices.Only, instead of computing those prices by running the tâtonnement process, we simulate simultaneous ascending auctions.Our procedure is simpler to implement than tâtonnement, yet achieves comparable performance, and runs sufficiently fast.In Section 5, we describe RoxyBot's optimization technique: sample average approximation.We argue that this approach is optimal in pseudo-auctions, an abstract model of auctions.In Section 6.1, we describe simulation experiments in a controlled testing environment which show that our combined approach-simultaneous ascending auctions for hotel price prediction and sample average approximation for bid optimization-performs well in practice in comparison with other reasonable bidding heuristics.In Section 6.2, we detail the results of the TAC-06 tournament, further validating the success of RoxyBot-06's strategy, and reporting statistics that shed light on the bidding strategies of other participating agents.Finally, in Section 7, we evaluate the collective behavior of the autonomous agents in the TAC finals since 2002.We find that the accuracy of competitive equilibrium calculations has varied from year to year and is highly dependent on the particular agent pool.Still, generally speaking, the collective appears to be moving toward competitive equilibrium behavior. RoxyBot-06

TAC Market Game: A Brief Summary
In this section, we summarize the TAC game.For more details, see http://www.sics.se/tac/.
Eight agents play the TAC game.Each is a simulated travel agent whose task is to organize itineraries for its clients to travel to and from "TACTown" during a five day (four night) period.In the time allotted (nine minutes), each agent's objective is to procure travel goods as inexpensively as possible, trading off against the fact that those goods are ultimately compiled into feasible trips that satisfy its client preferences to the greatest extent possible.The agents know the preferences of their own eight clients only, not the other 56.
Travel goods are sold in simultaneous auctions as follows: • Flight tickets are sold by "TACAir" in dynamic posted-pricing environments.There are flights both to and from TACTown on each applicable day.No resale of flight tickets by agents is permitted.
Flight price quotes are broadcast by the TAC server every ten seconds.
• Hotel reservations are sold by the "TAC seller" in multi-unit ascending call markets.Specifically, 16 hotel reservations are sold in each hotel auction to the 16 highest bidders at the 16th highest price.The TAC seller runs eight hotel auctions, one per night-hotel combination (recall that travel takes place during a four night period; moreover, there are two hotels: a good one and a bad one).No resale of hotel reservations by agents is permitted.Nor is bid withdrawal allowed.
More specifically, the eight hotel auctions clear on the minute with exactly one auction closing at each of minutes one through eight.(The precise auction to close is chosen at random, with all open auctions equally likely to be selected.)For the auction that closes, the TAC server broadcasts the final closing price, and informs each agent of its winnings.For the others, the TAC server reports the current ask price, and informs each agent of its "hypothetical quantity won" (HQW).
• Agents are allocated an initial endowment of entertainment tickets, which they trade among themselves in continuous double auctions (CDAs).There are three entertainment events scheduled each day.
Although the event auctions clear continuously, price quotes are broadcast only every 30 seconds.
One of the primary challenges posed by TAC is to design and build autonomous agents that bid effectively on interdependent (i.e., complementary or substitutable) goods that are sold in separate markets.Flight tickets and hotel reservations are complementary because flights are not useful to a client without the corresponding hotel reservations, nor vice versa.Tickets to entertainment events (e.g., the Boston Red Sox and the Boston Symphony Orchestra) are substitutable because a client cannot attend multiple events simultaneously.

RoxyBot-06's Architecture: A High-Level View
In our approach to the problem of bidding on interdependent goods in the separate TAC markets, we adopt some simplifying assumptions.Rather than tackle the game-theoretic problem of characterizing strategic equilibria, we focus on a single agent's (decision-theoretic) problem of optimizing its own bidding behavior, assuming the other agents' strategies are fixed.In addition, we assume that the environment can be modeled in terms of the agent's predictions about market clearing prices.These prices serve to summarize the relevant information hidden in other agents' bidding strategies.These two assumptions-fixed otheragent behaviors and market information encapsulated by prices-support the modular design of RoxyBot-06 and many other successful TAC agents, which consists of two key stages: (i) price prediction; and (ii) optimization.
The optimization problem faced by TAC agents is a dynamic one that incorporates aspects of sequentiality as well as simultaneity in auctions.The markets operate simultaneously, but in addition, prices are discovered incrementally over time.In principle, a clairvoyant agent-one with knowledge of future clearing prices-could justifiably employ an open-loop strategy: it could solve the TAC optimization problem once at the start of the game and place all its bids accordingly, never reconsidering those decisions.A more practical alternative (and the usual approach taken in TAC2 ), is to incorporate into an agent's architecture a closed loop, or bidding cycle, enabling the agent to condition its behavior on the evolution of prices.As price information is revealed, the agent improves its price predictions, and reoptimizes its bidding decisions, repeatedly.
One distinguishing feature of RoxyBot-06 is that it builds stochastic models of market clearing prices, rather than predicting clearing prices as point estimates.Given its stochastic price predictions, stochastic optimization lies at the heart of RoxyBot-06.Assuming time is RoxyBot-06 discretized into stages, or bid intervals, during each iteration of its bidding cycle, RoxyBot-06 faces an n-stage stochastic optimization problem, where n is the number of stages remaining in the game.The key input to this optimization problem is a sequence of n − 1 stochastic models of future prices, each one a joint probability distribution over all goods conditioned on past prices and past hotel closings.The solution to this optimization problem, and the output of each iteration of the bidding cycle, is a vector of bids, one per good (or auction).
Table 1 presents a high-level view of RoxyBot-06's architecture, emphasizing its bidding cycle.At the start of each bid interval, current prices and winnings are downloaded from the TAC server.Next, the key prediction and optimization routines are run.In the prediction module, stochastic models of flight, hotel, and entertainment prices are built.In the optimization module, bids are constructed as an approximate solution to an n-stage stochastic optimization problem.Prior to the end of each bid interval, the agents' bids are uploaded to the TAC server using three separate threads: (i) the flight thread bids on a flight only if its price is near its predicted minimum; (ii) the hotel thread bids on open hotels only if it is moments before the end of a minute; and (iii) the entertainment thread places bids immediately.
We discuss the details of RoxyBot-06's price prediction module first, and its optimization module second.

Price Prediction
In this section, we describe how RoxyBot-06 builds its stochastic models of flight, hotel, and event prices.Each model is a discrete probability distribution, represented by a set of "scenarios."Each scenario is a vector of "future" prices-prices at which goods can be bought and sold after the current stage.For flights, the price prediction model is not stochastic: the future buy price is simply RoxyBot-06's prediction of the expected minimum price during the current stage.For hotels, the future buy prices are predicted by Monte Carlo simulations of simultaneous ascending auctions to approximate competitive equilibrium prices.There are no current buy prices for hotels.For entertainment, RoxyBot-06 predicts future buy and sell prices based on historical data.Details of these price prediction methods are the focus of this section.

Flights
Efforts to deliberate about flight purchasing start with understanding the TAC model of flight price evolution.

TAC Flight Prices' Stochastic Process
Flight prices follow a biased random walk.They are initialized uniformly in the range [250,400], and constrained to remain in the range [150,800].At the start of each TAC game instance, a bound z on the final perturbation value is selected for each flight.These bounds are not revealed to the agents.What is revealed to the agents is a sequence of random flight prices.Every ten seconds, TACAir perturbs the price of each flight by a random value that depends on the hidden parameter z and the current time t as follows: given constants c, d ∈ R and T > 0, each (intermediate) bound on the perturbation value is a linear function of t: x The perturbation value at time t is drawn uniformly from one of the following ranges (see Algorithm 1): Observe that the expected perturbation value in each case is simply the average of the corresponding upper and lower bounds.In particular, • if x(t, z) > c, then the expected perturbation is positive; • if x(t, z) ∈ (0, c), then the expected perturbation is negative; • if x(t, z) ∈ (−c, 0), then the expected perturbation is positive; • otherwise, if x(t, z) ∈ {−c, 0, c}, then the expected perturbation is zero.
Moreover, using Equation 1, we can compute the expected perturbation value conditioned on z: ], so prices are expected not to increase; so prices are expected not to decrease; • if z ∈ [−c, 0], then x(t, z) ∈ [−c, c], so prices are expected not to increase while t ≤ cT c−z and they are expected not to decrease while t ≥ cT c−z .The TAC parameters are set as follows: c = 10, d = 30, T = 540, and z uniformly distributed in the range [−c, d].Based on the above discussion, we note the following: given no further information about z, TAC flight prices are expected to increase (i.e., the expected perturbation is positive); however, conditioned on z, TAC flight prices may increase or decrease (i.e., the expected perturbation can be positive or negative).

RoxyBot-06's Flight Prices Prediction Method
Although the value of the hidden parameter z is never revealed to the agents, recall that the agents do observe sample flight prices, say y 1 , . . ., y t , that depend on this value.This information can be used to model the probability distribution P t [z] ≡ P [z | y 1 , . . ., y t ].Such a probability distribution can be estimated using Bayesian updating.Before RoxyBot-06, agents Walverine (Cheng et al., 2005) and Mertacor (Toulis et al., 2006) took this approach.Walverine uses Bayesian updating to compute the next expected price perturbation and then compares that value to a threshold, postponing its flight purchases if prices are not expected to increase by more than that threshold.Mertacor uses Bayesian updating to estimate the time at which flight prices will reach their minimum value.RoxyBot uses Bayesian updating to compute the expected minimum price, as we now describe. where Equation 4 follows from the fact that future observations are independent of past observations; observations depend only on the hidden parameter z.
The only thing left to explain is how to set the values P [y i | z], for i = 1, . . ., t.As described in the pseudocode, this is done as follows: if y t+1 is within the appropriate range at that time, then this probability is set uniformly within the bounds of that range; otherwise, it is set to 0. Presumably, Walverine's and Mertacor's implementations of Bayesian updating are not very different from this one.3However, as alluded to above, how the agents make use of the ensuing estimated probability distributions does differ.
RoxyBot-06 predicts each flight's price to be its expected minimum price.This value is computed as follows (see Algorithm 3): for each possible value of the hidden parameter z, RoxyBot simulates an "expected" random walk, selects the minimum price along this walk, and then outputs as its prediction the expectation of these minima, averaging according to P t [z].We call this random walk "expected," since the perturbation value ∆ is an expectation (i.e., ∆ = b−a 2 ) instead of a sample (i.e., ∆ ∼ U [a, b]).By carrying out this computation, RoxyBot generates flight price predictions that are point estimates.The implicit decision to make only RoxyBot-06's hotel and event price predictions stochastic was made based on our intuitive sense of the time vs. accuracy tradeoffs in RoxyBot's optimization module, and hence warrants further study.

Hotels
In a competitive market where each individual's effect on prices is negligible, equilibrium prices are prices at which supply equals demand, assuming all producers are profitmaximizing and all consumers are utility-maximizing.RoxyBot-06 predicts hotel prices by simulating simultaneous ascending auctions (SimAA) (Cramton, 2006), in an attempt to approximate competitive equilibrium (CE) prices.This approach is inspired by Walverine's (Cheng et al., 2005), where the tâtonnement method (Walras, 1874) is used for the same purpose.

Simultaneous Ascending Auctions
Let p denote a vector of prices.If y( p) denotes the cumulative supply of all producers, and if x( p) denotes the cumulative demand of all consumers, then z( p) = x( p)− y( p) denotes the RoxyBot-06 excess demand in the market.The tâtonnement process adjusts the price vector at iteration n + 1, given the price vector at iteration n and an adjustment rate α n as follows: p n+1 = p n + α n z( p n ).SimAA adjusts the price vector as follows: p n+1 = p n + α max{ z( p n ), 0}, for some fixed value of α.Both of these processes continue until excess demand is non-positive: i.e., supply exceeds demand.
Although competitive equilibrium prices are not guaranteed to exist in TAC markets (Cheng et al., 2003), the SimAA adjustment process, is still guaranteed to converge: as prices increase, demand decreases while supply increases; hence, supply eventually exceeds demand.The only parameter to the SimAA method is the magnitude α of the price adjustment.The smaller this value, the more accurate the approximation (assuming CE prices exist), so the value of α can be chosen to be the lowest value that time permits.
The tâtonnement process, on the other hand, is more difficult to apply as it is not guaranteed to converge.The Walverine team dealt with the convergence issue by decaying an initial value of α.However, careful optimization was required to ensure convergence to a reasonable solution in a reasonable amount of time.In fact, Walverine found it helpful to set initial prices to certain non-zero values.This complexity is not present when using simultaneous ascending auctions to approximate competitive equilibrium prices.

Prediction Quality
In TAC, cumulative supply is fixed.Hence, the key to computing excess demand is to compute cumulative demand.Each TAC agent knows the preferences of its own clients, but must estimate the demand of the others.Walverine computes a single hotel price prediction (a point estimate) by considering its own clients' demands together with those of 56 "expected" clients.Briefly, the utility of an expected client is an average across travel dates and hotel types augmented with fixed entertainment bonuses that favor longer trips (see the paper by Cheng et al., 2005, for details).In contrast, RoxyBot-06 builds a stochastic model of hotel prices consisting of S scenarios by considering its own clients' demands together with S random samples of 56 clients.A (random or expected) client's demand is simply the quantity of each good in its optimal package, given current prices.The cumulative demand is the sum total of all client's individual demands.
In Figure 1, we present two scatter plots that depict the quality of various hotel price predictions at the beginning of the TAC 2002 final games.All price predictions are evaluated using two metrics: Euclidean distance and the "expected value of perfect prediction" (EVPP).Euclidean distance is a measure of the difference between two vectors, in this case the actual and the predicted prices.The value of perfect prediction (VPP) for a client is the difference between its surplus (value of its preferred package less price) based on actual and predicted prices.EVPP is the VPP averaged over the distribution of client preferences. 4n the left, we plot the predictions generated using the competitive equilibrium approximation methods, tâtonnement and SimAA, both with fixed α = 1 24 , making expected, random, and exact predictions.The "exact" predictions are computed based on the actual clients in the games, not just the client distribution; hence, they serve as a lower bound on the performance of these techniques on this data set.Under both metrics, and for both expected and random, SimAA's predictions outperform tâtonnement's.and SimAA with α = 1 24 ; expected, random, and exact) and the TAC 2002 agents' predictions in the 2002 finals (60 games).The plot on the left shows that SimAA's predictions are better than tâtonnement's and that expected's are better than random's.RoxyBot-06's method of hotel price prediction (SimAA, Random) is plotted again on the right.Note the differences in scales between the two plots.
Since α is fixed, and tâtonnement is not guaranteed to converge under this condition, this outcome is not entirely surprising.What is interesting, though, is that SimAA expected performs comparably to Walverine (see the right plot). 5 This is interesting because SimAA has fewer parameter settings than tâtonnement-only a single α value as compared to an initial α value together with a decay schedule-and moreover, we did not optimize its parameter setting.Walverine's parameter settings, on the other hand, were highly optimized.
We interpret each prediction generated using randomly sampled clients as a sample scenario, so that a set of such scenarios represents draws from a probability distribution over CE prices.The corresponding vector of predicted prices that is evaluated is actually the average of multiple (40) such predictions; that is, we evaluate an estimate of the mean of this probability distribution.The predictions generated using sets of random clients are not as good as the predictions with expected clients (see Figure 1 left), although with more than 40 sets of random clients, the results might improve.Still, the predictions with random clients comprise RoxyBot-06's stochastic model of hotel prices, which is key to its bidding strategy.Moreover, using random clients helps RoxyBot-06 make better interim predictions later in the game as we explain next.

Prediction Quality over Time: Interim Price Prediction
The graphs depicted in Figure 1 pertain to hotel price predictions made at the beginning of the game, when all hotel auctions are open.In those CE computations, prices are initialized to 0. As hotel auctions close, RoxyBot-06 updates the predicted prices of the hotel auctions 5.With the exception of the RoxyBot-06 data point (i.e., SimAA random), this plot was produced by the Walverine team (Wellman et al., 2004).
RoxyBot-06 that remain open.We experimented with two ways of constructing interim price predictions.
The first is to initialize and lower bound the prices in the hotel markets at their closing (for closed auctions) or current ask (for open auctions) prices while computing competitive equilibrium prices. 6The second differs in its treatment of closed auctions: we simulate a process of distributing the goods in the closed auctions to the clients who want them most, and then exclude the closed markets (i.e., fix prices at ∞) from further computations of competitive equilibrium prices.
Regarding the second method-the distribution method-we determine how to distribute goods by computing competitive equilibrium prices again.As explained in Algorithm 4, all hotels (in both open and closed auctions) are distributed to random clients by determining who is willing to pay the competitive equilibrium prices for what.It is not immediately obvious how to distribute goods to expected clients; hence, we enhanced only the prediction methods with random clients with distribution.
Figure 2, which depicts prediction quality over time, shows that the prediction methods enhanced with distribution are better than the predictions obtained by merely initializing the prices of closed hotel auctions at their closing prices.Hotels that close early tend to sell for less than hotels that close late; hence, the prediction quality of any method that makes decent initial predictions is bound to deteriorate if those predictions remain relatively constant throughout the game.

Run Time
Table 2 shows the run times of the CE prediction methods on the TAC 2002 (60 games) and TAC 2006 (165 games) finals data set at minute 0, as well as their run times during 6.At first blush, it may seem more sensible to fix the prices of closed hotels at their closing prices, rather than merely lower bound them (i.e., allow them to increase).If some hotel closed at an artificially low price, however, and if that price were not permitted to increase, then the predicted prices of the hotels complementing the hotel in question would be artificially high.
Algorithm 4 Distribute 1: for all hotel auctions h do 2: initialize price to 0 3: initialize supply to 16 4: end for 5: compute competitive equilibrium prices {Tâtonnement or SimAA} 6: for all closed hotel auctions h do 7: distribute units of h to those who demand them at the computed competitive equilibrium prices 8: distribute any leftover units of h uniformly at random 9: end for minutes 1-7 on the TAC 2006 finals data set.What the numbers in this table convey is that SimAA's run time, even with distribution, is reasonable.For example, at minute 0, SimAA sample takes on the order of 0.1 seconds.At minutes 1-7, this method without distribution runs even faster.This speed increase occurs because CE prices are bounded below by current ask prices and above by the maximum price a client is willing to pay for a hotel, and current ask prices increase over time, correspondingly reducing the size of the search space.SimAA sample with distribution at minutes 1-7 takes twice as long as SimAA sample without distribution at minute 0 because of the time it takes to distribute goods, but the run time is still only (roughly) 0.2 seconds.Our implementation of tâtonnement runs so slowly because we fixed α instead of optimizing the tradeoff between convergence rate and accuracy, so the process did not converge, and instead ran for the maximum number of iterations (10,000).In summary, SimAA is simpler than tâtonnement to implement, yet performs comparably to an optimized version of tâtonnement (i.e., Walverine), and runs sufficiently fast.

Summary
The simulation methods discussed in this section-the tâtonnement process and simultaneous ascending auctions-were employed to predict hotel prices only.(In our simulations, flight prices are fixed at their expected minima, and entertainment prices are fixed at 80.) In principle, competitive equilibrium (CE) prices could serve as predictions in all TAC markets.However, CE prices are unlikely to be good predictors of flight prices, since flight prices are determined exogenously.With regard to entertainment tickets, CE prices might RoxyBot-06 have predictive power; however, incorporating entertainment tickets into the tâtonnement and SimAA calculations would have been expensive.(In our simulations, following Wellman et al., 2004, client utilities are simply augmented with fixed entertainment bonuses that favor longer trips.)Nonetheless, in future work, it could be of interest to evaluate the success of these or related methods in predicting CDA clearing prices.
Finally, we note that we refer to our methods of computing excess demand as "clientbased" because we compute the demands of each client on an individual basis.In contrast, one could employ an "agent-based" method, whereby the demands of agents, not clients, would be calculated.Determining an agent's demands involves solving so-called completion, a deterministic (prices are known) optimization problem at the heart of RoxyBot-00's architecture (Greenwald & Boyan, 2005).As TAC completion is NP-hard, the agent-based method of predicting hotel prices is too expensive to be included in RoxyBot-06's inner loop.In designing RoxyBot-06, we reasoned that an architecture based on a stochastic pricing model generated using the client-based method and randomly sampled clients would outperform one based on a point estimate pricing model generated using the agent-based method and some form of expected clients, but we did not verify our reasoning empirically.

Entertainment
During each bid interval, RoxyBot-06 predicts current and future buy and sell prices for tickets to all entertainment events.These price predictions are optimistic: the agent assumes it can buy (or sell) goods at the least (or most) expensive prices that it expects to see before the end of the game.More specifically, each current price prediction is the best predicted price during the current bid interval.
RoxyBot-06's estimates of entertainment ticket prices are based on historical data from the past 40 games.To generate a scenario, a sample game is drawn at random from this collection, and the sequences of entertainment bid, ask, and transaction prices are extracted.Given such a history, for each auction a, let trade ai denote the price at which the last trade before time i transacted; this value is initialized to 200 for buying and 0 for selling.In addition, let bid ai denote the bid price at time i, and let ask ai denote the ask price at time i.
RoxyBot-06 predicts the future buy price in auction a after time t as follows: future buy at = min i=t+1,...,T min{trade ai , ask ai } (5) In words, the future buy price at each time i = t + 1, . . ., T is the minimum of the ask price after time i and the most recent trade price.The future buy price at time t is the minimum across the future buy prices at all later times.The future sell price after time t is predicted analogously: Arguably, RoxyBot-06's entertainment predictions are made in the simplest possible way: past data are future predictions.It is likely one could improve upon this naive approach by using a generalization technique capable of "learning" a distribution over these data, and then sampling from the learned distribution.

Summary
In this section, we described RoxyBot-06's price prediction methods.The key ideas, which may be transferable if not beyond TAC, at least to other TAC agents, are as follows: 1. RoxyBot makes stochastic price predictions.It does so by generating a set of so-called "scenarios," where each scenario is a vector of future prices.
2. For each flight, RoxyBot uses Bayesian updating to predict its expected minimum price.
3. For hotels, RoxyBot-uses a method inspired by Walverine's: it approximates competitive equilibrium prices by simulating simultaneous ascending auctions, rather than the usual tâtonnement process.

Optimization
Next, we characterize RoxyBot-06's optimization routine.It is (i) stochastic, (ii) global, and (iii) dynamic.It takes as input stochastic price predictions; it considers its flight, hotel, and entertainment bidding decisions in unison; and it simultaneously reasons about bids to be placed in both current and future stages of the game.

Abstract Auction Model
Recall that our treatment of bidding is decision-theoretic, rather than game-theoretic.In particular, we focus on a single agent's problem of optimizing its own bidding behavior, assuming the other agents' strategies are fixed.In keeping with our basic agent architecture, we further assume that the environment can be modeled in terms of the agent's predictions about market clearing prices.We introduce the term pseudo-auction to refer to a market mechanism defined by these two assumptions-fixed other-agent behaviors and market information encapsulated by prices.The optimization problem that RoxyBot solves is one of bidding in pseudo-auctions, not (true) auctions.In this section, we formally develop this abstract auction model and relate it to TAC auctions; in the next, we define and propose heuristics to solve various pseudo-auction bidding problems.

Basic Formalism
In this section, we formalize the basic concepts needed to precisely formulate bidding under uncertainty as an optimization problem, including: packages-sets of goods, possibly multiple units of each; a function that describes how much the agent values each package; pricelines-data structures in which to store the prices of each unit of each good; and bids-pairs of vectors corresponding to buy and sell offers.
Packages Let G denote an ordered set of n distinct goods and let N ∈ N n represent the multiset of these goods in the marketplace, with N g denoting the number of units of each good g ∈ G.A package M is a collection of goods, that is, a "submultiset" of N .We write M ⊆ N whenever M g ≤ N g for all g ∈ G.
It is instructive to interpret this notation in the TAC domain.The flights, hotel rooms, and entertainment events up for auction in TAC comprise an ordered set of 28 distinct A seller priceline for good g is a vector π g ∈ R Ng + .Much like a buyer priceline, the kth component of a seller priceline for g stores the marginal revenue that an agent could earn from the kth unit it sells.For example, if the market demands four units of good g, which can be sold at prices of $20, $15, $10, and $5, then the corresponding seller priceline is given by π g = 20, 15, 10, 5, 0, 0, 0, 0 .Analogously to buyer pricelines, the tail of zero revenues indicates that the market demands only four of those units.
We assume seller pricelines are nonincreasing.Note that this assumption is WLOG, since an optimizing agent sells more expensive goods before cheaper ones.
Given a set of seller pricelines Π = { π g | g ∈ G}, we define revenue additively, that is, the revenue associated with multiset Z ⊆ N is given by: Revenue(Z, Π) = g∈G Revenue g (Z, Π). ( 10) If a priceline is constant, we say that prices are linear.We refer to the constant value as a unit price.With linear prices, the cost of acquiring k units of good g is k times the unit price of good g.
Bids An agent submits a bid β expressing offers to buy or sell various units of the goods in the marketplace.We divide β into two components b, a , where for each good g the bid consists of a buy offer, b g = b g1 , . . ., b gNg , and a sell offer, a g = a g1 , . . ., a gNg .The bid price b gk ∈ R + (resp.a gk ∈ R + ) represents an offer to buy (sell) the kth unit of good g at that price.
By definition, the agent cannot buy (sell) the kth unit unless it also buys (sells) units 1, . . ., k − 1.To accommodate this fact, we impose the following constraint: Buy offers must be nonincreasing in k, and sell offers nondecreasing.In addition, an agent may not offer to sell a good for less than the price at which it is willing to buy that good: i.e., b g1 < a g1 .Otherwise, it would simultaneously buy and sell good g.We refer to these restrictions as bid monotonicity constraints.

Pseudo-Auction Rules
Equipped with this formalism, we can specify the rules that govern pseudo-auctions.As in a true auction, the outcome of a pseudo-auction dictates the quantity of each good to exchange, and at what prices, conditional on the agent's bid.The quantity issue is resolved by the winner determination rule whereas the price issue is resolved by the payment rule.Note that the monotonicity restrictions on bids ensure that the agent's offer is better than or equal to the price for every unit it exchanges, and that the agent does not simultaneously buy and sell any good.
There are at least two alternative payment rules an agent may face.In a first-price pseudo-auction, the agent pays its bid price (for buy offers, or is paid its bid price for sell offers) for each good it wins.In a second-price pseudo-auction, the agent pays (or is paid) the prevailing prices, as specified by the realized buyer and seller pricelines.This terminology derives by analogy from the standard first-and second-price sealed bid auctions (Krishna, 2002;Vickrey, 1961).In these mechanisms, the high bidder for a single item pays its bid (the first price), or the highest losing bid (the second price), respectively.The salient property is that in first-price pseudo-auctions, the price is set by the bid of the winner, whereas in second-price pseudo-auctions an agent's bid price determines whether or not it wins but not the price it pays.
In this paper, we focus on the second-price model.That is, our basic problem definitions presume second-price auctions; however, our bidding heuristics are not tailored to this case.As in true auctions, adopting the second-price model in pseudo-auctions simplifies the problem for the bidder.It also provides a reasonable approximation to the situation faced by TAC agents, as we now argue: • In TAC entertainment auctions, agents submit bids (i.e., buy and sell offers) of the form specified above.If we interpret an agent's buyer and seller pricelines as the current order book (not including the agent's own bid), then the agent's immediate winnings are as determined by the winner determination rule, and payments are according to the second-price rule (i.e., the order-book prices prevail).
• In TAC hotel auctions, only buy bids are allowed.Assuming once again an order book that reflects all outstanding bids other than the agent's own, an accurate buyer priceline would indicate that the agent can win k units of a good if it pays-for all k units-a price just above the (17 − k)th existing (other-agent) offer.The actual price it pays will be that of the 16th-highest unit offer (including its own offer).Since the agent's own bid may affect the price, 7 this situation lies between the first-and second-price characterizations of pseudo-auctions described above.
• In TAC flight auctions, agents may buy any number of units at the posted price.The situation at any given time is modeled exactly by the second-price pseudo-auction abstraction.

Bidding Problems
We are now ready to discuss the optimization module repeatedly employed by RoxyBot-06 within its bidding cycle to construct its bids.The key bidding decisions are: what goods to bid on, at what price, and when?
7. It can do so in two ways.First, the agent may submit the 16th-highest unit offer, in which case it sets the price.Second, when it bids for multiple units, the number it wins determines the price-setting unit, thus affecting the price for all winning units.Note that this second effect would be present even if the auction cleared at the 17th-highest price.
Although RoxyBot technically faces an n-stage stochastic optimization problem, it solves this problem by collapsing those n stages into only two relevant stages, "current" and "future," necessitating only one stochastic model of future prices (current prices are known).This simplification is achieved by ignoring the potentially useful information that hotel auctions close one by one in a random, unspecified order, and instead operating (like most TAC agents) under the assumption that all hotel auctions close at the end of the current stage.Hence, there is only one model of hotel prices: a stochastic model of future prices.Moreover, the only pressing decisions regarding hotels are what goods to bid on now and at what price.There is no need to reason about the timing of hotel bid placement.
contrast, since flight and entertainment auctions clear continuously, a trading agent should reason about the relevant tradeoffs in timing its placement of bids on these goods.Still, under the assumption that all hotel auctions close at the end of the current stage, in future stages, hotel prices, and hence hotel winnings, are known, so the only remaining decisions are what flight and entertainment tickets to buy.A rational agent will time its bids in these markets to capitalize on the "best" prices.(The best prices are the minima for buying and the maxima for selling.)Hence, it suffices for an agent's model of future prices in these markets to predict only the best prices (conditioned on current prices).That is, it suffices to consider only one stochastic pricing model.No further information is necessary.
Having established that it suffices for RoxyBot to pose and solve a two-stage, rather than an n-stage, stochastic optimization problem, we now proceed to define an abstract series of such problems that is designed to capture the essence of bidding under uncertainty in TAC-like hybrid markets that incorporate aspects of simultaneous and sequential, one-shot and continuously-clearing, auctions.More specifically, we formulate these problems as twostage stochastic programs with integer recourse (see the book by Birge & Louveaux, 1997, an introduction to stochastic programming).
In a two-stage stochastic program, there are two decision-making stages, and hence two sets of variables: first-and second-stage variables.The objective is to maximize the sum of the first-stage objectives (which depend only on the first-stage variables) and the expected value of the ensuing second-stage objectives (which can depend on both the first-and secondstage variables).The objective value in the second stage is called the recourse value, and if any of the second-stage variables are integer-valued, then the stochastic program is said to have integer At a high-level, the bidding problem can be formulated as a two-stage stochastic program as follows: in the first stage, when current prices are known but future prices are uncertain, bids are selected; in the second stage, all uncertainty is resolved, and goods are exchanged.The objective is to maximize the expected value of the second-stage objective, namely the sum of the inherent value of final holdings and any profits earned, less any first-stage costs.Since the second stage involves integer-valued decisions (the bidder decides what goods to buy and sell at known prices), the bidding problem is one with integer recourse.
In this section, we formulate a series of bidding problems as two-stage stochastic programs with integer recourse, each one tailored to a different type of auction mechanism, illustrating a different type of bidding decision.The mechanisms we study, inspired by TAC, are one-shot and continuously-clearing variants of second-price pseudo-auctions.In the former, bids can only be placed in the first stage; in the latter, there is an opportunity RoxyBot-06 for recourse.Ultimately, we combine all decision problems into one unified problem that captures what we mean by bidding under uncertainty.
In our formal problem statements, we rely on the following notation: • Variables: -Q 1 is a multiset of goods to buy now -Q 2 is a multiset of goods to buy later -R 1 is a multiset of goods to sell now -R 2 is a multiset of goods to sell later • Constants: -P 1 is a set of current buyer pricelines -P 2 is a set of future buyer pricelines -Π 1 is a set of current seller pricelines -Π 2 is a set of future seller pricelines Note that P 1 and Π 1 are always known, whereas P 2 and Π 2 are uncertain in the first stage but their uncertainty is resolved in the second stage.
Flight Bidding Problem An agent's task in bidding in flight auctions is to decide how many flights to buy now at current prices and later at the lowest future prices, given (known) current prices and a stochastic model of future prices.Although in TAC all units of each flight sell for the same price at any one time, we state the flight bidding problem more generally: we allow for different prices for different units of the same flight.
Definition 5.2 [Continuously-Clearing, Buying] Given a set of current buyer pricelines P 1 and a probability distribution f over future buyer pricelines P 2 , Note that there are two cost terms referring to future pricelines (Cost(•, P 2 )).The first of these terms adds the total cost of the goods bought in the first and second stages.The second term subtracts the cost of the goods bought in just the first stage.This construction ensures that, if an agent buys k units of a good now, any later purchases of that good incur the charges of units (k + 1, k + 2, ...) in the good's future priceline.
Entertainment Bidding Problem Abstractly, the entertainment buying problem is the same as the flight bidding problem.An agent must decide how many entertainment tickets to buy now at current prices and later at the lowest future prices.The entertainment selling problem is the opposite of this buying problem.An agent must decide how many tickets to sell now at current prices and later at the highest future prices.
Definition 5.3 [Continuously-Clearing, Buying and Selling] Given a set of current buyer and seller pricelines (P, Π) 1 and a probability distribution f over future buyer and seller pricelines (P, Π) 2 , The constraints ensure that an agent does not sell more units of any good than it buys.
Hotel Bidding Problem Hotel auctions close at fixed times, but in an unknown order.Hence, during each iteration of an agent's bidding cycle, one-shot auctions approximate these auctions well.Unlike in the continuous setup, where decisions are made in both the first and second stages, in the one-shot setup, bids can only be placed in the first stage; in the second stage, winnings are determined and evaluated.
Definition 5.4 [One-Shot, Buying] Given a probability distribution f over future buyer pricelines P 2 , HOT(f ) = max Hotel Bidding Problem, with Selling Although it is not possible for agents to sell TAC hotel auctions, one could imagine an analogous auction setup in which it were possible to sell goods as well as buy them.
Definition 5.5 [One-Shot, Buying and Selling] Given a probability distribution f over future buyer and seller pricelines (P, Π) 2 , max Bidding Problem Finally, we present (a slight generalization of) the TAC bidding problem by combining the four previous stochastic optimization problems into one.This abstract problem models bidding to buy and sell goods both via continuously-clearing and one-shot second-price pseudo-auctions, as follows: Definition 5.6 [Bidding Under Uncertainty] Given a set of current buyer and seller pricelines (P, Π) 1 and a probability distribution f over future buyer and seller pricelines (P, Π) 2 , RoxyBot-06 Once again, this bidding problem is (i) stochastic: it takes as input a stochastic model of future prices; (ii) global: it seamlessly integrates flight, hotel, and entertainment bidding decisions; and (iii) dynamic: it facilitates simultaneous reasoning about current and future stages of the game.
Next, we describe various heuristic approaches to solving the problem of bidding under uncertainty.

Bidding Heuristics
In this section, we discuss two heuristic solutions to the bidding problem: specifically, the expected value method (EVM), an approach that collapses stochastic information, and sample average approximation (SAA), an approach that exploits stochastic information and characterizes RoxyBot-06.

Expected Value Method
The expected value method (Birge & Louveaux, 1997) is a standard way of approximating the solution to a stochastic optimization problem.First, the given distribution is collapsed into a point estimate (e.g., the mean); then, a solution to the corresponding deterministic optimization problem is output as an approximate solution to the original stochastic optimization problem.Applying this idea to the problem of bidding under uncertainty yields: Definition 5.7 [Expected Value Method] Given a probability distribution f over buyer and seller pricelines, with expected values P 2 and Π2 , respectively, In practice, without full knowledge of the distribution f , we cannot implement the expected value method; in particular, we cannot compute P 2 or Π2 so we cannot solve BID EVM( P 2 , Π2 ) exactly.We can, however, solve an approximation of this problem in which the expected buyer and seller pricelines P 2 and Π2 are replaced by an average scenario ( P 2 , Π2 ) (i.e., average buyer and seller pricelines), defined as follows: Algorithm 5 EVM(G, N, f, S) Algorithm 6 SAA(G, N, f, S) 1: sample S scenarios (P, Π) 2 1 , . . ., (P, Π) 2 S ∼ f 2: β ⇐ BID SAA((P, Π) 2 1 , . . ., (P, Π) 2 S ) 3: return β Using the theory of large deviations, Ahmed and Shapiro (2002) establish the following result: as S → ∞, the probability that an optimal solution to the sample average approximation of a stochastic program with integer recourse is an optimal solution to the original stochastic optimization problem approaches 1 exponentially fast.Given hard time and space constraints, however, it is not always possible to sample sufficiently many scenarios to infer any reasonable guarantees about the quality of a solution to a sample average approximation.Hence, we propose a modified SAA heuristic, in which SAA is fed some tailor-made "important" scenarios, and we apply this idea to the bidding problem.

Modified Sample Average Approximation
The bids that SAA places are sample prices that appear in its scenarios.SAA never bids higher on any good than its highest sampled price, because as far as it knows, bidding that price is enough to win that good in all scenarios.However, there is some chance that the

RoxyBot-06
We built a test suite of agents, all of which predict using RoxyBot-06's SimAA random mechanism with distribution.The agents differ in their bidding strategies; the possibilities include SAA, 8 SAA*, and the six marginal-utility-based heuristics studied by Wellman et al. (2007), and summarized in Table 3.
Our experiments were conducted in a TAC Travel-like setting, modified to remove any aspects of the game that would obscure a controlled study of bidding.Specifically, we eliminated flight and entertainment trading, and endowed all agents with eight flights in and eight flights out on each day.Further, we assumed all hotels closed after one round of bidding (i.e., hotel auctions are one-shot, so that the ensuing bid optimization problem adheres to Definition 5.4).
We designed two sets of experiments: one decision-theoretic and one game-theoretic.In former, hotel clearing prices are the outcome of a simulation of simultaneous ascending auctions, but depend on the actual clients in each game, not a random sampling.(Our is more informed than the individual agents.)In the latter, hotel clearing prices are determined by the bids the agents submit using the same mechanism as in TAC Travel: the clearing price is the 16th highest bid (or zero, if fewer than 16 bids are submitted).
We first ran experiments with 8 agents per game, but found that hotel prices were often zero: i.e., there was insufficient competition.We then changed the setup to include a random number of agents drawn from a binomial distribution with n = 32 and p = 0.5, with the requisite number of agents sampled uniformly with replacement from the set of possible agents.The agents first sample the number of competitors from the binomial distribution, and then generate scenarios assuming the sampled number of competitors.
Because of the game-theoretic nature of TAC, an individual agent's performance can depend heavily on the other agents included in the agent pool.In our experiments, we attempted to mitigate any artificial effects of the specific agents we chose to include in our pool by sampling agents from the pool to play each game, with replacement.Thus, an agent's average score from the games is a measure of the agent's performance against various combinations of opponents.
In Figures 3(a) and 3(b), we plot the mean scores obtained by each agent type in each setting, along with 95% confidence intervals.These averages were computed based on 1000 independent observations, obtained by playing 1000 games.Scores were averaged across agent types in each game to account for any game dependencies.SAAB and SAAT 9 are the best performing agents in the game-theoretic experiments and among the best in the decision-theoretic setting.

TAC 2006 Competition Results
Table 4 lists the agents entered in TAC-06 and Table 5 summarizes the outcome.The TAC-06 finals comprised 165 games over three days, with the 80 games on the last day weighted 1.5 times as much as the 85 over the first two days.On the first day of the finals, RoxyBot finished third, behind Mertacor and Walverine-the top scorers in 2005.As it happens, RoxyBot's optimization routine, which was designed for stochastic hotel and entertainment 8.The particular implementation details explaining how RoxyBot-06 applied SAA in the TAC domain are relegated to Appendix A. 9. SAAB is SAA, and SAAT is a slight variant of SAA*.See the paper by Greenwald et al. (2008)   price predictions, was accidentally fed deterministic predictions (i.e., point price estimates) for entertainment.Moreover, these predictions were fixed, rather than adapted based on recent game history.On days 2 and 3, RoxyBot ran properly, basing its bidding in all auctions on stochastic information.Moreover, the agent was upgraded after day 1 to bid on flights not just once, but twice, during each minute.This enabled the agent to delay its bidding somewhat at the end of a game for flights whose prices are decreasing.No doubt this minor modification enabled RoxyBot to emerge victorious in 2006, edging out Walverine by a whisker, below the integer precision reported in Table 5.The actual margin was 0.22-a mere 22 parts in 400,000.Adjusting for control variates (Ross, 2002)    ulation.In this section, we assume this knowledge and view the output of the tâtonnement and SimAA calculations not as predictions but as ground truth.We compare the actual prices in the final games to this ground truth in respective years since 2002 to determine whether TAC market prices resemble CE prices.What we find is depicted in Figure 5.
Because of the nature of our methods, these calculations pertain to hotel prices only.
The results are highly correlated on both metrics (Euclidean distance and EVPP).We observe that the accuracy of CE price calculations has varied from year to year.2003 was the year in which TAC Supply Chain Management (SCM) was introduced.Many participants diverted their attention away from Travel towards SCM that year, perhaps leading to degraded performance in Travel.Things seem to improve in 2004 and 2005.We RoxyBot-06 cannot explain the setback in 2006, except by noting that performance is highly dependent on the particular agent pool, and in 2006 there were fewer agents in that pool.

Conclusion
The foremost aim of trading agent research is to develop a body of techniques for effective design and analysis of trading agents.Contributions to trading agent design include the invention of trading strategies, together with models and algorithms for realizing their computation and methods to measure and evaluate the performance of agents characterized by those strategies.Researchers seek both specific solutions to particular trading problems and general principles to guide the development of trading agents across market scenarios.This paper purports to contribute to this research agenda.We described the design and implementation of RoxyBot-06, an able trading agent as demonstrated by its performance in TAC-06.Although automated trading in electronic markets has not yet fully taken hold, the trend is well underway.Through TAC, the trading agent community is demonstrating the potential for autonomous bidders to make pivotal trading decisions in a most effective way.Such agents offer the potential to accelerate the automation of trading more broadly, and thus shape the future of commerce.
RoxyBot-06 q ∈ Q indexes the set of quantities (i.e., the units of each good in each auction).Equation ( 20) limits each client to one trip in each scenario.Equation ( 21) prevents the agent from allocating flights that it does not own or buy.Equation ( 22) prevents the agent from allocating hotels that it does not own or buy.Equation ( 23) prevents the agent from allocating event tickets that it does not own or buy and not sell.Equation ( 24) ensures the agent bids on at least HQW units in each hotel auction.Equation (25) prevents the agent from placing more than one buy offer per unit in each hotel auction.
An agent might also be constrained not to place sell offers on more units of each good than it owns, and/or not to place buy (sell) offers for more units of each good than the market supplies (demands).
Note that there is no need to explicitly enforce the bid monotonicity constraints in this ILP formulation: • "Buy offers must be nonincreasing in k, and sell offers nondecreasing." The ILP does not need this constraint because prices are assumed to be linear.In effect, the only decisions the ILP makes are how many units of each good to bid on.Hence, the bids (10, 15, 20) and (20, 15, 10) are equivalent.
• "An agent may not offer to sell for less than the price it is willing to buy."

REPEAT
{start bid interval } 0. Download current prices and winnings from server 1. predict: build stochastic models a. flights: Bayesian updating/learning b. hotels: simultaneous ascending auctions c. entertainment: sample historical data 2. optimize: sample average approximation 3. Upload current bids to server (three separate threads) {end bid interval } UNTIL game over

Figure 1 :
Figure1: EVPP and Euclidean Distance for the CE price prediction methods (tâtonnement and SimAA with α = 1 24 ; expected, random, and exact) and the TAC 2002 agents' predictions in the 2002 finals (60 games).The plot on the left shows that SimAA's predictions are better than tâtonnement's and that expected's are better than random's.RoxyBot-06's method of hotel price prediction (SimAA, Random) is plotted again on the right.Note the differences in scales between the two plots.
Definition 5.1 [Pseudo-Auction Winner Determination Rule] Given buyer and seller pricelines P and Π, and bid β = b, a , the agent buys the multiset of goods Buy(β, P ) and sells the multiset of goods Sell(β, Π), where Buy g (β, P ) = max k k such that b gk ≥ p gk Sell g (β, Π) = max k k such that a gk ≤ π gk

Figure 3 :
Figure 3: Mean scores and confidence intervals.

Figure 5 :
Figure 5: A comparison of the actual (hotel) prices to the output of competitive equilibrium price calculations in the final games since 2002.The label "exact" means: full knowledge of the client population.

sφ
∈ S indexes the set of scenarios.t ∈ T indexes the set of trips.A.2 Constants G at indicates the quantity of good a required to complete trip t.M a indicates the current buy price of a f , a e .N a indicates the current sell price of a e .Y as indicates the future buy price of a f , a h , a e in scenario s.Z as indicates the future sell price of a e in scenario s.H a indicates the hypothetical quantity won of hotel a h .O a indicates the quantity of good a the agent owns.U ct indicates client c's value for trip t.A.3 Decision Variables Γ = {γ cst } is a set of boolean variables indicating whether or not client c is allocated trip t in scenario s.Φ = {φ apq } is a set of boolean variables indicating whether to bid price p on the qth unit of a h .M = {µ a } is a set of integer variables indicating how many units of a f , a e to buy now.N = {ν a } is a set of integer variables indicating how many units of a e to sell now.Y = {υ as } is a set of integer variables indicating how many units of a f , a e to buy later in scenario s.Z = {ζ as } is a set of integer variables indicating how many units of a e to sell later in scenario s. + υ as ) ∀a ∈ A f , s ∈ S (21) allocation C,T γ cst G at ≤ own O a + buy Q,p≥Yas φ apq ∀a ∈ A h , s ∈ S apq ≥ H a ∀a ∈ A h (24) P φ apq ≤ 1 ∀a ∈ A h , q ∈ Q (25)

Table 2 :
Run times for the CE price prediction methods, in milliseconds.Experiments were run on AMD Athlon(tm) 64 bit 3800+ dual core processors with 2M of RAM.
Like the expected value method, sample average approximation is an intuitive way of approximating the solution to a stochastic optimization problem.The idea is simple: (i) generate a set of sample scenarios, and (ii) solve an approximation of the problem that incorporates only the sample scenarios.Applying the SAA heuristic (see Algorithm 6) involves solving the following approximation of the bidding problem: spreads the top two finishers a bit further. 10 Toulis et al., 2006;Kehagias et al., 2006 al., 2006;Kehagias et al., 200610.Kevin Lochner computed these adjustment factors using the method described byWellman et al. (2007,  ch.8).

Table 6 :
2006 Finals, Last day.Tabulated Statistics.We omit the first two days because agents can vary across days, but cannot vary within.Presumably, the entries on the last day are the teams' preferred versions of the agents.