Decentralized Control of Cooperative Systems: Categorization and Complexity Analysis

Decentralized control of cooperative systems captures the operation of a group of decision makers that share a single global objective. The difficulty in solving optimally such problems arises when the agents lack full observability of the global state of the system when they operate. The general problem has been shown to be NEXP-complete. In this paper, we identify classes of decentralized control problems whose complexity ranges between NEXP and P. In particular, we study problems characterized by independent transitions, independent observations, and goal-oriented objective functions. Two algorithms are shown to solve optimally useful classes of goal-oriented decentralized processes in polynomial time. This paper also studies information sharing among the decision-makers, which can improve their performance. We distinguish between three ways in which agents can exchange information: indirect communication, direct communication and sharing state features that are not controlled by the agents. Our analysis shows that for every class of problems we consider, introducing direct or indirect communication does not change the worst-case complexity. The results provide a better understanding of the complexity of decentralized control problems that arise in practice and facilitate the development of planning algorithms for these problems.


Introduction
Markov decision processes have been widely studied as a mathematical framework for sequential decision-making in stochastic domains. In particular, single-agent planning problems in stochastic domains were modeled as partially observable Markov decision processes (POMDPs) or fully-observable MDPs (Dean, Kaelbling, Kirman, & Nicholson, 1995;Kaelbling, Littman, & Cassandra, 1998;Boutilier, Dearden, & Goldszmidt, 1995). Borrowing from Operations Research techniques, optimal plans can be computed for these planning problems by solving the corresponding Markov decision problem. There has been a vast amount of progress in solving individual MDPs by exploiting domain structure (e.g., Boutilier et al., 1995;Feng & Hansen, 2002). Approximations of MDPs have also been studied, for example, by Guestrin et al. (2003), assuming that the reward function can be decomposed into local reward functions each depending on only a small set of variables.
We are interested in a single Markov decision process that is collaboratively controlled by multiple decision-makers. The group of agents cooperate in the sense that they all want to maximize a single global objective (or minimize the cost of achieving it). Nevertheless, the decision-makers do not have full observability of the whole system at the time of execution. These processes can be found in many application domains such as multi-robot problems, flexible manufacturing and information gathering. For example, consider a group of space exploration rovers, like those sent by NASA to Mars. 1 These rovers could be assigned a set of experiments to perform on the planet before they need to meet. They may have a limited amount of time to perform these experiments. Then, the robots need to decide what experiments to perform and how much time they should invest in each one given the available battery power and remaining time. Decentralized cooperative problems also include information gathering systems, where several software agents with access to different servers may provide answers to a user's query. These agents' global objective is to give the user the best answer as early as possible, given the load on their servers and the preferences given by the user. Controlling the operation of the agents in such examples is not trivial because they face uncertainty regarding the state of the environment (e.g., the load on the communication link between the servers in the information gathering example may vary), the outcome of their actions (e.g., the rovers on Mars may be uncertain about their location and time needed to move from one target location to another), and the accuracy of their observations. All these types of uncertainty are taken into account when solving such decentralized problems.
The processes described above are examples of decentralized partially-observable Markov decision processes (Dec-POMDPs) or decentralized Markov decision processes (Dec-MDPs). 2 The complexity of solving these problems has been studied recently (Bernstein, Givan, Immerman, & Zilberstein, 2002;Pynadath & Tambe, 2002). Bernstein et al. have shown that solving optimally a Dec-MDP is NEXP-complete by reducing the control problem to the tiling problem. Rabinovich et al. (2003) have shown that even approximating the off-line optimal solution of a Dec-MDP remains NEXP. Nair et al. (2003) have presented the Joint Equilibrium-based Search for Policies (JESP) algorithm that finds a locally-optimal joint solution. Researchers have attempted to approximate the coordination problem by proposing on-line learning procedures. Peshkin et al. (2000) have studied how to approximate the decentralized solution based on a gradient descent approach for on-line learning (when the agents do not know the model). Schneider et al. (1999) assume that each decision-maker is assigned a local optimization problem. Their analysis shows how to approximate the global optimal value function when agents may exchange information about their local values at no cost. Neither convergence nor bounds have been established for this approach. Wolpert et al. (1999) assume that each agent runs a predetermined reinforcement learning algorithm and transforms the coordination problem into updating the local reward function so as to maximize the global reward function. Again, this is an approximation algorithm for online learning that does not guarantee convergence. Agents in this model may communicate freely. Guestrin et al. study off-line approximations (a centralized approach (Guestrin, Koller, & Parr, 2001) and a distributed approach (Guestrin & Gordon, 2002)), where a known structure of the agents' action dependencies induces a message passing structure. In this context, agents choose their actions in turns and communication is free. The solution is based on the assumption that the value function of the system can be represented by a set of compact basis functions, which are then approximated. The complexity of the algorithm is exponential in the width of the coordination graph. The order of elimination is needed beforehand because it has a great effect on the result. games where only one agent can send a single message at no cost. In contrast, we study sequential decision-making problems where all the agents can send various types of messages, which incur some cost. Our objective is to analyze the complexity of the problem and formulate algorithms that can find optimal policies of behavior as well as communication.
Our work focuses on decentralized cooperative MAS. Agents in cooperative MAS typically have limited ability to share information during execution (due to the distributed nature of the system). However, due to the cooperative nature of such systems, these constraints rarely apply to the pre-execution stage. Thus, we focus on cooperative agents that can share information fully at the off-line planning stage. Unlike the centralized approach, these agents will be acting in real-time in a decentralized manner. They must take this into account while planning off-line.
Sub-classes of Dec-POMDPs can be characterized based on how the global states, transition function, observation function, and reward function relate to the partial view of each of the controlling agents. In the simplest case, the global states can be factored, the probability of transitions and observations are independent, the observations combined determine the global state, and the reward function can be easily defined as the sum of local reward functions. In this extreme case we can say that the Dec-POMDP is equivalent to the combination of n independent MDPs. This simple case is solvable by combining all the optimal solutions of the independent MDPs. We are interested in more complex Dec-POMDPs, in which some or all of these assumptions are violated. In particular, we characterize Dec-POMDPs, which may be jointly fully-observable, may have independent transitions and observations and may result in goal-oriented behavior. We analyze the complexity of solving these classes off-line and show that it ranges from NEXP to P. We also identify different forms of information sharing and show that when direct communication is possible, exchanging local observations is sufficient to attain optimal decentralized control.
The contributions of the paper are as follows: formalizing special classes of decentralized control problems (Section 2), identifying classes of decentralized control that are critical in decreasing the complexity of the problem (Section 3), designing algorithms for controlling optimally a decentralized process with goal-oriented behavior (Section 4), and extending the formal framework by introducing direct communication, considering the trade-off between its cost and the value of the information acquired, and analyzing the complexity of solving optimally such problems (Section 5).

The Dec-POMDP Model
We are interested in a stochastic process that is cooperatively controlled by a group of decision-makers who lack a central view of the global state. Nevertheless, these agents share a set of objectives and all of them are interested in maximizing the utility of the system. The process is decentralized because none of the agents can control the whole process and none of the agents has a full view of the global state. The formal framework in which we study such decentrally controlled processes, called Dec-POMDPs, is presented below (originally presented by Bernstein et al., 2002). For simplicity of exposition, the formal model is presented for two agents, although it can be extended to any number. M = < S, A 1 , A 2 , P, R, Ω 1 , Ω 2 , O, T > where: • S is a finite set of world states with a distinguished initial state s 0 .
• A 1 and A 2 are finite sets of control actions. a i denotes an action performed by agent i.
• P is the transition probability function. P (s ′ |s, a 1 , a 2 ) is the probability of moving from state s ∈ S to state s ′ ∈ S when agents 1 and 2 perform actions a 1 and a 2 respectively. We note that the transition model is stationary, i.e., it is independent of time.
• R is the global reward function. R(s, a 1 , a 2 , s ′ ) represents the reward obtained by the system as a whole, when agent 1 executes action a 1 and agent 2 executes action a 2 in state s resulting in a transition to state s ′ .
• O is the observation function. O(o 1 , o 2 |s, a 1 , a 2 , s ′ ) is the probability of observing o 1 and o 2 (respectively by the two agents) when in state s agent 1 takes action a 1 and agent 2 takes action a 2 , resulting is state s ′ .
• If the Dec-POMDP has a finite horizon, it is represented by a positive integer T .
We will illustrate our definitions and results through the Meeting under Uncertainty example. In this scenario, we assume for simplicity that there are two robots operating on a two-dimensional grid. The state of the system is given by the locations of each one of the robots, s = [(x 1 , y 1 )(x 2 , y 2 )]. In a more general example, this state may include other features such as the topology of the terrain, which no rover may be able to sense (e.g., due to lack of equipment). The robots cannot observe each other, and the movement actions they can perform have uncertain outcomes (e.g., each robot will successfully move to the next location with some probability, but it may remain at the same location where it took the action). In a more uncertain case, the topology of the terrain, although hidden from the agents' observations may affect the resulting location of a moving action. The robots' objective is to minimize the time to meet. The observation of robot i corresponds to i's x and y coordinates. Solving optimally such a decentralized problem means finding the sequence of moves for each agent such that they meet as soon as possible.
Given the Dec-POMDP model, a local policy of action for a single agent is given by a mapping from sequences of observations to actions. In our example, a robot's local policy instructs it to take a certain movement action given the sequence of locations it has observed so far. A joint policy is a tuple composed of these local policies, one for each agent. To solve a decentralized POMDP problem one must find the optimal joint policy that is, the one with maximum value (for example given by the maximum expected accumulated global reward). By associating a small negative reward with each action, maximizing reward coincides with the objective of minimizing time to meet. Notice that the agents' observations can be dependent on each other, allowing the agents to know what other agents are observing and in some sense enabling the agents to obtain full observability of the system state. That is, even though the agents may not communicate directly, when the observations are dependent, agents may be able to obtain information about the others without receiving direct messages. For example, assume that in our scenario there are certain locations, which can host only one robot at a time. If one robot observes that it is located at any one of these sites, then it knows that the other robot cannot be located there even though this robot does not actually see the other nor receive any information from it.
In the next section, we characterize certain properties that a decentralized process may have. These properties will play an important role when analyzing the complexity of solving different classes of decentrally controlled cooperative problems.

Classes of Dec-POMDPs
It is known that solving optimally general decentralized problems is very hard. We are interested in identifying interesting subclasses of the general problem and their characteristics. As we show in Section 3, this classification reveals interesting complexity results and facilitates the design of algorithms. The first two categories that we define involve independence of the transitions or the observations of the agents. Figure 1 presents a snapshot of the system at times t and t + 1 that illustrates these categories. We assume that the global states are factored and we denote by s i the features of the world that may be observed fully by agent i. 3 Specifically, agent i senses observation o i , which in some cases can be identical to the local state of the agent. As a result of agent i performing action a i , when the system is at local state s i , the system transitions to state s ′ i , where the agent observes o i . The resulting state s ′ i is not affected by the resulting state of the other agent s ′ j . Similarly, the observations in these resulting states do not influence each other given the corresponding local information. The formal definitions for decentralized processes with independent transitions and observations follow.
In other words, the transition probability P of the Dec-POMDP can be represented as P = P 1 × P 2 , where P 1 = P r(s ′ 1 |s 1 , a 1 ) and P 2 = P r(s ′ 2 |s 2 , a 2 ).
The Meeting under Uncertainty example can be modified to present a problem with dependent transitions. For example, the scenario may include obstacles that can be moved by the agents when performing a move-obstacle action. The outcome of this action is that the rover's location is updated based on the direction of the move. For example, assume that one rover is located at (1,1) and finds an obstacle in location (1,2) which blocks its movement towards some goal. The rover decides to perform a move-obstacle action on that obstacle to the east. This action may have a stochastic outcome, for example with some probability P obstacle the rover succeeds and with probability 1 − P obstacle it remains in the same location (1,1). With this modification, the problem has dependent transitions. For example, if both rovers decide to apply a move-obstacle action to the same obstacle, standing each on different sides of the obstacle, then the resulting state of each rover depends on the other rover also performing a move-obstacle action.
Moreover, the observations of the agents can be independent, i.e., each agent's own observations are independent of the other agents' actions.
In the Meeting under Uncertainty example, if a robot's observation of its current location depends only on its transition from its previous location and on the action it performed, then the observations are independent. But more complex problems can arise if each agent's observation depends also on the other agent's location or action. For example, assume that in addition to the move actions, the rovers can also turn a flashlight that they hold on or off. In such cases, one rover's observation may be affected by the light or the lack of light in the area where the rover is observing. That is, the probability of a certain observation o depends on both rovers' actions and states.
Throughout the paper, when we refer to a Dec-POMDP with independent transitions and observations, we assume the same decomposition of the global states into S 1 and S 2 . We refer to S i as the partial view of agent i.
There are cases where agents may observe some common features of the global state, leading to dependent observations. When these common features have no impact on the transition model or reward, the problem can be reformulated to satisfy the property of independent observations. Such reformulation is shown to reduce the complexity of the problem in Section 3.
One of the main difficulties in solving Dec-POMDPs optimally results from the lack of full observability of the complete global state. An agent has full observability if it can determine with certainty the global state of the world from its local observation. For example, each time a robot observes where it is located, it also observes the other robot's location. Knowing both locations enables both agents to make the optimal decision about their next move in order to eventually meet sooner.
Definition 3 (Fully-observable Dec-POMDP) A Dec-POMDP is fully observable if there exists a mapping for each agent i, F i : This paper analyzes decentralized problems where full observability is not satisfied. Instead, we distinguish between two classes of problems with restricted system observability: (1) combining both agents' partial views leads to the complete global state, and (2) each agent's own partial view s i is fully observable. 4 We say that Dec-POMDPs with property (1) are jointly fully observable.
Definition 4 (Jointly Fully-observable Dec-POMDP) A Dec-POMDP is jointly fully observable (also referred to as a Dec-MDP) if there exists a mapping J : Notice that both Definitions 1 and 2 apply to Dec-MDPs as well as to Dec-POMDPs. The Meeting under Uncertainty scenario presented in Section 2 when the state includes only the rovers' locations (and it does not include the terrain topology) is actually a Dec-MDP. The global state is given by the two pairs of coordinates. There is no other feature in the system state that is hidden from the agents. Notice that even though the combination of the agents' observations results in the global state, each agent may still be uncertain about its own current partial view. Each agent may have a belief about its actual location. We define another class of problems, locally fully-observable Dec-POMDPs, where each agent is certain about its observations. General Dec-MDPs consider only the combination of the agents' observations, but the definition does not say anything about each agent's observation.
Definition 5 (Locally Fully-observable Dec-POMDP) A Dec-POMDP with independent transitions is locally fully observable if there exists a mapping for each agent i, The Meeting under Uncertainty example is locally fully observable because each robot knows with certainty where it is located. We may think of more realistic robots, which may be uncertain about their actual location due to hardware inaccuracies.
Notice that a jointly fully-observable process, which is also locally fully observable is not necessarily fully observable. In decentralized control problems we generally do not have full observability of the system. In the problems we consider, at best, the observations of all the agents combined determine with certainty the global state, and each such observation determines with certainty the partial view of each agent. The Meeting scenario is jointly fully observable and locally fully observable, but none of the agents know the complete state of the system. Our next lemmas show some interesting relations between the classes identified so far. These lemmas will help us show in Section 3 that certain classes of Dec-POMDPs are easier to solve. The classes identified so far correspond to practical real-world scenarios such as multi-rover scenarios, multi-agent mapping and manufacturing where loosely-coupled robots act to achieve a global objective.
Lemma 1 If a Dec-MDP has independent observations and transitions, then the Dec-MDP is locally fully observable.
Proof. We use properties of conditional probabilities established by Pearl (1988). The notation I(X, Y |Z) means that the set of variables X is independent of the set of variables Y given the set of variables Z. The properties we use are: • Decomposition -I(X, Y W |Z) ⇒ I(X, Y |Z) ∧ I(X, W |Z).
Using this notation, the property of independent transitions implies I({s ′ 1 }, {s 2 , a 2 , s ′ 2 }|{s 1 , a 1 }). The property of independent observations can be stated as I({o 2 }, {s 1 , a 1 , s ′ 1 , o 1 }|{s 2 , a 2 , s ′ 2 }). From the independent observations and the weak union properties, we obtain . From the independent observations and decomposition property, we obtain: . From the independent transitions, Equation 2 and the contraction properties, we obtain:

Applying the symmetry property to Equation 4, we obtain
Applying the symmetry property to Equation 1 and contracting with Equation 5, we obtain If we apply all of the above equations to the other agent (i.e., replace the index 2 with the index 1 and vice versa), then the following equation holds: Applying the decomposition property to Equation The lemma assumes a Dec-MDP, that is: P r(s ′ |o 1 , o 2 ) = 1. Since this probability is one, it is also true that P r(s ′ |s, a 1 , a 2 , o 1 , o 2 ) = 1. The lemma assumes independent transitions and observations, therefore the set of states is factored. Following conditional probabilities rules, we obtain: Equation 6 means that P r(s ′ 1 |s 1 , a 1 , o 1 , s 2 , a 2 , s ′ 2 , o 2 ) = P r(s ′ 1 |s 1 , a 1 , o 1 ). Equation 8 is equivalent to P r(s ′ 2 |s 1 , a 1 , o 1 , s 2 , a 2 , o 2 ) = P r(s ′ 2 |s 2 , a 2 , o 2 ). Therefore, So, each agent's partial view is determined with certainty by its observation and own transition, i.e., the Dec-MDP is locally fully observable. ✷ For the classes of problems studied in this lemma (Dec-MDPs with independent transitions and observations), a local policy for agent i in a locally fully-observable Dec-MDP is a mapping from sequences of states in agent i's partial view to actions. This differs from the general Dec-MDP case, where local policies are mappings from sequences of observations to actions. Formally, δ i : S * i → A i where S i corresponds to the decomposition of global states assumed in Definitions 1 and 2 for Dec-MDPs with independent transitions and observations. Moreover, we can show that an agent does not need to map a sequence of partial views to actions, but rather that it is sufficient to remember only the current partial view. This is shown in the next lemma.
Lemma 2 The current partial view of a state s observed by agent i (s i ) is a sufficient statistic for the past history of observations (o i ) of a locally fully-observable Dec-MDP with independent transitions and observations.
Proof. Without loss of generality we do all the computations for agent 1. We define I 1 t as all the information about the Dec-MDP process available to agent 1 at the end of the control interval t. This is done similarly to Smallwood and Sondik's original proof for classical POMDPs (Smallwood & Sondik, 1973). I 1 t is given by the action a 1t that agent 1 chose to perform at time t, the current resulting state s 1t , which is fully observable by agent 1 (s 1t = i 1 ), and the previous information I 1 t−1 . We assume a certain policy for agent 2, π 2 is known and fixed. π 2 (s t ) is the action taken by agent 2 at the end of control interval t.
We compute the belief-state of agent 1, which is the probability that the system is at global state j assuming only the information available to agent 1 (I 1 t ). This computation allows us to build a belief-state MDP for agent 1. Agent 1's optimal local policy is the solution that obtains the highest value over all the solutions resulting from solving all the belief-state MDPs built for each possible policy for agent 2.
We compute the probability that the system is in state s t = j = (j 1 , j 2 ) at time t, given the information available to agent 1:P r(s t = j|I 1 t ) = P r(s t = (j 1 , j 2 )| < a 1t , s 1t , π 2 (s t ), I 1 t−1 >). Applying Bayes rule leads to the following result: .
Since the Dec-MDP is locally fully observable, the denominator is equal to one. We expand the numerator by summing over all the possible states that could have lead to the current state j. P r(s t = (j 1 , j 2 ), s 1t = i 1 |a 1t , π 2 (s t ), I 1 t−1 ) = Σ s t−1 P r(s t−1 |a 1t , π 2 (s t ), I 1 t−1 )P r(s t = j|s t−1 , a 1t , π 2 (s t ), I 1 t−1 )P r(s 1t = i 1 |s t = j, s t−1 , a 1t , π 2 (s t ), I 1 t−1 ) The actions taken by the agents at time t do not affect the state of the system at time t − 1, therefore the first probability term is not conditioned on the values of the actions. The second probability term is exactly the transition probability of the Dec-MDP. Since the Dec-MDP has independent transitions, we can decompose the system transition probability into two corresponding probabilities P 1 and P 2 , following Definition 1. The last term is equal to one because the Dec-MDP is locally fully observable. Therefore, we obtain: . Since agent 1 fully observes s 1 = i 1 at time t, then the probability that the system is at state j and its first component j 1 is not i 1 is zero.
. Agent 1 can compute the last term for the fixed policy for agent 2. We conclude that the probability of the system being at state j at time t depends on the belief-state at time t−1. ✷ Following this lemma, s i , the current partial view of agent i is a sufficient statistic for the history of observations. Therefore, an agent does not need to remember sequences of observations in order to decide which actions to perform.
Corollary 1 Agent i's optimal local policy in a Dec-MDP with independent transitions and observations can be expressed as a mapping from agent i's current partial view and current time to actions. For the finite horizon case: The Meeting under Uncertainty scenario as described corresponds to a Dec-MDP with independent transitions and observations, therefore it is locally-fully observable. In such a case, for every possible location, a robot's decision about its next optimal move is not affected by the previous locations where the robot moved through.
We continue our classification of decentralized problems considering two additional dimensions: one is whether agents can share information and the other is whether the agents' behavior is goal-oriented. These classes are further described in the next two sections.

Information Sharing
We distinguish among three possible ways in which agents can share information: indirect communication, direct communication, and common uncontrollable features.
1. Indirect Communication -In the most general view, an action (a i ∈ A i ) performed by an agent can result in three different consequences, and thus it serves any of the following three purposes: information gathering, changing the environment and indirect communication. Agent i's actions can affect the observations made by agent j; these observations can serve as messages transmitted by agent i. Let's assume, for example, that a robot determines its location relative to the other robot's location. Then, the agents may have agreed on a meeting place based on their respective locations: If robot 1 sees robot 2 in location A, then they will meet at meeting place MA otherwise they will meet at meeting place MB. Even though the agents do not communicate directly, the dependencies between the observations can carry information that is shared by these agents.
It should be noted that when assuming dependencies between observations that result in information sharing through indirect communication, the general decentralized control problem already includes the problem of what to communicate and when. That is, indirect communication is established as a consequence of an action performed by an agent and the observations sensed by the other agents as a result. Independent of the policy, this type of communication is limited to transferring only information about the features of the state. In a more general context, the meaning of the communication can be embedded in the policy. That is, each time that it makes an observation, each agent can infer what was meant by the communication in the domain and in the policy. This type of communication is assuming that the observations of the agents are indeed dependent and this dependency is actually the means that enables each agent to transmit information.
2. Direct Communication -Information can be shared by the agents if they can send messages directly to each other (e.g., robot 1 sends a message to robot 2: "Bring tool T to location (x,y)"). In this case, the observations can be either dependent or independent. We analyze decentralized processes with direct communication further in Section 5.
3. Common Uncontrollable Features -This is knowledge about environmental features that can be acquired by both agents but are not affected by any of these agents' actions. This common knowledge exists when there are features in the system state that are affected by the environment independent of the agents' actions. An example of such feature is the weather (assuming that neither of the agents can have any effect on whether it rains or it shines). Then, information about the weather can be made available to both agents if they share the same feature. Agents can then act upon the conditions of the weather and thus coordinate their actions without exchanging messages directly. They may have already decided that when the sun shines they meet at location MA, and otherwise at location MB.
Given that the global set of states S is factored, a common feature S k is a feature of the global state that is included in the partial views of both agents.
Definition 6 (Common Uncontrollable Features) A common feature is uncontrollable if: It is important to note that the classes of problems that allow for uncontrollable features present an open problem that may be different from the categories studied in this paper due to the kind of dependencies that this knowledge may cause.
In this paper, we focus on either indirect communication or direct communication when we allow information sharing. We exclude from the discussion uncontrollable state features because this knowledge could provide a form of dependency between the agents that is beyond the scope of this paper.
Assumption 1 We assume that every change in the system results necessarily from the agents' actions. 5 Finally, the next section presents our last classification of decentralized problems that have goal-oriented behavior. This classification is practical in many areas where the agents' actions may incur some cost while trying to achieve a goal and may attain a global reward only when the goal is reached. This is different from most of the studies done on single-agent MDPs where a reward is obtained for every action performed.
5. Deterministic features that never change their values, or change their values in a deterministic way (such as time that increases in each step) are allowed.

Goal-oriented Behavior
We characterize decentralized processes in which the agents aim to reach specific global goal states. The Meeting under Uncertainty problem satisfies this requirement since the agents' goal is to meet at some location. Other practical scenarios may include assembling a machine, transferring objects from one location to a final destination, and answering to a query.
Definition 7 (Finite-horizon Goal-oriented Dec-MDPs (GO-Dec-MDP)) A finitehorizon Dec-MDP is goal-oriented if the following conditions hold: 1. There exists a special subset G of S of global goal states. At least one of the global goal states g ∈ G is reachable by some joint policy.
2. The process ends at time T (the finite horizon of the problem).
3. All actions in A incur a cost, C(a i ) < 0. For simplicity, we assume in this paper that the cost of an action depends only on the action. In general, this cost may also depend on the state.
5. If at time T , the system is in a state s ∈ G there is an additional reward JR(s) ∈ ℜ that is awarded to the system for reaching a global goal state.
A goal-oriented Dec-MDP has uniform cost when the costs of all actions are the same. When a goal oriented Dec-MDP has independent transitions and observations, we assume that there is a distinct action NOP with cost zero that has no effect on the local state and can only be performed at a global goal state component g i . That is, NOP ∈ A such that C(NOP) = 0 and P 1 (s ′ 1 |s 1 , NOP) is one when s ′ 1 = s 1 = g i 1 for some global goal g i . 6 When the GO-Dec-MDP has uniform cost and the set of actions include the NOP action, the uniform cost refers to all the actions different from NOP, i.e., ∀a i , a j ∈ A \ NOP, C(a i ) = C(a j ).
Solving a GO-Dec-MDP is the problem of finding a joint policy that maximizes the global value. The definition is concerned with global goal-oriented behavior; it does not necessarily imply that each agent separately must achieve certain goals. In the next section, we analyze the complexity of various classes of Dec-POMDPs based on the above characterization.

A Taxonomy of Decentralized POMDPs: Complexity Results
We have distinguished between Dec-POMDPs and Dec-MDPs (where joint full observability is assumed). In neither case do the agents have full observability of the global state (at most they have joint full observability and local full observability, which are different from full observability). Therefore, each one of the agents has a belief about the global state of the system, represented as a probability distribution over global states. Table 1 presents the information that each agent needs in order to update its belief about the global state of the 6. These NOP actions are necessary for agents that reach a component of a global goal state to "wait" for the other agent to complete its task.  Table 1: A summary of the information upon which an optimal local policy is conditioned. IT stands for independent transitions and IO for independent observations. system. Since each agent can solve its belief-state MDP assuming a fixed and known policy for the other agent, the information required by an agent to update each belief-state reflects the complexity of solving each class of corresponding decentralized control problems. All the complexity results presented in this section apply to decentralized processes controlled by n agents.
Conjecture 1 An optimal policy of a Dec-POMDP with independent transitions and observations depends on the entire sequence of observations.
Conjecture 2 An optimal policy of a Dec-MDP with independent transitions (but no independent observations) depends on the entire sequence of observations.
It is an open question whether any belief-update scheme must memorize all the observations, for example, when there is partial observability because the process is not jointly fully observable or because the observations are dependent (see the first three cases in Table 1). It should be noted that only for the last case in the table have we shown that s i is a sufficient statistic (see Lemma 2).
This section studies to what extent the classes characterized in the previous section differ in complexity. In all the results shown below we refer to the complexity of finding an optimal joint policy for the decentralized control problems handled in the lemmas. The lemmas are stated for the corresponding decision problems (i.e., given the decentralized process assumed in each one of the lemmas, the decision problem is to decide whether there is a joint policy whose value is equal or larger than a given constant K). Nevertheless, finding a solution cannot be easier than deciding the same problem.
All the results in this section correspond to problems given with a finite horizon T . It is already known that deciding a finite-horizon decentralized MDP is NEXP-complete (Bernstein et al., 2002). In Section 2.2, we described indirect communication, i.e., situations in which information can be shared when the observations are dependent. Therefore, the same complexity result applies to the decentralized control problem with indirect-communication, as stated in the next corollary.
Corollary 2 Deciding a Dec-MDP or a Dec-POMDP with indirect communication (i.e., allowing the agents to communicate by acting and observing, when the observations are dependent) is NEXP-complete.
We show in the next lemma that adding only goal-oriented behavior to a general decentralized process does not change the complexity of the problem. In other words, a goal-oriented decentralized problem is not easier than the general problem.
Lemma 3 Deciding a goal-oriented Dec-MDP is NEXP-complete.
Proof. This case can be proved through the same reduction applied by Bernstein et al. (2002). We can reduce the general goal-oriented Dec-MDP problem to the tiling problem by adding a goal state to the last state of the Dec-MDP defined in the reduction. The agents reach this new goal state and receive a reward of zero if the tiling is consistent. Otherwise the agents obtain a reward of -1 and do not reach the goal state (but they do reach a terminal state and the process ends).
The main reason for this complexity result relies on the fact that each agent needs to remember a sequence of observations that it has sensed (see Table 1). Adding only goal states to the decentralized process (without assuming any further assumptions) does not make the control problem any easier. ✷ Since a Dec-POMDP is more general than a Dec-MDP, the same lower bound for the Dec-MDP is valid.

Corollary 3 Deciding a goal-oriented Dec-POMDP is NEXP-complete.
The next lemma shows that by assuming that the transitions and observations are independent and that the agents have joint full observability, the problem of solving a decentralized cooperative system becomes easier (the lemma does not assume goal-oriented behavior).

Lemma 4 Deciding a Dec-MDP with independent transitions and observations is NPcomplete.
Proof. The Dec-MDP is locally fully observable because it has independent transitions and observations (see Lemma 1). We have shown in Lemma 2 that for such Dec-MDPs, the current partial view of an agent is a sufficient statistic. Therefore, a local policy of an agent i is of size polynomial in |S i |T . 7 There are |A i | |S i |T policies (mappings from S i and time to A i ). Each agent i needs to build a belief-state MDP with a number of states that is polynomial in |S i | (for a fixed and known policy for the other agent). Evaluating one such local policy can be done in polynomial time (by running dynamic programming on the belief-state MDP), but there is an exponential number of such policies for which this should be done. Therefore, the upper bound for the decision problem stated in this lemma is NP. Figure 2 shows schematically the differences in the policies, which lead to the difference in the complexity classes of the control problems when independent transitions and observations are assumed. When no assumptions are made (as in the leftmost figure), a local policy is represented by a tree, where each node corresponds to a possible action and each 7. We proved that si is a sufficient statistic, i.e., the current local state summarizes all the observations seen so far. However, since we are studying finite-horizon problems, a local policy of action is not stationary. Therefore, the time is indeed needed to decide upon an action. edge corresponds to a possible observation (i.e., a possible transition). In each local policy, the agent needs to remember a sequence of observations as opposed to just the last observation as in the rightmost figure. In the belief-state MDP that each agent builds, there is an exponential number of states that correspond to all the possible sequences of observations (this number is |Ω i | T , T is the finite horizon). Each such policy (of exponential size) can be evaluated with dynamic programming. There are a total of |A i | |Ω i | T local policies for which such a belief-state MDP need to be built by a brute-force search algorithm. 8 00 00 11 11 00 00 11 11 00 00 11 11 00 00 11 11 0 1 00 11 00 11 00 11 0 1 00 11 00 11 00 11 0 10 10 1 0 1 0 10 10 1 It is already known that a simple decentralized decision-making problem for two agents is NP-hard (where |A i | ≥ 2 and |A j | ≥ 3) (Papadimitriou & Tsitsiklis, 1982, 1986. Therefore, the lower bound for the problem class stated in the lemma is also NP. ✷ It is an open question whether a Dec-POMDP with independent transitions and observations (without joint full observability) results in a complexity class lower than NEXP.
Easier problems are obtained when the class of decentralized problems is restricted to goal-oriented behavior and does not include any type of information sharing, while maintaining the independence assumptions.
Lemma 5 Deciding a goal-oriented Dec-MDP with independent transitions and observations, with a single global goal state and with uniform cost is P-complete.
Proof. We argue that each agent should follow locally the policy that minimizes the cost to g i by solving a single-agent MDP. Under the uniform cost assumption, this is equivalent to minimizing the number of steps. Because computing these policies can be done with dynamic programming, the problem is P-complete.
In a given local state, in general, agents could follow the shortest path to g i or abandon it and do something more beneficial. Abandoning the goal is never beneficial because every course of action is equally valuable (there are no intermediate rewards and all actions' costs are the same). ✷ We extend this result to the case involving multiple global goal states. Unfortunately, it is not possible to avoid the need to change local goal states even when no information exchange is possible in GO-Dec-MDPs with independent transitions and observations. However, we will provide a necessary and sufficient condition for agents so that the need to change a goal never arises. Under this condition, the best strategy is to minimize the cost of reaching the local components of the single most beneficial goal. The analysis that we perform is only for the case of uniform cost of actions (i.e., all actions different from NOP incur the same cost). Without this assumption, it is sometimes beneficial to "waste time" and move away from goals because of the high cost/low likelihood to reach a goal.
Let δ i 1 be the best local policy for agent 1 to reach g i 1 (computed as if g i is the only goal). Similarly, we define δ i 2 for agent 2 to reach its corresponding component of global goal g i .
Let α s 1 i/j be the probability of reaching g j 1 from some state s 1 , when executing the policy δ i 1 . s 1 is some state reachable by policy δ i 1 . Similarly, let β s 2 i/j be the corresponding probability for agent 2.
Let C(s 1 , δ i 1 ) be the expected cost incurred by agent 1 when it executes policy δ i 1 starting from state s 1 . Similarly, C(s 2 , δ i 2 ) is defined for agent 2.
Property 1 (No Benefit to Change Local Goals) Let g i be a global goal state such that the joint policy (δ i 1 , δ i 2 ) has the highest value over all global goal states. A GO-Dec-MDP satisfies the NBCLG property if and only if the following two conditions hold: 1. ∀j = i, s 1 reachable f rom s 0 1 by δ i 1 , Note that the property guarantees that whenever an agent is at an intermediate local state, if the agent reevaluates the value of continuing to optimize its path to its goal i versus switching to another goal j, the value of continuing the policy to the original goal always remains the highest. Hence, it is never beneficial to change local goals. When the condition is not met, however, it is beneficial to change a local goal despite the fact that there is no information exchange between the agents.
Lemma 6 Deciding a goal-oriented Dec-MDP with independent transitions and observations with at least one global goal state and with uniform cost is P-complete when the NBCLG property is satisfied.
Sketch of Proof. For each global goal state g k ∈ G, agent 1 can compute its optimal local policy δ k 1 to reach the local component g k 1 . Similarly, agent 2 can compute its optimal local policy δ k 2 for every goal g k . This is achieved by solving single agent MDPs, each aimed at reaching a local component of a global goal state: M DP k 1 =< S 1 , A 1 , P 1 , R k 1 > and M DP k 2 =< S 2 , A 2 , P 2 , R k 2 >. S 1 and S 2 are given from the factored representation of the global states of the system. P 1 and P 2 result from the independent transitions assumption. The local reward R k 1 (s 1 , a 1 , s ′ 1 ) is the sum of the cost of an action, C(a 1 ) and an additional arbitrary reward attained at time T if the state s ′ 1 reached at T is g k 1 . R 2 is defined similarly for agent 2. We denote by g i the global goal state that attains the highest value when the agents follow the joint policy (δ i 1 , δ i 2 ). The NBCLG property implies that when agent 1 executes its optimal local policy δ i 1 , it cannot reach a state in which it is beneficial to abandon g i 1 and pursue a different local component g j 1 . (Without the property, this situation may arise despite the fact that agent 2 may continue to pursue g i 2 .) Note that this remains true even though the local policy δ i 1 may bring agent 1 eventually to a component of a goal that is different from the goal i. Since this property includes the same condition for both agents, it is also true that agent 2 will have no incentive to switch from an optimal local policy to goal g i , δ i 2 , once it starts executing it at the initial state. Therefore, the joint policy (δ i 1 , δ i 2 ) is the optimal solution to the given GO-Dec-MDP. Since computing δ i 1 and δ i 2 respectively can be done with dynamic programming, a GO-Dec-MDP with independent transitions and observations and with uniform cost that satisfies the NBCLG is P-complete. ✷ Assume that the Meeting scenario is given with a single meeting location (e.g., the lander in the rovers' case). Then, given the other independence assumptions, we can solve this problem optimally by building single-agent MDPs, each designed to achieve its component of the goal state (in the example, each robot needs to reach the lander). If there is a larger set of global goal states (e.g., when there is a finite number of possible meeting sites such as the lander, the space station on the planet and some other particular site) then following the lemma, we can let each agent find its local optimal policy to its corresponding component in these goal states. The optimal joint policy is the pair of local policies with the highest value.
A summary of the complexity results presented in this section appears in Figure 3.

Algorithms for Decentralized Control with No Information Sharing
So far, the only known algorithms for solving optimally decentralized control problems are the generalized version of dynamic programming for Dec-POMDPs (Hansen et al., 2004) and the Coverage-set algorithm (Becker et al., 2003) for Dec-MDPs with independent transitions and observations. The first algorithm solves optimally a general Dec-POMDP. Its practicality is restricted by the complexity of these problems (NEXP-complete). The Coverage-set algorithm assumes that the agents' actions could result in super-additive or sub-additive joint rewards as follows. In the first case, the reward obtained by the system from agents doing certain actions is larger than the sum of each agent's local reward for those actions.
In the second case, sub-additive joint rewards will be attained when the agents are penalized for doing redundant actions. As an example, we can look at a modified version of the Meeting scenario, where robots can move and also run experiments at different sites. Then, a process may lead to sub-additive rewards if both agents run the same experiment, wasting their resources instead of doing non-overlapping tasks. In other cases, the system may benefit when both robots perform the same tasks. For example, both agents run the same Dec-MDP Approx.

NEXP-C IO and IT
NP-C [Bernstein et al. 2002] [Rabinovich et al. 2003] [  [Bernstein et al. 2002] [Rabinovich et al. 2003] [  Figure 3: A summary of the complexity analysis for classes of decentralized control processes. We use the notation IT and IO for independent transitions and observations respectively. experiment at different times in the day, collecting eventually results with better quality.
The class of problems handled by the Coverage-set algorithm does not include necessarily goal-oriented decentralized processes. In this section, we present two tractable algorithms for controlling optimally Dec-MDPs with independent transitions and observations, which are also goal-oriented. A summary of the algorithms that are known to solve decentralized control problems optimally is presented in Table 2.

Single-goal, Goal-oriented Dec-MDPs
Following Lemma 5, the optimal solution for a GO-Dec-MDP with a single global goal state is computed by solving single agent MDPs aimed at the corresponding components of the given global goal state. Algorithm Opt1Goal is shown in Figure 5. Each agent i's MDP finds the least cost path to g i . Because the cost of actions (different from NOP) is uniform any value set to GR(g i ) is suitable to solve the problems.

Many-goals, Goal-oriented Dec-MDPs
Due to the uncertainty about the outcomes of actions, an agent may decide to change its intention with respect to the global goal state it is planning to reach. The algorithm that optimally and decentrally solves a goal-oriented Dec-MDP problem with many global goal states is OptN Goals, presented in Figure 5. Assuming the conditions of Lemma 6, this algorithm is optimal because the lemma guarantees that changing local goals is not beneficial.
9. No algorithm was proposed short of full search with complexity NP as shown in Lemma 9.
It is possible to extend the algorithm to non-distinctive global goal states, by running the algorithm over all possible subsets of goals G 1 , G 2 such that ∀s 1 ∈ G 1 , s 2 ∈ G 2 : (s 1 , s 2 ) ∈ G.
Run Value Iteration to compute V δ (s) for all s ∈ S:

Decentralized Control with Communication
Direct communication can be beneficial in decentralized control because the agents lack full observability of the global state. That is, the value of the optimal joint policy that allows communication may be larger than the value of the optimal joint policy without communication. We are interested in solving a decentralized control problem off-line taking into account direct communication. Agents will consider this expected information while computing their optimal joint policy, thus deriving a policy for when and what to communicate.
If we assume that direct communication leads to full observability of the system state that direct communication is free and that the observations are independent then obviously the agents will benefit most by communicating constantly. This results in a fully observable decentralized process, which is equivalent to an MMDP (Boutilier, 1999). This problem is known to be P-complete (Papadimitriou & Tsitsiklis, 1987).
In real-world scenarios, it is reasonable to assume that direct communication has indeed an additional cost associated with it; the cost may reflect the risk of revealing information to competitive agents, the bandwidth necessary for the transmission or even the complexity of computing the information to be transferred. Therefore, communication may not be possible or even desirable at all times.
We extend the model of decentralized partially-observable Markov decision process to include an explicit language of communication with an associated cost. 10 We call this model Dec-POMDP-Com. It is given by the following tuple: < S, A 1 , A 2 , Σ, C Σ , P, R, Ω 1 , Ω 2 , O, T >.
Σ denotes the alphabet of messages and σ i ∈ Σ represents an atomic message sent by agent i (i.e., σ i is a letter in the language). σ i denotes a sequence of atomic messages. A special message that belongs to Σ is the null message, which is denoted by ǫ σ . This message is sent by an agent that does not want to transmit anything to the other agents. 11 C Σ is the cost of transmitting an atomic message: C Σ : Σ → ℜ. The cost of transmitting a null message is zero. Communication cost models determine the flow of the information exchange and the cost of this communication. These models may include, for example, one-way communication models, in which the cost C Σ is incurred by each agent that sends information to another agent and two-way communication models, where agents exchange messages when at least one of them initiates communication, and the cost is incurred only once each time (we refer to these agents as jointly exchanging messages). Other models may require additional messages like acknowledgments that may incur additional costs.
We restrict ourselves in this paper to communication cost models based on joint exchange of messages and to communication that leads to full observability of the global state. The agents send messages by broadcasting, and only one message is sent at each time. The agents in the system share the same language of communication. In a separate line of research, we are addressing the question of agents controlling a decentralized process where the agents develop a mutual understanding of the messages exchanged along the process (Goldman, Allen, & Zilberstein, 2004). Direct communication is the only means of achieving full observability when the observations are independent and when there is no common uncontrollable features (Assumption 1).
We define a Dec-MDP-Com as a Dec-POMDP-Com with joint full observability, as we did with Dec-POMDPs and Dec-MDPs in Section 2.1. The Dec-POMDP-Com model can have independent transitions, independent observations, be locally fully observable, and goal-oriented as the basic model presented in Section 2. 10. The model was presented by Goldman and Zilberstein (2003). 11. We omit in this paper the details of the communication network that may be necessary to implement the transmission of the messages.
We describe the interaction among the agents as a process in which agents perform an action, then observe their environment, and then send a message that is instantaneously received by the other agent. 12 Then, we can define the local policies of the controlling agents as well as the resulting joint policy whose value we are interested in optimizing. A local policy δ i is composed of two policies: δ A i that determines the actions of the agents and δ Σ i that states the communication policy. Notice that δ A i allows indirect communication if the observations of the agents are dependent and that δ Σ i allows direct communication even when the observations are dependent.
Definition 8 (Local Policy for Action) A local policy for action for agent i, δ A i , is a mapping from local histories of observations o i = o i 1 , . . . , o it over Ω i and histories of messages σ j = σ j 1 , . . . , σ jt , 13 received (j = i) since the last time the agents exchanged information to actions in A i .
Definition 9 (Local Policy for Communication) A local policy for communication for agent i, δ Σ i , is a mapping from local histories of observations o i = o i 1 , . . . , o it and o, the last observation perceived after performing the last local action, over Ω i and histories of messages σ j = σ j 1 , . . . , σ jt , received (j = i) since the last time the agents exchanged information to messages in Σ.
More complex cases result if the agents could communicate partial information about their partial views. This is left for future work.
Definition 10 (Joint Policy) A joint policy δ = (δ 1 , δ 2 ) is defined as a pair of local policies, one for each agent, where each δ i is composed of the communication and the action policies for agent i.
The optimal joint policy that stipulates for each decision-maker how it should behave and when it should communicate with other agents is the policy that maximizes the value of the initial state of the Dec-POMDP-Com. We first study the general problem, when no particular assumptions are made on the class of the Dec-POMDP-Com. We will then study certain classes of this problem as we did with the case without communication.
12. When agents exchange information there is a question whether information is obtained instantaneously or there are delays. For simplicity of exposition we assume no delays in the system. 13. The notation o = o1, . . . , ot and oo represents the sequence o1, . . . , oto. Similarly, the notation for sequences of messages: σiσ represents the sequence σi 1 , . . . , σi t σ.
Then, the value of the initial state s 0 of the Dec-POMDP-Com after following a joint policy δ for T steps can be defined as follows: Definition 12 (Value of an Initial State Given a Policy) The value V T δ (s 0 ) after following policy δ = (δ 1 , δ 2 ) from state s 0 for T steps is given by: where the observation and the message sequences are of length at most T − 1, and both sequences of observations are of the same length l. The sequences of messages are of length l + 1 because they considered the last observation resulting from the control action prior to communicating.
The problem of decentralized control with direct communication is to find an optimal joint policy δ * for action and for communication such that δ * = argmax δ V T δ (s 0 ).

Languages of Communication
We start showing that under some circumstances the language of observations is as good as any other communication language. In the Meeting scenario, no matter what are the tasks assigned to the system, agents that exchange their current coordinates are guaranteed to find the optimal solution to the decentralized problem.
Theorem 1 Given a Dec-MDP-Com with constant message cost, the value of the optimal joint policy δ * with respect to any Σ, V T δ * ,Σ (s 0 ) is not greater than the value of the optimal joint policy with respect to the language of observations (Σ = Ω). That is: Proof. The decentralized process is jointly fully observable. Therefore, it is not beneficial for the agents to send any information in addition to their observations (we assume joint exchange of messages). Thus, combining both agents' observations results in the complete global state. Moreover, the theorem assumes a constant cost for every message, i.e., all non-null messages incur the same cost: there are not any messages that are either more expensive or cheaper to transmit than others. Therefore, the agents cannot benefit from exchanging information that is a strict subset of their partial views because the cost of sending any message is equal. ✷ We note that the theorem does not hold when different messages may incur different costs. In this case, sending less information might be cheaper, but equally valuable. For example, when agents observe their respective x and y coordinates, they may benefit from sending only one coordinate if it costs less than sending the complete location. Agents may also benefit from sending functions of their observations if this incurs a smaller cost. For example, agents may benefit from exchanging information about the Manhattan distance between their current location and some mutually-known location.
In general, it seems reasonable to introduce a language of communication to reduce complexity, but as the theorem shows, this cannot guarantee optimality when the language is comprised of messages different from the agents' observations. Examples of such messages include: 1) commitments, which are constraints on the future behavior of the message sender, 2) instructions, which are constraints on the future behavior of the message receiver, and 3) feedback that is an encouraging or punishing signal that is sent to another agent. The study of Dec-POMDP-Com problems with languages of communication different from observations is left for future work. Similarly, certain protocols of communication can restrict the optimal value of the policy of communication but may be easier to implement.

The Effect of Communicating on the Complexity Analysis
The complexity results we obtained in Section 3 apply also for the same classes of problems when direct communication is possible. Although agents achieve full observability each time they exchange information, the problem of finding the policy of communication offline (when there is a cost associated with each communication act) remains as hard as the general problem with no communication. In the worst case, transmitting the messages can be prohibitively expensive. Therefore, adding direct communication does not simplify the problem. For all the cases shown to be in NEXP, adding direct communication cannot make them more difficult. The complexity of deciding a Dec-MDP when observations are independent and direct communication is allowed remains the same as when direct communication is not assumed, as shown in Lemma 9. The impact of direct communication on the classes of Dec-POMDPs with independent transitions and observations and with possible goal-oriented behavior remains an open question.
It is interesting to note that the decentralized control problem with direct communication can be reduced to the same problem with indirect communication when the observations are dependent. We assume that transmitting messages incur the same cost and that the language of messages is the language of observations. If the language of communication is different then the reduction does not apply. Theorem 1 showed that exchanging observations is sufficient to guarantee that the joint policy computed will be optimal. The next lemma shows that the optimal policy of communication will instruct the agent to transmit only its current observation or the null message.
Lemma 8 Given a Dec-MDP-Com with constant message cost, there is an optimal policy of communication such that whenever a non-null message is sent, it must be the agent's last observation.
Proof. This lemma results from Theorem 1. In a jointly-fully observable process, sending a non-null message that is an observation different from the last one cannot provide more information about the current state of the process than the last observation does. ✷ Since the current global state becomes fully observable each time that the agents communicate, all the necessary information is stored in the synchronized state s; the agents do not need to remember all the messages received so far when the decentralized process is jointly fully observable. When the problem is framed as a Dec-MDP-Com with independent transitions and observations, the current local state is fully observable. Thus, the local policies of action and communication for such a Dec-MDP-Com can be formalized as follows: Corollary 4 An optimal local policy of action for this problem, δ A i , can be represented as a mapping from synchronized states, current partial views, and time to actions.
Similarly, an optimal local policy of communication δ Σ i can be represented as a mapping from synchronized states, current partial views, and time to two possible messages: either the current partial view or the null message.
Agents need to remember only their current partial view and the last synchronized information to decide on their next action. This is a primary observation that affects the complexity of deciding Dec-MDP-Com with independent transitions and observations as shown in the next lemma.
Lemma 9 Deciding a Dec-MDP-Com with independent transitions and observations is NPcomplete.
Proof. Following Corollary 4, each agent's policy is of size polynomial in |S|, and the number of possible policies is 2 |S| 2 T × |A| |S| 2 T . In the worst case, a brute force algorithm can go through all the possible policies for agent 1 and for each one of them compute the optimal policy for agent 2. Agent 2 builds its belief-state MDP, where each node is the agent's belief that the global state is a state s. There is an edge for any possible action and message that the agents can choose. Agent 2 can choose any action a 2 ∈ A 2 and it can either send a null message, or a message with its last observation (s 2 ). For any possible policy of action and communication of agent 1, agent 2 can build such a belief-state MDP and solve it. This can also be done in time that is polynomial in the number of the belief-states. Whenever an agent sends a non-null message, then the belief-state MDP has a transition to a state that is fully observable with probability one. In any case, each agent needs only to remember its last current partial view, so the complexity of solving a Dec-MDP-Com with independent observations and transitions is in the NP class. In the worst case, the cost of sending a message can be prohibitively large, therefore policies that do not send a message need also to be evaluated. It is already known (Papadimitriou & Tsitsiklis, 1982, 1986) that a simple decentralized decision-making problem for two agents is NP-hard (where |A i | ≥ 2 and |A j | ≥ 3). Therefore, the lower bound for the problem class stated in the lemma is also NP. ✷ The proof of this lemma is based on the number of policies that need to be evaluated by a naive algorithm. Limiting ourselves to GO-Dec-MDP with direct communication does not change the number of policies. Therefore, we obtain the same complexity for the goaloriented case with direct communication and independent transitions and observations: Corollary 5 Deciding a GO-Dec-MDP-Com with independent transitions and observations is NP-complete.
To address the complexity of decentralized control with communication, we propose a practical approximation technique . This approach is based on meta-level control of communication, motivated by a similar decision-theoretic approach to meta-level reasoning that was developed by Russell and Wefald (1991). We assume that the designer of the system also designs a mechanism for communication. This mechanism stipulates how to decompose the global problem into local (single-agent) and temporary problems that depend on the information held by the agents whenever they exchange messages.

Discussion
Decentralized control problems are very intriguing from a theoretical point of view as well as from a practical point of view. From a theoretical perspective, decentralized partiallyobservable Markov decision processes serve as a formal framework to study the foundations of multi-agent systems (e.g., Becker et al., 2003;Hansen et al., 2004;Guestrin & Gordon, 2002;Peshkin et al., 2000;Pynadath & Tambe, 2002;Claus & Boutilier, 1998). Our study focuses on computing off-line decentralized policies of control for cooperative systems. This paper analyzes the complexity of solving these problems optimally for certain classes of decentralized control. We found critical transitions in complexity between classes of problems that range from NEXP to P. The critical transition occurs when we assume that the decentralized problems have independent transitions and observations. Adding only goal-oriented behavior to a decentralized problem does not simplify it, unless the transitions and observations are independent. We extended the decentralized process model to enable direct communication among the agents, taking into account that this communication incurs a certain cost. Communication allows the agents to synchronize their knowledge and thus eliminate the uncertainty about the global state of the world (at least at certain times). We also analyzed the complexity of the decentralized problems with indirect and direct communication.
From a practical perspective, decentralized control problems appear frequently in realworld applications where the decision-makers may be robots placed at separate geographical locations or computational processes distributed in information space. The classes of Dec-POMDPs that we identify seem to match many practical applications. Independent transitions and observations arise in examples such as multi-agent mapping, flexible manufacturing, and multiple-rovers working on data-collection in uncertain terrains, when the agents' actions are not strongly coupled. Goal-oriented behavior is relevant in these examples when the agents' behavior is aimed at reaching specific states, for example the assembly of a particular machine in a manufacturing process, or the retrieval of information.
We analyzed the notion of information sharing in decentralized systems by distinguishing among three possible sources for information: indirect communication attained by agents observing dependent observations, direct communication achieved by adding an external language of communication, and common uncontrollable features. The typical distinction previously made in the literature is between systems with no communication and systems with a predefined language of communication, which typically does not incur any costs, overlooking the fact that dependent observations offer yet another form of communication (Pynadath & Tambe, 2002;Decker & Lesser, 1992;Grosz & Kraus, 1996;Durfee, 1988;Roth, Vail, & Veloso, 2003). The problem of combining communication acts into the decision problem of a group of cooperative agents was addressed by Xuan et al. (2001). Their framework is similar to ours but their approach is heuristic. We proved that the language of the observations is sufficient in order to reach an optimal decentralized solution (assuming all the messages incur the same cost). This leads to the understanding that any other type of communication can serve as an approximation to the optimal solution, which may be easier to obtain.
In addition to presenting a formal framework of decentralized control, we introduced tractable algorithms for solving optimally certain classes of Dec-MDPs. The Coverageset algorithm that solves optimally decentralized MDPs with a certain reward structure appeared in (Becker et al., 2003). Here, we add two optimal algorithms aimed at goaloriented decentralized control.
The contribution of this paper is in framing and categorizing fundamental issues in decentralized control of cooperative systems. In particular, we characterized and studied the complexity of goal-oriented behavior, jointly fully-observable processes and independent transitions and observations, which result in important and practical classes of control problems. We also studied three sources for information sharing in such decentralized systems and provided algorithms that compute optimal solutions. Future research will focus on algorithms for decentralized control with direct communication of messages that are different from the observations. In particular, we will examine more general models of communication that allow exchange of partial information and involve unreliable communication.