We study the online estimation of the optimal policy of a Markov decision process (MDP). We propose a class of Stochastic Primal-Dual (SPD) methods which exploit the inherent minimax duality of Bellman equations. The SPD methods update a few coordinates of the value and policy estimates as a new state transition is observed. These methods use small storage and has low computational complexity p...

Hurricanes Katrina and Rita showed the vulnerability of coastal communities and how human activities that caused deterioration of the Mississippi Deltaic Plain (MDP) exacerbated this vulnerability. The MDP formed by dynamic interactions between river and coast at various temporal and spatial scales, and human activity has reduced these interactions at all scales. Restoration efforts aim to re-e...

In this paper we study the Call Admission Control (CAC) and routing issue for ATM networks which carry integrated CBR/VBR and ABR traffic. The integration of CBR/VBR and ABR traffic is assumed to be based on the max-min fairness criterion. The CAC and routing task is formulated as a Markov decision problem (MDP) where the objective is to maximize the revenue from carried calls. The MDP is solve...

We consider discrete-time factorial Markov Decision Processes (MDPs) in multiple decision-makers environment for infinite horizon average reward criterion with a general joint reward structure but a factorial joint state transition structure. We introduce the “localization” concept that a global MDP is localized for each agent such that each agent needs to consider a local MDP defined only with...

We study the computational complexity of the infinite-horizon discounted-reward Markov Decision Problem (MDP) with a finite state space S and a finite action space A. We show that any randomized algorithm needs a running time at least Ω(|S||A|) to compute an -optimal policy with high probability. We consider two variants of the MDP where the input is given in specific data structures, including...

Numerous communications have indicated that specific binding proteins for high density lipoprotein (HDL) exist in addition to the well characterized candidate HDL receptor SR-BI, but structural information was presented only in a few cases, and most of the work was aimed at the liver and steroidogenic glands. In this study, we purified two HDL-binding proteins by standard procedures from rat lu...

Liposomes containing encapsulated lymphokines or muramyl dipeptide (MDP), when injected i.v. into C57BL/6 mice, pro duce significant destruction of established lung and lymph node métastasesfrom a s.c. highly metastatic B16-BL6 mela noma. We present evidence that eradication of the métastases is mediated by the activation of host macrophages to the tumoricidal state. Results from three separa...

Spatial puzzles composed of rigid objects, flexible strings and holes offer interesting challenges for reasoning about spatial entities that are common in the human daily-life’s activities. This motivates use as domains study this work. The goal paper is to investigate automated solution kind problems by extending an algorithm combines Answer Set Programming (ASP) with Markov Decision Process (...

The SmartWheeler project aims at developing an intelligent wheelchair for handicapped people. In this paper, we model the dialogue manager of SmartWheeler in MDP and POMDP frameworks using its collected dialogues. First, we learn the model components of the dialogue MDP based on our previous works. Then, we extend the dialogue MDP to a dialogue POMDP, by proposing two observation models learned...

This paper proposes a new model, the EMDP (Evidential Markov Decision Process). It is a MDP (Markov Decision Process) for belief functions in which rewards are defined for each state transition, like in a classical MDP, whereas the transitions are modeled as in an EMC (Evidential Markov Chain), i.e. they are sets transitions instead of states transitions. The EMDP can fit to more applications t...