Certain discrete-time Markov processes on locally compact metric spaces which arise from graphdirected constructions of fractal sets with place-dependent probabilities are studied. Such systems naturally extend finite Markov chains and inherit some of their properties. It is shown that the Markov operator defined by such a system has a unique invariant probability measure in the irreducible cas...

We discuss the temporal-difference learning algorithm, as applied to approximating the cost-to-go function of an infinite-horizon discounted Markov chain. The algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic Markov chain with a finite or infinite state space. We present a proof of convergence (with pr...

We discuss the temporal-di erence learning algorithm, as applied to approximating the cost-to-go function of an in nite-horizon discounted Markov chain. The algorithm we analyze updates parameters of a linear function approximator on{line, during a single endless trajectory of an irreducible aperiodic Markov chain with a nite or in nite state space. We present a proof of convergence (with proba...

We discuss the temporal di erence learning algorithm as applied to approximating the cost to go function of an in nite horizon discounted Markov chain The algorithm we analyze updates parameters of a linear function approximator on line during a single endless traject ory of an irreducible aperiodic Markov chain with a nite or in nite state space We present a proof of convergence with probabili...

Consider a monoecious diploid population with nonoverlapping generations, whose size varies with time according to an irreducible, aperiodic Markov chain with states x1N, . . . , xKN , where K N . It is assumed that all matings except for selfing are possible and equally probable. At time 0 a random sample of n N genes is taken. Given two successive population sizes xjN and xiN , the numbers of...

The stationary distribution is uniform: πi = 1/(kn) for i ∈ V . The Markov chain is connected and therefore irreducible, moreover since it has self loops and is undirected it is also aperiodic. Hence, it converges to π. To apply the flow method, we have to route one unit of flow, for each pair (u, v) of vertices. We do this as follows: if u and v are connected via an edge e, then we route the u...

The problem of maximizing the input-output mutual information rate for a Markov channel is cast as a problem of controlling a partially observed controlled Markov chain on a suitable state space with time-averaged reward. This establishes the equivalence of its channel capacity with the value of a certain dynamic program. 1. Introduction. The problem of characterizing and computing the capacity...

We illustrate how a technique from the theory of random iterations of functions can be used within the theory of products of matrices. Using this technique we give a simple proof of a basic theorem about the asymptotic behavior of (deterministic) “backwards products” of row-stochastic matrices and present an algorithm for perfect sampling from the limiting common rowvector (interpreted as a pro...

Abstract Under a compactness assumption, we show that a φ-irreducible and aperiodic MetropolisHastings chain is geometrically ergodic if and only if its rejection probability is bounded away from unity. In the particular case of the Independence Metropolis-Hastings algorithm, we obtain that the whole spectrum of the induced operator is contained in (and in many cases equal to) the essential ran...

Let P = [pij] be the probability probability transition matrix of Markov chain, so that pij is the probability that node i contacts node j. We assume that P is aperiodic, irreducible. It may or may not be symmetric. P is double–stochastic matrix, i.e. the sum of elements in each row or column is equal to 1. By the Perron–Frobenius theorem, P has a stationary distribution π = [πi] π = πP (1) whe...