Chapter 2 reviewed some aspects of the elementary theory of general nite irreducible Markov chains. In this chapter we specialize to reversible chains, treating the discrete and continuous-time cases in parallel. After section 3 we shall assume that we are dealing with reversible chains without continually repeating this assumption, and shall instead explicitly say \general" to mean not-necessa...

Canonical forms of Markovian distributions and processes provide an efficient way of describing these structures by eliminating the redundancy of general representations. Canonical forms of order 2 stationary Markov arrival processes (MAPs) have already been established for both continuous and discrete time. In this paper we present canonical form of continuous time non-stationary MAPs of order...

Random walk graph and Markov chain based models are used heavily in many data and system analysis domains, including web, bioinformatics, and queuing. These models enable the description and analysis of various behaviors of stochastic systems. If the system being modeled has certain properties, such as if it is irreducible and aperiodic, close form formulations corresponding to its stationary b...

We obtain universal estimates on the convergence to equilibrium and the times of coupling for continuous time irreducible reversible finite-state Markov chains, both in the total variation and in the L norms. The estimates in total variation norm are obtained using a novel identity relating the convergence to equilibrium of a reversible Markov chain to the increase in the entropy of its one-dim...

Motivated by the recent applications of the conjugate residual method to nonsymmetric linear systems by Sogabe, Sugihara and Zhang [An extension of the conjugate residual method to nonsymmetric linear systems. J. Comput. Appl. Math., Vol. 266, 2009, pp. 103–113], this paper describes two conjugate direction methods, BiCR and BiCG, and attempts to extend their applications to compute the station...

We study the classical problem of noisy constrained capacity in the case of the binary symmetric channel (BSC), namely, the capacity of a BSC whose inputs are sequences chosen from a constrained set. Motivated by a result of Ordentlich and Weissman [In Proceedings of IEEE Information Theory Workshop (2004) 117–122], we derive an asymptotic formula (when the noise parameter is small) for the ent...

In this paper, we consider a class of new accelerated multilevel aggregation methods using two polynomial-type vector extrapolation methods, namely the reduced rank extrapolation (RRE) and the generalization of quadratic extrapolation (GQE) methods. We show how to combine the multilevel aggregation methods with the RRE and GQE algorithms on the finest level in order to speed up the numerical co...

Randomized search heuristics are widely used for solving combinatorial optimization problems. Randomized search heuristics can often be modeled, either directly, or after some simple modifications, as families of ergodic Markov chains, where each chain has a goal state that corresponds to the optimum. Such a family is termed successful if every chain, starting from any state, reaches its goal s...

We consider a dynamic version of sender–receiver games, where the sequence of states follows an irreducible Markov chain observed by the sender. Under mild assumptions, we provide a simple characterization of the limit set of equilibrium payoffs, as players become very patient. Under these assumptions, the limit set depends on the Markov chain only through its invariant measure. The (limit) equ...

One of the proposed solutions to the equilibrium selection problem for agents learning in repeated games is obtained via the notion of stochastic stability. Learning algorithms are perturbed so that the Markov chain underlying the learning dynamics is necessarily irreducible and yields a unique stable distribution. The stochastically stable distribution is the limit of these stable distribution...