In this paper, we obtain the Rényi entropy rate for irreducible-aperiodic Markov chains with countable state space, using the theory of countable nonnegative matrices. We also obtain the bound for the rate of Rényi entropy of an irreducible Markov chain. Finally, we show that the bound for the Rényi entropy rate is the Shannon entropy rate.

In this paper, the two parameter ADK entropy, as a generalized of Re'nyi entropy, is considered and some properties of it, are investigated. We will see that the ADK entropy for continuous random variables is invariant under a location and is not invariant under a scale transformation of the random variable. Furthermore, the joint ADK entropy, conditional ADK entropy, and chain rule of this ent...

Consider a discrete-time Markov chain (Xn : n > 0) ∼ Markov(λ, P ) with countable state space S. In the last lecture, we proved the following theorem for aperiodic Markov chains. Theorem (Aperiodic case). Assume P is irreducible and positive recurrent with unique invariant measure π. If, in addition, P is aperiodic, then P(Xn = j)→ πj as n→∞ for all j ∈ S. In particular, p (n) ij → πj as n→∞ fo...

We develop in finite case sufficient conditions for weak ergodicity of a nonstationary Markov chain r-states having transition matrices (Pn) n≥1 with Pn → P, where P has 2 irreducible and aperiodic closed classes and,possibly, transient states.

In this paper, the ambiguity of nite state irreducible Markov chain trajectories is reminded and is obtained for two state Markov chain. I give an applicable example of this concept in President election

We consider an irreducible and aperiodic Markov chain {kn}n=0 over the finite state space E = {1, . . . , p} with positive regular transition matrix P = {pij} and additive component {Un} such that {Sn} = {(kn, Un)} is also a Markov chain over the state space E1 = E × R. We prove a central and a local limit theorem for this chain when the probability density functions of {Sn}, conditional on the...

It is well known that any irreducible and aperiodic Markov chain has exactly one stationary distribution, and for any arbitrary initial distribution, the sequence of distributions at time n converges to the stationary distribution, that is, the Markov chain is approaching equilibrium as n → ∞. In this paper, a characterization of the aperiodicity in existential terms of some state is given. At ...

The important ideas related to a Markov chain can be understood by just studying its graph, which has nodes corresponding to states and edges corresponding to nonzero entries in the transition matrix. Figure 1 helps us to summarize key ideas. The first part of this figure shows an irreducible Markov chain on states A,B,C. The graph in this case is strongly connected, i.e., one can move from any...

We propose a variant of temporal-difference learning that approximates average and differential costs of an irreducible aperiodic Markov chain. Approximations are comprised of linear combinations of fixed basis functions whose weights are incrementally updated during a single endless trajectory of the Markov chain. We present a proof of convergence (with probability 1), and a characterization o...

We propose a variant of temporal di erence learning that approximates average and di erential costs of an irreducible aperiodic Markov chain Approximations are comprised of linear combinations of xed basis functions whose weights are incrementally updated during a single endless trajectory of the Markov chain We present a proof of convergence with probability and a characterization of the limit...