We propose and analyze an algorithm that approximates solutions to the problem of optimal stopping in a discounted irreducible aperiodic Markov chain. The scheme involves the use of linear combinations of fixed basis functions to approximate a Q-function. The weights of the linear combination are incrementally updated through an iterative process similar to Q-Iearning, involving simulation of t...

We consider a Markov chain model for population growth subject to rare catastrophic events. In this model, the moves of the process are getting algebraically rare (as from x−λ) when the process visits large heights x, and given a move occurs and the height is large, the chain grows by one unit with large probability or undergoes a rare catastrophic event with small complementary probability ∼ γ...

A Markov chain with memory is no different from the conventional Markov chain on the product state space. Such a Markovianization, however, increases the dimensionality exponentially. Instead, Markov chain with memory can naturally be represented as a tensor, whence the transitions of the state distribution and the memory distribution can be characterized by specially defined tensor products. I...

In this paper, the two parameter ADK entropy, as a generalized of Re’nyi entropy, is considered and some properties of it, are investigated. We will see that the ADK entropy for continuous random variables is invariant under a location and is not invariant under a scale transformation of the random variable. Furthermore, the joint ADK entropy, conditional ADK entropy, and chain rule of this ent...

Suppose C is a closed recurrent communicating class for a Markov chain, if we restrict the chain on this class, the result is still a Markov chain with state space C. The resulting Markov chain is irreducible and recurrent. In this lecture, we will focus on the long-time behavior of irreducible and recurrent Markov chain. First, we have the following related definition. Definition 10.1. For a M...

Suppose C is a closed recurrent communicating class for a Markov chain, if we restrict the chain on this class, the result is still a Markov chain with state space C. The resulting Markov chain is irreducible and recurrent. In this lecture, we will focus on the long-time behavior of irreducible and recurrent Markov chain. First, we have the following related definition. Definition 10.1. For a M...

A sequence ofreal numbers (xn) is Benford if the significands, i.e., the fraction parts in the floating-point representation of (x ), are distributed logarithmically. Similarly, a discrete-time irreducible and aperiodic fi­ n nite-state Markov chain with transition probability matrix P and limiting matrix P' is Benford if every com­ ponent of both sequences of matrices (pn P') and (pn+1 pn) is ...

For many applications it is useful to sample from a nite set of objects in accordance with some particular distribution. One approach is to run an ergodic (i.e., irreducible aperiodic) Markov chain whose stationary distribution is the desired distribution on this set; after the Markov chain has run for M steps, with M suuciently large, the distribution governing the state of the chain approxima...

For many applications it is useful to sample from a nite set of objects in accordance with some particular distribution. One approach is to run an ergodic (i.e., irreducible aperiodic) Markov chain whose stationary distribution is the desired distribution on this set; after the Markov chain has run for M steps, with M suuciently large, the distribution governing the state of the chain approxima...