In this lecture, we will introduce Markov chains and show a potential algorithmic use of Markov chains for sampling from complex distributions. For a finite state space Ω, we say a sequence of random variables (Xt) on Ω is a Markov chain if the sequence is Markovian in the following sense, for all t, all x0, . . . , xt, y ∈ Ω, we require Pr(Xt+1 = y|X0 = x0, X1 = x1, . . . , Xt = xt) = Pr(Xt+1 ...

This paper describes the use of spatially-sparse inputs to influence global changes in the behavior of Dynamic Movement Primitives (DMPs). The dynamics of DMPs are analyzed through the framework of contraction theory as networked hierarchies of contracting or transversely contracting systems. Within this framework, sparsely-inhibited rhythmic DMPs (SI-RDMPs) are introduced to both inhibit or en...

This paper is devoted to establishing sharp bounds for deviation probabilities of partial sums Σi=1f(Xi), where X = (Xn)n2N is a positive recurrent Markov chain and f is a real valued function defined on its state space. Combining the regenerative method to the Esscher transformation, these estimates are shown in particular to generalize probability inequalities proved in the i.i.d. case to the...

This paper is concerned with long-run average cost minimization of a stochastic inventory problem with Markovian demand, xed ordering cost, and convex surplus cost. The states of the Markov chain represent di erent possible states of the environment. Using a vanishing discount approach, a dynamic programming equation and the corresponding veri cation theorem are established. Finally, the existe...

Abstract. We introduce a class of time-inhomogeneous transition operators of Feynman-Kac type that can be considered as a generalization of symmetric Markov semigroups to the case of a time-dependent reference measure. Applying weighted Poincaré and logarithmic Sobolev inequalities, we derive L → L and L → L estimates for the transition operators. Since the operators are not Markovian, the esti...

Motivated by problems arising in time-dependent queues and dynamic systems with random environment, this work develops moderate deviations principles for dynamic systems driven by a fast-varying nonhomogeneous Markov chain in continuous time. A distinct feature is that the Markov chain is time dependent or inhomogeneous so are the dynamic systems. Under irreducibility of the nonhomogeneous Mark...

I will present an overview on the entropy rate of hidden Markov processes (HMP’s), information and coding-theoretic motivation for its study, and some of its connections to dynamical systems, to non-linear filtering, and to statistical physics. Particular attention will be given to: – Alternative representations: via the Blackwell measure, as a Lyapunov exponent, and as a partition function in ...

In a networked control system scenario, the packet dropout is usually modeled by time-invariant (homogeneous) Markov chain (MC) process. However, from practical point of view, probabilities loss can vary in time and/or probability parameter dependency. Therefore, to design fault detection filter (FDF) implemented semi-reliable communication network, it important consider variation network param...

We predict a mechanism for achieving complete population inversion in continuously driven InAs/GaAs semiconductor quantum dot featuring $V$-type transitions. This highly nonequilibrium steady state is enabled by the interplay between interband transitions and non-Markovian decoherence mechanism, introduced acoustic phonons. The trapping generalized to chain of coupled emitters. Exploiting inver...