We discuss a generalization of the classical Gittins Index for a Markov chain and propose a transparent recursive algorithm for its calculation. The foundation for this algorithm is a modified version of the Elimination algorithm proposed earlier by the author to solve the problem of optimal stopping of a Markov chain in discrete time and a finite or countable state space.

A discrete-time control problem of a finite-state hidden Markov chain partially observed in a fractional Gaussian process is discussed using filtering. The control problem is then recast as a separated problem with information variables given by the unnormalized conditional probabilities of the whole path of the hidden Markov chain. A dynamic programming result and a minimum principle are obtai...

The major goal of this study was to create a continuous time Markov chain (CTMC) models of voltage gating of gap junction (GJ) channels formed of connexin protein. This goal was achieved by using the Piece Linear Aggregate (PLA) formalism to describe the function of GJs and transforming PLA into Markov process. Infinitesimal generator of CTMC was used to automate construction of Markov chain mo...

In this lecture we will discuss Markov Chains in continuous time. Continuous time Markov Chains are used to represent population growth, epidemics, queueing models, reliability of mechanical systems, etc. In Continuous time Markov Process, the time is perturbed by exponentially distributed holding times in each state while the succession of states visited still follows a discrete time Markov ch...

We consider a simple discrete-time Markov chain with values in [0,∞)Zd . The Markov chain describes various interesting examples such as oriented percolation, directed polymers in random environment, time discretizations of binary contact path process and the voter model. We study the phase transition for the growth rate of the “total number of particles” in this framework. The main results are...

We introduce the concepts of lumpability and commutativity of a continuous time discrete state space Markov process, and provide a necessary and sufficient condition for a lumpable Markov process to be commutative. Under suitable conditions we recover some of the basic quantities of the original Markov process from the jump chain of the lumped Markov process.

We consider a simple discrete-time Markov chain with values in [0,∞)Zd . The Markov chain describes various interesting examples such as oriented percolation, directed polymers in random environment, time discretizations of binary contact path process and the voter model. We study the phase transition for the growth rate of the “total number of particles” in this framework. The main results are...

Hidden Markov models have proved to be a very exible class of models, with many and diverse applications. Recently Markov chain Monte Carlo (MCMC) techniques have provided powerful computational tools to make inferences about the parameters of hidden Markov models, and about the unobserved Markov chain, when the chain is deened in discrete time. We present a general algorithm, based on reversib...

2 Continuous-Time Markov Chains 3 2.1 Time-homogeneous chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Time-inhomogeneous chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.3 Embedded Discrete-Time Markov Chain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.4 Conditional Expectations . . . . . . . . . . . . . . . . . . ...