Excerpted Section A. MARKOV CHAIN APPROXIMATION Another approach to finite differences is the well developed Markov Chain Approximation (MCA) of Kushner [3, 4]. Recent developments are surveyed and further advanced by Kushner [5], and by Kushner and Dupuis [6], with special attention to methods for jump and reflected diffusions. This method applies a Markov chain approximation to continuous tim...

Hidden Markov models have proved to be a very exible class of models, with many and diverse applications. Recently Markov chain Monte Carlo (MCMC) techniques have provided powerful computational tools to make inferences about the parameters of hidden Markov models, and about the unobserved Markov chain, when the chain is deened in discrete time. We present a general algorithm, based on reversib...

Motivated by problems arising in time-dependent queues and dynamic systems with random environment, this work develops moderate deviations principles for dynamic systems driven by a fast-varying nonhomogeneous Markov chain in continuous time. A distinct feature is that the Markov chain is time dependent or inhomogeneous so are the dynamic systems. Under irreducibility of the nonhomogeneous Mark...

Abstract Two GI/M/1 type Markov chains associated with the queue length are often used in analyzing the discrete time MAP/PH/K queue. The first Markov chain is introduced by tracking service phases for servers; a method we call TPFS. The transition probability matrix of the Markov chain can be constructed in a straightforward manner. The second Markov chain is introduced by counting servers for...

We introduce the concepts of lumpability and commutativity of a continuous time discrete state space Markov process, and provide a necessary and sufficient condition for a lumpable Markov process to be commutative. Under suitable conditions we recover some of the basic quantities of the original Markov process from the jump chain of the lumped Markov process.

Usually, in monitoring a proportion p < /em>, the binary observations are considered independent; however, in many real cases, there is a continuous stream of autocorrelated binary observations in which a two-state Markov chain model is applied with first-order dependence. On the other hand, the Bernoulli CUSUM control chart which is not robust to autocorrelation can be applied two-sided co...

In this note we explore an aspect of the relationship between the notion of equilibrium of a continuous time Markov chain (CTMC) and that of the traditional concept of termination in rewriting systems. Unlike in deterministic dynamical systems, a Markov chain equilibrium is not a definite state, but rather a probability over the state space which is invariant under the Markov semigroup, and sat...

In this paper algorithms for model checking CSL (continuous stochastic logic) against infinite-state continuous-time Markov chains of so-called quasi birth-death type are developed. In doing so we extend the applicability of CSL model checking beyond the recently proposed case for finite-state continuous-time Markov chains, to an important class of infinite-state Markov chains. We present synta...