Jump Markov linear systems (JMLSs) are linear systems whose parameters evolve with time according to a finite state Markov chain. Given a set of observations, our aim is to estimate the states of the finite state Markov chain and the continuous (in space) states of the linear system. In this paper, we present original deterministic and stochastic iterative algorithms for optimal state estimatio...

Consider a continuous time, finite state Markov chain X = {Xt, t ∈ [0, T ]}. We identify the states of this process with the unit vectors ei in R N , where N is the number of states of the chain. We consider stochastic processes defined on the filtered probability space (Ω, F , {Ft}, P), where {Ft} is the completed natural filtration generated by the σ-fields Ft = σ({Xu, u ≤ t}, F ∈ FT : P(F ) ...

This paper presents the analysis of a renewal input  finite buffer queue wherein the customers can decide either to  join the queue with a probability or balk. The service process is Markovian service process ($MSP$) governed  by an underlying $m$-state Markov chain. Employing the supplementary  variable and imbedded Markov chain techniques,   the steady-state system length distributions at pre...

A new stochastic epidemic model, that is, a general continuous time birth and death chain model, is formulated based on a deterministic model including vaccination. We use continuous time Markov chain to construct the birth and death process. Through the Kolmogorov forward equation and the theory of moment generating function, the corresponding population expectations are studied. The theoretic...

For each n let Y n t be a continuous time symmetric Markov chain with state space n −1 Z d. A condition in terms of the conductances is given for the convergence of the Y n t to a symmetric Markov process Yt on R d. We have weak convergence of {Y n t : t ≤ t0} for every t0 and every starting point. The limit process Y has a continuous part and may also have jumps.

A brief introduction to the formulation of various types of stochastic epidemic models is presented based on the well-known deterministic SIS and SIR epidemic models. Three different types of stochastic model formulations are discussed: discrete time Markov chain, continuous time Markov chain and stochastic differential equations. Properties unique to the stochastic models are presented: probab...

The n-coalescent is a continuous-time Markov chain on a finite set of states, which describes the family relationships among a sample of n members drawn from a large haploid population. Its transition probabilities can be calculated from a factorization of the chain into two independent components, a pure death process and a discrete-time jump chain. For a deeper study, it is useful to construc...

One of the basic facts known for discrete-time Markov decision processes is that, if probability distribution an initial state fixed, then every policy it easy to construct a (randomized) with same marginal distributions state-action pairs as original policy. This equality implies that values major objective criteria, including expected discounted total costs and average rewards per unit time, ...