We examine project scheduling with net-present-value objective and exponential activity durations, using a continuous-time Markov decision chain. Based on a judicious partitioning of the state space, we achieve a significant performance improvement compared to the existing algorithms.

In the context of Markov decision processes running in continuous time, one of the most intriguing challenges is the efficient approximation of finite horizon reachability objectives. A multitude of sophisticated model checking algorithms have been proposed for this. However, no proper benchmarking has been performed thus far. This paper presents a novel and yet simple solution: an algorithm, o...

In this paper, the ambiguity of nite state irreducible Markov chain trajectories is reminded and is obtained for two state Markov chain. I give an applicable example of this concept in President election

A. A stochastic process is a collection of random variables {X t , t ∈ T }. B. A sample path or realization of a stochastic process is the collection of values assumed by the random variables in one realization of the random process, e.g. C. The state space is the collection of all possible values the random variables can take on, i.e. it is the sample space of the random variables. For example...

We study special Quantum Markov chains on a Fock space related to iterated beam splittings as introduced in [23]. Besides a characterizatioin of the position distributions of the chain, we show some kind of weak convergence of such discrete time Quantum Markov chains to a kind of continuous time Quantum Markov process. Furthermore, we provide existence and uniqueness for the solution of a quant...

In many applications, for example when computing statistics of fast subsystems in a multiscale setting, we wish to find the stationary distributions of systems of continuous-time Markov chains. Here we present a class of models that appears naturally in certain averaging approaches whose stationary distributions can be computed explicitly. In particular, we study continuous-time Markov chain mo...

This paper is concerned with distributed H∞ filtering for a class of continuous-time linear plants over sensor networks with multiple communication channels (MCCs). A practical framework is presented to optimize communication over MCCs with uncertain delays and switching characteristics. The channel switching is assumed to follow a continuous-time Markov chain and a Markov jump linear system (M...

It is the object of this paper to draw together certain lines of research which during the last decade have grown out of the problem of characterizing the functions which can arise as transition probabilities of continuous time Markov chains. This problem is now solved (see Sections 8 and 9). although as usual its solution has thrown up further problems which demand attention. The evolution of ...