Markov decision process (MDP) provides the foundations for a number of problems, such as artificial intelligence studying, automated planning and reinforcement learning. MDP can be solved efficiently in theory. However, for large scenarios, more investigations are needed to reveal practical algorithms. Algorithms for solving MDP have a natural concurrency. In this paper, we present parallel alg...

The TASEP (totally asymmetric simple exclusion process) studied here is a Markov chain on cyclic words over the alphabet {1, 2, . . . , n} given by at each time step sorting an adjacent pair of letters chosen uniformly at random. For example, from the word 3124 one may go to 1324, 3124, 3124, 4123 by sorting the pair 31, 12, 24, or 43. Two words have the same type if they are permutations of ea...

In this article, a novel acceptance-sampling plan is proposed to decide whether to accept or reject a receiving batch of items. In this plan, the items in the receiving batch are inspected until a nonconforming item is found. When the sum of two consecutive values of the number of conforming items between two successive nonconforming items falls underneath of a lower control threshold, the batc...

We discuss a class of Markov processes that occur, somewhat unexpectedly, in the construction of wavelet bases obtained from multiresolution analyses (MRA). The processes in question have been around for a long time. One of the first references that should be cited is a paper by Doeblin and Fortêt [10] (1937), entitled “Sur les chaînes à liaisons complètes”. In English, they are sometimes calle...

This paper proposes a new model—the Markov-modulated Hawkes process with stepwise decay (MMHPSD)—to investigate the variation in seismicity rate during a series of earthquake sequences including multiple main shocks. The MMHPSD is a self-exciting process which switches among different states, in each of which the process has distinguishable background seismicity and decay rates. Parameter estim...

Several physical and biological populations exhibit a branching character in their evolutionary behaviour in the sense that each individual (also called a particle ) of one type lives for some amount of time and then splits into several individuals of the same type or of different types. These processes are called cascade processes or reproductive processes or branching processes. Two important...

A probabilistic approach to the kinetics of the polymerase chain reaction (PCR) is developed. The approach treats the primer extension step of PCR as a microscopic Markov process in which the molecules of deoxy-nucleoside triphosphate (dNTP) are bound to the 3' end of the primer strand one at a time. The binding probability rates are prescribed by combinatorial rules in accord with the microsco...

We seek to find the robust policy that maximizes the expected cumulative reward for the worst case when a partially observable Markov decision process (POMDP) has uncertain parameters whose values are only known to be in a given region. We prove that the robust value function, which represents the expected cumulative reward that can be obtained with the robust policy, is convex with respect to ...

Markov switching models can be used to study heterogeneous populations that are observed over time. This paper explores modeling the group characteristics nonparametrically, under both homogeneous and nonhomogeneous Markov switching for group probabilities. The model formulation involves a finite mixture of conditionally independent Dirichlet process mixtures, with a Markov chain defining the m...

In the paper the integration of available results on SemiMarkov Decision Processes and on Markov Regenerative Processes is attempted, in order to de ne the mathematical framework for solving decision problems where the underlying structure state process is a Markov Regenerative Process, referred to as Markov Regenerative Decision Process. The essential question investigated here is which descri...