Recent research indicates that Markov decision processes (MDPs) can be viewed from a sensitivity point of view; and perturbation analysis (PA), MDPs, and reinforcement learning (RL) are three closely related areas in optimization of discrete-event dynamic systems that can be modeled as Markov processes. The goal of this paper is two-fold. First, we develop PA theory for semi-Markov processes (S...

Received We show that the category whose objects are famidclies of Markov processes on Polish spaces, with a given transition kernel, and whose morphisms are transition probability preserving, surjective continuous maps has semi-pullbacks, i.e. for any pair of morphisms fi : Si ! S (i = 1; 2), there exists an object V and morphisms i : V ! Si (i = 1; 2) such that f1 1 = f2 2. This property hold...

An Edgeworth expansion of a Studentized statistic for an ergodic regenerative strong Markov process is validated. A specific nonparametric bootstrap method is proposed and proved to be second-order correct in the light of the Edgeworth expansion, which is a variant of the regenerative block bootstrap designed for discrete-time Markov processes. One-dimensional diffusions and semi-Markov process...

In this paper, finite phase semi-Markov processes are introduced. By introducing variables and a simple transformation, every finite phase semi-Markov process can be transformed to a finite Markov chain which is called its associated Markov chain. A consequence of this is that every phase semi-Markovian switching system may be equivalently expressed as its associated Markovian switching system....

This article gives a short presentation of random evolutions. At first, the following two examples are presented: dynamical stochastic systems and increment processes both in Markov media. After, an introduction to semi-Markov Random evolution in a Banach space is given, where the previous evolutionary systems are obtained as particular cases. Finally, two abstract limit theorems of average and...

In this paper, we extend to our knowledge, for the first case, some reliability results obtained using homogeneous semi-Markov processes to the case of non-homogenous semi-Markov modelisation. Moreover, we apply some of our preceding results to give the numerical solutions and so the possibility to treat real life problems for which non-homogeneity in time is important.

This paper provides sufficient conditions when certain information about the past of a stochastic decision processes can be ignored by a controller. We illustrate the results with particular applications to queueing control, control of semi-Markov decision processes with iid sojourn times, and uniformization of continuous-time Markov decision processes.

With the increasing complexity of multiprocessor and distributed processing systems, the need to develop efficient and accurate modeling methods is evident. Fault tolerance and degradable performance of such systems has given rise to considerable interest in models for the combined evaluation of performance and reliability [l], [2]. Markov or semi-Markov reward models can be used to evaluate th...

Semi-Markov processes have found increasing applications in modeling the kinetics of single enzyme molecules. Detailed balance is a widely accepted condition for Markov models of closed chemical systems and well known to be equivalent to the reversibility of a stationary Markov process. We show that for a semi-Markov process detailed balance is only a necessary condition, but not sufficient, fo...