This paper presents an iterative technique for the transient analysis of large structurally unrestricted semi-Markov processes (SMPs), which builds on our previous work on iterative passage time calculation. The method is based on the calculation and subsequent numerical inversion of Laplace transforms. We demonstrate our technique on a Markovian process algebra model of a web-server with 69 44...

This paper addresses the problem of dialogue optimization on large search spaces. For such a purpose, in this paper we propose to learn dialogue strategies using multiple Semi-Markov Decision Processes and hierarchical reinforcement learning. This approach factorizes state variables and actions in order to learn a hierarchy of policies. Our experiments are based on a simulated flight booking di...

Semi-Markov decision processes are used to formulate many control problems and also play a key role in hierarchical reinforcement learning. In this chapter we show how to translate the decision making problem into a form that can instead be solved by inference and learning techniques. In particular, we will establish a formal connection between planning in semiMarkov decision processes and infe...

One general strategy for approximately solving large Markov decision processes is \divide-and-conquer": the original problem is decomposed into sub-problems which interact with each other, but yet can be solved independently by taking into account the nature of the interaction. In this paper we focus on a class of \policy-coupled" semi-Markov decision processes (SMDPs), which arise in many nons...

We consider the problem of minimizing the long-run average expected cost per unit time in a semiMarkov decision process with arbitrary state and action space. Using the idea of successive approximations, sufficient conditions for the existence of an optimal stationary policy are given. These results are applied to solve the replacement problem with a semi-Markov shock model.

Continuous time random walks (CTRWs) are versatile models for anomalous diffusion processes that have found widespread application in the quantitative sciences. Their scaling limits are typically non-Markovian, and the computation of their finitedimensional distributions is an important open problem. This paper develops a general semi-Markov theory for CTRW limit processes in d-dimensional Eucl...

The notion of infinite order Markov process is introduced and the Markov property of the flow of information is established.

This work deals with the asymptotic properties of maximum likelihood estimators for semi-Markov processes parametric sojourn time distributions. It is motivated by comparison, via a two-sample test procedure, distribution two panels qualitative trajectories modeled and observed over random number transitions. Considering first one panel growing size, we derive, under classical conditions, conve...