There are situations where decisions must be made in sequence, and the result of each decision is not clear to the decision maker. These situations can be formulated mathematically as Markov decision processes, and given the probabilities of each value, it is possible to determine a policy that maximizes the expected outcome of a sequence of decisions. This tutorial explains Markov decision pro...

In this paper we introduce the calculus of interactive generalized semi-Markov processes (IGSMPs), a stochastic process algebra which can express probabilistic timed delays with general distributions and synchronizable actions with zero duration, and where choices may be probabilistic, non-deterministic and prioritized. IGSMP is equipped with a structural operational semantics which generates s...

Semi-Markov and modulated renewal processes provide a large class of multi-state models which can be used for analysis of longitudinal failure time data. In biomedical applications, models of this kind are often used to describe evolution of a disease and assume that patient may move among a finite number of states representing different phases in the disease progression. Several authors propos...

In this paper we propose a new method for identifying processing stages in human information processing. Since the 1860s scientists have used different methods to identify processing stages, usually based on reaction time (RT) differences between conditions. To overcome the limitations of RT-based methods we used hidden semi-Markov models (HSMMs) to analyze EEG data. This HSMM-EEG methodology c...

Hidden-Mode Markov Decision Processes (HM-MDPs) were proposed to represent sequential decision-making problems in non-stationary environments that evolve according to a Markov chain. We introduce in this paper Hidden-Semi-Markov-Mode Markov Decision Processes (HS3MDPs), a generalization of HM-MDPs to the more realistic case of non-stationary environments evolving according to a semi-Markov chai...

Markov regenerative processes (MRGPs) constitute a more general class of stochastic processes than traditional Markov processes. Markovian dependency, the first-order dependency, is the simplest and most important dependency in stochastic processes. Past history of a Markov chain is summarized in the current state and the behavior of the system thereafter only depends on the current state. Sojo...

The aim of this paper is to present a systematic modeling of reliability and related measures: availability, maintainability, failure rate, rate of occurrence of failures, mean times, etc., known in the literature under the term dependability. This model includes the continuous and discrete time semi-Markov processes with general state space. This is one of the most general models in reliabilit...

The notion of N-ple Markov process is defined in a quite general framework and it is shown that N-ple Markov processes-arel inear combinationso f some martingales