The completion time analysis of Markov reward models with partial incremental loss is provided in this paper. The complexity of the model behaviour requires the use of an extra (supplementary) variable.

In robust Markov decision processes (MDPs), the uncertainty in transition kernel is addressed by finding a policy that optimizes worst-case performance over an set of MDPs. While much literature has focused on discounted MDPs, average-reward MDPs remain largely unexplored. this paper, we focus where goal to find average reward set. We first take approach approximates using prove value function ...

This paper presents a model-checking approach for analyzing discrete-time Markov reward models. For this purpose, the temporal logic probabilistic CTL is extended with reward constraints. This allows to formulate complex measures – involving expected as well as accumulated rewards – in a precise and succinct way. Algorithms to efficiently analyze such formulae are introduced. The approach is il...

MRMSolve is an analysis tool developed for the evaluation of large Markov Reward Models (MRM). The previous version of MRMSolve [8] provided only the moments of MRMs at arbitrary transient instant of time. This paper presents a new version of MRMSolve with new analysis features and software environment. The most important new functionality of MRMSolve is that it also makes distribution estimati...

Abstract. We extend the population continuous time Markov chain formalism so that the state space is augmented with continuous variables accumulated over time as functions of component populations. System feedback can be expressed using accumulations that in turn can influence the Markov chain behaviour via functional transition rates. We show how to obtain mean-field differential equations cap...

In this paper, we propose a framework to analyze Markov reward models, which are commonly used in system performability analysis. The framework builds on a set of analytical tools developed for a class of stochastic processes referred to as Stochastic Hybrid Systems (SHS). The state space of an SHS is comprised of: i) a discrete state that describes the possible configurations/modes that a syst...

Temporal-difference learning (TD) models explain most responses of primate dopamine neurons in appetitive conditioning. But because existing models are based in the simple formal setting of Markov processes, they do not provide a realistic account of the partial observability of the state of the world, nor of variation in event timing. For instance, the TD model of Montague et al. (1996) mispre...

Analysis of Markov Reward Models (MRM) with preemptive resume (prs) policy usually results in a double transform expression, whose solution is based on the inverse transformations both in time and reward variable domain. This paper discusses the case when the reward rates can be either 0 or positive, and analyses the completion time of MRMs. We present a symbolic expression of moments of the co...