General additive functions called rewards are defined on a "regular" finite-state Markov-renewal process. The asymptotic form of the mean total reward in [0,t] has previously been obtained, and it is known that the total rewards are joint-normally distributed as t -► oo. This paper finds the dominant asymptotic term in the covariance of the total rewetrds as a simple function of the moments of ...

Markov Decision Processes (MDPs) model controlled stochastic systems. Like Markov chains, an MDP consists of states and probabilistic transitions; unlike Markov chains, there is assumed to be an outside controller who chooses an action (with its associated transition matrix) at each step of the process, according to some strategy or policy. In addition, each state and action pair has an associa...

We consider a class of discrete time, dynamic decision-making models which we refer to as Periodically Time-Inhomogeneous Markov Decision Processes (PTMDPs). In these models, the decision-making horizon can be partitioned into intervals, called slow scale cycles, of N +1 epochs. The transition law and reward function are time-homogeneous over the first N epochs of each slow scale cycle, but dis...

context-dependent modeling is a well-known approach to increase modeling accuracy in continuous speech recognition. the most common way to implement this approach is via triphone modeling. nevertheless, the large number of such models results in several problems in model training, whilst the robust training of such models is often hardly obtained. one approach to solve this problem is via param...

Many embedded systems behave as discrete-time semi-Markov processes (DTSMPs). For those systems, performability measures, especially when specified as an accumulated reward, are often difficult to evaluate analytically. In this article, we informally describe an approach that uses a recurrence-relation-based (RRB) reward model for performability evaluation of systems exhibiting DTSMP behavior. ...

The classical model of Markov decision processes with costs or rewards, while widely used to formalize optimal decision making, cannot capture scenarios where there are multiple objectives for the agent during the system evolution, but only one of these objectives gets actualized upon termination. We introduce the model of Markov decision processes with alternative objectives (MDPAO) for formal...

In this paper, we study the call admission control (CAC) and routing issue in multi-service networks. Two categories of calls are considered: a narrow-band (NB) with blocked calls cleared and a wide-band (WB) with blocked calls delayed. The objective function is formulated as reward maximisation with penalty for delay. The optimisation is subject to quality of service (QoS) constraints and, pos...

Abstract We consider a decision network on an undirected graph in which each node corresponds to a decision variable, and each node and edge of the graph is associated with a reward function whose value depends only on the variables of the corresponding nodes. The goal is to construct a decision vector which maximizes the total reward. This decision problem encompasses a variety of models, incl...

Telecommunication systems are large and complex, consisting of multiple intelligent modules in shelves, multiple shelves in frames, and multiple frames to compose a single network element. In the availability and performability analysis of such a complex system, combinatorial models are computationally efficient but have limited expressive power. State-based models are expressive but computatio...