We study properties and parameter estimation of finite-state homogeneous continuous-time bivariate Markov chains. Only one of the two processes of the bivariate Markov chain is observable. The general form of the bivariate Markov chain studied here makes no assumptions on the structure of the generator of the chain, and hence, neither the underlying process nor the observable process is necessa...

In this study we model the daily rainfall occurrence using Markov Chain Analogue Yearmodel (MCAYM) and the intensity or amount of daily rainfall using three different probability distributions; gamma, exponential and mixed exponential distributions. Combining the occurrence and intensity model we obtain Markov Chain Analogue Year gamma model (MCAYGM), Markov Chain Analogue Year exponentia...

We discuss a generalization of the classical Gittins Index for a Markov chain and propose a transparent recursive algorithm for its calculation. The foundation for this algorithm is a modified version of the Elimination algorithm proposed earlier by the author to solve the problem of optimal stopping of a Markov chain in discrete time and a finite or countable state space.

 In this paper we study the relative entropy rate between a homogeneous Markov chain and a hidden Markov chain defined by observing the output of a discrete stochastic channel whose input is the finite state space homogeneous stationary Markov chain. For this purpose, we obtain the relative entropy between two finite subsequences of above mentioned chains with the help of the definition of...

Hidden Markov models have proved to be a very exible class of models, with many and diverse applications. Recently Markov chain Monte Carlo (MCMC) techniques have provided powerful computational tools to make inferences about the parameters of hidden Markov models, and about the unobserved Markov chain, when the chain is deened in discrete time. We present a general algorithm, based on reversib...

This paper considers the problem of paging under the assumption that the sequence of pages accessed is generated by a Markov chain. We use this model to study the fault-rate of paging algorithms. We first draw on the theory of Markov decision processes to characterize the paging algorithm that achieves optimal fault-rate on any Markov chain. Next, we address the problem of devising a paging str...

The PageRank is used by search engines to reflect the popularity and importance of a page based on its reference ranking. Since the web changes very fast, the PageRank has to be regularly updated. Such updates is an challenging task due to the huge size of the World Wide Web. Consequently, the analysis of the PageRank has become a hot topic with vast literature ranging from the original paper b...

Jump Markov linear systems (JMLS) are linear systems whose parameters evolve with time according to a finite state Markov chain. We present three original deterministic and stochastic iterative algorithms for optimal state estimation of JMLS whose computational complexity at each iteration is linear in the data length. The first algorithm yields conditional mean estimates. The second algorithm ...