2. The Metropolis-Hastings Algorithm. Metropolis’ idea is to start with a Markov chain Xn on the state space X with a fairly arbitrary Markov transition density q(x, y)dy and then modify it to define a Markov chain X∗ n that has π(x) as a stationary measure. By definition, q(x, y) is a Markov transition density if q(x, y) ≥ 0 and ∫ y∈X q(x, y)dy = 1. If the transformed random walk X ∗ n is irre...

Likelihood inference for discretely observed Markov jump processes with finite state space is investigated. The existence and uniqueness of the maximum likelihood estimator of the intensity matrix are investigated. This topic is closely related to the imbedding problem for Markov chains. It is demonstrated that the maximum likelihood estimator can be found either by the EM algorithm or by a Mar...

In this paper we consider the problem of policy evaluation in reinforcement learning, i.e., learning the value function of a fixed policy, using the least-squares temporal-difference (LSTD) learning algorithm. We report a finite-sample analysis of LSTD. We first derive a bound on the performance of the LSTD solution evaluated at the states generated by the Markov chain and used by the algorithm...

We analyse Pollack and Blair’s HC-Gammon backgammon program using a new technique that performs Monte Carlo simulations to derive a Markov Chain model for Imperfect Comparison ALgorithms, called the MICAL method, which models the behavior of the algorithm using a Markov chain, each of whose states represents a class of players of similar strength. The Markov chain transition matrix is populated...

We analyse Pollack and Blair’s HC-Gammon backgammon program using a new technique that performs Monte Carlo simulations to derive a Markov Chain model for Imperfect Comparison ALgorithms, called the MICAL method, which models the behavior of the algorithm using a Markov chain, each of whose states represents a class of players of similar strength. The Markov chain transition matrix is populated...

These notes provide an introduction to Markov chain Monte Carlo methods and their applications to both Bayesian and frequentist statistical inference. Such methods have revolutionized what can be achieved computationally, especially in the Bayesian paradigm. The account begins by discussing ordinary Monte Carlo methods: these have the same goals as the Markov chain versions but can only rarely ...

We discuss the rejection-free event-chain Monte-Carlo algorithm and several applications to dense soft matter systems. Event-chain is an alternative standard local Markov-chain schemes, which are based on detailed balance, for example well-known Metropolis-Hastings algorithm. a Markov chain scheme that uses so-called lifting moves achieve global balance without rejections (maximal balance). It ...

We consider connected Markov chain for sampling 3 × 3 × K contingency tables having fixed two-dimensional marginal totals. Such sampling arises in performing various tests of the hypothesis of no three-factor interactions. Markov chain algorithm is a valuable tool for evaluating p values, especially for sparse data sets where large-sample theory does not work well. For constructing a connected ...

abstract background: identify talented people to donate predisposed a major role in supplying healthy blood. if the time interval between donations decrease, the chance that voluntary of blood donation be a regular donor increase. the aim of this study was to determine factors affecting the interval between blood donations in a sample of first-time blood donors. methods: data of a cross-section...