We define the concept of an ‘open’ Markov process, or more precisely, continuous-time Markov chain, which is one where probability can flow in or out of certain states called ‘inputs’ and ‘outputs’. One can build up a Markov process from smaller open pieces. This process is formalized by making open Markov processes into the morphisms of a dagger compact category. We show that the behavior of a...

This paper studies a single-stage production system, two-stage production system where specification limits are designed for inspection. When quality characteristics fall below a lower threshold or above an upper threshold, a decision is made to rework or scrap the item. The target is to determine the optimum target for a process based on rework or scrap costs. In contrast to previous studies, ...

Existence and uniqueness are established for a translation-invariant Gibbs measure corresponding to a spatial point process that has, in addition to inhibition and clustering, the new feature of penalizing isolated points. It is shown that this point process has the so-called two-step Markov property, and a theorem is proved that characterizes the more general m-step Markov density functions. T...

In the course lectures, we have discussed a lot regarding unconstrained Markov Decision Process (MDP). The dynamic programming decomposition and optimal policies with MDP are also given. However, in this report we are going to discuss a different MDP model, which is constrained MDP. There are many realistic demand of studying constrained MDP. For instance, in the wireless sensors networks, each...

Outline Objective Background: Stochastic tools used in healthcare MDP in healthcare Preliminaries Optimality Equations and the Principle of Optimality Solving MDPs Examples References Objective: To discuss the construction and evaluation of Markov Decision Process (MDP) To investigate the role of MDP in healthcare. To identify the most appropriate solution techniques for finite and infinite-hor...

Given a set of sequential data in an unsupervised setting, we often aim to infer meaningful states present in the data along with characteristics that describe and distinguish those states. For example, in a speaker diarization (or who-spoke-when) problem, we are given a single audio recording of a meeting and wish to infer the number of speakers present, when they speak, and some characteristi...

A general description of Bernstein processes, a class of diffusion processes, relevant to the probabilistic counterpart of quantum theory known as Euclidean Quantum Mechanics, is given. It is compatible with finite or infinite dimensional state spaces and singular interactions. Although the relations with statistical physics concepts (Gibbs measure, entropy,. . . ) is stressed here, recent deve...

In this paper, we considered decision processes in which one decision is made in each process. We incorporate the utility function concept into the decision process, derived the utility function in fuzzy events and by the max-product operation obtained the utility possibility measure of the fuzzy events. In cases with numerous decision processes, the optimum action can be determined from the re...

This paper presents the Markov decision process extraction network, which is a data-efficient, automatic state estimation approach for discrete-time reinforcement learning (RL) based on recurrent neural networks. The architecture is designed to model the minimal relevant dynamics of an environment, capable of condensing large sets of continuous observables to a compact state representation and ...

In this paper, we deal with a multicriteria competitive Markov decision process. In the decision process there are two decision makers with a competitive behaviour, so they are usually called players. Their rewards are coupled because depend on the actions chosen by both players in each state of the process. We propose as solution of this game the set of Pareto-optimal security strategies for a...