Reactive (memoryless) policies are sufficient in completely observable Markov decision processes (MDPs), but some kind of memory is usually necessary for optimal control of a partially observable MDP. Policies with finite memory can be represented as finite-state automata. In this paper, we extend Baird and Moore’s VAPS algorithm to the problem of learning general finite-state automata. Because...

Several formalisms for concurrent computation have been proposed in recent years that incorporate means to express stochastic continuous-time dynamics and non-determinism. In this setting, some obscure phenomena are known to exist, related to the fact that schedulers may yield too pessimistic verification results, since current nondeterminism can surprisingly be resolved based on prophesying th...

We present an unsupervised learning algorithm (GenESeSS) to infer the causal structure of quantized stochastic processes, defined as stochastic dynamical systems evolving over discrete time, and producing quantized observations. Assuming ergodicity and stationarity, GenESeSS infers probabilistic finite state automata models from a sufficiently long observed trace. Our approach is abductive; att...

We propose a formal definition for the general notion of stochastic transducer, called stochastic λ-transducer. Our definition is designed with two objectives in mind: (i) to extend naturally the established notion of stochastic automaton with output – as defined in the classic books of Paz (1971) and Starke (1972) – by permitting pairs of input-output words of different lengths; (ii) to be com...

LANGVILLE, AMY N. Preconditioning for Stochastic Automata Networks. (Under the direction of William J. Stewart.) Many very large Markov chains can be modeled efficiently as Stochastic Automata Networks (SANs). A SAN is composed of individual automata that, for the most part, act independently, requiring only infrequent interaction. SANs represent the generator matrix Q of the underlying Markov ...

It is well known that language models are effective for increasing accuracy of speech and handwriting recognizers, but large language models are often required to achieve low model perplexity (or entropy) and still have adequate language coverage. We study three efficient methods for stochastic language modeling in the context of the stochastic pattern recognition problem and give results of a ...

We investigate a stochastic version of cellular automata used for simulating hydrodynamical flows, e. g. the HPP and FHP models. The extra stochasticity consists of "random exchanges" between neighboring cells which conserve momentum. We prove that, in suitable limits, these models satisfy the appropriate continuous Boltzmann and hydrodynamic equations, the same as those conjectured for the ori...

1. Introduction. In a probabilistic automaton, as defined e.g. in [2] and [l], the initial vector as well as the transition matrices are stochastic, and the final vector consists of O's and l's only. This paper is concerned with the so-called generalized automata where the elements of the initial vector, of the final vector and of the matrices are allowed to be arbitrary real numbers. Our purpo...

We propose a formal definition for the general notion of stochastic transducer, called stochastic λ-transducer. Our definition is designed with two objectives in mind: (i) to extend naturally the established notion of stochastic automaton with output – as defined in the classic books of Paz (1971) and Starke (1972) – by permitting pairs of input-output words of different lengths; (ii) to be com...

Stochastic computing has a broad range of applications, yet electronic computers realize its basic step, stochastic choice between alternative computation paths, in a cumbersome way. Biomolecular computers use a different computational paradigm and hence afford novel designs. We constructed a stochastic molecular automaton in which stochastic choice is realized by means of competition between a...