Source coding theorems for general sources are presented. For a source , which is assumed to be a probability measure on all strings of infinite-length sequence with a finite alphabet, the notion of almost-sure sup entropy rate is defined; it is an extension of the Shannon entropy rate. When both an encoder and a decoder know that a sequence is generated by , the following two theorems can be p...

The entropy rate of independent and identically distributed events can on average be encoded by H(X) bits per source symbol. However, in reality, series of events (or processes) are often randomly distributed and there can be arbitrary dependence between each event. Such processes with arbitrary dependence between variables are called stochastic processes. This report shows how to calculate the...

We present a constancy rate principle governing language generation. We show that this principle implies that local measures of entropy (ignoring context) should increase with the sentence number. We demonstrate that this is indeed the case by measuring entropy in three different ways. We also show that this effect has both lexical (which words are used) and non-lexical (how the words are used)...

We introduce a method for quantifying the inherent unpredictability of a continuous-valued time series via an extension of the differential Shannon entropy rate. Our extension, the specific entropy rate, quantifies the amount of predictive uncertainty associated with a specific state, rather than averaged over all states. We relate the specific entropy rate to popular ‘complexity’ measures such...

A task is randomly drawn from a finite set of tasks and is described using a fixed number of bits. All the tasks that share its description must be performed. Upper and lower bounds on the minimum ρ-th moment of the number of performed tasks are derived. The case where a sequence of tasks is produced by a source and n tasks are jointly described using nR bits is considered. If R is larger than ...

Lactic fermentation and respiration are important metabolic pathways on which life is based. Here, the rate of entropy in a cell associated to fermentation and respiration processes in glucose catabolism of living systems is calculated. This is done for both internal and external heat and matter transport according to a thermodynamic approach based on Prigogine’s formalism. It is shown that the...

A task is randomly drawn from a finite set of tasks and is described using a fixed number of bits. All the tasks that share its description must be performed. Upper and lower bounds on the minimum ρ-th moment of the number of performed tasks are derived. The key is an analog of the Kraft Inequality for partitions of finite sets. When a sequence of tasks is produced by a source of a given Rényi ...