Shannon entropy is the most crucial foundation of Information Theory, which has been proven to be effective in many fields such as communications. Rényi entropy and Chernoff information are other two popular measures of information with wide applications. The mutual information is effective to measure the channel information for the fact that it reflects the relation between output variables an...

Let p be an unknown probability distribution on [n] := {1, 2, . . . n} that we can access via two kinds of queries: A SAMP query takes no input and returns x ∈ [n] with probability p[x]; a PMF query takes as input x ∈ [n] and returns the value p[x]. We consider the task of estimating the entropy of p to within ±∆ (with high probability). For the usual Shannon entropy H(p), we show that Ω(log n/...

We consider the problem of approximating the empirical Shannon entropy of a high-frequency data stream when space limitations make exact computation infeasible. It is known that αdependent quantities such as the Rényi and Tsallis entropies can be estimated efficiently and unbiasedly from low-dimensional α-stable data sketches. An approximation to the Shannon entropy can be obtained from either ...

We introduce an extension of the notion of Shannon conditional entropy to a more general form of conditional entropy that captures both the conditional Shannon entropy and a similar notion related to the Gini index. The proposed family of conditional entropies generates a collection of metrics over the set of partitions of finite sets, which can be used to construct decision trees. Experimental...

We introduce an extension of the notion of Shannon conditional entropy to a more general form of conditional entropy that captures both the conditional Shannon entropy and a similar notion related to the Gini index. The proposed family of conditional entropies generates a collection of metrics over the set of partitions of finite sets, which can be used to construct decision trees. Experimental...

Nowadays, multiagent systems became a widely used technology in everyday life. More studies are needed to evaluate these systems from different aspects such as evaluating agent dialogues, the participants to these dialogues, and the protocols governing the dialogues, etc. In this paper, we define new measures for dialogue games from an external agent’s point of view. In particular, two measurem...

Efficient estimation of the moments and Shannon entropy of data streams is an important task in modern machine learning and data mining. To estimate the Shannon entropy, it suffices to accurately estimate the α-th moment with ∆ = |1 − α| ≈ 0. To guarantee that the error of estimated Shannon entropy is within a ν-additive factor, the method of symmetric stable random projections requires O ( 1 ν...

today, along with enhancement of social well-being andeconomic development, demand for transportation is growing increasingly.many countries are seeking for providing transportation servicesat a reasonable price. therefore, optimization of carrying goods on nationalor international roads is highly important in terms of costs andtransportation indicator, so that the goods are shipped at minimumc...