Shannon's entropy revisited

I consider the effect of a finite sample size on the entropy of a sample of independent events. I propose formula for entropy which satisfies Shannon’s axioms, and which reduces to Shannon’s entropy when sample size is infinite. I discuss the physical meaning of the difference between two formulas, including some practical implications, such as maximum achievable channel utilization, and minimum achievable communication protocol overhead, for a given message size. The classic formula (Shannon, 1948) (1) is accepted as a measure of entropy for a sample of independent events. Other measures, such as (Renyi, 1960) and (Tsallis, 1988) entropies, and more (Sharma B.D., 1975; Masi, 2005; Gorban, 2002) have been proposed for situations when events are correlated, as e.g. in systems with long-range interactions. These alternative measures, generally, are expected to have properties (Wehrl, 1978), different from Shannon’s entropy: non-additivity, convexity, etc., while converging to (1) when correlations disappear. The underlying assumption is that when events are independent formula (1) still gives the correct measure. Here I take a second look at the original premise that formula (1) and its equivalents in statistical (Landau, 1980) and quantum (Neumann, 1955) mechanics are the correct measures of entropy in the absence of correlations. I show that if the sample size is finite there is an inherently smaller amount of information which can be encoded per event, than given by Shannon’s formula. I propose a formula for entropy which takes into account the sample size and which reduces to (1) when sample size is infinite. Consider independent events, with possible outcomes for each event, having probabilities . Independence of events implies only the frequencies of events describe the sample, while the order of events is irrelevant. The probability density function for the sample is then given by multinomial distribution: