Non-extensive entropy from incomplete knowledge

of Shannon entropy?
F. Sattin
Consorzio RFX, Associazione EURATOM-Enea per la fusione
Corso Stati Uniti 4, 35127 Padova - ITALY
e-mail: [Link]@[Link]

Abstract
In this paper we give an interpretation of Tsallis nonextensive statistical mechanics
based upon the information-theoretic point of view of Luzzi et al. [cond-mat/0306217;
cond-mat/0306247; cond-mat/0307325], suggesting Tsallis entropy to be not a
fundamental concept but a derived one, stemming from an incomplete knowledge of the
system, not taking properly into account its interaction with the environment. This
interpretation seems to avoid some problems occuring with the original interpretation of
Tsallis statistics.

PACS: [Link], 05.40.-a, 47.27.-i

Nonextensive statistical mechanics (NESM) introduced by Tsallis [1] has been raising
considerable interest for its ability in interpreting several experimental results (for a
review of the applications of Tsallis formalism, see the updated website:
[Link]
The standard way of introducing NESM is through the definition of a generalized entropy
functional:

1
Sq =

piq
i

q 1

(1)

from which a whole generalized thermodynamics can be recovered (see, e.g. [2]).
With the usual procedure of constrained entropy maximimization, one can deduce from
(1) the equilibrium Probability Distribution Function (PDF) for any given system. Indeed,
it is possible to show that the set of probabilities pi maximizing Sq within the canonical
ensemble are

pi =

1
1/(1q )
1 (1 q) i ]
[
Zq

(2)

with i energy of the i-th microstate and the Lagrange multiplier that plays the role of
generalized inverse temperature. This expression enjoys so widespread consideration that
Tsallis NESM has become often sinonimous for power-law PDFs. Actually, there is a
huge and ever-increasing number of systems where PDFs of some quantity are measured
and found to be consistent with power laws (2).
It is important to point out that Eq. (1) is a sufficient condition for (2) to hold, but by no
means a necessary one. There are other different definitions for the entropy which lead to
the same extremizing pi-distribution (e.g., the Rnyi entropy). Even more important,
PDFs of the type (2) can be recovered without any explicit recourse to a definition of
entropy whatsoever: they can be seen as the equilibrium or quasi-equilibrium
distributions for systems interacting with thermal baths subject to fluctuations. This fact
was first empirically found for a particular system by Wilk and Wlodarczyk [3], then
Beck [4] provided a theoretical interpretation (see also Sattin and Salasnich [5]), pointing
it out that, in principle, fluctuations of the thermal bath could lead to several classes of
PDFs, of which Eq. (2) represents only a particular case. Finally, Beck and Cohen [6]
enframed all these findings within the unifying concept of superstatistics.
A matter of debate is, therefore, the role that must be assigned to the generalized entropy
Sq (Eq. 1): has it a physical meaning just like standard Shannon entropy has in
thermostatistics, or is just a useful compact expression to derive non-exponential PDFs?
Indeed, some authors [7] have pointed out that Sq should have a meaning just within an
information-thoretic approach, as an informational entropy, in situations where the
observer cannot have access to a complete characterization of the system (i.e., the states
of the system cannot be properly characterized, for example systems with fractal
structure, or with long range interactions).
In this work we will be providing some considerations about this question, following
essentially the path of Luzzi et al [7]. We do not pretend to give a definite answer here; in
particular we must admit the severe constraint that our calculations hold only for the
nonextensive exponent close to unity. This covers many of the cases tracted in literature,

but there are also several exceptions. However, we will provide the reader with some
comments and analytical calculations supporting Luzzi et als viewpoint..
We will try, hence, to give Eq. (1) a different, information-theoretic interpretation: Tsallis
entropy is looked like a partial information entropy, arising from incomplete information
over the whole system under consideration. The starting point is the empirical
observation of the existence of PDFs of the kind (2) (or, more generally, non-Gaussian),
together with the explanation of their existence as due the interaction of the system
with a fluctuating thermal bath B. From this, expressions of the form (1) can be
recovered, but they do not express any fundamental property of the system. Instead, they
are simply approximate expressions arising when we have an incomplete knowledge of
the whole system +B. We remind that this fact is perhaps implicit in Beck's
superstatistics formalism, and was recently verified through numerical experiments by
Potiguar and Costa [8]. These authors showed that PDFs (2) necessarily arise when our
system (endowed with ordinary Maxwell-Boltzmann statistics) interacts with a thermal
bath made of a finite number of degrees of freedom, hence fluctuacting in its
macroscopic quantities (e.g., temperature).
In the following we will use the unnormalized form of Tsallis statistics. Nowadays, there
is evidence in support of a normalized form, where mean quantities (observables) must be
divided by

p q . However, we do not expect the use of one form rather than another

may affect the present results.

Let us consider the (Shannon) entropy of the total system + B:
S =

bi s( j | i )ln[bis ( j | i)]

(3)

ij

where bi is the probability for the bath B to be in the i-th state, and s(j|i) is the conditional
probability for the system to be in the state j if B is in the state i. If +B is an isolated
system, its equilibrium distribution can be found by extremizing (3) with the
bis ( j | i) = 1 , that is

normalization condition
ij

F S

bis( j | i) = 0 bi s( j | i) = const

(4)

ij

Note that this does not imply s = const, that is, is +B to follow microcanonical statistics,
not .
Let us now suppose that there is a large (almost but not exactly unitary) probability for
the bath to be just in one single state, say I; thus write
bi ~ iI + (1 iI )(1 )di

(5)

In the expression above (1-) is just a small parameter, useful for doing perturbative
expansions. Its meaning is that of an effective width of the fluctuations around the most
probable state I. If we insert Eq. (5) into (3) we can write
s( j | I) ln[s( j | I)] (1 )

S ~

i I, j

di s( j | i)ln[di s( j | i)]

(6)

and the functional F defined in (4) is rewritten

s( j | I)ln[s( j | I)] (1 )

F~

i I , j

dis( j | i) ln[di s(j | i)]

s( j | I) (1 )
i I , j

di s( j | i)

(7)
The first term in the r.h.s of (6) would be the ordinary entropy for the system , if it were
isolated.
Let us now imagine a researcher wishing to determine the equilibrium PDF for ,
ignoring that it is interacting with B; hence, he would assume s( j | i) s(j) i , and try to
extremize the microcanonical-ensemble functional
s( j)ln[s( j)]

F =
j

s( j)

(8)

or, allowing for a weak energy exchange, the canonical-ensemble functional

s( j)ln[s(j)]

FC =
j

s( j)

j s( j)

(9)

Of course, in both cases, he would not find agreement with experiment, since
distributions extremizing (8) or (9) generally do not extremize (7). At this point, one
could be tempted to modify the definition of the entropy. Actually, if we write the
equivalent of (9) using Tsallis rules, we get the functional
Fq = Sq

ss
i

isiq

(10)

(Note the use of sq instead of s in the Lagrange multiplier relative to energy).

Supposing q ~ 1 and expanding (10) to first order in (1 - q), we find

s j ln( sj )

Fq ~
j

sj
j

js j (1 q)
j

1
2

sj ln 2 (sj ) +
j

j s j ln( sj ) + O(1 q)2

j

(11)
Eqns. (7) and (11) can be identified provided the following relations hold:
s( j | I)ln[s( j | I)]

s j ln(s j )
j

s( j | I)
j

sj
j

(1 )

dis( j | i)
i I , j

(1 )

(12)

j sj
j

s j ln 2 (s j )

di s( j | i)ln[di s( j | i)] (1 q)
i I, j

+ j s j ln(sj )

The fourth line suggests obviously to identify and q. Notice that the effect of using
Tsallis entropy is to make an effective diagonalization, replacing in (12) the sums over
the index i by a single term. Since these are the sums over the heat bath states, this means
that we are replacing the -B interaction with a sort of mean-field approximation. In
conclusion, the appearance of non-ortodox entropy appears related just to the fact that
we are trying to get correct results using the wrong functional (FC -Eq. 9-instead of F-Eq.
7).
We have shown that it is possible to identify the non-extensivity parameter q with . But
the latter (or, better, 1 - ) is simply a measure of the fluctuations of the heat bath around
its most probable state. Therefore q is by no means an intrinsic property of and instead
is completely determined by B. If no fluctuations are possible, = 1 and the system is
endowed with Shannon entropy. This can happen either because and B do not interact
at all, so each of them is described within the microcanonical ensemble; or because B has
a huge number of degrees of freedom (thermodynamic limit) so that its most probable
state is overwhemingly probable.
We point out some consequences of this interpretation. Let us consider two systems 1
and 2, modelled through non-ordinary statistics with different non-extensivity exponents
(i.e. q1 q2 and both 1). One question sometimes found in literature is: what is the
statistics of the combined system? There does not appear to be a simple way of writing an
expression of the kind (1) from two subsystems which, individually, are described by two

different non-extensivity exponents. Indeed, there are several papers by Wang and other
authors [9] where this issue is debated, without reaching a satisfactory answer. We may
mention also the paper [10], where some steps in this directions were made, but without
conclusive results (strictly speaking, this paper states that zeroth-law of thermodynamics
applies to quasiequilibrium systems with long range interactions. This is not the same as
saying that it holds within nonextensive statistical mechanics, although some connections
undoubtedly may be seen). We see instead that now, within the present framework, the
question itself becomes ill-posed. Infact, either 1 and 2 are in contact with the same
heat bath, and in this case they are forced to have the same q, or they are in contact with
two different heat baths B1, B2. But in this latter case the two independent systems to be
considered are (1 + B1), (2 + B2). These composite systems follow ordinary extensive
statistics, hence ordinary addition of independent entropies applies: It is not possible to
have two systems interacting with different q indices.
A related issue is: let us suppose to put into thermal contact the two heat baths B1, B2 ,
isolated from the outside. We have therefore the whole system B = B1 + B2 , and B1, B2
could be alternatively regarded as the heat bath or the system studied. Let us suppose that
a system be endowed with q1 (q2) statistics if it were separately interacting with B1
(B2). Then, there appears an (apparent) contradiction within our scheme: infact, if we
chose B1 as heat bath, and = B2 , then B2 should follow q1-statistics, while, in the
opposite case it is B1 to follow q2-statistics. But is this a real contradiction? Not at all.
Infact, it is necessary to remind the distinction between thermal bath and system: from an
operational point of view, they are distinct by the fact that the former acts on the latter,
but no feedback is possible; the state-and the statistics-of B are not appreciably
influenced by the presence of . We must therefore ask how B1, B2 relate to each other.
If, qualitatively speaking, one of them-say, B1-is of macroscopical size with respect to the
other, we are still in the situation of one heat bath (B1) and a system (B2). Previous
equations are not symmetrical with respect to the interchange bath-system: no conclusion
about the statistics of B1 can be drawn; indeed, its statistics is given as known. Therefore,
we simply expect the dynamics of B2 to be distorted to the extent that now it follows q1statistics. Let us suppose, instead, that B1 , B2 are of comparable magnitude and able to
appreciably affect each the dynamics of the other. In this case there is no reason to

pretend that the unperturbed statistics q1 , q2 still hold. Instead, the whole system will
follow an "intermediate" statistics. We can still use the heat bath-system picture, but now
each of the two systems will be endowed with the same statistics. The intermediate
statistics will not likely even be Tsallis-like. This result is important since it is exactly
what has been found in the investigation [11], prior to this study, which is therefore an
independent confirmation. Also, a different proof of the impossibility of merging two
Tsallis distributions into one, which does not make use of entropy, is provided in
Appendix.
A comment is in order about the earlier statement according to which finite heat baths
would be suitable generators of power-law statistics. Rigorously, this cannot be true: if B
is finite, and (B + ) of course also are, thus are endowed with a total finite energy.
Hence, arbitrarily high velocities are not allowed and the velocity PDF must have a finite
support (We refer here, for simplicity, to energy and velocity PDFs. Of course, the same
can be said for any other quantity). Indeed, the results shown in this paper rely upon a
truncation to first order of a Taylor expansion in powers of (1 - q) of the correct
functional. The true statistics arises from the full expression, hence power-law PDFs
necessarily are only approximations of the true PDFs. However, differences appear when
sampling extreme tails of the PDF, which experimentally is hard to perform.
Furthermore, the quality of the approximation is good as long as (1 - q) is a small
parameter. This has a number of consequences: 1) first of all, only for finite heat baths
characterized by large numbers of degrees of freedom (i.e., q ~ 1) we expect Tsallis-like
statistics to yield a good description of reality and, in all cases-again-it must break down
for very high velocity. 2) Our formalism can indeed accomodate several nonstandard
statistics, all of them parameterizable by the single parameter q (and, of course, it could
even be generalized to multiparameterized statistics). However, since q ~ 1 must hold, all
the statistics must be very similar between them since they must all approach the
common Gaussian limit (q = 1).
Finally, some further comments are in order:
I) in this work we have focussed on Tsallis' NESM since, as we have already told, it is
the most famous one. Of course, the above reasoning apply equally well to any non-

extensive statistics (see the above paragraph: all of them count among Luzzi et als
Unconventional Statistical Mechanics).
II) Recently, it has been understood that Tsallis statistics is not relevant to equilbria of
simple systems but to quasi-equilibria, or non-equilibrium stationary states, of complex
systems (see, e.g., [12]). We want to point out that this does not make any difference for
the present work. Indeed, regardless of the fact that we are speaking about equilibria or
just quasi-equilibria, in the standard formalism of NESM, the generalized entropy is the
fundamental quantity from which everything is derived.

Appendix
A well known problem with Tsallis NESM concerns the zeroth law of thermodynamics
in presence of two or more systems characterized by different entropic q indices: a
precise definition of a common temperature for such systems is still missing. Intensive
investigations have been carried on in the past in particular by Wang et al to address this
issue [9], but only with limited success: indeed, papers [9] are somewhat successful in
deriving the zeroth law, but only at the expense of further ad-hoc postulates.
In this appendix we provide a fairly simple argument suggesting that approaches la
Wang do appear unfruitful: a zeroth law of thermodynamics, apparently, can only be
written down for systems with the same index q.
As a first step, let us remind the standard form for the probability density function (PDF)
in a system ruled by Tsallis statistics, with the energy E as independent variable:
p( E ) =

1
1
Z (1 + (q 1) E ) q11

(A1)

We identify three regimes, depending upon the value of E: I) first, the low-energy
region, (q 1) E 1 . This region is scarcely interesting from our point of view: in this
range, regardless of its precise dependence on E, p can be expanded into a Taylor series
truncated to first order, hence no real information about the true shape of the PDF can be
got. II) Then, there is the usual intermediate range, (q 1) E 1 , where the precise
analytical form (exponential, stretched exponential, power-law, etc...) for p(E) is looked
for. III) Any physical system is, however, finite, hence there must be an extreme-value

(very high E) region, where p(E) falls sharply to zero. We will stay away also from the
region affected by these finite-size effects, and consider only the intermediate region
(II).
Under this hypothesis, we can discard the unity term in the denominator of (A1) and
write

1
1
1
1
dE
exp X
1 dX
1
1
Z ( (q 1) E ) q 1
q 1
q 1
Z (q 1)
C q exp( X )dX
.

p ( E )dE

(A2)
where we have set E = exp( X ), = 1 /( q 1) 1 .
Eq. (A2) admits a straightforward interpretation: at least in a restricted energy region (but
it is the most important region for this kind of studies), the probability distribution of a
system ruled by Tsallis statistics can be mapped into that of a system ruled by ordinary
Maxwell-Gibbs statistics, in thermal equilibrium with its surroundings, with energy X
and temperature 1/B. Notice, however, that writing p(E) in the form (A2) is not
necessary to our purposes, just slightly simplifies the following computations.
Let us now move to the next step, and consider two systems S1, S2, parameterized with
the nonextensivity indices q1, q2 respectively. In order to give an intuitive picture of the
problem, we may think of each system as a box containing a large number of particles,
respectively N1 and N2. Each particle is given a value: its energy E. Apart for the
individual E label, particles are identical between them. The statistical distribution of E
follows Tsallis statistics, with the corresponding value for q, in each box. In other words,
the probability of randomly picking up a particle labelled with energy E from box i (i
=1,2) is given by Eq. (A1) with q = q1, q2 .
The attempt of describing the combined system (S1 + S2) in terms of a single distribution
is tantamount to pouring all the particles into a single box. The resulting combined
statistics will come out by randomly picking a large number of particles out of this box.
Let us ask how much the probability is of extracting from the urn a particle labelled with
energy E. The answer is, trivially:

p ( S1 + S 2 , E ) =

N1
N2
p ( S1 , E ) +
p(S 2 , E )
N1 + N 2
N1 + N 2

(A3)

In this expression, Ni/(N1+N2) is the probability of choosing a particle out of those

previously in the box i, and p(Si,E) is the probability that this particle had energy E. Since
we are interested to the high-energy tail of the distribution, we can replace p(Si,E) by its
form (A2), which yields:

p ( S1 + S 2 , X ) = exp( B1 X 1 ) + exp( B2 X 2 )

(A4)

where i = ln ( N i /( N 1 + N 2 ) ) ln C qi is a sort of chemical potential.

The l.h.s. of (A4) must be, in its turn, a Tsallis distribution, hence, we get the final result
exp( B1+ 2 X 1+ 2 ) = exp( B1 X 1 ) + exp( B2 X 2 )

(A5)

It is just obvious that Eq. (A5) cannot be fulfilled for arbitrary X, given generic Bi, i , the
exception being, as expected, the equal-q-case, for which B1 = B2 = B1+2, 1 = 2 = 1/ln(2)
1+2 . This concludes our proof.
Other interesting consequences can be drawn from Eq. (A4). In order to fix ideas, let us
set B1 < B2 . As long as it does not hold 1 >> 2, the former exponential on the r.h.s of
(A4) will prevail over the latter, hence the asymptotic behaviour of the combined
distribution will essentially match a q1-statistics. On the other hand, widely differing
values between 1 and 2 can locally override this trend and give birth to an extremely
complicated behaviour. Since different s can be made roughly correspond to hugely
different numbers of particles (although there is also a q-dependent contribution from the
partition function, C q ), this statement is equivalent to saying anything but that the overall
statistics of the system is driven by the larger-in-size of the two subsystems.
Some final comments are: the paper [13] has recently appeared, in which the possibility
of defining an unique effective temperature for non-equilibrium critical systems is
criticized. These systems are exactly between those though amenable to study using
nonextensive statistics. It is also remarkable that the impossibility of defining a common
temperature, in that work, is directly related to the existence of fluctuations away from
the Gaussian limit.
Secondly, within Becks superstatistics interpretation of NESM [14], previous results do
appear fairly reasonable: since, there, the entropic index of a system is a measure of the

10

fluctuations of some generalized temperature, it does seems difficult to relate within one
and the same formulation fluctuations from two completely uncorrelated systems.

References

[1] C. Tsallis, J. Stat. Phys. 52 (1988) 479.

[2] C. Tsallis and Andre M.C. Souza, Phys. Rev. E 67 (2003) 026106.
[3] G. Wilk and Z. Wlodarczyk, Phys. Rev. Lett. 84 (2000) 2770.
[4] C. Beck, Phys. Rev. Lett. 87 (2001) 180601.
[5] F. Sattin and L. Salasnich, Phys. Rev. E 65 (2002) R035106.
[6] C. Beck and E.G.D. Cohen, Physica A 322 (2003) 267.
[7] R. Luzzi, .R. Vasconcellos, J. Galvo Ramos, cond-mat/0306217; .R.
Vasconcellos, J. Galvo Ramos, R. Luzzi, cond-mat/0306247; R. Luzzi, .R.
Vasconcellos, J. Galvo Ramos, cond-mat/0307325.
[8] F.Q. Potiguar and U.M.S. Costa, Physica A 321 (2003) 482; ibid 342 (2004) 145
[9] Q.A. Wang, cond-mat/0310056; Q.A. Wang et al, Chaos, Solitons & Fractals, to
appear [cond-mat/0305398]; Q.A. Wang et al, Europhys. Lett. 65 (2004) 606; K.
Sasaki and M. Hotta, cond-mat/9912454; K. Sasaki and M. Hotta, cond-mat/0004158.
[10] L.G. Moyano, F. Baldovin, C. Tsallis, cond-mat/0305091.
[11] F. Sattin, J. Phys. A: Math. Gen. 36 (2003) 1583.
[12] C. Tsallis, in "Nonextensive statistical mechanics and its applications", S. Abe and
Y. Okamoto (eds), Springer (2001)
[13] P. Calabrese and A. Gambassi, J. Stat. Mech.: Theor. Exp. P07013 (2004)
[14] C. Beck, Continuum Mechanics and Thermodynamics 16 (2004) 293

11