[Fis] Re: Shannon Entropy

Dear Michel,
let me try to answer your complicated questions:

On Saturday 12 Jun 2004 10:57 pm, you wrote:
> Dear Werner,
>
> Thanks for your helpful, informative message. May I use this avenue to
> reply now? I can?t make a posting to the FIS discussion until next week.
> I regret I missed an acknowledgment of your communication in my latest
> epistle.
>
> So far, we?ve heard in our FIS discussion that Shannon entropy is pure
> mathematics
Thats true, it is a mathematical expression which can be applied to many
systems having probability dostributions.
and therefore completely separate from the entropy described
> by Boltzmann et al.
For me not completely separated, but realted !!!
> Also, that some form of ?Shannon entropy? employed
> in research outside physics may actually decrease with increasing
> freedom degrees, and that in some probability analyses of biological
> systems, the probability values in the distribution may become negative.

Negative probabilities are of course not possible, but Shannon entropies
are necessarily positive only for discrete cases, if the undelying
space is continuous, there are some difficulties and we need
a more general formalism.

> I interpret all this as a misappropriation of the name Shannon entropy
> for analyses which do not, in fact, depict the sort of genuine
> probability distributions that physical systems exhibit. I?ve argued,
> from Shannon?s own derivation of the equation, that if the analyzed
> property is not monotonically increasing with N, it cannot be Shannon
> entropy.
Applying Shannons formula to physical systems not necessarily gives
thermodyanmic entropy, we have to take distributions on the space of
coordinates and momenta (as Boltzmann actually did) or
more general densities of quantum states or
still more general quantum density operators (as did von Neumann).
There are applications to physical systems as e.g. to
electrical systems or to nonlinear dynamics, which lead to some
entropy but not to the measurable thermodynamic entropy.
>
> I do recognize that Shannon entropy is an appropriate name for
> probability distributions of nonphysical systems (perhaps linguistics,
> and economics and such) which satisfy Shannon?s three axioms for such
> distributions. But, Shannon?s derivation of the equation actually
> described a physical system; the electrical signals in a communication
> cable. And would you agree that one might just as appropriately utilize
> Boltzmann?s H equation for nonphysical distributions of a suitable type?

The H-theorem of Boltzmann does not apply to any system, its true only
for special probabilistic (stochastic) systems. However for probabilistic
systems with Markov character
one can find some generalization of the H-theorem which is valid
for the so-called Kullback-Leibler entropy (this is to be found
in many books on stochastic theory, I could give references).
 
> Are you familiar with Szilard?s engine? I continue to believe it?s the
> best model for demonstrating the direct connection between Clausius?
> form of entropy and information.
Yes Szilard worked at my University.
My interpretation is, that any exchange of information betrween two systems
is connected with some exchange of thermodynamic entropy.
>
> Very best wishes,
>
> Michael Devereux
By the way I do not claim to know the full truth,
entropy is one of the most difficult concepts of science !!!!
In reply best wishes
Werner Ebeling

-- 
-----------------------------------------------------------
Prof. W. Ebeling                ebeling@physik.hu-berlin.de
Humboldt-Universitaet Berlin    phone:  +49/(0)30-2093 7636
Institut fuer Physik            fax:    +49/(0)30-2093 7638
Invalidenstrasse 110
D-10115 Berlin                                                           
-----------------------------------------------------------
_______________________________________________
fis mailing list
fis@listas.unizar.es
http://webmail.unizar.es/mailman/listinfo/fis