RE: [Fis] Information and communication

Hello all.

Michel considers time and space to be mixed via stochastic processes in
non-relativistic models, and points out that sigma-algebras are needed in
Minkowski spaces before we can speak of probability. I agree with his
concerns, but I'm not a specialist of relativity either. I think we can
model any function y = f(x) using a more general concept of a probability
function P(x,y) using the equivalence between P(x,y|f) > 0 and y=f(x);
P(x,y|f)=0 and y <> f(x).

Loet is working on rather well-defined problems that only capture a few
aspects of reality, but not everything. In such a context, it is not
necessary to ponder physical models, but just define some probability mass
function that models the relationship between the given variables. I agree
with this simplification, but I've encountered a few issues which are
connected directly with Michel's concerns.

1. Probability mass functions are what we need to properly deal with
Shannon's entropy. As you know, entropy can be defined on probability
density functions, but this entropy has different properties from the
entropy defined on probability mass functions. For that reason we refer to
entropy on a density function as "differential entropy" (Cover&Thomas), it
can be negative or zero, and the value of entropy changes when the space is
transformed. Although one can do various interesting things with
differential entropy, let us try to focus on probability mass functions.

2. A probability mass function gives the probability of occurrence of a
certain concrete configuration. The configuration can also be considered an
event. To someone using entropy, the underlying alphabet of concrete
configurations has to be completely clear. For thermodynamics, the
configuration is a microstate of a physical system given a particular energy
level (so we speak of the probability of a particular configuration of
microparticles). For Shannon's application of entropy to communication, the
configuration is a letter from an alphabet, or a message composed of several
letters. In my applications of entropy to medicine, the configuration is a
particular patient. In Loet's application of entropy to analysis of the
economy, the configuration is a particular company. The meaning of entropy
would change if I instead considered an employee to be a configuration
instead. It would further change if I instead considered a Euro measured at
the 20th of May, 2004, to be a configuration.

3. It was proven in statistical thermodynamics that, at the same energy
level, (changes into) the high entropy configurations are a lot more likely
than (changes into) low entropy configurations. But whatever was proved for
thermodynamical models has not been proved for other probabilistic models
that we can compute entropy on. Quite the opposite! Therefore, we cannot
infer ever-increasing entropy for Shannon's probabilistic model of
communication, because this model has nothing to do with thermodynamical
models. Other fields introduce entropy and refer to thermodynamical laws,
but often neglect to show that their underlying models show the same
properties as those of statistical mechanics.

4. Energy level can be considered to be a single variable. If we include
other variables, we can investigate the contribution to the entropy that
arises from interactions between there variables and the energy level using
mutual information. This way, mutual information (and its generalizations)
can be considered to be a decomposition of entropy. Inferring interaction or
causal influence from mutual information is possible, but may be tricky:
there are situations where variables that interact have zero mutual
information (parity problem), while variables that do not interact have
non-zero mutual information (Simpson's paradox). There are further problems
associated with probabilistic modeling: is some mutual information due to
true interaction or merely due to the "noise" or random perturbation.

Therefore, although I endorse the use of mutual information to infer
interactions between variables (it's my research topic after all!), I
recommend a careful approach, as the methodology has not been polished yet.

Best regards,

                Aleks

--
mag. Aleks Jakulin
http://www.ailab.si/aleks/
Artificial Intelligence Laboratory, 
Faculty of Computer and Information Science, University of Ljubljana. 
_______________________________________________
fis mailing list
fis@listas.unizar.es
http://webmail.unizar.es/mailman/listinfo/fis