Download Information-Theoretic Variable Selection and Network

47
Name
Kullback-Leibler divergence
Entropy
Conditional entropy
Mutual information
Conditional Mutual information
Interaction information
Multiinformation
Notation
KL(p; q)
H(X)
H(Y |X)
I(X; Y )
I(X; Y |Z)
C(X; Y ; Z)
R(X; Y ; Z)
Definition
2.1
2.4
2.6
2.7
2.8
2.10
2.11
Table 2.3: Information-theoretic measures of the chapter
variance σ 2 of the distribution,
j
k the number of samples m and the range (b − a) of Xi .
(b−a)
σ
More precisely, |Xi | =
where h = 3, 5 ∗ √
3 m . The motivation for this criterion can
h
be found in [122].
2.8
Conclusion
Information theory deals with various functions of probability distributions to measure
predictability and dependency of random variables (see Tab. 2.3). The Kullback-Leibler divergence, entropy, mutual information, multiinformation and interaction information have
been defined. In the chapter 3, these notions are used to define relevance, redundancy and
complementarity of variables.
In a finite sample size setting, a probability distribution can be estimated by identifying
the family, the number and the values of the parameters θm to describe a target distribution.
In other words, a statistical model is built by carrying out two steps:
1. a structural identification which consists in finding the “best” class Λ of models for
the target (the family and the number of parameters).
2. a parametric identification whose goal is to find the “best” model from the chosen
class of models (the values of the parameters θ).
Hence, in this approach, not only the values of the parameters but also the number of the
parameters θ depend on the data. As the number of parameters increases, the bias of a
model is reduced but the variance is increased. A higher number of parameters may then
reduce the estimation accuracy.
We have seen three machine learning algorithms (naive Bayes, nearest neighbors and
SVM), five entropy estimators (empirical, Miller-Madow, shrink, Schurmann-Grassberger,
Gaussian) and two discretization methods (equal frequency and equal width). In chapter 4,
these algorithms are used in the experimental sessions. These methods all deal, explicitly or