Dual total correlation
In-game article clicks load inline without leaving the challenge.
In information theory, dual total correlation, information rate, excess entropy, or binding information is one of several known non-negative generalizations of mutual information. While total correlation is bounded by the sum entropies of the n elements, the dual total correlation is bounded by the joint-entropy of the n elements. Although well behaved, dual total correlation has received much less attention than the total correlation. A measure known as "TSE-complexity" defines a continuum between the total correlation and dual total correlation.
Definition
For a set of n random variables { X 1 , … , X n } {\displaystyle \{X_{1},\ldots ,X_{n}\}}, the dual total correlation D ( X 1 , … , X n ) {\displaystyle D(X_{1},\ldots ,X_{n})} is given by
D ( X 1 , … , X n ) = H ( X 1 , … , X n ) − ∑ i = 1 n H ( X i ∣ X 1 , … , X i − 1 , X i + 1 , … , X n ) , {\displaystyle D(X_{1},\ldots ,X_{n})=H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right),}
where H ( X 1 , … , X n ) {\displaystyle H(X_{1},\ldots ,X_{n})} is the joint entropy of the variable set { X 1 , … , X n } {\displaystyle \{X_{1},\ldots ,X_{n}\}} and H ( X i ∣ ⋯ ) {\displaystyle H(X_{i}\mid \cdots )} is the conditional entropy of variable X i {\displaystyle X_{i}}, given the rest.
Normalized
The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value H ( X 1 , … , X n ) {\displaystyle H(X_{1},\ldots ,X_{n})},
N D ( X 1 , … , X n ) = D ( X 1 , … , X n ) H ( X 1 , … , X n ) . {\displaystyle ND(X_{1},\ldots ,X_{n})={\frac {D(X_{1},\ldots ,X_{n})}{H(X_{1},\ldots ,X_{n})}}.}
Relationship with Total Correlation
Dual total correlation is non-negative and bounded above by the joint entropy H ( X 1 , … , X n ) {\displaystyle H(X_{1},\ldots ,X_{n})}.
0 ≤ D ( X 1 , … , X n ) ≤ H ( X 1 , … , X n ) . {\displaystyle 0\leq D(X_{1},\ldots ,X_{n})\leq H(X_{1},\ldots ,X_{n}).}
Secondly, Dual total correlation has a close relationship with total correlation, C ( X 1 , … , X n ) {\displaystyle C(X_{1},\ldots ,X_{n})}, and can be written in terms of differences between the total correlation of the whole, and all subsets of size N − 1 {\displaystyle N-1}:
D ( X ) = ( N − 1 ) C ( X ) − ∑ i = 1 N C ( X − i ) {\displaystyle D({\textbf {X}})=(N-1)C({\textbf {X}})-\sum _{i=1}^{N}C({\textbf {X}}^{-i})}
where X = { X 1 , … , X n } {\displaystyle {\textbf {X}}=\{X_{1},\ldots ,X_{n}\}} and X − i = { X 1 , … , X i − 1 , X i + 1 , … , X n } {\displaystyle {\textbf {X}}^{-i}=\{X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\}}
Furthermore, the total correlation and dual total correlation are related by the following bounds:
C ( X 1 , … , X n ) n − 1 ≤ D ( X 1 , … , X n ) ≤ ( n − 1 ) C ( X 1 , … , X n ) . {\displaystyle {\frac {C(X_{1},\ldots ,X_{n})}{n-1}}\leq D(X_{1},\ldots ,X_{n})\leq (n-1)\;C(X_{1},\ldots ,X_{n}).}
Finally, the difference between the total correlation and the dual total correlation defines a novel measure of higher-order information-sharing: the O-information:
Ω ( X ) = C ( X ) − D ( X ) {\displaystyle \Omega ({\textbf {X}})=C({\textbf {X}})-D({\textbf {X}})}.
The O-information (first introduced as the "enigmatic information" by James and Crutchfield is a signed measure that quantifies the extent to which the information in a multivariate random variable is dominated by synergistic interactions (in which case Ω ( X ) < 0 {\displaystyle \Omega ({\textbf {X}})<0}) or redundant interactions (in which case Ω ( X ) > 0 {\displaystyle \Omega ({\textbf {X}})>0}, and have found multiple applications in neuroscience.
History
Han (1978) originally defined the dual total correlation as,
D ( X 1 , … , X n ) ≡ [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] − ( n − 1 ) H ( X 1 , … , X n ) . {\displaystyle {\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv {}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\;.\end{aligned}}}
However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:
D ( X 1 , … , X n ) ≡ [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] − ( n − 1 ) H ( X 1 , … , X n ) = [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] + ( 1 − n ) H ( X 1 , … , X n ) = H ( X 1 , … , X n ) + [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) − H ( X 1 , … , X n ) ] = H ( X 1 , … , X n ) − ∑ i = 1 n H ( X i ∣ X 1 , … , X i − 1 , X i + 1 , … , X n ) . {\displaystyle {\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv {}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\\={}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]+(1-n)\;H(X_{1},\ldots ,X_{n})\\={}&H(X_{1},\ldots ,X_{n})+\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})-H(X_{1},\ldots ,X_{n})\right]\\={}&H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right)\;.\end{aligned}}}
See also
- Interaction information– Generalization of mutual information for more than two variables
- Mutual information– Measure of dependence between two variables
- Total correlation
- Hilberg's hypothesis– Power law growth of entropy of language or a stochastic process