Skip to content

Commit d22ce17

Browse files
JoramSochJoramSoch
authored
added 2 proofs
1 parent f05ec30 commit d22ce17

2 files changed

Lines changed: 245 additions & 0 deletions

File tree

‎P/dent-addvec.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2021-10-07 09:10:00
9+
10+
title: "Addition of the differential entropy upon multiplication with invertible matrix"
11+
chapter: "General Theorems"
12+
section: "Information theory"
13+
topic: "Differential entropy"
14+
theorem: "Addition upon matrix multiplication"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2021
19+
title: "Differential entropy"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2021-10-07"
22+
url: "https://en.wikipedia.org/wiki/Differential_entropy#Properties_of_differential_entropy"
23+
24+
proof_id: "P261"
25+
shortcut: "dent-addvec"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** Let $X$ be a [continuous](/D/rvar-disc) [random vector](/D/rvec). Then, the [differential entropy](/D/dent) of $X$ increases additively when multiplied with an invertible matrix $A$:
31+
32+
$$ \label{eq:dent-addvec}
33+
\mathrm{h}(AX) = \mathrm{h}(X) + \log |A| \; .
34+
$$
35+
36+
37+
**Proof:** By definition, the [differential entropy](/D/dent) of $X$ is
38+
39+
$$ \label{eq:X-dent}
40+
\mathrm{h}(X) = - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x
41+
$$
42+
43+
where $f_X(x)$ is the [probability density function](/D/pdf) of $X$ and $\mathcal{X}$ is the set of possible values of $X$.
44+
45+
The [probability density function of a linear function of a continuous random vector](/P/pdf-linfct) $Y = g(X) = \Sigma X + \mu$ is
46+
47+
$$ \label{eq:pdf-linfct}
48+
f_Y(y) = \left\{
49+
\begin{array}{rl}
50+
\frac{1}{\left| \Sigma \right|} f_X(\Sigma^{-1}(y-\mu)) \; , & \text{if} \; y \in \mathcal{Y} \\
51+
0 \; , & \text{if} \; y \notin \mathcal{Y}
52+
\end{array}
53+
\right.
54+
$$
55+
56+
where $\mathcal{Y} = \left\lbrace y = \Sigma x + \mu: x \in \mathcal{X} \right\rbrace$ is the set of possible outcomes of $Y$.
57+
58+
Therefore, with $Y = g(X) = AX$, i.e. $\Sigma = A$ and $\mu = 0_n$, the [probability density function](/D/pdf) of $Y$ is given by
59+
60+
$$ \label{eq:Y-pdf}
61+
f_Y(y) = \left\{
62+
\begin{array}{rl}
63+
\frac{1}{\left| A \right|} f_X(A^{-1}y) \; , & \text{if} \; y \in \mathcal{Y} \\
64+
0 \; , & \text{if} \; y \notin \mathcal{Y}
65+
\end{array}
66+
\right.
67+
$$
68+
69+
where $\mathcal{Y} = \left\lbrace y = A x: x \in \mathcal{X} \right\rbrace$.
70+
71+
Thus, the [differential entropy](/D/dent) of $Y$ is
72+
73+
$$ \label{eq:Y-dent-s1}
74+
\begin{split}
75+
\mathrm{h}(Y) &\overset{\eqref{eq:X-dent}}{=} - \int_{\mathcal{Y}} f_Y(y) \log f_Y(y) \, \mathrm{d}y \\
76+
&\overset{\eqref{eq:Y-pdf}}{=} - \int_{\mathcal{Y}} \left[ \frac{1}{\left| A \right|} f_X(A^{-1}y) \right] \log \left[ \frac{1}{\left| A \right|} f_X(A^{-1}y) \right] \, \mathrm{d}y \; .
77+
\end{split}
78+
$$
79+
80+
Substituting $y = Ax$ into the integral, we obtain
81+
82+
$$ \label{eq:Y-dent-s2}
83+
\begin{split}
84+
\mathrm{h}(Y) &= - \int_{\mathcal{X}} \left[ \frac{1}{\left| A \right|} f_X(A^{-1}Ax) \right] \log \left[ \frac{1}{\left| A \right|} f_X(A^{-1}Ax) \right] \, \mathrm{d}(Ax) \\
85+
&= - \frac{1}{\left| A \right|} \int_{\mathcal{X}} f_X(x) \log \left[ \frac{1}{\left| A \right|} f_X(x) \right] \, \mathrm{d}(Ax) \; .
86+
\end{split}
87+
$$
88+
89+
Using the differential $\mathrm{d}(Ax) = \left|A\right| \mathrm{d}x$, this becomes
90+
91+
$$ \label{eq:Y-dent-s3}
92+
\begin{split}
93+
\mathrm{h}(Y) &= - \frac{\left| A \right|}{\left| A \right|} \int_{\mathcal{X}} f_X(x) \log \left[ \frac{1}{\left| A \right|} f_X(x) \right] \, \mathrm{d}x \\
94+
&= - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x - \int_{\mathcal{X}} f_X(x) \log \frac{1}{\left| A \right|} \, \mathrm{d}x \; .
95+
\end{split}
96+
$$
97+
98+
Finally, employing [the fact](/D/pdf) that $\int_{\mathcal{X}} f_X(x) \, \mathrm{d}x = 1$, we can derive the [differential entropy](/D/dent) of $Y$ as
99+
100+
$$ \label{eq:Y-dent-s4}
101+
\begin{split}
102+
\mathrm{h}(Y) &= - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x + \log \left| A \right| \int_{\mathcal{X}} f_X(x) \, \mathrm{d}x \\
103+
&\overset{\eqref{eq:X-dent}}{=} \mathrm{h}(X) + \log |A| \; .
104+
\end{split}
105+
$$

‎P/dent-noninv.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2021-10-07 10:39:00
9+
10+
title: "Non-invariance of the differential entropy under change of variables"
11+
chapter: "General Theorems"
12+
section: "Information theory"
13+
topic: "Differential entropy"
14+
theorem: "Non-invariance and transformation"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2021
19+
title: "Differential entropy"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2021-10-07"
22+
url: "https://en.wikipedia.org/wiki/Differential_entropy#Properties_of_differential_entropy"
23+
- authors: "Bernhard"
24+
year: 2016
25+
title: "proof of upper bound on differential entropy of f(X)"
26+
in: "StackExchange Mathematics"
27+
pages: "retrieved on 2021-10-07"
28+
url: "https://math.stackexchange.com/a/1759531"
29+
- authors: "peek-a-boo"
30+
year: 2019
31+
title: "How to come up with the Jacobian in the change of variables formula"
32+
in: "StackExchange Mathematics"
33+
pages: "retrieved on 2021-08-30"
34+
url: "https://math.stackexchange.com/a/3239222"
35+
- authors: "Wikipedia"
36+
year: 2021
37+
title: "Jacobian matrix and determinant"
38+
in: "Wikipedia, the free encyclopedia"
39+
pages: "retrieved on 2021-10-07"
40+
url: "https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant#Inverse"
41+
- authors: "Wikipedia"
42+
year: 2021
43+
title: "Inverse function theorem"
44+
in: "Wikipedia, the free encyclopedia"
45+
pages: "retrieved on 2021-10-07"
46+
url: "https://en.wikipedia.org/wiki/Inverse_function_theorem#Statement"
47+
- authors: "Wikipedia"
48+
year: 2021
49+
title: "Determinant"
50+
in: "Wikipedia, the free encyclopedia"
51+
pages: "retrieved on 2021-10-07"
52+
url: "https://en.wikipedia.org/wiki/Determinant#Properties_of_the_determinant"
53+
54+
proof_id: "P262"
55+
shortcut: "dent-noninv"
56+
username: "JoramSoch"
57+
---
58+
59+
60+
**Theorem:** The [differential entropy](/D/dent) is not invariant under change of variables, i.e. there exist random variables $X$ and $Y = g(X)$, such that
61+
62+
$$ \label{eq:dent-noninv}
63+
\mathrm{h}(Y) \neq \mathrm{h}(X) \; .
64+
$$
65+
66+
In particular, for an invertible transformation $g: X \rightarrow Y$ from a random vector $X$ to another random vector of the same dimension $Y$, it holds that
67+
68+
$$ \label{eq:dent-trans}
69+
\mathrm{h}(Y) = \mathrm{h}(X) + \int_{\mathcal{X}} f_X(x) \log \left| J_g(x) \right| \, \mathrm{d}x \; .
70+
$$
71+
72+
where $J_g(x)$ is the Jacobian matrix of the vector-valued function $g$ and $\mathcal{X}$ is the set of possible values of $X$.
73+
74+
75+
**Proof:** By definition, the [differential entropy](/D/dent) of $X$ is
76+
77+
$$ \label{eq:X-dent}
78+
\mathrm{h}(X) = - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x
79+
$$
80+
81+
where $f_X(x)$ is the [probability density function](/D/pdf) of $X$.
82+
83+
The [probability density function of an invertible function of a continuous random vector](/P/pdf-invfct) $Y = g(X)$ is
84+
85+
$$ \label{eq:pdf-invfct}
86+
f_Y(y) = \left\{
87+
\begin{array}{rl}
88+
f_X(g^{-1}(y)) \, \left| J_{g^{-1}}(y) \right| \; , & \text{if} \; y \in \mathcal{Y} \\
89+
0 \; , & \text{if} \; y \notin \mathcal{Y}
90+
\end{array}
91+
\right.
92+
$$
93+
94+
where $\mathcal{Y} = \left\lbrace y = g(x): x \in \mathcal{X} \right\rbrace$ is the set of possible outcomes of $Y$ and $J_{g^{-1}}(y)$ is the Jacobian matrix of $g^{-1}(y)$
95+
96+
$$ \label{eq:jac}
97+
J_{g^{-1}}(y) = \left[ \begin{matrix}
98+
\frac{\mathrm{d}x_1}{\mathrm{d}y_1} & \ldots & \frac{\mathrm{d}x_1}{\mathrm{d}y_n} \\
99+
\vdots & \ddots & \vdots \\
100+
\frac{\mathrm{d}x_n}{\mathrm{d}y_1} & \ldots & \frac{\mathrm{d}x_n}{\mathrm{d}y_n}
101+
\end{matrix} \right] \; .
102+
$$
103+
104+
Thus, the [differential entropy](/D/dent) of $Y$ is
105+
106+
$$ \label{eq:Y-dent-s1}
107+
\begin{split}
108+
\mathrm{h}(Y) &\overset{\eqref{eq:X-dent}}{=} - \int_{\mathcal{Y}} f_Y(y) \log f_Y(y) \, \mathrm{d}y \\
109+
&\overset{\eqref{eq:pdf-invfct}}{=} - \int_{\mathcal{Y}} \left[ f_X(g^{-1}(y)) \, \left| J_{g^{-1}}(y) \right| \right] \log \left[ f_X(g^{-1}(y)) \, \left| J_{g^{-1}}(y) \right| \right] \, \mathrm{d}y \; .
110+
\end{split}
111+
$$
112+
113+
Substituting $y = g(x)$ into the integral and applying $J_{f^{-1}}(y) = J_f^{-1}(x)$, we obtain
114+
115+
$$ \label{eq:Y-dent-s2}
116+
\begin{split}
117+
\mathrm{h}(Y) &= - \int_{\mathcal{X}} \left[ f_X(g^{-1}(g(x))) \, \left| J_{g^{-1}}(y) \right| \right] \log \left[ f_X(g^{-1}(g(x))) \, \left| J_{g^{-1}}(y) \right| \right] \, \mathrm{d}[g(x)] \\
118+
&= - \int_{\mathcal{X}} \left[ f_X(x) \, \left| J_g^{-1}(x) \right| \right] \log \left[ f_X(x) \, \left| J_g^{-1}(x) \right| \right] \, \mathrm{d}[g(x)] \; .
119+
\end{split}
120+
$$
121+
122+
Using the relations $y = f(x) \Rightarrow \mathrm{d}y = \left| J_f(x) \right| \, \mathrm{d}x$ and $\left|A\right|\left|B\right| = \left|AB\right|$, this becomes
123+
124+
$$ \label{eq:Y-dent-s3}
125+
\begin{split}
126+
\mathrm{h}(Y) &= - \int_{\mathcal{X}} \left[ f_X(x) \, \left| J_g^{-1}(x) \right| \left| J_g(x) \right| \right] \log \left[ f_X(x) \, \left| J_g^{-1}(x) \right| \right] \, \mathrm{d}x \\
127+
&= - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x - \int_{\mathcal{X}} f_X(x) \log \left| J_g^{-1}(x) \right| \, \mathrm{d}x \; .
128+
\end{split}
129+
$$
130+
131+
Finally, employing [the fact](/D/pdf) that $\int_{\mathcal{X}} f_X(x) \, \mathrm{d}x = 1$ and the determinant property $\left|A^{-1}\right| = 1/\left|A\right|$, we can derive the [differential entropy](/D/dent) of $Y$ as
132+
133+
$$ \label{eq:Y-dent-s4}
134+
\begin{split}
135+
\mathrm{h}(Y) &= - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x - \int_{\mathcal{X}} f_X(x) \log \frac{1}{\left| J_g(x) \right|} \, \mathrm{d}x \\
136+
&\overset{\eqref{eq:X-dent}}{=} \mathrm{h}(X) + \int_{\mathcal{X}} f_X(x) \log \left| J_g(x) \right| \, \mathrm{d}x \; .
137+
\end{split}
138+
$$
139+
140+
Because there exist $X$ and $Y$, such that the integral term in \eqref{eq:Y-dent-s4} is non-zero, this also demonstrates that there exist $X$ and $Y$, such that \eqref{eq:dent-noninv} is fulfilled.

0 commit comments

Comments
 (0)