|
| 1 | +--- |
| 2 | +layout: proof |
| 3 | +mathjax: true |
| 4 | + |
| 5 | +author: "Joram Soch" |
| 6 | +affiliation: "BCCN Berlin" |
| 7 | +e_mail: "joram.soch@bccn-berlin.de" |
| 8 | +date: 2021-10-07 10:39:00 |
| 9 | + |
| 10 | +title: "Non-invariance of the differential entropy under change of variables" |
| 11 | +chapter: "General Theorems" |
| 12 | +section: "Information theory" |
| 13 | +topic: "Differential entropy" |
| 14 | +theorem: "Non-invariance and transformation" |
| 15 | + |
| 16 | +sources: |
| 17 | + - authors: "Wikipedia" |
| 18 | + year: 2021 |
| 19 | + title: "Differential entropy" |
| 20 | + in: "Wikipedia, the free encyclopedia" |
| 21 | + pages: "retrieved on 2021-10-07" |
| 22 | + url: "https://en.wikipedia.org/wiki/Differential_entropy#Properties_of_differential_entropy" |
| 23 | + - authors: "Bernhard" |
| 24 | + year: 2016 |
| 25 | + title: "proof of upper bound on differential entropy of f(X)" |
| 26 | + in: "StackExchange Mathematics" |
| 27 | + pages: "retrieved on 2021-10-07" |
| 28 | + url: "https://math.stackexchange.com/a/1759531" |
| 29 | + - authors: "peek-a-boo" |
| 30 | + year: 2019 |
| 31 | + title: "How to come up with the Jacobian in the change of variables formula" |
| 32 | + in: "StackExchange Mathematics" |
| 33 | + pages: "retrieved on 2021-08-30" |
| 34 | + url: "https://math.stackexchange.com/a/3239222" |
| 35 | + - authors: "Wikipedia" |
| 36 | + year: 2021 |
| 37 | + title: "Jacobian matrix and determinant" |
| 38 | + in: "Wikipedia, the free encyclopedia" |
| 39 | + pages: "retrieved on 2021-10-07" |
| 40 | + url: "https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant#Inverse" |
| 41 | + - authors: "Wikipedia" |
| 42 | + year: 2021 |
| 43 | + title: "Inverse function theorem" |
| 44 | + in: "Wikipedia, the free encyclopedia" |
| 45 | + pages: "retrieved on 2021-10-07" |
| 46 | + url: "https://en.wikipedia.org/wiki/Inverse_function_theorem#Statement" |
| 47 | + - authors: "Wikipedia" |
| 48 | + year: 2021 |
| 49 | + title: "Determinant" |
| 50 | + in: "Wikipedia, the free encyclopedia" |
| 51 | + pages: "retrieved on 2021-10-07" |
| 52 | + url: "https://en.wikipedia.org/wiki/Determinant#Properties_of_the_determinant" |
| 53 | + |
| 54 | +proof_id: "P262" |
| 55 | +shortcut: "dent-noninv" |
| 56 | +username: "JoramSoch" |
| 57 | +--- |
| 58 | + |
| 59 | + |
| 60 | +**Theorem:** The [differential entropy](/D/dent) is not invariant under change of variables, i.e. there exist random variables $X$ and $Y = g(X)$, such that |
| 61 | + |
| 62 | +$$ \label{eq:dent-noninv} |
| 63 | +\mathrm{h}(Y) \neq \mathrm{h}(X) \; . |
| 64 | +$$ |
| 65 | + |
| 66 | +In particular, for an invertible transformation $g: X \rightarrow Y$ from a random vector $X$ to another random vector of the same dimension $Y$, it holds that |
| 67 | + |
| 68 | +$$ \label{eq:dent-trans} |
| 69 | +\mathrm{h}(Y) = \mathrm{h}(X) + \int_{\mathcal{X}} f_X(x) \log \left| J_g(x) \right| \, \mathrm{d}x \; . |
| 70 | +$$ |
| 71 | + |
| 72 | +where $J_g(x)$ is the Jacobian matrix of the vector-valued function $g$ and $\mathcal{X}$ is the set of possible values of $X$. |
| 73 | + |
| 74 | + |
| 75 | +**Proof:** By definition, the [differential entropy](/D/dent) of $X$ is |
| 76 | + |
| 77 | +$$ \label{eq:X-dent} |
| 78 | +\mathrm{h}(X) = - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x |
| 79 | +$$ |
| 80 | + |
| 81 | +where $f_X(x)$ is the [probability density function](/D/pdf) of $X$. |
| 82 | + |
| 83 | +The [probability density function of an invertible function of a continuous random vector](/P/pdf-invfct) $Y = g(X)$ is |
| 84 | + |
| 85 | +$$ \label{eq:pdf-invfct} |
| 86 | +f_Y(y) = \left\{ |
| 87 | +\begin{array}{rl} |
| 88 | +f_X(g^{-1}(y)) \, \left| J_{g^{-1}}(y) \right| \; , & \text{if} \; y \in \mathcal{Y} \\ |
| 89 | +0 \; , & \text{if} \; y \notin \mathcal{Y} |
| 90 | +\end{array} |
| 91 | +\right. |
| 92 | +$$ |
| 93 | + |
| 94 | +where $\mathcal{Y} = \left\lbrace y = g(x): x \in \mathcal{X} \right\rbrace$ is the set of possible outcomes of $Y$ and $J_{g^{-1}}(y)$ is the Jacobian matrix of $g^{-1}(y)$ |
| 95 | + |
| 96 | +$$ \label{eq:jac} |
| 97 | +J_{g^{-1}}(y) = \left[ \begin{matrix} |
| 98 | +\frac{\mathrm{d}x_1}{\mathrm{d}y_1} & \ldots & \frac{\mathrm{d}x_1}{\mathrm{d}y_n} \\ |
| 99 | +\vdots & \ddots & \vdots \\ |
| 100 | +\frac{\mathrm{d}x_n}{\mathrm{d}y_1} & \ldots & \frac{\mathrm{d}x_n}{\mathrm{d}y_n} |
| 101 | +\end{matrix} \right] \; . |
| 102 | +$$ |
| 103 | + |
| 104 | +Thus, the [differential entropy](/D/dent) of $Y$ is |
| 105 | + |
| 106 | +$$ \label{eq:Y-dent-s1} |
| 107 | +\begin{split} |
| 108 | +\mathrm{h}(Y) &\overset{\eqref{eq:X-dent}}{=} - \int_{\mathcal{Y}} f_Y(y) \log f_Y(y) \, \mathrm{d}y \\ |
| 109 | +&\overset{\eqref{eq:pdf-invfct}}{=} - \int_{\mathcal{Y}} \left[ f_X(g^{-1}(y)) \, \left| J_{g^{-1}}(y) \right| \right] \log \left[ f_X(g^{-1}(y)) \, \left| J_{g^{-1}}(y) \right| \right] \, \mathrm{d}y \; . |
| 110 | +\end{split} |
| 111 | +$$ |
| 112 | + |
| 113 | +Substituting $y = g(x)$ into the integral and applying $J_{f^{-1}}(y) = J_f^{-1}(x)$, we obtain |
| 114 | + |
| 115 | +$$ \label{eq:Y-dent-s2} |
| 116 | +\begin{split} |
| 117 | +\mathrm{h}(Y) &= - \int_{\mathcal{X}} \left[ f_X(g^{-1}(g(x))) \, \left| J_{g^{-1}}(y) \right| \right] \log \left[ f_X(g^{-1}(g(x))) \, \left| J_{g^{-1}}(y) \right| \right] \, \mathrm{d}[g(x)] \\ |
| 118 | +&= - \int_{\mathcal{X}} \left[ f_X(x) \, \left| J_g^{-1}(x) \right| \right] \log \left[ f_X(x) \, \left| J_g^{-1}(x) \right| \right] \, \mathrm{d}[g(x)] \; . |
| 119 | +\end{split} |
| 120 | +$$ |
| 121 | + |
| 122 | +Using the relations $y = f(x) \Rightarrow \mathrm{d}y = \left| J_f(x) \right| \, \mathrm{d}x$ and $\left|A\right|\left|B\right| = \left|AB\right|$, this becomes |
| 123 | + |
| 124 | +$$ \label{eq:Y-dent-s3} |
| 125 | +\begin{split} |
| 126 | +\mathrm{h}(Y) &= - \int_{\mathcal{X}} \left[ f_X(x) \, \left| J_g^{-1}(x) \right| \left| J_g(x) \right| \right] \log \left[ f_X(x) \, \left| J_g^{-1}(x) \right| \right] \, \mathrm{d}x \\ |
| 127 | +&= - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x - \int_{\mathcal{X}} f_X(x) \log \left| J_g^{-1}(x) \right| \, \mathrm{d}x \; . |
| 128 | +\end{split} |
| 129 | +$$ |
| 130 | + |
| 131 | +Finally, employing [the fact](/D/pdf) that $\int_{\mathcal{X}} f_X(x) \, \mathrm{d}x = 1$ and the determinant property $\left|A^{-1}\right| = 1/\left|A\right|$, we can derive the [differential entropy](/D/dent) of $Y$ as |
| 132 | + |
| 133 | +$$ \label{eq:Y-dent-s4} |
| 134 | +\begin{split} |
| 135 | +\mathrm{h}(Y) &= - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x - \int_{\mathcal{X}} f_X(x) \log \frac{1}{\left| J_g(x) \right|} \, \mathrm{d}x \\ |
| 136 | +&\overset{\eqref{eq:X-dent}}{=} \mathrm{h}(X) + \int_{\mathcal{X}} f_X(x) \log \left| J_g(x) \right| \, \mathrm{d}x \; . |
| 137 | +\end{split} |
| 138 | +$$ |
| 139 | + |
| 140 | +Because there exist $X$ and $Y$, such that the integral term in \eqref{eq:Y-dent-s4} is non-zero, this also demonstrates that there exist $X$ and $Y$, such that \eqref{eq:dent-noninv} is fulfilled. |
0 commit comments