added 2 proofs

Author: JoramSoch

P/dent-add.md

@@ -0,0 +1,88 @@
+---
+layout: proof
+mathjax: true
+
+author: "Joram Soch"
+affiliation: "BCCN Berlin"
+e_mail: "joram.soch@bccn-berlin.de"
+date: 2020-12-02 16:39:00
+
+title: "Addition of the differential entropy upon multiplication with a constant"
+chapter: "General Theorems"
+section: "Information theory"
+topic: "Differential entropy"
+theorem: "Addition upon multiplication"
+
+sources:
+  - authors: "Wikipedia"
+    year: 2020
+    title: "Differential entropy"
+    in: "Wikipedia, the free encyclopedia"
+    pages: "retrieved on 2020-02-12"
+    url: "https://en.wikipedia.org/wiki/Differential_entropy#Properties_of_differential_entropy"
+
+proof_id: "P200"
+shortcut: "dent-add"
+username: "JoramSoch"
+---
+
+
+**Theorem:** Let $X$ be a [continuous](/D/rvar-disc) [random variable](/D/rvar). Then, the [differential entropy](/D/dent) of $X$ increases additively upon multiplication with a constant:
+
+$$ \label{eq:dent-add}
+\mathrm{h}(aX) = \mathrm{h}(X) + \log |a| \; .
+$$
+
+
+**Proof:** By definition, the [differential entropy](/D/dent) of $X$ is
+
+$$ \label{eq:X-dent}
+\mathrm{h}(X) = - \int_{\mathcal{X}} p(x) \log p(x) \, \mathrm{d}x
+$$
+
+where $p(x) = f_X(x)$ is the [probability density function](/D/pdf) of $X$.
+
+Define the mappings between $X$ and $Y = aX$ as
+
+$$ \label{eq:X-Y}
+Y = g(X) = aX \quad \Leftrightarrow \quad X = g^{-1}(Y) = \frac{Y}{a} \; .
+$$
+
+If $a > 0$, then $g(X)$ is a [strictly increasing function, such that the probability density function](/P/pdf-sifct) of $Y$ is
+
+$$ \label{eq:Y-pdf-c1}
+f_Y(y) = f_X(g^{-1}(y)) \, \frac{\mathrm{d}g^{-1}(y)}{\mathrm{d}y} \overset{\eqref{eq:X-Y}}{=} \frac{1}{a} \, f_X\left(\frac{y}{a}\right) \; ,
+$$
+
+and if $a < 0$, then $g(X)$ is a [strictly decreasing function, such that the probability density function](/P/pdf-sifct) of $Y$ is
+
+$$ \label{eq:Y-pdf-c2}
+f_Y(y) = - f_X(g^{-1}(y)) \, \frac{\mathrm{d}g^{-1}(y)}{\mathrm{d}y} \overset{\eqref{eq:X-Y}}{=} -\frac{1}{a} \, f_X\left(\frac{y}{a}\right) \; ,
+$$
+
+such that we can write
+
+$$ \label{eq:Y-pdf}
+f_Y(y) = \left| \frac{1}{a} \right| \, f_X\left(\frac{y}{a}\right) \; .
+$$
+
+Writing down the differential entropy for $Y$, we have:
+
+$$ \label{eq:Y-dent-s1}
+\begin{split}
+\mathrm{h}(Y) &= - \int_{\mathcal{Y}} f_Y(y) \log f_Y(y) \, \mathrm{d}y \\
+&\overset{\eqref{eq:Y-pdf}}{=} - \int_{\mathcal{Y}} \left| \frac{1}{a} \right| \, f_X\left(\frac{y}{a}\right) \log \left[ \left| \frac{1}{a} \right| \, f_X\left(\frac{y}{a}\right) \right] \, \mathrm{d}y
+\end{split}
+$$
+
+Substituting $x = y/a$, such that $y = ax$, this yields:
+
+$$ \label{eq:Y-dent-s2}
+\begin{split}
+\mathrm{h}(Y) &= - \int_{\left\lbrace y/a \,|\, y \in {\mathcal{Y}} \right\rbrace} \left| \frac{1}{a} \right| \, f_X\left(\frac{ax}{a}\right) \log \left[ \left| \frac{1}{a} \right| \, f_X\left(\frac{ax}{a}\right) \right] \, \mathrm{d}(ax) \\
+&= - \int_{\mathcal{X}} f_X(x) \log \left[ \left| \frac{1}{a} \right| \, f_X(x) \right] \, \mathrm{d}x \\
+&= - \int_{\mathcal{X}} f_X(x) \left[ \log f_X(x) - \log |a| \right] \, \mathrm{d}x \\
+&= - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x + \log |a| \int_{\mathcal{X}} f_X(x) \, \mathrm{d}x \\
+&\overset{\eqref{eq:X-dent}}{=} \mathrm{h}(X) + \log |a| \; .
+\end{split}
+$$

P/dent-inv.md

@@ -0,0 +1,74 @@
+---
+layout: proof
+mathjax: true
+
+author: "Joram Soch"
+affiliation: "BCCN Berlin"
+e_mail: "joram.soch@bccn-berlin.de"
+date: 2020-12-02 16:11:00
+
+title: "Invariance of the differential entropy under addition of a constant"
+chapter: "General Theorems"
+section: "Information theory"
+topic: "Differential entropy"
+theorem: "Invariance under addition"
+
+sources:
+  - authors: "Wikipedia"
+    year: 2020
+    title: "Differential entropy"
+    in: "Wikipedia, the free encyclopedia"
+    pages: "retrieved on 2020-02-12"
+    url: "https://en.wikipedia.org/wiki/Differential_entropy#Properties_of_differential_entropy"
+
+proof_id: "P199"
+shortcut: "dent-inv"
+username: "JoramSoch"
+---
+
+
+**Theorem:** Let $X$ be a [continuous](/D/rvar-disc) [random variable](/D/rvar). Then, the [differential entropy](/D/dent) of $X$ remains constant under addition of a constant:
+
+$$ \label{eq:dent-inv}
+\mathrm{h}(X + c) = \mathrm{h}(X) \; .
+$$
+
+
+**Proof:** By definition, the [differential entropy](/D/dent) of $X$ is
+
+$$ \label{eq:X-dent}
+\mathrm{h}(X) = - \int_{\mathcal{X}} p(x) \log p(x) \, \mathrm{d}x
+$$
+
+where $p(x) = f_X(x)$ is the [probability density function](/D/pdf) of $X$.
+
+Define the mappings between $X$ and $Y = X + c$ as
+
+$$ \label{eq:X-Y}
+Y = g(X) = X + c \quad \Leftrightarrow \quad X = g^{-1}(Y) = X - c \; .
+$$
+
+Note that $g(X)$ is a [strictly increasing function, such that the probability density function](/P/pdf-sifct) of $Y$ is
+
+$$ \label{eq:Y-pdf}
+f_Y(y) = f_X(g^{-1}(y)) \, \frac{\mathrm{d}g^{-1}(y)}{\mathrm{d}y} \overset{\eqref{eq:X-Y}}{=} f_X(y-c) \; .
+$$
+
+Writing down the differential entropy for $Y$, we have:
+
+$$ \label{eq:Y-dent-s1}
+\begin{split}
+\mathrm{h}(Y) &= - \int_{\mathcal{Y}} f_Y(y) \log f_Y(y) \, \mathrm{d}y \\
+&\overset{\eqref{eq:Y-pdf}}{=} - \int_{\mathcal{Y}} f_X(y-c) \log f_X(y-c) \, \mathrm{d}y
+\end{split}
+$$
+
+Substituting $x = y - c$, such that $y = x + c$, this yields:
+
+$$ \label{eq:Y-dent-s2}
+\begin{split}
+\mathrm{h}(Y) &= - \int_{\left\lbrace y-c \,|\, y \in {\mathcal{Y}} \right\rbrace} f_X(x+c-c) \log f_X(x+c-c) \, \mathrm{d}(x+c) \\
+&= - \int_{\mathcal{X}} f_X(x) \log f_X(x) \, \mathrm{d}x \\
+&\overset{\eqref{eq:X-dent}}{=} \mathrm{h}(X) \; .
+\end{split}
+$$