Merge pull request #34 from StatProofBook/master

JoramSoch · web-flow · commit 2ed393befaf0 · 2020-10-08T07:10:09.000+02:00
update to master
diff --git a/D/cdf.md b/D/cdf.md
@@ -16,7 +16,7 @@ definition: "Cumulative distribution function"
 sources:
   - authors: "Wikipedia"
     year: 2020
-    title: "Probability density function"
+    title: "Cumulative distribution function"
     in: "Wikipedia, the free encyclopedia"
     pages: "retrieved on 2020-02-17"
     url: "https://en.wikipedia.org/wiki/Cumulative_distribution_function#Definition"
@@ -27,8 +27,13 @@ username: "JoramSoch"
 ---
 
 
-**Definition:**
+**Definition:** The cumulative distribution function (CDF) of a [random variable](/D/rvar) $X$ at a given value $x$ is defined as the [probability](/D/prob) that $X$ is smaller than $x$:
 
+$$ \label{eq:cdf}
+F_X(x) = \mathrm{Pr}(X \leq x) \; .
+$$
+
+<br>
 1) Let $X$ be a discrete [random variable](/D/rvar) with possible outcomes $\mathcal{X}$ and the [probability mass function](/D/pmf) $f_X(x)$. Then, the function $F_X(x): \mathbb{R} \to [0,1]$ with
 
 $$ \label{eq:cdf-disc}
diff --git a/D/mom.md b/D/mom.md
@@ -30,5 +30,5 @@ username: "JoramSoch"
 **Definition:** Let $X$ be a [random variable](/D/rvar) and let $n$ be a positive integer. Then, the $n$-th moment of $X$, also called ($n$-th) "raw moment" or "crude moment", is defined as the [expected value](/D/mean) of the $n$-th power of $X$:
 
 $$ \label{eq:mom}
-\mu_n = \mathrm{E}[X^n] \; .
+\mu_n' = \mathrm{E}[X^n] \; .
 $$
diff --git a/I/Definition_by_Author.md b/I/Definition_by_Author.md
@@ -4,7 +4,7 @@ title: "Definition by Author"
 ---
 
 
-### JoramSoch (92 definitions)
+### JoramSoch (93 definitions)
 
 - [Akaike information criterion](/D/aic)
 - [Bayesian information criterion](/D/bic)
@@ -19,6 +19,7 @@ title: "Definition by Author"
 - [Conditional differential entropy](/D/dent-cond)
 - [Conditional entropy](/D/ent-cond)
 - [Conditional probability distribution](/D/dist-cond)
+- [Constant](/D/const)
 - [Continuous uniform distribution](/D/cuni)
 - [Correlation](/D/corr)
 - [Correlation matrix](/D/corrmat)
diff --git a/I/Definition_by_Number.md b/I/Definition_by_Number.md
@@ -102,3 +102,4 @@ title: "Definition by Number"
 | D93 | encm | [Encompassing model](/D/encm) | tomfaulkenberry | 2020-09-02 |
 | D94 | std | [Standard deviation](/D/std) | JoramSoch | 2020-09-03 |
 | D95 | wald | [Wald distribution](/D/wald) | tomfaulkenberry | 2020-09-04 |
+| D96 | const | [Constant](/D/const) | JoramSoch | 2020-09-09 |
diff --git a/I/Definition_by_Topic.md b/I/Definition_by_Topic.md
@@ -26,6 +26,7 @@ title: "Definition by Topic"
 - [Conditional differential entropy](/D/dent-cond)
 - [Conditional entropy](/D/ent-cond)
 - [Conditional probability distribution](/D/dist-cond)
+- [Constant](/D/const)
 - [Continuous uniform distribution](/D/cuni)
 - [Correlation](/D/corr)
 - [Correlation matrix](/D/corrmat)
diff --git a/I/Proof_by_Author.md b/I/Proof_by_Author.md
@@ -4,7 +4,7 @@ title: "Proof by Author"
 ---
 
 
-### JoramSoch (159 proofs)
+### JoramSoch (163 proofs)
 
 - [(Non-)Multiplicativity of the expected value](/P/mean-mult)
 - [Additivity of the Kullback-Leibler divergence for independent distributions](/P/kl-add)
@@ -44,7 +44,9 @@ title: "Proof by Author"
 - [Expected value of a non-negative random variable](/P/mean-nnrvar)
 - [Exponential distribution is a special case of gamma distribution](/P/exp-gam)
 - [Expression of the cumulative distribution function of the normal distribution without the error function](/P/norm-cdfwerf)
+- [First central moment is zero](/P/momcent-1st)
 - [Full width at half maximum for the normal distribution](/P/norm-fwhm)
+- [Gibbs' inequality](/P/gibbs-ineq)
 - [Invariance of the Kullback-Leibler divergence under parameter transformation](/P/kl-inv)
 - [Invariance of the variance under addition of a constant](/P/var-inv)
 - [Joint likelihood is the product of likelihood function and prior density](/P/jl-lfnprior)
@@ -63,6 +65,7 @@ title: "Proof by Author"
 - [Log model evidence for multinomial observations](/P/mult-lme)
 - [Log model evidence for multivariate Bayesian linear regression](/P/mblr-lme)
 - [Log model evidence for the Poisson distribution with exposure values](/P/poissexp-lme)
+- [Log sum inequality](/P/logsum-ineq)
 - [Log-odds and probability in logistic regression](/P/logreg-lonp)
 - [Logarithmic expectation of the gamma distribution](/P/gam-logmean)
 - [Marginal distributions of the multivariate normal distribution](/P/mvn-marg)
@@ -93,6 +96,7 @@ title: "Proof by Author"
 - [Moment-generating function of the normal distribution](/P/norm-mgf)
 - [Monotonicity of the expected value](/P/mean-mono)
 - [Non-negativity of the Kullback-Leibler divergence](/P/kl-nonneg)
+- [Non-negativity of the Kullback-Leibler divergence](/P/kl-nonneg2)
 - [Non-negativity of the Shannon entropy](/P/ent-nonneg)
 - [Non-negativity of the expected value](/P/mean-nonneg)
 - [Non-negativity of the variance](/P/var-nonneg)
@@ -170,9 +174,12 @@ title: "Proof by Author"
 
 - [Proof Template](/P/-temp-)
 
-### tomfaulkenberry (4 proofs)
+### tomfaulkenberry (7 proofs)
 
 - [Encompassing Prior Method for computing Bayes Factors](/P/bf-ep)
+- [Mean of the Wald distribution](/P/wald-mean)
+- [Moment-generating function of the Wald distribution](/P/wald-mgf)
 - [Probability density function of the Wald distribution](/P/wald-pdf)
 - [Savage-Dickey Density Ratio for computing Bayes Factors](/P/bf-sddr)
 - [Transitivity of Bayes Factors](/P/bf-trans)
+- [Variance of the Wald distribution](/P/wald-var)
diff --git a/I/Proof_by_Number.md b/I/Proof_by_Number.md
@@ -170,3 +170,10 @@ title: "Proof by Number"
 | P161 | mblr-lme | [Log model evidence for multivariate Bayesian linear regression](/P/mblr-lme) | JoramSoch | 2020-09-03 |
 | P162 | wald-pdf | [Probability density function of the Wald distribution](/P/wald-pdf) | tomfaulkenberry | 2020-09-04 |
 | P163 | bf-trans | [Transitivity of Bayes Factors](/P/bf-trans) | tomfaulkenberry | 2020-09-07 |
+| P164 | gibbs-ineq | [Gibbs' inequality](/P/gibbs-ineq) | JoramSoch | 2020-09-09 |
+| P165 | logsum-ineq | [Log sum inequality](/P/logsum-ineq) | JoramSoch | 2020-09-09 |
+| P166 | kl-nonneg2 | [Non-negativity of the Kullback-Leibler divergence](/P/kl-nonneg2) | JoramSoch | 2020-09-09 |
+| P167 | momcent-1st | [First central moment is zero](/P/momcent-1st) | JoramSoch | 2020-09-09 |
+| P168 | wald-mgf | [Moment-generating function of the Wald distribution](/P/wald-mgf) | tomfaulkenberry | 2020-09-13 |
+| P169 | wald-mean | [Mean of the Wald distribution](/P/wald-mean) | tomfaulkenberry | 2020-09-13 |
+| P170 | wald-var | [Variance of the Wald distribution](/P/wald-var) | tomfaulkenberry | 2020-09-13 |
diff --git a/I/Proof_by_Topic.md b/I/Proof_by_Topic.md
@@ -63,8 +63,13 @@ title: "Proof by Topic"
 
 ### F
 
+- [First central moment is zero](/P/momcent-1st)
 - [Full width at half maximum for the normal distribution](/P/norm-fwhm)
 
+### G
+
+- [Gibbs' inequality](/P/gibbs-ineq)
+
 ### I
 
 - [Invariance of the Kullback-Leibler divergence under parameter transformation](/P/kl-inv)
@@ -94,6 +99,7 @@ title: "Proof by Topic"
 - [Log model evidence for multinomial observations](/P/mult-lme)
 - [Log model evidence for multivariate Bayesian linear regression](/P/mblr-lme)
 - [Log model evidence for the Poisson distribution with exposure values](/P/poissexp-lme)
+- [Log sum inequality](/P/logsum-ineq)
 - [Log-odds and probability in logistic regression](/P/logreg-lonp)
 - [Logarithmic expectation of the gamma distribution](/P/gam-logmean)
 
@@ -108,6 +114,7 @@ title: "Proof by Topic"
 - [Maximum likelihood estimator of variance is biased](/P/resvar-bias)
 - [Mean of the Bernoulli distribution](/P/bern-mean)
 - [Mean of the Poisson distribution](/P/poiss-mean)
+- [Mean of the Wald distribution](/P/wald-mean)
 - [Mean of the binomial distribution](/P/bin-mean)
 - [Mean of the categorical distribution](/P/cat-mean)
 - [Mean of the continuous uniform distribution](/P/cuni-mean)
@@ -124,12 +131,14 @@ title: "Proof by Topic"
 - [Mode of the normal distribution](/P/norm-mode)
 - [Moment in terms of moment-generating function](/P/mom-mgf)
 - [Moment-generating function of linear combination of independent random variables](/P/mgf-lincomb)
+- [Moment-generating function of the Wald distribution](/P/wald-mgf)
 - [Moment-generating function of the normal distribution](/P/norm-mgf)
 - [Monotonicity of the expected value](/P/mean-mono)
 
 ### N
 
 - [Non-negativity of the Kullback-Leibler divergence](/P/kl-nonneg)
+- [Non-negativity of the Kullback-Leibler divergence](/P/kl-nonneg2)
 - [Non-negativity of the Shannon entropy](/P/ent-nonneg)
 - [Non-negativity of the expected value](/P/mean-nonneg)
 - [Non-negativity of the variance](/P/var-nonneg)
@@ -219,6 +228,7 @@ title: "Proof by Topic"
 ### V
 
 - [Variance of constant is zero](/P/var-const)
+- [Variance of the Wald distribution](/P/wald-var)
 - [Variance of the gamma distribution](/P/gam-var)
 - [Variance of the linear combination of two random variables](/P/var-lincomb)
 - [Variance of the normal distribution](/P/norm-var)
diff --git a/I/Proofs_without_Source.md b/I/Proofs_without_Source.md
@@ -25,6 +25,7 @@ title: "Proofs without Source"
 - [Maximum likelihood estimation for Poisson-distributed data](/P/poiss-mle)
 - [Maximum likelihood estimation for multiple linear regression](/P/mlr-mle)
 - [Maximum likelihood estimation for the general linear model](/P/glm-mle)
+- [Mean of the Wald distribution](/P/wald-mean)
 - [Mean of the categorical distribution](/P/cat-mean)
 - [Mean of the continuous uniform distribution](/P/cuni-mean)
 - [Mean of the multinomial distribution](/P/mult-mean)
@@ -62,5 +63,6 @@ title: "Proofs without Source"
 - [Relationship between signal-to-noise ratio and R²](/P/snr-rsq)
 - [Transitivity of Bayes Factors](/P/bf-trans)
 - [Transposition of a matrix-normal random variable](/P/matn-trans)
+- [Variance of the Wald distribution](/P/wald-var)
 - [Weighted least squares for multiple linear regression](/P/mlr-wls2)
 - [Weighted least squares for the general linear model](/P/glm-wls)
diff --git a/I/Table_of_Contents.md b/I/Table_of_Contents.md
@@ -226,16 +226,16 @@ title: "Table of Contents"
    &emsp;&ensp; 3.2.2. *[Standard normal distribution](/D/snorm)* <br>
    &emsp;&ensp; 3.2.3. **[Relation to standard normal distribution](/P/norm-snorm)** <br>
    &emsp;&ensp; 3.2.4. **[Probability density function](/P/norm-pdf)** <br>
-   &emsp;&ensp; 3.2.5. **[Cumulative distribution function](/P/norm-cdf)** <br>
-   &emsp;&ensp; 3.2.6. **[Cumulative distribution function without error function](/P/norm-cdfwerf)** <br>
-   &emsp;&ensp; 3.2.7. **[Quantile function](/P/norm-qf)** <br>
-   &emsp;&ensp; 3.2.8. **[Mean](/P/norm-mean)** <br>
-   &emsp;&ensp; 3.2.9. **[Median](/P/norm-med)** <br>
-   &emsp;&ensp; 3.2.10. **[Mode](/P/norm-mode)** <br>
-   &emsp;&ensp; 3.2.11. **[Variance](/P/norm-var)** <br>
-   &emsp;&ensp; 3.2.12. **[Full width at half maximum](/P/norm-fwhm)** <br>
-   &emsp;&ensp; 3.2.13. **[Differential entropy](/P/norm-dent)** <br>
-   &emsp;&ensp; 3.2.14. **[Moment-generating function](/P/norm-mgf)** <br>
+   &emsp;&ensp; 3.2.5. **[Moment-generating function](/P/norm-mgf)** <br>
+   &emsp;&ensp; 3.2.6. **[Cumulative distribution function](/P/norm-cdf)** <br>
+   &emsp;&ensp; 3.2.7. **[Cumulative distribution function without error function](/P/norm-cdfwerf)** <br>
+   &emsp;&ensp; 3.2.8. **[Quantile function](/P/norm-qf)** <br>
+   &emsp;&ensp; 3.2.9. **[Mean](/P/norm-mean)** <br>
+   &emsp;&ensp; 3.2.10. **[Median](/P/norm-med)** <br>
+   &emsp;&ensp; 3.2.11. **[Mode](/P/norm-mode)** <br>
+   &emsp;&ensp; 3.2.12. **[Variance](/P/norm-var)** <br>
+   &emsp;&ensp; 3.2.13. **[Full width at half maximum](/P/norm-fwhm)** <br>
+   &emsp;&ensp; 3.2.14. **[Differential entropy](/P/norm-dent)** <br>
 
    3.3. Gamma distribution <br>
    &emsp;&ensp; 3.3.1. *[Definition](/D/gam)* <br>
@@ -264,6 +264,9 @@ title: "Table of Contents"
    3.6. Wald distribution <br>
    &emsp;&ensp; 3.6.1. *[Definition](/D/wald)* <br>
    &emsp;&ensp; 3.6.2. **[Probability density function](/P/wald-pdf)** <br>
+   &emsp;&ensp; 3.6.3. **[Moment-generating function](/P/wald-mgf)** <br>
+   &emsp;&ensp; 3.6.4. **[Mean](/P/wald-mean)** <br>
+   &emsp;&ensp; 3.6.5. **[Variance](/P/wald-var)** <br>   
 
 4. Multivariate continuous distributions
 
diff --git a/P/bma-lme.md b/P/bma-lme.md
@@ -37,7 +37,7 @@ $$
 where $p(\theta \vert m_i,y)$ is the posterior distributions over $\theta$ obtained using $m_i$.
 
 
-**Proof:** According to the [law of marginal probability](/D/prob-marg), the probability of the shared parameters $\theta$ conditional on the measured data $y$ [can be obtained](/D/bma-der) by marginalizing over the discrete variable model $m$:
+**Proof:** According to the [law of marginal probability](/D/prob-marg), the probability of the shared parameters $\theta$ conditional on the measured data $y$ [can be obtained](/P/bma-der) by marginalizing over the discrete variable model $m$:
 
 $$ \label{eq:BMA-PMP}
 p(\theta|y) = \sum_{i=1}^{M} p(\theta|m_i,y) \cdot p(m_i|y) \; ,
diff --git a/P/ent-nonneg.md b/P/ent-nonneg.md
@@ -36,23 +36,23 @@ $$
 **Proof:** The [entropy of a discrete random variable](/D/ent) is defined as
 
 $$ \label{eq:ent}
-\mathrm{H}(X) = - \sum_{i=1}^{k} p(x_i) \cdot \log_b p(x_i)
+\mathrm{H}(X) = - \sum_{x \in \mathcal{X}} p(x) \cdot \log_b p(x)
 $$
 
 The minus sign can be moved into the sum:
 
 $$ \label{eq:ent-dev}
-\mathrm{H}(X) = \sum_{i=1}^{k} \left[ p(x_i) \cdot \left( - \log_b p(x_i) \right) \right]
+\mathrm{H}(X) = \sum_{x \in \mathcal{X}} \left[ p(x) \cdot \left( - \log_b p(x) \right) \right]
 $$
 
 Because the co-domain of [probability mass functions](/D/pmf) is $[0,1]$, we can deduce:
 
 $$ \label{eq:nonneg}
 \begin{array}{rcccl}
-0 &\leq &p(x_i) &\leq &1 \\
--\infty &\leq &\log_b p(x_i) &\leq &0 \\
-0 &\leq &-\log_b p(x_i) &\leq &+\infty \\
-0 &\leq &p(x_i) \cdot \left(-\log_b p(x_i)\right) &\leq &+\infty \; .
+0 &\leq &p(x) &\leq &1 \\
+-\infty &\leq &\log_b p(x) &\leq &0 \\
+0 &\leq &-\log_b p(x) &\leq &+\infty \\
+0 &\leq &p(x) \cdot \left(-\log_b p(x)\right) &\leq &+\infty \; .
 \end{array}
 $$
 
diff --git a/P/gibbs-ineq.md b/P/gibbs-ineq.md
@@ -43,32 +43,32 @@ $$
 Let $I$ be the of all $x$ for which $p(x)$ is non-zero. Then, proving \eqref{eq:Gibbs-ineq} requires to show that
 
 $$ \label{eq:Gibbs-ineq-s1}
-\sum_{x \in I} p(x) \, \frac{\ln p(x)}{\ln q(x)} \geq 0 \; .
+\sum_{x \in I} p(x) \, \ln \frac{p(x)}{q(x)} \geq 0 \; .
 $$
 
 Because $\ln x \leq x - 1$, i.e. $-\ln x \geq 1 - x$, for all $x > 0$, with equality only if $x = 1$, we can say about the left-hand side that
 
 $$ \label{eq:Gibbs-ineq-s2}
 \begin{split}
-\sum_{x \in I} p(x) \, \frac{\ln p(x)}{\ln q(x)} &\geq \sum_{x \in I} p(x) \left( 1 - \frac{p(x)}{q(x)} \right) \\
+\sum_{x \in I} p(x) \, \ln \frac{p(x)}{q(x)} &\geq \sum_{x \in I} p(x) \left( 1 - \frac{p(x)}{q(x)} \right) \\
 &= \sum_{x \in I} p(x) - \sum_{x \in I} q(x) \; .
 \end{split}
 $$
 
-Finally, since $p(x)$ and $q(x)$ are [probability mass functions](/D/pmf), it holds that $0 \leq p(x),q(x) \leq 1$ and we also have
+Finally, since $p(x)$ and $q(x)$ are [probability mass functions](/D/pmf), we have
 
 $$ \label{eq:p-q-pmf}
 \begin{split}
-\sum_{x \in I} p(x) &= 1 \\
-\sum_{x \in I} q(x) &\leq 1 \; ,
+0 \leq p(x) \leq 1, \quad \sum_{x \in I} p(x) &= 1 \quad \text{and} \\
+0 \leq q(x) \leq 1, \quad \sum_{x \in I} q(x) &\leq 1 \; ,
 \end{split}
 $$
 
 such that it follows from \eqref{eq:Gibbs-ineq-s2} that
 
 $$ \label{eq:Gibbs-ineq-s3}
 \begin{split}
-\sum_{x \in I} p(x) \, \frac{\ln p(x)}{\ln q(x)} &\geq \sum_{x \in I} p(x) - \sum_{x \in I} q(x) \\
+\sum_{x \in I} p(x) \, \ln \frac{p(x)}{q(x)} &\geq \sum_{x \in I} p(x) - \sum_{x \in I} q(x) \\
 &= 1 - \sum_{x \in I} q(x) \geq 0 \; .
 \end{split}
 $$
diff --git a/P/kl-nonneg.md b/P/kl-nonneg.md
@@ -39,13 +39,13 @@ with $\mathrm{KL}[P \vert \vert Q] = 0$, if and only if $P = Q$.
 **Proof:** The discrete [Kullback-Leibler divergence](/D/kl) is defined as
 
 $$ \label{eq:KL}
-\mathrm{KL}[P||Q] = \sum_{x \in \mathcal{X}} p(x) \cdot \log \frac{p(x)}{q(x)} \, \mathrm{d}x
+\mathrm{KL}[P||Q] = \sum_{x \in \mathcal{X}} p(x) \cdot \log \frac{p(x)}{q(x)}
 $$
 
 which can be reformulated into
 
 $$ \label{eq:KL-dev}
-\mathrm{KL}[P||Q] = \sum_{x \in \mathcal{X}} p(x) \cdot \log p(x) - \sum_{x \in \mathcal{X}} p(x) \cdot \log q(x) \, \mathrm{d}x \; .
+\mathrm{KL}[P||Q] = \sum_{x \in \mathcal{X}} p(x) \cdot \log p(x) - \sum_{x \in \mathcal{X}} p(x) \cdot \log q(x) \; .
 $$
 
 [Gibbs' inequality](/P/gibbs-ineq) states that the [entropy](/D/ent) of a probability distribution is always less than or equal to the [cross-entropy](/D/ent-cross) with another probability distribution – with equality only if the distributions are identical –,
diff --git a/P/kl-nonneg2.md b/P/kl-nonneg2.md
@@ -50,12 +50,12 @@ $$
 
 where $a_1, \ldots, a_n$ and $b_1, \ldots, b_n$ be non-negative real numbers and $a = \sum_{i=1}^{n} a_i$ and $b = \sum_{i=1}^{n} b_i$. Because $p(x)$ and $q(x)$ are [probability mass functions](/D/pmf), such that
 
-\begin{equation} \label{eq:p-q-pmf}
+$$ \label{eq:p-q-pmf}
 \begin{split}
 p(x) \geq 0, \quad \sum_{x \in \mathcal{X}} p(x) &= 1 \quad \text{and} \\
 q(x) \geq 0, \quad \sum_{x \in \mathcal{X}} q(x) &= 1 \; ,
 \end{split}
-\end{equation}
+$$
 
 theorem \eqref{eq:KL-nonneg} is simply a special case of \eqref{eq:logsum-ineq}, i.e.
 
diff --git a/P/momcent-1st.md b/P/momcent-1st.md
@@ -5,7 +5,7 @@ mathjax: true
 author: "Joram Soch"
 affiliation: "BCCN Berlin"
 e_mail: "joram.soch@bccn-berlin.de"
-date: 2020-08-19 07:51:00
+date: 2020-09-09 07:51:00
 
 title: "First central moment is zero"
 chapter: "General Theorems"
@@ -42,11 +42,11 @@ $$
 
 Due to the [linearity of the expected value](/P/mean-lin) and by plugging in $\mu = \mathrm{E}(X)$, we have
 
-\begin{equation} \label{eq:momcent-1st-qed}
+$$ \label{eq:momcent-1st-qed}
 \begin{split}
 \mu_1 &= \mathrm{E}\left[ X-\mu \right] \\
 &= \mathrm{E}(X) - \mu \\
 &= \mathrm{E}(X) - \mathrm{E}(X) \\
 &= 0 \; .
 \end{split}
-\end{equation}
+$$
diff --git a/P/wald-mean.md b/P/wald-mean.md
diff --git a/P/wald-mgf.md b/P/wald-mgf.md
diff --git a/P/wald-var.md b/P/wald-var.md
diff --git a/S/Model_Selection.md b/S/Model_Selection.md
diff --git a/S/Probability_Distributions.md b/S/Probability_Distributions.md
diff --git a/S/Statistical_Models.md b/S/Statistical_Models.md
