Skip to content

Commit 6aa01e5

Browse files
authored
Merge pull request #93 from StatProofBook/master
update to master
2 parents f1d46db + dc5ac2e commit 6aa01e5

20 files changed

Lines changed: 86 additions & 39 deletions

D/mult.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,4 +33,4 @@ $$ \label{eq:mult}
3333
X \sim \mathrm{Mult}(n, \left[p_1, \ldots, p_k \right]) \; ,
3434
$$
3535

36-
if $X$ are the numbers of observations belonging to $k$ distinct categories in $n$ [independent](/D/ind) trials, where each trial has [$k$ possible outcomes](/D/cat) and the category probabilities are identical across trials.
36+
if $X$ are the numbers of observations belonging to $k$ distinct categories in $n$ [independent](/D/ind) trials, where each trial has $k$ [possible outcomes](/D/cat) and the category probabilities are identical across trials.

I/PbA.md

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ title: "Proof by Author"
88

99
- [Covariance matrix of the multinomial distribution](/P/mult-cov)
1010

11-
### JoramSoch (321 proofs)
11+
### JoramSoch (326 proofs)
1212

1313
- [Accuracy and complexity for the univariate Gaussian](/P/ug-anc)
1414
- [Accuracy and complexity for the univariate Gaussian with known variance](/P/ugkv-anc)
@@ -44,7 +44,9 @@ title: "Proof by Author"
4444
- [Corrected Akaike information criterion in terms of maximum log-likelihood](/P/aicc-mll)
4545
- [Correlation always falls between -1 and +1](/P/corr-range)
4646
- [Correlation coefficient in terms of standard scores](/P/corr-z)
47+
- [Covariance matrices of the matrix-normal distribution](/P/matn-cov)
4748
- [Covariance matrix of the categorical distribution](/P/cat-cov)
49+
- [Covariance matrix of the multivariate normal distribution](/P/mvn-cov)
4850
- [Covariance of independent random variables](/P/cov-ind)
4951
- [Cross-validated log Bayes factor for the univariate Gaussian with known variance](/P/ugkv-cvlbf)
5052
- [Cross-validated log model evidence for the univariate Gaussian with known variance](/P/ugkv-cvlme)
@@ -140,6 +142,7 @@ title: "Proof by Author"
140142
- [Log sum inequality](/P/logsum-ineq)
141143
- [Log-odds and probability in logistic regression](/P/logreg-lonp)
142144
- [Logarithmic expectation of the gamma distribution](/P/gam-logmean)
145+
- [Marginal distributions for the matrix-normal distribution](/P/matn-marg)
143146
- [Marginal distributions of the multivariate normal distribution](/P/mvn-marg)
144147
- [Marginal distributions of the normal-gamma distribution](/P/ng-marg)
145148
- [Marginal likelihood is a definite integral of joint likelihood](/P/ml-jl)
@@ -162,7 +165,9 @@ title: "Proof by Author"
162165
- [Mean of the continuous uniform distribution](/P/cuni-mean)
163166
- [Mean of the exponential distribution](/P/exp-mean)
164167
- [Mean of the gamma distribution](/P/gam-mean)
168+
- [Mean of the matrix-normal distribution](/P/matn-mean)
165169
- [Mean of the multinomial distribution](/P/mult-mean)
170+
- [Mean of the multivariate normal distribution](/P/mvn-mean)
166171
- [Mean of the normal distribution](/P/norm-mean)
167172
- [Mean of the normal-gamma distribution](/P/ng-mean)
168173
- [Mean of the normal-Wishart distribution](/P/nw-mean)
@@ -279,7 +284,7 @@ title: "Proof by Author"
279284
- [Relation of continuous mutual information to joint and conditional differential entropy](/P/cmi-jcde)
280285
- [Relation of continuous mutual information to marginal and conditional differential entropy](/P/cmi-mcde)
281286
- [Relation of continuous mutual information to marginal and joint differential entropy](/P/cmi-mjde)
282-
- [Relation of Kullback-Leibler divergence to entropy](/P/kl-ent)
287+
- [Relation of discrete Kullback-Leibler divergence to Shannon entropy](/P/kl-ent)
283288
- [Relation of mutual information to joint and conditional entropy](/P/dmi-jce)
284289
- [Relation of mutual information to marginal and conditional entropy](/P/dmi-mce)
285290
- [Relation of mutual information to marginal and joint entropy](/P/dmi-mje)

I/PbN.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ title: "Proof by Number"
119119
| P110 | gam-logmean | [Logarithmic expectation of the gamma distribution](/P/gam-logmean) | JoramSoch | 2020-05-25 |
120120
| P111 | norm-snorm | [Relationship between normal distribution and standard normal distribution](/P/norm-snorm) | JoramSoch | 2020-05-26 |
121121
| P112 | gam-sgam | [Relationship between gamma distribution and standard gamma distribution](/P/gam-sgam) | JoramSoch | 2020-05-26 |
122-
| P113 | kl-ent | [Relation of Kullback-Leibler divergence to entropy](/P/kl-ent) | JoramSoch | 2020-05-27 |
122+
| P113 | kl-ent | [Relation of discrete Kullback-Leibler divergence to Shannon entropy](/P/kl-ent) | JoramSoch | 2020-05-27 |
123123
| P114 | kl-dent | [Relation of continuous Kullback-Leibler divergence to differential entropy](/P/kl-dent) | JoramSoch | 2020-05-27 |
124124
| P115 | kl-inv | [Invariance of the Kullback-Leibler divergence under parameter transformation](/P/kl-inv) | JoramSoch | 2020-05-28 |
125125
| P116 | kl-add | [Additivity of the Kullback-Leibler divergence for independent distributions](/P/kl-add) | JoramSoch | 2020-05-31 |
@@ -344,3 +344,8 @@ title: "Proof by Number"
344344
| P336 | cat-ent | [Entropy of the categorical distribution](/P/cat-ent) | JoramSoch | 2022-09-09 |
345345
| P337 | mult-ent | [Entropy of the multinomial distribution](/P/mult-ent) | JoramSoch | 2022-09-09 |
346346
| P338 | cat-cov | [Covariance matrix of the categorical distribution](/P/cat-cov) | JoramSoch | 2022-09-09 |
347+
| P339 | mvn-mean | [Mean of the multivariate normal distribution](/P/mvn-mean) | JoramSoch | 2022-09-15 |
348+
| P340 | mvn-cov | [Covariance matrix of the multivariate normal distribution](/P/mvn-cov) | JoramSoch | 2022-09-15 |
349+
| P341 | matn-mean | [Mean of the matrix-normal distribution](/P/matn-mean) | JoramSoch | 2022-09-15 |
350+
| P342 | matn-cov | [Covariance matrices of the matrix-normal distribution](/P/matn-cov) | JoramSoch | 2022-09-15 |
351+
| P343 | matn-marg | [Marginal distributions for the matrix-normal distribution](/P/matn-marg) | JoramSoch | 2022-09-15 |

I/PbT.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,10 @@ title: "Proof by Topic"
4747
- [Corrected Akaike information criterion in terms of maximum log-likelihood](/P/aicc-mll)
4848
- [Correlation always falls between -1 and +1](/P/corr-range)
4949
- [Correlation coefficient in terms of standard scores](/P/corr-z)
50+
- [Covariance matrices of the matrix-normal distribution](/P/matn-cov)
5051
- [Covariance matrix of the categorical distribution](/P/cat-cov)
5152
- [Covariance matrix of the multinomial distribution](/P/mult-cov)
53+
- [Covariance matrix of the multivariate normal distribution](/P/mvn-cov)
5254
- [Covariance of independent random variables](/P/cov-ind)
5355
- [Cross-validated log Bayes factor for the univariate Gaussian with known variance](/P/ugkv-cvlbf)
5456
- [Cross-validated log model evidence for the univariate Gaussian with known variance](/P/ugkv-cvlme)
@@ -173,6 +175,7 @@ title: "Proof by Topic"
173175

174176
### M
175177

178+
- [Marginal distributions for the matrix-normal distribution](/P/matn-marg)
176179
- [Marginal distributions of the multivariate normal distribution](/P/mvn-marg)
177180
- [Marginal distributions of the normal-gamma distribution](/P/ng-marg)
178181
- [Marginal likelihood is a definite integral of joint likelihood](/P/ml-jl)
@@ -195,7 +198,9 @@ title: "Proof by Topic"
195198
- [Mean of the continuous uniform distribution](/P/cuni-mean)
196199
- [Mean of the exponential distribution](/P/exp-mean)
197200
- [Mean of the gamma distribution](/P/gam-mean)
201+
- [Mean of the matrix-normal distribution](/P/matn-mean)
198202
- [Mean of the multinomial distribution](/P/mult-mean)
203+
- [Mean of the multivariate normal distribution](/P/mvn-mean)
199204
- [Mean of the normal distribution](/P/norm-mean)
200205
- [Mean of the normal-gamma distribution](/P/ng-mean)
201206
- [Mean of the normal-Wishart distribution](/P/nw-mean)
@@ -337,7 +342,7 @@ title: "Proof by Topic"
337342
- [Relation of continuous mutual information to joint and conditional differential entropy](/P/cmi-jcde)
338343
- [Relation of continuous mutual information to marginal and conditional differential entropy](/P/cmi-mcde)
339344
- [Relation of continuous mutual information to marginal and joint differential entropy](/P/cmi-mjde)
340-
- [Relation of Kullback-Leibler divergence to entropy](/P/kl-ent)
345+
- [Relation of discrete Kullback-Leibler divergence to Shannon entropy](/P/kl-ent)
341346
- [Relation of mutual information to joint and conditional entropy](/P/dmi-jce)
342347
- [Relation of mutual information to marginal and conditional entropy](/P/dmi-mce)
343348
- [Relation of mutual information to marginal and joint entropy](/P/dmi-mje)

I/PwS.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ title: "Proofs without Source"
4646
- [Log model evidence for the Poisson distribution with exposure values](/P/poissexp-lme)
4747
- [Log model evidence for the univariate Gaussian with known variance](/P/ugkv-lme)
4848
- [Log model evidence in terms of prior and posterior distribution](/P/lme-pnp)
49+
- [Marginal distributions for the matrix-normal distribution](/P/matn-marg)
4950
- [Marginal distributions of the multivariate normal distribution](/P/mvn-marg)
5051
- [Marginal distributions of the normal-gamma distribution](/P/ng-marg)
5152
- [Marginal likelihood is a definite integral of joint likelihood](/P/ml-jl)

P/kl-ent.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ affiliation: "BCCN Berlin"
77
e_mail: "joram.soch@bccn-berlin.de"
88
date: 2020-05-27 23:20:00
99

10-
title: "Relation of Kullback-Leibler divergence to entropy"
10+
title: "Relation of discrete Kullback-Leibler divergence to Shannon entropy"
1111
chapter: "General Theorems"
1212
section: "Information theory"
1313
topic: "Kullback-Leibler divergence"

P/matn-marg.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,8 @@ where $m_{ij}$ is the $(i,j)$-th entry of $M$.
6969
$$ \label{eq:A}
7070
A \in \mathbb{R}^{\lvert I \rvert \times n}, \quad \text{s.t.} \quad a_{ij} = \left\{
7171
\begin{array}{rl}
72-
0 \; , & \text{if} \; I_i = j \\
73-
1 \; , & \text{otherwise}
72+
1 \; , & \text{if} \; I_i = j \\
73+
0 \; , & \text{otherwise}
7474
\end{array}
7575
\right.
7676
$$
@@ -80,8 +80,8 @@ and define a selector matrix $B$, such that $b_{ij} = 1$, if the $j$-th column i
8080
$$ \label{eq:B}
8181
B \in \mathbb{R}^{p \times \lvert J \rvert}, \quad \text{s.t.} \quad b_{ij} = \left\{
8282
\begin{array}{rl}
83-
0 \; , & \text{if} \; J_j = i \\
84-
1 \; , & \text{otherwise} \; .
83+
1 \; , & \text{if} \; J_j = i \\
84+
0 \; , & \text{otherwise} \; .
8585
\end{array}
8686
\right.
8787
$$

P/matn-samp.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ $$
3737
**Proof:** If all entries of $X$ are independent and [standard normally distributed](/D/snorm)
3838

3939
$$ \label{eq:xij-dist}
40-
x_{ij} \sim \mathcal{N}(0, 1) \quad \text{ind. for all} \quad i = 1,\ldots,n \quad \text{and} \quad j = 1,\ldots,p \; ,
40+
x_{ij} \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, 1) \quad \text{for all} \quad i = 1,\ldots,n \quad \text{and} \quad j = 1,\ldots,p \; ,
4141
$$
4242

4343
this [implies a multivariate normal distribution with diagonal covariance matrix](/P/mvn-ind):

P/mult-cov.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,13 @@ $$
4646
which [has the variance](/P/bin-var) $\mathrm{Var}(X_i) = n p_i(1-p_i) = n (p_i - p_i^2)$, constituting the elements of the main diagonal in $\mathrm{Cov}(X)$ in \eqref{eq:mult-cov}. To prove $\mathrm{Cov}(X_i, X_j) = -n p_i p_j$ for $i \ne j$ (which constitutes the off-diagonal elements of the covariance matrix), we first recognize that
4747

4848
$$ \label{eq:bin-sum}
49-
X_i = \sum_{k=1}^n \mathbb{I}_i(k), \quad \text{with} \quad \mathbb{I}_i(k) := \begin{cases}
49+
X_i = \sum_{k=1}^n \mathbb{I}_i(k), \quad \text{with} \quad \mathbb{I}_i(k) = \begin{cases}
5050
1 & \text{if $k$-th draw was of category $i$}, \\
51-
0 & \text{otherwise},
51+
0 & \text{otherwise} \; ,
5252
\end{cases}
5353
$$
5454

55-
the indicator function $\mathbb{I}_i$ being a [Bernoulli-distributed](/D/bern) random variable with the [expected value](/P/bern-mean) $p_i$. Then, we have
55+
where the indicator function $\mathbb{I}_i$ is a [Bernoulli-distributed](/D/bern) random variable with the [expected value](/P/bern-mean) $p_i$. Then, we have
5656

5757
$$ \label{eq:mult-cov-qed}
5858
\begin{split}

P/mvn-cov.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ $$ \label{eq:mvn-cov-qed}
102102
\begin{split}
103103
\mathrm{Cov}(x) &= \mathrm{Cov}( Az + \mu ) \\
104104
&\overset{\eqref{eq:cov-inv}}{=} \mathrm{Cov}(Az) \\
105-
&\overset{\eqref{eq:cov-scal}}{=} A \mathrm{Cov}(z) A^\mathrm{T} \\
105+
&\overset{\eqref{eq:cov-scal}}{=} A \, \mathrm{Cov}(z) A^\mathrm{T} \\
106106
&\overset{\eqref{eq:z-cov}}{=} A I_n A^\mathrm{T} \\
107107
&= A A^\mathrm{T} \\
108108
&= \Sigma \; .

0 commit comments

Comments
 (0)