Skip to content

Commit 384e448

Browse files
JoramSochJoramSoch
authored
added 5 proofs
1 parent 6aa01e5 commit 384e448

5 files changed

Lines changed: 371 additions & 0 deletions

File tree

‎P/covmat-inv.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-22 11:29:00
9+
10+
title: "Invariance of the covariance matrix under addition of constant vector"
11+
chapter: "General Theorems"
12+
section: "Probability theory"
13+
topic: "Covariance"
14+
theorem: "Invariance under addition of vector"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2022
19+
title: "Covariance matrix"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2022-09-22"
22+
url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties"
23+
24+
proof_id: "P347"
25+
shortcut: "covmat-inv"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** The [covariance matrix](/D/covmat) $\Sigma_{XX}$ of a [random vector](/D/rvec) $X$ is invariant under addition of a [constant vector](/D/const) $a$:
31+
32+
$$ \label{eq:covmat-inv}
33+
\Sigma(X+a) = \Sigma(X) \; .
34+
$$
35+
36+
37+
**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows:
38+
39+
$$ \label{eq:covmat}
40+
\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] \; .
41+
$$
42+
43+
Using this and the [linearity of the expected value](/P/mean-lin), we can derive \eqref{eq:covmat-inv} as follows:
44+
45+
$$ \label{eq:covmat-inv-qed}
46+
\begin{split}
47+
\Sigma(X+a) &\overset{\eqref{eq:covmat}}{=} \mathrm{E}\left[ ([X+a]-\mathrm{E}[X+a]) ([X+a]-\mathrm{E}[X+a])^\mathrm{T} \right] \\
48+
&= \mathrm{E}\left[ (X + a - \mathrm{E}[X] - a) (X + a - \mathrm{E}[X] - a)^\mathrm{T} \right] \\
49+
&= \mathrm{E}\left[ (X - \mathrm{E}[X]) (X - \mathrm{E}[X])^\mathrm{T} \right] \\
50+
&\overset{\eqref{eq:covmat}}{=} \Sigma(X) \; .
51+
\end{split}
52+
$$

‎P/covmat-scal.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-22 11:45:00
9+
10+
title: "Scaling of the covariance matrix upon multiplication with constant matrix"
11+
chapter: "General Theorems"
12+
section: "Probability theory"
13+
topic: "Covariance"
14+
theorem: "Scaling upon multiplication with matrix"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2022
19+
title: "Covariance matrix"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2022-09-22"
22+
url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties"
23+
24+
proof_id: "P348"
25+
shortcut: "covmat-scal"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** The [covariance matrix](/D/covmat) $\Sigma_{XX}$ of a [random vector](/D/rvec) $X$ scales upon multiplication with a [constant matrix](/D/const) $A$:
31+
32+
$$ \label{eq:covmat-scal}
33+
\Sigma(AX) = A \, \Sigma(X) A^\mathrm{T} \; .
34+
$$
35+
36+
37+
**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows:
38+
39+
$$ \label{eq:covmat}
40+
\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] \; .
41+
$$
42+
43+
Using this and the [linearity of the expected value](/P/mean-lin), we can derive \eqref{eq:covmat-scal} as follows:
44+
45+
$$ \label{eq:covmat-scal-qed}
46+
\begin{split}
47+
\Sigma(AX) &\overset{\eqref{eq:covmat}}{=} \mathrm{E}\left[ ([AX]-\mathrm{E}[AX]) ([AX]-\mathrm{E}[AX])^\mathrm{T} \right] \\
48+
&= \mathrm{E}\left[ (A[X-\mathrm{E}[X]]) (A[X-\mathrm{E}[X]])^\mathrm{T} \right] \\
49+
&= \mathrm{E}\left[ A (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} A^\mathrm{T} \right] \\
50+
&= A \, \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] A^\mathrm{T} \\
51+
&\overset{\eqref{eq:covmat}}{=} A \, \Sigma(X) A^\mathrm{T} \; .
52+
\end{split}
53+
$$

‎P/matn-dent.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-22 08:39:00
9+
10+
title: "Differential entropy for the matrix-normal distribution"
11+
chapter: "Probability Distributions"
12+
section: "Matrix-variate continuous distributions"
13+
topic: "Matrix-normal distribution"
14+
theorem: "Differential entropy"
15+
16+
sources:
17+
18+
proof_id: "P344"
19+
shortcut: "matn-dent"
20+
username: "JoramSoch"
21+
---
22+
23+
24+
**Theorem:** Let $X$ be an $n \times p$ [random matrix](/D/rmat) following a [matrix-normal distribution](/D/matn)
25+
26+
$$ \label{eq:matn}
27+
X \sim \mathcal{MN}(M, U, V) \; .
28+
$$
29+
30+
Then, the [differential entropy](/D/dent) of $X$ in nats is
31+
32+
$$ \label{eq:matn-dent}
33+
\mathrm{h}(X) = \frac{np}{2} \ln(2\pi) + \frac{n}{2} \ln|V| + \frac{p}{2} \ln|U| + \frac{np}{2} \; .
34+
$$
35+
36+
37+
**Proof:** The [matrix-normal distribution is equivalent to the multivariate normal distribution](/P/matn-mvn),
38+
39+
$$ \label{eq:matn-mvn}
40+
X \sim \mathcal{MN}(M, U, V) \quad \Leftrightarrow \quad \mathrm{vec}(X) \sim \mathcal{N}(\mathrm{vec}(M), V \otimes U) \; ,
41+
$$
42+
43+
and the [differential entropy for the multivariate normal distribution](/P/mvn-dent) in nats is
44+
45+
$$ \label{eq:mvn-dent}
46+
X \sim \mathcal{N}(\mu, \Sigma) \quad \Rightarrow \quad \mathrm{h}(X) = \frac{n}{2} \ln(2\pi) + \frac{1}{2} \ln|\Sigma| + \frac{1}{2} n
47+
$$
48+
49+
where $X$ is an $n \times 1$ [random vector](/D/rvec).
50+
51+
Thus, we can plug the distribution parameters from \eqref{eq:matn} into the differential entropy in \eqref{eq:mvn-dent} using the relationship given by \eqref{eq:matn-mvn}
52+
53+
$$ \label{eq:matn-dent-s1}
54+
\mathrm{h}(X) = \frac{np}{2} \ln(2\pi) + \frac{1}{2} \ln|V \otimes U| + \frac{1}{2} np \; .
55+
$$
56+
57+
Using the Kronecker product property
58+
59+
$$ \label{eq:kron-det}
60+
|A \otimes B| = |A|^m \, |B|^n \quad \text{where} \quad A \in \mathbb{R}^{n \times n} \quad \text{and} \quad B \in \mathbb{R}^{m \times m} \; ,
61+
$$
62+
63+
the differential entropy from \eqref{eq:matn-dent-s1} becomes:
64+
65+
$$ \label{eq:matn-dent-s2}
66+
\begin{split}
67+
\mathrm{h}(X) &= \frac{np}{2} \ln(2\pi) + \frac{1}{2} \ln\left(|V|^n |U|^p\right) + \frac{1}{2} np \\
68+
&= \frac{np}{2} \ln(2\pi) + \frac{n}{2} \ln|V| + \frac{p}{2} \ln|U| + \frac{np}{2} \; .
69+
\end{split}
70+
$$

‎P/ng-cov.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-22 09:17:00
9+
10+
title: "Covariance and variance of the normal-gamma distribution"
11+
chapter: "Probability Distributions"
12+
section: "Multivariate continuous distributions"
13+
topic: "Normal-gamma distribution"
14+
theorem: "Covariance"
15+
16+
sources:
17+
18+
proof_id: "P345"
19+
shortcut: "ng-cov"
20+
username: "JoramSoch"
21+
---
22+
23+
24+
**Theorem:** Let $x \in \mathbb{R}^n$ and $y > 0$ follow a [normal-gamma distribution](/D/ng):
25+
26+
$$ \label{eq:ng}
27+
x,y \sim \mathrm{NG}(\mu, \Lambda, a, b) \; .
28+
$$
29+
30+
Then,
31+
32+
1) the [covariance](/D/cov) of $x$, [conditional](/D/dist-cond) on $y$ is
33+
34+
$$ \label{eq:ng-cov-cond}
35+
\mathrm{Cov}[x|y] = \frac{1}{y} \Lambda^{-1} \; ;
36+
$$
37+
38+
2) the [covariance](/D/cov) of $x$, [unconditional](/D/dist-marg) on $y$ is
39+
40+
$$ \label{eq:ng-cov-x}
41+
\mathrm{Cov}[x] = \frac{b}{a-1} \Lambda^{-1} \; ;
42+
$$
43+
44+
3) the [variance](/D/var) of $y$ is
45+
46+
$$ \label{eq:ng-var-y}
47+
\mathrm{Var}[y] = \frac{a}{b^2} \; .
48+
$$
49+
50+
51+
**Proof:**
52+
53+
1) According to the [definition of the normal-gamma distribution](/D/ng), the distribution of $x$ given $y$ is a [multivariate normal distribution](/D/mvn):
54+
55+
$$ \label{eq:ng-mvn}
56+
x \vert y \sim \mathcal{N}(\mu, (y \Lambda)^{-1}) \; .
57+
$$
58+
59+
The [covariance of the multivariate normal distribution](/P/mvn-cov) is
60+
61+
$$ \label{eq:mvn-cov}
62+
x \sim \mathcal{N}(\mu, \Sigma) \quad \Rightarrow \quad \mathrm{Cov}(x) = \Sigma \; ,
63+
$$
64+
65+
such that we have:
66+
67+
$$ \label{eq:ng-cov-cond-qed}
68+
\mathrm{Cov}[x|y] = (y \Lambda)^{-1} = \frac{1}{y} \Lambda^{-1} \; .
69+
$$
70+
71+
2) The [marginal distribution of the normal-gamma distribution](/P/ng-marg) with respect to $x$ is a [multivariate t-distribution](/D/mvt):
72+
73+
$$ \label{eq:ng-marg-x}
74+
x \sim t\left( \mu, \left(\frac{a}{b} \Lambda \right)^{-1}, 2a \right) \; .
75+
$$
76+
77+
The [covariance of the multivariate t-distribution](/P/mvt-cov) is
78+
79+
$$ \label{eq:mvt-cov}
80+
x \sim t(\mu, \Sigma, \nu) \quad \Rightarrow \quad \mathrm{Cov}(x) = \frac{\nu}{\nu-2} \Sigma \; ,
81+
$$
82+
83+
such that we have:
84+
85+
$$ \label{eq:ng-cov-x-qed}
86+
\mathrm{Cov}[x] = \frac{2a}{2a-2} \left(\frac{a}{b} \Lambda \right)^{-1} = \frac{a}{a-1} \, \frac{b}{a} \, \Lambda^{-1} = \frac{b}{a-1} \Lambda^{-1} \; .
87+
$$
88+
89+
3) The [marginal distribution of the normal-gamma distribution](/P/ng-marg) with respect to $y$ is a [univariate gamma distribution](/D/gam):
90+
91+
$$ \label{eq:ng-marg-y}
92+
y \sim \mathrm{Gam}(a, b) \; .
93+
$$
94+
95+
The [variance of the gamma distribution](/P/gam-var) is
96+
97+
$$ \label{eq:gam-var}
98+
x \sim \mathrm{Gam}(a, b) \quad \Rightarrow \quad \mathrm{Var}(x) = \frac{a}{b^2} \; ,
99+
$$
100+
101+
such that we have:
102+
103+
$$ \label{eq:ng-var-y-qed}
104+
\mathrm{Var}[y] = \frac{a}{b^2} \; .
105+
$$

‎P/ng-samp.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-22 11:10:00
9+
10+
title: "Sampling from the normal-gamma distribution"
11+
chapter: "Probability Distributions"
12+
section: "Multivariate continuous distributions"
13+
topic: "Normal-gamma distribution"
14+
theorem: "Drawing samples"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2022
19+
title: "Normal-gamma distribution"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2022-09-22"
22+
url: "https://en.wikipedia.org/wiki/Normal-gamma_distribution#Generating_normal-gamma_random_variates"
23+
24+
proof_id: "P346"
25+
shortcut: "ng-samp"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** Let $Z_1 \in \mathbb{R}^{n \times 1}$ be a [random vector](/D/rvec) with all entries independently following a [standard normal distribution](/D/snorm) and let $Z_2 \in \mathbb{R}$ be a [random variable](/D/rvar) following a [standard gamma distribution](/D/sgam) with shape $a$. Moreover, let $A \in \mathbb{R}^{n \times n}$ be a matrix such that, such that $A A^\mathrm{T} = \Lambda^{-1}$.
31+
32+
Then, $X = \mu + A Z_1 / \sqrt{Z_2/b}$ and $Y = Z_2/b$ jointly follow a [normal-gamma distribution](/D/ng) with [mean vector](/D/mean-rvec) $\mu$, [precision matrix](/D/precmat) $\Lambda$, shape parameter $a$ and rate parameter $b$:
33+
34+
$$ \label{eq:ng-samp}
35+
\left( X = \mu + A Z_1 / \sqrt{Z_2/b}, \; Y = Z_2/b \right) \sim \mathrm{NG}(\mu, \Lambda, a, b) \; .
36+
$$
37+
38+
39+
**Proof:** If all entries of $Z_1$ are independent and [standard normally distributed](/D/snorm)
40+
41+
$$ \label{eq:zi-dist}
42+
z_{1i} \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, 1) \quad \text{for all} \quad i = 1,\ldots,n \; ,
43+
$$
44+
45+
this [implies a multivariate normal distribution with diagonal covariance matrix](/P/mvn-ind):
46+
47+
$$ \label{eq:Z1-dist}
48+
Z_1 \sim \mathcal{N}\left(0_n, I_n \right)
49+
$$
50+
51+
where $0_n$ is an $n \times 1$ matrix of zeros and $I_n$ is the $n \times n$ identity matrix.
52+
53+
If the distribution of $Z_2$ is a [standard gamma distribution](/D/sgam)
54+
55+
$$ \label{eq:Z2-dist}
56+
Z_2 \sim \mathrm{Gam}(a, 1) \; ,
57+
$$
58+
59+
then due to the [relationship between gamma and standard gamma distribution distribution](/P/gam-sgam), we have:
60+
61+
$$ \label{eq:Y-dist}
62+
Y = \frac{Z_2}{b} \sim \mathrm{Gam}(a,b) \; .
63+
$$
64+
65+
Moreover, using the [linear transformation theorem for the multivariate normal distribution](/P/mvn-ltt), it follows that:
66+
67+
$$ \label{eq:X-dist}
68+
\begin{split}
69+
Z_1 &\sim \mathcal{N}\left(0_n, I_n \right) \\
70+
X = \mu + \frac{1}{\sqrt{Z_2/b}} A Z_1 &\sim \mathcal{N}\left(\mu + \frac{1}{\sqrt{Z_2/b}} A \, 0_n, \left( \frac{1}{\sqrt{Z_2/b}} A \right) I_n \left( \frac{1}{\sqrt{Z_2/b}} A \right)^\mathrm{T} \right) \\
71+
X &\sim \mathcal{N}\left(\mu + 0_n, \left( \frac{1}{\sqrt{Y}} \right)^2 A A^\mathrm{T} \right) \\
72+
X &\sim \mathcal{N}\left(\mu, \left( Y \Lambda \right)^{-1} \right) \; .
73+
\end{split}
74+
$$
75+
76+
Thus, $Y$ follows a [gamma distribution](/D/gam) and the distribution of $X$ conditional on $Y$ is a [multivariate normal distribution](/D/mvn):
77+
78+
$$ \label{eq:mvn-gam}
79+
\begin{split}
80+
X \vert Y &\sim \mathcal{N}(\mu, (Y \Lambda)^{-1}) \\
81+
Y &\sim \mathrm{Gam}(a, b) \; .
82+
\end{split}
83+
$$
84+
85+
This means that, [by definition](/D/ng), $X$ and $Y$ jointly follow a [normal-gamma distribution](/D/ng):
86+
87+
$$ \label{eq:ng-samp-qed}
88+
X,Y \sim \mathrm{NG}(\mu, \Lambda, a, b) \; ,
89+
$$
90+
91+
Thus, given $Z_1$ defined by \eqref{eq:zi-dist} and $Z_2$ defined by \eqref{eq:Z2-dist}, $X$ and $Y$ defined by \eqref{eq:ng-samp} are a [sample](/D/samp) from $\mathrm{NG}(\mu, \Lambda, a, b)$.

0 commit comments

Comments
 (0)