Skip to content

Commit b012b82

Browse files
JoramSochJoramSoch
authored
added 5 proofs
1 parent ae8c6a2 commit b012b82

5 files changed

Lines changed: 323 additions & 0 deletions

File tree

‎P/cov-symm.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-26 12:14:00
9+
10+
title: "Symmetry of the covariance"
11+
chapter: "General Theorems"
12+
section: "Probability theory"
13+
topic: "Covariance"
14+
theorem: "Symmetry"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2022
19+
title: "Covariance"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2022-09-26"
22+
url: "https://en.wikipedia.org/wiki/Covariance#Covariance_of_linear_combinations"
23+
24+
proof_id: "P353"
25+
shortcut: "cov-symm"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** The [covariance](/D/cov) of two [random variables](/D/rvar) is a symmetric function:
31+
32+
$$ \label{eq:cov-symm}
33+
\mathrm{Cov}(X,Y) = \mathrm{Cov}(Y,X) \; .
34+
$$
35+
36+
37+
**Proof:** The [covariance](/D/cov) of [random variables](/D/rvar) $X$ and $Y$ is defined as:
38+
39+
$$ \label{eq:cov}
40+
\mathrm{Cov}(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \; .
41+
$$
42+
43+
Switching $X$ and $Y$ in \eqref{eq:cov}, we can easily see:
44+
45+
$$ \label{eq:cov-symm-qed}
46+
\begin{split}
47+
\mathrm{Cov}(Y,X) &\overset{\eqref{eq:cov}}{=} \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X]) \right] \\
48+
&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \\
49+
&= \mathrm{Cov}(X,Y) \; .
50+
\end{split}
51+
$$

‎P/cov-var.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-26 12:08:00
9+
10+
title: "Self-covariance equals variance"
11+
chapter: "General Theorems"
12+
section: "Probability theory"
13+
topic: "Covariance"
14+
theorem: "Self-covariance"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2022
19+
title: "Covariance"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2022-09-26"
22+
url: "https://en.wikipedia.org/wiki/Covariance#Covariance_with_itself"
23+
24+
proof_id: "P352"
25+
shortcut: "cov-var"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** The [covariance](/D/cov) of a [random variable](/D/rvar) with itself is equal to the [variance](/D/var):
31+
32+
$$ \label{eq:cov-var}
33+
\mathrm{Cov}(X,X) = \mathrm{Var}(X) \; .
34+
$$
35+
36+
37+
**Proof:** The [covariance](/D/cov) of [random variables](/D/rvar) $X$ and $Y$ is defined as:
38+
39+
$$ \label{eq:cov}
40+
\mathrm{Cov}(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \; .
41+
$$
42+
43+
Inserting $X$ for $Y$ in \eqref{eq:cov}, the result is the [variance](/D/var) of $X$:
44+
45+
$$ \label{eq:cov-var-qed}
46+
\begin{split}
47+
\mathrm{Cov}(X,X) &\overset{\eqref{eq:cov}}{=} \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X]) \right] \\
48+
&= \mathrm{E}\left[ (X-\mathrm{E}[X])^2 \right] \\
49+
&= \mathrm{Var}(X) \; .
50+
\end{split}
51+
$$

‎P/covmat-psd.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-26 11:26:00
9+
10+
title: "Positive semi-definiteness of the covariance matrix"
11+
chapter: "General Theorems"
12+
section: "Probability theory"
13+
topic: "Covariance"
14+
theorem: "Positive semi-definiteness"
15+
16+
sources:
17+
- authors: "hkBattousai"
18+
year: 2013
19+
title: "What is the proof that covariance matrices are always semi-definite?"
20+
in: "StackExchange Mathematics"
21+
pages: "retrieved on 2022-09-26"
22+
url: "https://math.stackexchange.com/a/327872"
23+
- authors: "Wikipedia"
24+
year: 2022
25+
title: "Covariance matrix"
26+
in: "Wikipedia, the free encyclopedia"
27+
pages: "retrieved on 2022-09-26"
28+
url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties"
29+
30+
proof_id: "P351"
31+
shortcut: "covmat-psd"
32+
username: "JoramSoch"
33+
---
34+
35+
36+
**Theorem:** Each [covariance matrix](/D/covmat) is positive semi-definite:
37+
38+
$$ \label{eq:covmat-symm}
39+
a^\mathrm{T} \Sigma_{XX} a \geq 0 \quad \text{for all} \quad a \in \mathbb{R}^n \; .
40+
$$
41+
42+
43+
**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows
44+
45+
$$ \label{eq:covmat}
46+
\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right]
47+
$$
48+
49+
A positive semi-definite matrix is a matrix whose eigenvalues are all non-negative or, equivalently,
50+
51+
$$ \label{eq:psd}
52+
M \; \text{pos. semi-def.} \quad \Leftrightarrow \quad x^\mathrm{T} M x \geq 0 \quad \text{for all} \quad x \in \mathbb{R}^n \; .
53+
$$
54+
55+
Here, for an arbitrary real column vector $a \in \mathbb{R}^n$, we have:
56+
57+
$$ \label{eq:covmat-symm-s1}
58+
a^\mathrm{T} \Sigma_{XX} a \overset{\eqref{eq:covmat}}{=} a^\mathrm{T} \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] a \; .
59+
$$
60+
61+
Because the [expected value is a linear operator](/P/mean-lin), we can write:
62+
63+
$$ \label{eq:covmat-symm-s2}
64+
a^\mathrm{T} \Sigma_{XX} a = \mathrm{E}\left[ a^\mathrm{T} (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} a \right] \; .
65+
$$
66+
67+
Now define the [scalar random variable](/D/rvar)
68+
69+
$$ \label{eq:Y-X}
70+
Y = a^\mathrm{T} (X-\mu_X) \; .
71+
$$
72+
73+
where $\mu_X = \mathrm{E}[X]$ and note that
74+
75+
$$ \label{eq:YT-Y}
76+
a^\mathrm{T} (X-\mu_X) = (X-\mu_X)^\mathrm{T} a \; .
77+
$$
78+
79+
Thus, combing \eqref{eq:covmat-symm-s2} with \eqref{eq:Y-X}, we have:
80+
81+
$$ \label{eq:covmat-symm-s3}
82+
a^\mathrm{T} \Sigma_{XX} a = \mathrm{E}\left[ Y^2 \right] \; .
83+
$$
84+
85+
Because $Y^2$ is a random variable that cannot become negative and the [expected value of a strictly non-negative random variable is also non-negative](/P/mean-nonneg), we finally have
86+
87+
$$ \label{eq:covmat-symm-s4}
88+
a^\mathrm{T} \Sigma_{XX} a \geq 0
89+
$$
90+
91+
for any $a \in \mathbb{R}^n$.

‎P/covmat-sum.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-26 10:37:00
9+
10+
title: "Covariance of the sum of two random vectors"
11+
chapter: "General Theorems"
12+
section: "Probability theory"
13+
topic: "Covariance"
14+
theorem: "Covariance matrix of a sum"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2022
19+
title: "Covariance matrix"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2022-09-26"
22+
url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties"
23+
24+
proof_id: "P349"
25+
shortcut: "covmat-sum"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** The [covariance matrix](/D/covmat) of the sum of two [random vectors](/D/rvec) of the same dimension equals the sum of the covariances of those random vectors, plus the sum of their [cross-covariances](/D/covmat-cross):
31+
32+
$$ \label{eq:covmat-sum}
33+
\Sigma(X+Y) = \Sigma_{XX} + \Sigma_{YY} + \Sigma_{XY} + \Sigma_{YX} \; .
34+
$$
35+
36+
37+
**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows
38+
39+
$$ \label{eq:covmat}
40+
\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right]
41+
$$
42+
43+
and the [cross-covariance matrix](/D/covmat-cross) of $X$ and $Y$ can similarly be written as
44+
45+
$$ \label{eq:covmat-cross}
46+
\Sigma_{XY} = \Sigma(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right]
47+
$$
48+
49+
Using this and the [linearity of the expected value](/P/mean-lin) as well as the definitions of [covariance matrix](/D/covmat) and [cross-covariance matrix](/D/covmat-cross), we can derive \eqref{eq:covmat-sum} as follows:
50+
51+
$$ \label{eq:covmat-sum-qed}
52+
\begin{split}
53+
\Sigma(X+Y) &\overset{\eqref{eq:covmat}}{=} \mathrm{E}\left[ ([X+Y]-\mathrm{E}[X+Y]) ([X+Y]-\mathrm{E}[X+Y])^\mathrm{T} \right] \\
54+
&= \mathrm{E}\left[ ([X-\mathrm{E}(X)] + [Y-\mathrm{E}(Y)]) ([X-\mathrm{E}(X)] + [Y-\mathrm{E}(Y)])^\mathrm{T} \right] \\
55+
&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} + (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} + (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} + (Y-\mathrm{E}[Y]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] \\
56+
&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] + \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] \\
57+
&\overset{\eqref{eq:covmat}}{=} \Sigma_{XX} + \Sigma_{YY} + \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} \right] \\
58+
&\overset{\eqref{eq:covmat-cross}}{=} \Sigma_{XX} + \Sigma_{YY} + \Sigma_{XY} + \Sigma_{YX} \; .
59+
\end{split}
60+
$$

‎P/covmat-symm.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-09-26 10:54:00
9+
10+
title: "Symmetry of the covariance matrix"
11+
chapter: "General Theorems"
12+
section: "Probability theory"
13+
topic: "Covariance"
14+
theorem: "Symmetry"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2022
19+
title: "Covariance matrix"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2022-09-26"
22+
url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties"
23+
24+
proof_id: "P350"
25+
shortcut: "covmat-symm"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** Each [covariance matrix](/D/covmat) is symmetric:
31+
32+
$$ \label{eq:covmat-symm}
33+
\Sigma_{XX}^\mathrm{T} = \Sigma_{XX} \; .
34+
$$
35+
36+
37+
**Proof:** The [covariance matrix](/D/covmat) of a [random vector](/D/rvec) $X$ is defined as
38+
39+
$$ \label{eq:covmat}
40+
\Sigma_{XX} =
41+
\begin{bmatrix}
42+
\mathrm{Cov}(X_1,X_1) & \ldots & \mathrm{Cov}(X_1,X_n) \\
43+
\vdots & \ddots & \vdots \\
44+
\mathrm{Cov}(X_n,X_1) & \ldots & \mathrm{Cov}(X_n,X_n)
45+
\end{bmatrix} \; .
46+
$$
47+
48+
A symmetric matrix is a matrix whose transpose is equal to itself. The transpose of $\Sigma_{XX}$ is
49+
50+
$$ \label{eq:covmat-trans}
51+
\Sigma_{XX}^\mathrm{T} =
52+
\begin{bmatrix}
53+
\mathrm{Cov}(X_1,X_1) & \ldots & \mathrm{Cov}(X_n,X_1) \\
54+
\vdots & \ddots & \vdots \\
55+
\mathrm{Cov}(X_1,X_n) & \ldots & \mathrm{Cov}(X_n,X_n)
56+
\end{bmatrix} \; .
57+
$$
58+
59+
Because the [covariance is a symmetric function](/P/cov-symm), i.e. $\mathrm{Cov}(X,Y) = \mathrm{Cov}(Y,X)$, this matrix is equal to
60+
61+
$$ \label{eq:covmat-symm-qed}
62+
\Sigma_{XX}^\mathrm{T} =
63+
\begin{bmatrix}
64+
\mathrm{Cov}(X_1,X_1) & \ldots & \mathrm{Cov}(X_1,X_n) \\
65+
\vdots & \ddots & \vdots \\
66+
\mathrm{Cov}(X_n,X_1) & \ldots & \mathrm{Cov}(X_n,X_n)
67+
\end{bmatrix}
68+
$$
69+
70+
which is equivalent to our original definition in \eqref{eq:covmat}.

0 commit comments

Comments
 (0)