|
| 1 | +--- |
| 2 | +layout: proof |
| 3 | +mathjax: true |
| 4 | + |
| 5 | +author: "Joram Soch" |
| 6 | +affiliation: "BCCN Berlin" |
| 7 | +e_mail: "joram.soch@bccn-berlin.de" |
| 8 | +date: 2022-09-26 11:26:00 |
| 9 | + |
| 10 | +title: "Positive semi-definiteness of the covariance matrix" |
| 11 | +chapter: "General Theorems" |
| 12 | +section: "Probability theory" |
| 13 | +topic: "Covariance" |
| 14 | +theorem: "Positive semi-definiteness" |
| 15 | + |
| 16 | +sources: |
| 17 | + - authors: "hkBattousai" |
| 18 | + year: 2013 |
| 19 | + title: "What is the proof that covariance matrices are always semi-definite?" |
| 20 | + in: "StackExchange Mathematics" |
| 21 | + pages: "retrieved on 2022-09-26" |
| 22 | + url: "https://math.stackexchange.com/a/327872" |
| 23 | + - authors: "Wikipedia" |
| 24 | + year: 2022 |
| 25 | + title: "Covariance matrix" |
| 26 | + in: "Wikipedia, the free encyclopedia" |
| 27 | + pages: "retrieved on 2022-09-26" |
| 28 | + url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties" |
| 29 | + |
| 30 | +proof_id: "P351" |
| 31 | +shortcut: "covmat-psd" |
| 32 | +username: "JoramSoch" |
| 33 | +--- |
| 34 | + |
| 35 | + |
| 36 | +**Theorem:** Each [covariance matrix](/D/covmat) is positive semi-definite: |
| 37 | + |
| 38 | +$$ \label{eq:covmat-symm} |
| 39 | +a^\mathrm{T} \Sigma_{XX} a \geq 0 \quad \text{for all} \quad a \in \mathbb{R}^n \; . |
| 40 | +$$ |
| 41 | + |
| 42 | + |
| 43 | +**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows |
| 44 | + |
| 45 | +$$ \label{eq:covmat} |
| 46 | +\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] |
| 47 | +$$ |
| 48 | + |
| 49 | +A positive semi-definite matrix is a matrix whose eigenvalues are all non-negative or, equivalently, |
| 50 | + |
| 51 | +$$ \label{eq:psd} |
| 52 | +M \; \text{pos. semi-def.} \quad \Leftrightarrow \quad x^\mathrm{T} M x \geq 0 \quad \text{for all} \quad x \in \mathbb{R}^n \; . |
| 53 | +$$ |
| 54 | + |
| 55 | +Here, for an arbitrary real column vector $a \in \mathbb{R}^n$, we have: |
| 56 | + |
| 57 | +$$ \label{eq:covmat-symm-s1} |
| 58 | +a^\mathrm{T} \Sigma_{XX} a \overset{\eqref{eq:covmat}}{=} a^\mathrm{T} \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] a \; . |
| 59 | +$$ |
| 60 | + |
| 61 | +Because the [expected value is a linear operator](/P/mean-lin), we can write: |
| 62 | + |
| 63 | +$$ \label{eq:covmat-symm-s2} |
| 64 | +a^\mathrm{T} \Sigma_{XX} a = \mathrm{E}\left[ a^\mathrm{T} (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} a \right] \; . |
| 65 | +$$ |
| 66 | + |
| 67 | +Now define the [scalar random variable](/D/rvar) |
| 68 | + |
| 69 | +$$ \label{eq:Y-X} |
| 70 | +Y = a^\mathrm{T} (X-\mu_X) \; . |
| 71 | +$$ |
| 72 | + |
| 73 | +where $\mu_X = \mathrm{E}[X]$ and note that |
| 74 | + |
| 75 | +$$ \label{eq:YT-Y} |
| 76 | +a^\mathrm{T} (X-\mu_X) = (X-\mu_X)^\mathrm{T} a \; . |
| 77 | +$$ |
| 78 | + |
| 79 | +Thus, combing \eqref{eq:covmat-symm-s2} with \eqref{eq:Y-X}, we have: |
| 80 | + |
| 81 | +$$ \label{eq:covmat-symm-s3} |
| 82 | +a^\mathrm{T} \Sigma_{XX} a = \mathrm{E}\left[ Y^2 \right] \; . |
| 83 | +$$ |
| 84 | + |
| 85 | +Because $Y^2$ is a random variable that cannot become negative and the [expected value of a strictly non-negative random variable is also non-negative](/P/mean-nonneg), we finally have |
| 86 | + |
| 87 | +$$ \label{eq:covmat-symm-s4} |
| 88 | +a^\mathrm{T} \Sigma_{XX} a \geq 0 |
| 89 | +$$ |
| 90 | + |
| 91 | +for any $a \in \mathbb{R}^n$. |
0 commit comments