Skip to content

Commit c4ff5bf

Browse files
JoramSochJoramSoch
authored
added 10 proofs
1 parent 74c5c7b commit c4ff5bf

10 files changed

Lines changed: 926 additions & 0 deletions

‎P/slr-comp.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2021-10-27 12:52:00
9+
10+
title: "The regression line goes through the center of mass point"
11+
chapter: "Statistical Models"
12+
section: "Univariate normal data"
13+
topic: "Simple linear regression"
14+
theorem: "Regression line includes center of mass"
15+
16+
sources:
17+
- authors: "Wikipedia"
18+
year: 2021
19+
title: "Simple linear regression"
20+
in: "Wikipedia, the free encyclopedia"
21+
pages: "retrieved on 2021-10-27"
22+
url: "https://en.wikipedia.org/wiki/Simple_linear_regression#Numerical_properties"
23+
24+
proof_id: "P275"
25+
shortcut: "slr-comp"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** In [simple linear regression](/D/slr), the [regression line](/D/regline) estimated using [ordinary least squares](/P/slr-ols) includes the point $M(\bar{x},\bar{y})$.
31+
32+
**Proof:** The [fitted regression line](/D/regline) is described by the equation
33+
34+
$$ \label{eq:slr-ols-regline}
35+
y = \hat{\beta}_0 + \hat{\beta}_1 x \quad \text{where} \quad x,y \in \mathbb{R} \; .
36+
$$
37+
38+
Plugging in the coordinates of $M$ and the [ordinary least squares estimate of the intercept](/P/slr-ols), we obtain
39+
40+
$$ \label{eq:slr-ols}
41+
\begin{split}
42+
\bar{y} &= \hat{\beta}_0 + \hat{\beta}_1 \bar{x} \\
43+
\bar{y} &= \bar{y} - \hat{\beta}_1 \bar{x} + \hat{\beta}_1 \bar{x} \\
44+
\bar{y} &= \bar{y} \; .
45+
\end{split}
46+
$$
47+
48+
which is a true statement. Thus, the [regression line](/D/regline) goes through the center of mass point $(\bar{x},\bar{y})$, if [the model](/D/slr) includes an intercept term $\beta_0$.

‎P/slr-corr.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2021-10-27 14:58:00
9+
10+
title: "Relationship between correlation coefficient and slope estimate in simple linear regression"
11+
chapter: "Statistical Models"
12+
section: "Univariate normal data"
13+
topic: "Simple linear regression"
14+
theorem: "Correlation coefficient in terms of slope estimate"
15+
16+
sources:
17+
- authors: "Penny, William"
18+
year: 2006
19+
title: "Relation to correlation"
20+
in: "Mathematics for Brain Imaging"
21+
pages: "ch. 1.2.3, p. 18, eq. 1.27"
22+
url: "https://ueapsylabs.co.uk/sites/wpenny/mbi/mbi_course.pdf"
23+
- authors: "Wikipedia"
24+
year: 2021
25+
title: "Simple linear regression"
26+
in: "Wikipedia, the free encyclopedia"
27+
pages: "retrieved on 2021-10-27"
28+
url: "https://en.wikipedia.org/wiki/Simple_linear_regression#Fitting_the_regression_line"
29+
30+
proof_id: "P279"
31+
shortcut: "slr-corr"
32+
username: "JoramSoch"
33+
---
34+
35+
36+
**Theorem:** Assume a [simple linear regression model](/D/slr) with independent observations
37+
38+
$$ \label{eq:slr}
39+
y = \beta_0 + \beta_1 x + \varepsilon, \; \varepsilon_i \sim \mathcal{N}(0, \sigma^2), \; i = 1,\ldots,n
40+
$$
41+
42+
and consider estimation using [ordinary least squares](/P/slr-ols). Then, [correlation coefficient](/D/corr) and the estimated value of the [slope parameter](/D/slr) are related to each other via the sample [standard deviations](/D/std):
43+
44+
$$ \label{eq:slr-corr}
45+
r_{xy} = \frac{s_x}{s_y} \, \hat{\beta}_1 \; .
46+
$$
47+
48+
49+
**Proof:** The [ordinary least squares estimate of the slope](/P/slr-ols) is given by
50+
51+
$$ \label{eq:slr-ols-sl}
52+
\hat{\beta}_1 = \frac{s_{xy}}{s_x^2} \; .
53+
$$
54+
55+
Using the [relationship between covariance and correlation](/D/cov-corr)
56+
57+
$$ \label{eq:cov-corr}
58+
\mathrm{Cov}(X,Y) = \sigma_X \, \mathrm{Corr}(X,Y) \, \sigma_Y
59+
$$
60+
61+
which also holds for sample [correlation](/D/corr) and [sample covariance](/D/cov-samp)
62+
63+
$$ \label{eq:cov-corr-samp}
64+
s_{xy} = s_x \, r_{xy} \, s_y \; ,
65+
$$
66+
67+
we get the final result:
68+
69+
$$ \label{eq:slr-corr-qed}
70+
\begin{split}
71+
\hat{\beta}_1 &= \frac{s_{xy}}{s_x^2} \\
72+
\hat{\beta}_1 &= \frac{s_x \, r_{xy} \, s_y}{s_x^2} \\
73+
\hat{\beta}_1 &= \frac{s_y}{s_x} \, r_{xy} \\
74+
\Leftrightarrow \quad r_{xy} &= \frac{s_x}{s_y} \, \hat{\beta}_1 \; .
75+
\end{split}
76+
$$

‎P/slr-meancent.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2021-10-27 12:38:00
9+
10+
title: "Effects of mean-centering on parameter estimates for simple linear regression"
11+
chapter: "Statistical Models"
12+
section: "Univariate normal data"
13+
topic: "Simple linear regression"
14+
theorem: "Effects of mean-centering"
15+
16+
sources:
17+
18+
proof_id: "P274"
19+
shortcut: "slr-meancent"
20+
username: "JoramSoch"
21+
---
22+
23+
24+
**Theorem:** In [simple linear regression](/D/slr), when the independent variable $y$ and/or the dependent variable $x$ are [mean-centered](/D/mean), the [ordinary least squares](/P/slr-ols) estimate for the intercept changes, but that of the slope does not.
25+
26+
**Proof:**
27+
28+
1) Under unaltered $y$ and $x$, [ordinary least squares estimates for simple linear regression](/P/slr-ols) are
29+
30+
$$ \label{eq:slr-ols}
31+
\begin{split}
32+
\hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} \\
33+
\hat{\beta}_1 &= \frac{\sum_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{s_{xy}}{s_x^2}
34+
\end{split}
35+
$$
36+
37+
with [sample means](/D/mean-samp) $\bar{x}$ and $\bar{y}$, [sample variance](/D/var-samp) $s_x^2$ and [sample covariance](/D/cov-samp) $s_{xy}$, such that $\beta_0$ estimates "the mean $y$ at $x = 0$".
38+
39+
<br>
40+
2) Let $\tilde{x}$ be the mean-centered [covariate vector](/D/slr):
41+
42+
$$ \label{eq:slr-meancent-x}
43+
\tilde{x}_i = x_i - \bar{x} \quad \Rightarrow \quad \bar{\tilde{x}} = 0 \; .
44+
$$
45+
46+
Under this condition, the parameter estimates become
47+
48+
$$ \label{eq:slr-ols-meancent-x}
49+
\begin{split}
50+
\hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{\tilde{x}} \\
51+
&= \bar{y} \\
52+
\hat{\beta}_1 &= \frac{\sum_{i=1}^n (\tilde{x}_i - \bar{\tilde{x}}) (y_i - \bar{y})}{\sum_{i=1}^n (\tilde{x}_i - \bar{\tilde{x}})^2} \\
53+
&= \frac{\sum_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{s_{xy}}{s_x^2}
54+
\end{split}
55+
$$
56+
57+
and we can see that $\hat{\beta}_1(\tilde{x},y) = \hat{\beta}_1(x,y)$, but $\hat{\beta}_0(\tilde{x},y) \neq \hat{\beta}_0(x,y)$, specifically $\beta_0$ now estimates "the mean $y$ at the mean $x$".
58+
59+
60+
<br>
61+
3) Let $\tilde{y}$ be the mean-centered [data vector](/D/slr):
62+
63+
$$ \label{eq:slr-meancent-y}
64+
\tilde{y}_i = y_i - \bar{y} \quad \Rightarrow \quad \bar{\tilde{y}} = 0 \; .
65+
$$
66+
67+
Under this condition, the parameter estimates become
68+
69+
$$ \label{eq:slr-ols-meancent-y}
70+
\begin{split}
71+
\hat{\beta}_0 &= \bar{\tilde{y}} - \hat{\beta}_1 \bar{x} \\
72+
&= - \hat{\beta}_1 \bar{x} \\
73+
\hat{\beta}_1 &= \frac{\sum_{i=1}^n (x_i - \bar{x}) (\tilde{y}_i - \bar{\tilde{y}})}{\sum_{i=1}^n (x_i - \bar{x})^2} \\
74+
&= \frac{\sum_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{s_{xy}}{s_x^2}
75+
\end{split}
76+
$$
77+
78+
and we can see that $\hat{\beta}_1(x,\tilde{y}) = \hat{\beta}_1(x,y)$, but $\hat{\beta}_0(x,\tilde{y}) \neq \hat{\beta}_0(x,y)$, specifically $\beta_0$ now estimates "the mean $x$, multiplied with the negative slope".
79+
80+
<br>
81+
4) Finally, consider mean-centering both $x$ and $y$::
82+
83+
$$ \label{eq:slr-meancent-xy}
84+
\begin{split}
85+
\tilde{x}_i = x_i - \bar{x} \quad &\Rightarrow \quad \bar{\tilde{x}} = 0 \\
86+
\tilde{y}_i = y_i - \bar{y} \quad &\Rightarrow \quad \bar{\tilde{y}} = 0 \; .
87+
\end{split}
88+
$$
89+
90+
Under this condition, the parameter estimates become
91+
92+
$$ \label{eq:slr-ols-meancent-xy}
93+
\begin{split}
94+
\hat{\beta}_0 &= \bar{\tilde{y}} - \hat{\beta}_1 \bar{\tilde{x}} \\
95+
&= 0 \\
96+
\hat{\beta}_1 &= \frac{\sum_{i=1}^n (\tilde{x}_i - \bar{\tilde{x}}) (\tilde{y}_i - \bar{\tilde{y}})}{\sum_{i=1}^n (\tilde{x}_i - \bar{\tilde{x}})^2} \\
97+
&= \frac{\sum_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{s_{xy}}{s_x^2}
98+
\end{split}
99+
$$
100+
101+
and we can see that $\hat{\beta}_1(\tilde{x},\tilde{y}) = \hat{\beta}_1(x,y)$, but $\hat{\beta}_0(\tilde{x},\tilde{y}) \neq \hat{\beta}_0(x,y)$, specifically $\beta_0$ is now forced to become zero.

‎P/slr-ols.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2021-10-27 08:56:00
9+
10+
title: "Ordinary least squares for simple linear regression"
11+
chapter: "Statistical Models"
12+
section: "Univariate normal data"
13+
topic: "Simple linear regression"
14+
theorem: "Ordinary least squares"
15+
16+
sources:
17+
- authors: "Penny, William"
18+
year: 2006
19+
title: "Linear regression"
20+
in: "Mathematics for Brain Imaging"
21+
pages: "ch. 1.2.2, pp. 14-16, eqs. 1.24/1.25"
22+
url: "https://ueapsylabs.co.uk/sites/wpenny/mbi/mbi_course.pdf"
23+
- authors: "Wikipedia"
24+
year: 2021
25+
title: "Proofs involving ordinary least squares"
26+
in: "Wikipedia, the free encyclopedia"
27+
pages: "retrieved on 2021-10-27"
28+
url: "https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Derivation_of_simple_linear_regression_estimators"
29+
30+
proof_id: "P271"
31+
shortcut: "slr-ols"
32+
username: "JoramSoch"
33+
---
34+
35+
36+
**Theorem:** Given a [simple linear regression model](/D/slr) with independent observations
37+
38+
$$ \label{eq:slr}
39+
y = \beta_0 + \beta_1 x + \varepsilon, \; \varepsilon_i \sim \mathcal{N}(0, \sigma^2), \; i = 1,\ldots,n \; ,
40+
$$
41+
42+
the parameters minimizing the [residual sum of squares](/D/rss) are given by
43+
44+
$$ \label{eq:slr-ols}
45+
\begin{split}
46+
\hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} \\
47+
\hat{\beta}_1 &= \frac{s_{xy}}{s_x^2}
48+
\end{split}
49+
$$
50+
51+
where $\bar{x}$ and $\bar{y}$ are the [sample means](/D/mean-samp), $s_x^2$ is the [sample variance](/D/var-samp) of $x$ and $s_{xy}$ is the [sample covariance](/D/cov-samp) between $x$ and $y$.
52+
53+
54+
**Proof:** The [residual sum of squares](/D/rss) is defined as
55+
56+
$$ \label{eq:rss}
57+
\mathrm{RSS}(\beta_0,\beta_1) = \sum_{i=1}^n \varepsilon_i^2 = \sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i)^2 \; .
58+
$$
59+
60+
The derivatives of $\mathrm{RSS}(\beta_0,\beta_1)$ with respect to $\beta_0$ and $\beta_1$ are
61+
62+
$$ \label{eq:rss-der}
63+
\begin{split}
64+
\frac{\mathrm{d}\mathrm{RSS}(\beta_0,\beta_1)}{\mathrm{d}\beta_0} &= \sum_{i=1}^n 2 (y_i - \beta_0 - \beta_1 x_i) (-1) \\
65+
&= -2 \sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i) \\
66+
\frac{\mathrm{d}\mathrm{RSS}(\beta_0,\beta_1)}{\mathrm{d}\beta_1} &= \sum_{i=1}^n 2 (y_i - \beta_0 - \beta_1 x_i) (-x_i) \\
67+
&= -2 \sum_{i=1}^n (x_i y_i - \beta_0 x_i - \beta_1 x_i^2)
68+
\end{split}
69+
$$
70+
71+
and setting these derivatives to zero
72+
73+
$$ \label{eq:rss-der-zero}
74+
\begin{split}
75+
0 &= -2 \sum_{i=1}^n (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) \\
76+
0 &= -2 \sum_{i=1}^n (x_i y_i - \hat{\beta}_0 x_i - \hat{\beta}_1 x_i^2)
77+
\end{split}
78+
$$
79+
80+
yields the following equations:
81+
82+
$$ \label{eq:slr-norm-eq}
83+
\begin{split}
84+
\hat{\beta}_1 \sum_{i=1}^n x_i + \hat{\beta}_0 \cdot n &= \sum_{i=1}^n y_i \\
85+
\hat{\beta}_1 \sum_{i=1}^n x_i^2 + \hat{\beta}_0 \sum_{i=1}^n x_i &= \sum_{i=1}^n x_i y_i \; .
86+
\end{split}
87+
$$
88+
89+
From the first equation, we can derive the estimate for the intercept:
90+
91+
$$ \label{eq:slr-ols-int}
92+
\begin{split}
93+
\hat{\beta}_0 &= \frac{1}{n} \sum_{i=1}^n y_i - \hat{\beta}_1 \cdot \frac{1}{n} \sum_{i=1}^n x_i \\
94+
&= \bar{y} - \hat{\beta}_1 \bar{x} \; .
95+
\end{split}
96+
$$
97+
98+
From the second equation, we can derive the estimate for the slope:
99+
100+
$$ \label{eq:slr-ols-sl}
101+
\begin{split}
102+
\hat{\beta}_1 \sum_{i=1}^n x_i^2 + \hat{\beta}_0 \sum_{i=1}^n x_i &= \sum_{i=1}^n x_i y_i \\
103+
\hat{\beta}_1 \sum_{i=1}^n x_i^2 + \left( \bar{y} - \hat{\beta}_1 \bar{x} \right) \sum_{i=1}^n x_i &\overset{\eqref{eq:slr-ols-int}}{=} \sum_{i=1}^n x_i y_i \\
104+
\hat{\beta}_1 \left( \sum_{i=1}^n x_i^2 - \bar{x} \sum_{i=1}^n x_i \right) &= \sum_{i=1}^n x_i y_i - \bar{y} \sum_{i=1}^n x_i \\
105+
\hat{\beta}_1 &= \frac{\sum_{i=1}^n x_i y_i - \bar{y} \sum_{i=1}^n x_i}{\sum_{i=1}^n x_i^2 - \bar{x} \sum_{i=1}^n x_i} \; .
106+
\end{split}
107+
$$
108+
109+
Note that the numerator can be rewritten as
110+
111+
$$ \label{eq:slr-ols-sl-num}
112+
\begin{split}
113+
\sum_{i=1}^n x_i y_i - \bar{y} \sum_{i=1}^n x_i &= \sum_{i=1}^n x_i y_i - n \bar{x} \bar{y} \\
114+
&= \sum_{i=1}^n x_i y_i - n \bar{x} \bar{y} - n \bar{x} \bar{y} + n \bar{x} \bar{y} \\
115+
&= \sum_{i=1}^n x_i y_i - \bar{y} \sum_{i=1}^n x_i - \bar{x} \sum_{i=1}^n y_i + \sum_{i=1}^n \bar{x} \bar{y} \\
116+
&= \sum_{i=1}^n \left( x_i y_i - x_i \bar{y} - \bar{x} y_i + \bar{x} \bar{y} \right) \\
117+
&= \sum_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})
118+
\end{split}
119+
$$
120+
121+
and that the denominator can be rewritten as
122+
123+
$$ \label{eq:slr-ols-sl-den}
124+
\begin{split}
125+
\sum_{i=1}^n x_i^2 - \bar{x} \sum_{i=1}^n x_i &= \sum_{i=1}^n x_i^2 - n \bar{x}^2 \\
126+
&= \sum_{i=1}^n x_i^2 - 2 n \bar{x} \bar{x} + n \bar{x}^2 \\
127+
&= \sum_{i=1}^n x_i^2 - 2 \bar{x} \sum_{i=1}^n x_i - \sum_{i=1}^n \bar{x}^2 \\
128+
&= \sum_{i=1}^n \left( x_i^2 - 2 \bar{x} x_i + \bar{x}^2 \right) \\
129+
&= \sum_{i=1}^n (x_i - \bar{x})^2 \; .
130+
\end{split}
131+
$$
132+
133+
With \eqref{eq:slr-ols-sl-num} and \eqref{eq:slr-ols-sl-den}, the estimate from \eqref{eq:slr-ols-sl} can be simplified as follows:
134+
135+
$$ \label{eq:slr-ols-sl-qed}
136+
\begin{split}
137+
\hat{\beta}_1 &= \frac{\sum_{i=1}^n x_i y_i - \bar{y} \sum_{i=1}^n x_i}{\sum_{i=1}^n x_i^2 - \bar{x} \sum_{i=1}^n x_i} \\
138+
&= \frac{\sum_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} \\
139+
&= \frac{\frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})}{\frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2} \\
140+
&= \frac{s_{xy}}{s_x^2} \; .
141+
\end{split}
142+
$$
143+
144+
Together, \eqref{eq:slr-ols-int} and \eqref{eq:slr-ols-sl-qed} constitute the ordinary least squares parameter estimates for simple linear regression.

0 commit comments

Comments
 (0)