Skip to content

Commit 20144f0

Browse files
JoramSochJoramSoch
authored
added 3 proofs
1 parent 230df1f commit 20144f0

3 files changed

Lines changed: 412 additions & 0 deletions

File tree

‎P/anova1-f.md‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-11-06 13:05:00
9+
10+
title: "F-test for main effect in one-way analysis of variance"
11+
chapter: "Statistical Models"
12+
section: "Univariate normal data"
13+
topic: "Analysis of variance"
14+
theorem: "F-test for main effect in one-way ANOVA"
15+
16+
sources:
17+
- authors: "Denziloe"
18+
year: 2018
19+
title: "Derive the distribution of the ANOVA F-statistic under the alternative hypothesis"
20+
in: "StackExchange CrossValidated"
21+
pages: "retrieved on 2022-11-06"
22+
url: "https://stats.stackexchange.com/questions/355594/derive-the-distribution-of-the-anova-f-statistic-under-the-alternative-hypothesi"
23+
24+
proof_id: "P370"
25+
shortcut: "anova1-f"
26+
username: "JoramSoch"
27+
---
28+
29+
30+
**Theorem:** Assume the [one-way analysis of variance](/D/anova1) model
31+
32+
$$ \label{eq:anova1}
33+
y_{ij} = \mu_i + \varepsilon_{ij}, \; \varepsilon_{ij} \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, \sigma^2), \; i = 1, \ldots, k, \; j = 1, \dots, n_i \; ,
34+
$$
35+
36+
and consider the [null](/D/h0) and [alternative](/D/h1) hypothesis
37+
38+
$$ \label{eq:anova1-h0}
39+
\begin{split}
40+
H_0: &\; \mu_1 = \ldots = \mu_K \\
41+
H_1: &\; \mu_i \neq \mu_j \quad \text{for at least one} \quad i,j \in \left\lbrace 1, \ldots, k \right\rbrace, \; i \neq j \; .
42+
\end{split}
43+
$$
44+
45+
Then, the [test statistic](/D/tstat)
46+
47+
$$ \label{eq:anova1-f}
48+
F = \frac{\frac{1}{k-1} \sum_{i=1}^{k} n_i (\bar{y}_i - \bar{y})^2}{\frac{1}{n-k} \sum_{i=1}^{k} \sum_{j=1}^{n_i} (y_{ij} - \bar{y}_i)^2}
49+
$$
50+
51+
follows an [F-distribution](/D/f) under the null hypothesis:
52+
53+
$$ \label{eq:anova1-f-h0}
54+
F \sim \mathrm{F}(k-1, n-k), \; \text{if} \; H_0 \; .
55+
$$
56+
57+
58+
**Proof:** Let $\mu$ be the common [mean](/D/mean) under the [null hypothesis](/D/h0) $\mu_1 = \ldots = \mu_K = \mu$. Under $H_0$, we have:
59+
60+
$$ \label{eq:yij-h0}
61+
y_{ij} \sim \mathcal{N}(\mu, \sigma^2) \quad \text{for all} \quad i = 1, \ldots, k, \; j = 1, \ldots, n_i \; .
62+
$$
63+
64+
Thus, the [random variable](/D/rvar) $U_{ij} = (y_{ij} - \mu)/\sigma$ [follows a standard normal distribution](/P/norm-snorm)
65+
66+
$$ \label{eq:Uij-h0}
67+
U_{ij} = \frac{y_{ij} - \mu}{\sigma} \sim \mathcal{N}(0, 1) \; .
68+
$$
69+
70+
Now consider the following sum:
71+
72+
$$ \label{eq:sum-Uij-s1}
73+
\begin{split}
74+
\sum_{i=1}^{k} \sum_{j=1}^{n_i} U_{ij}^2 &= \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( \frac{y_{ij} - \mu}{\sigma} \right)^2 \\
75+
&= \frac{1}{\sigma^2} \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( (y_{ij} - \bar{y}_i) + (\bar{y}_i - \bar{y}) + (\bar{y} - \mu) \right)^2 \\
76+
&= \frac{1}{\sigma^2} \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left[ (y_{ij} - \bar{y}_i)^2 + (\bar{y}_i - \bar{y})^2 + (\bar{y} - \mu)^2 + 2 (y_{ij} - \bar{y}_i) (\bar{y}_i - \bar{y}) + 2 (y_{ij} - \bar{y}_i) (\bar{y} - \mu) + 2 (\bar{y}_i - \bar{y}) (\bar{y} - \mu) \right] \; .
77+
\end{split}
78+
$$
79+
80+
Because the following sum over $j$ is zero for all $i$
81+
82+
$$ \label{eq:sum-yij}
83+
\begin{split}
84+
\sum_{j=1}^{n_i} (y_{ij} - \bar{y}_i) &= \sum_{j=1}^{n_i} y_{ij} - n_i \bar{y}_i \\
85+
&= \sum_{j=1}^{n_i} y_{ij} - n_i \cdot \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij} \\
86+
&= 0, \; i = 1, \ldots, k
87+
\end{split}
88+
$$
89+
90+
and the following sum over $i$ and $j$ is also zero
91+
92+
$$ \label{eq:sum-yib}
93+
\begin{split}
94+
\sum_{i=1}^{k} \sum_{j=1}^{n_i} (\bar{y}_i - \bar{y}) &= \sum_{i=1}^{k} n_i (\bar{y}_i - \bar{y}) \\
95+
&= \sum_{i=1}^{k} n_i \bar{y}_i - \bar{y} \sum_{i=1}^{k} n_i \\
96+
&= \sum_{i=1}^{k} n_i \cdot \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij} - n \cdot \frac{1}{n} \sum_{i=1}^{k} \sum_{j=1}^{n_i} y_{ij} \\
97+
&= 0 \; ,
98+
\end{split}
99+
$$
100+
101+
where $n = \sum_{i=1}^{k} n_i$, the sum in \eqref{eq:sum-Uij-s1} reduces to
102+
103+
$$ \label{eq:sum-Uij-s2}
104+
\begin{split}
105+
\sum_{i=1}^{k} \sum_{j=1}^{n_i} U_{ij}^2 &= \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left[ \left( \frac{y_{ij} - \bar{y}_i}{\sigma} \right)^2 + \left( \frac{\bar{y}_i - \bar{y}}{\sigma} \right)^2 + \left( \frac{\bar{y} - \mu}{\sigma} \right)^2 \right] \\
106+
&= \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( \frac{y_{ij} - \bar{y}_i}{\sigma} \right)^2 + \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( \frac{\bar{y}_i - \bar{y}}{\sigma} \right)^2 + \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( \frac{\bar{y} - \mu}{\sigma} \right)^2 \; .
107+
\end{split}
108+
$$
109+
110+
[Cochran's theorem](/P/snorm-cochran) states that, if a sum of squared [standard normal](/D/snorm) [random variables](/D/rvar) can be written as a sum of squared forms
111+
112+
$$ \label{eq:cochran-p1}
113+
\begin{split}
114+
\sum_{i=1}^{n} U_i^2 = \sum_{j=1}^{m} Q_j \quad &\text{where} \quad Q_j = U^\mathrm{T} B^{(j)} U \\
115+
&\text{with} \quad \sum_{j=1}^{m} B^{(j)} = I_n \\
116+
&\text{and} \quad r_j = \mathrm{rank}(B^{(j)}) \; ,
117+
\end{split}
118+
$$
119+
120+
then the terms $Q_j$ are [independent](/D/ind) and each term $Q_j$ follows a [chi-squared distribution](/D/chi2) with $r_j$ degrees of freedom:
121+
122+
$$ \label{eq:cochran-p2}
123+
Q_j \sim \chi^2(r_j), \; j = 1, \ldots, m \; .
124+
$$
125+
126+
Let $U$ be the $n \times 1$ column vector of all observations
127+
128+
$$ \label{eq:U}
129+
U = \left[ \begin{matrix} u_1 \\ \vdots \\ u_k \end{matrix} \right]
130+
$$
131+
132+
where the group-wise $n_i \times 1$ column vectors are
133+
134+
$$ \label{yi}
135+
u_1 = \left[ \begin{matrix} (y_{1,1}-\mu)/\sigma \\ \vdots \\ (y_{1,n_1}-\mu)/\sigma \end{matrix} \right], \quad \ldots, \quad u_k = \left[ \begin{matrix} (y_{k,1}-\mu)/\sigma \\ \vdots \\ (y_{k,n_k}-\mu)/\sigma \end{matrix} \right] \; .
136+
$$
137+
138+
Then, we observe that the sum in \eqref{eq:sum-Uij-s2} can be represented in the form of \eqref{eq:cochran-p1} using the matrices
139+
140+
$$ \label{eq:sum-Uij-s3-Bj}
141+
\begin{split}
142+
B^{(1)} &= I_n - \mathrm{diag}\left( \frac{1}{n_1} J_{n_1}, \; \ldots, \; \frac{1}{n_K} J_{n_K} \right) \\
143+
B^{(2)} &= \mathrm{diag}\left( \frac{1}{n_1} J_{n_1}, \; \ldots, \; \frac{1}{n_K} J_{n_K} \right) - \frac{1}{n} J_n \\
144+
B^{(2)} &= \frac{1}{n} J_n
145+
\end{split}
146+
$$
147+
148+
where $J_n$ is an $n \times n$ matrix of ones and $\mathrm{diag}\left( A_1, \ldots, A_n \right)$ denotes a block-diagonal matrix composed of $A_1, \ldots, A_n$. The matrices in \eqref{eq:sum-Uij-s3-Bj} fulfill $B^{(1)} + B^{(2)} + B^{(3)} = I_n$ and their ranks are given by:
149+
150+
$$ \label{eq:sum-Uij-s3-Bj-rk}
151+
\begin{split}
152+
\mathrm{rank}\left( B^{(1)} \right) &= n-k \\
153+
\mathrm{rank}\left( B^{(2)} \right) &= k-1 \\
154+
\mathrm{rank}\left( B^{(3)} \right) &= 1 \; .
155+
\end{split}
156+
$$
157+
158+
Let's write down the [explained sum of squares](/D/ess) and the [residual sum of squares](/D/rss) for [one-way analysis of variance](/D/anova1) as
159+
160+
$$ \label{eq:ess-rss}
161+
\begin{split}
162+
\mathrm{ESS} &= \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( \bar{y}_i - \bar{y} \right)^2 \\
163+
\mathrm{RSS} &= \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( y_{ij} - \bar{y}_i \right)^2 \; .
164+
\end{split}
165+
$$
166+
167+
Then, using \eqref{eq:sum-Uij-s2}, \eqref{eq:cochran-p1}, \eqref{eq:cochran-p2}, \eqref{eq:sum-Uij-s3-Bj} and \eqref{eq:sum-Uij-s3-Bj-rk}, we find that
168+
169+
$$ \label{eq:ess-rss-dist}
170+
\begin{split}
171+
\frac{\mathrm{ESS}}{\sigma^2} = \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( \frac{\bar{y}_i - \bar{y}}{\sigma} \right)^2 &= Q_2 = U^\mathrm{T} B^{(2)} U \sim \chi^2(k-1) \\
172+
\frac{\mathrm{RSS}}{\sigma^2} = \sum_{i=1}^{k} \sum_{j=1}^{n_i} \left( \frac{y_{ij} - \bar{y}_i}{\sigma} \right)^2 &= Q_1 = U^\mathrm{T} B^{(1)} U \sim \chi^2(n-k) \; .
173+
\end{split}
174+
$$
175+
176+
Because $\mathrm{ESS}/\sigma^2$ and $\mathrm{RSS}/\sigma^2$ are also independent by \eqref{eq:cochran-p2}, the F-statistic from \eqref{eq:anova1-f} is equal to the ratio of two independent [chi-squared distributed](/D/chi2) [random variables](/D/rvar) divided by their degrees of freedom
177+
178+
$$ \label{eq:anova1-f-ess-tss}
179+
\begin{split}
180+
F &= \frac{(\mathrm{ESS}/\sigma^2)/(k-1)}{(\mathrm{RSS}/\sigma^2)/(n-k)} \\
181+
&= \frac{\mathrm{ESS}/(k-1)}{\mathrm{RSS}/(n-k)} \\
182+
&= \frac{\frac{1}{k-1} \sum_{i=1}^{k} \sum_{j=1}^{n_i} (\bar{y}_i - \bar{y})^2}{\frac{1}{n-k} \sum_{i=1}^{k} \sum_{j=1}^{n_i} (y_{ij} - \bar{y}_i)^2} \\
183+
&= \frac{\frac{1}{k-1} \sum_{i=1}^{k} n_i (\bar{y}_i - \bar{y})^2}{\frac{1}{n-k} \sum_{i=1}^{k} \sum_{j=1}^{n_i} (y_{ij} - \bar{y}_i)^2}
184+
\end{split}
185+
$$
186+
187+
which, [by definition of the F-distribution](/D/f), is distributed as:
188+
189+
$$ \label{eq:anova1-f-qed}
190+
F \sim \mathrm{F}(k-1, n-k), \; \text{if} \; H_0 \; .
191+
$$

‎P/anova1-ols.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
layout: proof
3+
mathjax: true
4+
5+
author: "Joram Soch"
6+
affiliation: "BCCN Berlin"
7+
e_mail: "joram.soch@bccn-berlin.de"
8+
date: 2022-11-06 11:18:00
9+
10+
title: "Ordinary least squares for one-way analysis of variance"
11+
chapter: "Statistical Models"
12+
section: "Univariate normal data"
13+
topic: "Analysis of variance"
14+
theorem: "Ordinary least squares for one-way ANOVA"
15+
16+
sources:
17+
18+
proof_id: "P369"
19+
shortcut: "anova1-ols"
20+
username: "JoramSoch"
21+
---
22+
23+
24+
**Theorem:** Given the [one-way analysis of variance](/D/anova1) assumption
25+
26+
$$ \label{eq:anova1}
27+
y_{ij} = \mu_i + \varepsilon_{ij}, \; \varepsilon_{ij} \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, \sigma^2), \; i = 1, \ldots, k, \; j = 1, \dots, n_i \; ,
28+
$$
29+
30+
the parameters minimizing the [residual sum of squares](/D/rss) are given by
31+
32+
$$ \label{eq:anova1-ols}
33+
\hat{\mu}_i = \bar{y}_i
34+
$$
35+
36+
where $\bar{y}_i$ is the [sample mean](/D/mean-samp) of all observations in [group](/D/anova1) $i$:
37+
38+
$$ \label{eq:mean-samp}
39+
\hat{\mu}_i = \bar{y}_i = \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij} \; .
40+
$$
41+
42+
43+
**Proof:** The [residual sum of squares](/D/rss) for this model is
44+
45+
$$ \label{eq:rss}
46+
\mathrm{RSS}(\mu) = \sum_{i=1}^{k} \sum_{j=1}^{n_i} \varepsilon_{ij}^2 = \sum_{i=1}^{k} \sum_{j=1}^{n_i} (y_{ij} - \mu_i)^2
47+
$$
48+
49+
and the derivatives of $\mathrm{RSS}$ with respect to $\mu_i$ are
50+
51+
$$ \label{eq:rss-der}
52+
\begin{split}
53+
\frac{\mathrm{d}\mathrm{RSS}(\mu)}{\mathrm{d}\mu_i} &= \sum_{j=1}^{n_i} \frac{\mathrm{d}}{\mathrm{d}\mu_i} (y_{ij} - \mu_i)^2 \\
54+
&= \sum_{j=1}^{n_i} 2 (y_{ij} - \mu_i) (-1) \\
55+
&= 2 \sum_{j=1}^{n_i} (\mu_i - y_{ij}) \\
56+
&= 2 n_i \mu_i - 2 \sum_{j=1}^{n_i} y_{ij} \quad \text{for} \quad i = 1, \ldots, k \; .
57+
\end{split}
58+
$$
59+
60+
Setting these derivatives to zero, we obtain the estimates of $\mu_i$:
61+
62+
$$ \label{eq:rss-der-zero}
63+
\begin{split}
64+
0 &= 2 n_i \hat{\mu}_i - 2 \sum_{j=1}^{n_i} y_{ij} \\
65+
\hat{\mu}_i &= \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij} \quad \text{for} \quad i = 1, \ldots, k \; .
66+
\end{split}
67+
$$

0 commit comments

Comments
 (0)