Merge pull request #170 from StatProofBook/master

update to master

Author: JoramSoch

D/prob-marg.md

@@ -47,4 +47,4 @@ $$ \label{eq:prob-marg-cont}
 p(A) = \int_{\mathcal{X}} p(A,x) \, \mathrm{d}x \; .
 $$
 
-The law of marginal probability can be motivated from the [law of total probability](/D/prob-tot) which follows from the [Kolmogorov axioms of probability](/D/prob-ax).
+The law of marginal probability can be motivated from the [law of total probability](/P/prob-tot) which follows from the [Kolmogorov axioms of probability](/D/prob-ax).

D/vb.md

@@ -47,7 +47,7 @@ username: "JoramSoch"
 ---
 
 
-**Definition:** Let $m$ be a [generative model](/D/gm) with model parameters $\theta$ implying the [likelihood function](/D/lf) $p(y \vert \theta, m)$ and [prior distribution](/D/prior) $p(\theta \vert m)$. Then, a Variational Bayes treatment of $m$, also referred to as "approximate inference" or "variational inference", consists in
+**Definition:** Let $m$ be a [generative model](/D/gm) with [model parameters](/D/para) $\theta \in \Theta$ implying the [likelihood function](/D/lf) $p(y \vert \theta, m)$ and [prior distribution](/D/prior) $p(\theta \vert m)$. Then, a Variational Bayes treatment of $m$, also referred to as "approximate inference" or "variational inference", consists in
 
 <br>
 1) constructing an approximate [posterior distribution](/D/post)
@@ -60,14 +60,14 @@ $$
 2) evaluating the [variational free energy](/D/vblme)
 
 $$ \label{eq:FE}
-F_q(m) = \int q(\theta) \log p(y|\theta,m) \, \mathrm{d}\theta - \int q(\theta) \frac{q(\theta)}{p(\theta|m)} \, \mathrm{d}\theta
+\mathrm{F}_m[q(\theta)] = \int_{\Theta} q(\theta) \log \frac{p(\theta \vert y, m)}{q(\theta)} \, \mathrm{d}\theta
 $$
 
 <br>
 3) and maximizing this function with respect to $q(\theta)$
 
 $$ \label{eq:VB}
-\hat{q}(\theta) = \operatorname*{arg\,max}_{q} F_q(m) \; .
+\hat{q}(\theta) = \operatorname*{arg\,max}_{q} \mathrm{F}_m[q(\theta)]
 $$
 
 for [Bayesian inference](/P/bayes-th), i.e. obtaining the [posterior distribution](/D/post) (from eq. \eqref{eq:VB}) and approximating the [marginal likelihood](/D/ml) (by plugging eq. \eqref{eq:VB} into eq. \eqref{eq:FE}).

D/vblme.md

@@ -34,20 +34,10 @@ username: "JoramSoch"
 ---
 
 
-**Definition:** Let $m$ be a [generative model](/D/gm) with model parameters $\theta$ implying the [likelihood function](/D/lf) $p(y \vert \theta, m)$ and [prior distribution](/D/prior) $p(\theta \vert m)$. Moreover, assume an [approximate](/D/vb) [posterior distribution](/D/post) $q(\theta)$. Then, the [Variational Bayesian](/D/vb) [log model evidence](/D/lme), also referred to as the "negative free energy", is the expectation of the [log-likelihood function](/D/llf) with respect to the approximate posterior, minus the [Kullback-Leibler divergence](/D/kl) between approximate posterior and the prior distribution:
+**Definition:** Let $m$ be a [generative model](/D/gm) with [model parameters](/D/para) $\theta \in \Theta$ implying the [likelihood function](/D/lf) $p(y \vert \theta, m)$ and [prior distribution](/D/prior) $p(\theta \vert m)$. Moreover, assume an [approximate](/D/vb) [posterior distribution](/D/post) $q(\theta)$. Then, the [Variational Bayesian](/D/vb) [log model evidence](/D/lme), also referred to as the "variational free energy", is defined as the expected logarithm of the likelihood function, divided by the approximate posterior:
 
 $$ \label{eq:vbLME}
-\mathrm{vbLME}(m) = \left\langle \log p(y \vert \theta, m) \right\rangle_{q(\theta)} - \mathrm{KL}\left[q(\theta) || p(\theta \vert m)\right]
+\mathrm{vbLME}(m) = \mathrm{F}_m[q(\theta)] = \int_{\Theta} q(\theta) \log \frac{p(\theta \vert y, m)}{q(\theta)} \, \mathrm{d}\theta \; .
 $$
 
-where
-
-$$ \label{eq:ELL}
-\left\langle \log p(y \vert \theta, m) \right\rangle_{q(\theta)} = \int q(\theta) \log p(y \vert \theta, m) \, \mathrm{d}\theta
-$$
-
-and
-
-$$ \label{eq:KL}
-\mathrm{KL}\left[q(\theta) || p(\theta \vert m)\right] = \int q(\theta) \log \frac{q(\theta)}{p(\theta \vert m)} \, \mathrm{d}\theta  \; .
-$$
+The variational free energy can be decomposed into the [difference between log model evidence and KL divergence of approximate from true posterior](/P/fren-dec) or, alternatively, as the [difference of expected log-likelihood and KL divergence of approximate posterior from prior](/P/fren-dec).

I/PbA.md

@@ -16,7 +16,7 @@ title: "Proof by Author"
 
 - [Posterior predictive distribution is a marginal distribution of the joint likelihood](/P/postpred-jl)
 
-### JoramSoch (479 proofs)
+### JoramSoch (480 proofs)
 
 - [Accuracy and complexity for Bayesian linear regression](/P/blr-anc)
 - [Accuracy and complexity for Bayesian linear regression with known covariance](/P/blrkc-anc)
@@ -472,6 +472,7 @@ title: "Proof by Author"
 - [The regression line goes through the center of mass point](/P/slr-comp)
 - [The residuals and the covariate are uncorrelated in simple linear regression](/P/slr-rescorr)
 - [The sum of residuals is zero in simple linear regression](/P/slr-ressum)
+- [The variational free energy is a lower bound on the log model evidence](/P/fren-lme)
 - [Transformation matrices for ordinary least squares](/P/mlr-mat)
 - [Transformation matrices for simple linear regression](/P/slr-mat)
 - [Transposition of a matrix-normal random variable](/P/matn-trans)

I/PbN.md

@@ -522,3 +522,4 @@ title: "Proof by Number"
 | P514 | jens-ineq | [Jensen's inequality](/P/jens-ineq) | JoramSoch | 2025-09-25 |
 | P515 | kl-nonneg3 | [Non-negativity of the Kullback-Leibler divergence](/P/kl-nonneg3) | JoramSoch | 2025-09-25 |
 | P516 | fren-dec | [Decompositions of the variational free energy](/P/fren-dec) | JoramSoch | 2025-09-25 |
+| P517 | fren-lme | [The variational free energy is a lower bound on the log model evidence](/P/fren-lme) | JoramSoch | 2025-09-25 |

I/PbT.md

@@ -546,6 +546,7 @@ title: "Proof by Topic"
 - [The regression line goes through the center of mass point](/P/slr-comp)
 - [The residuals and the covariate are uncorrelated in simple linear regression](/P/slr-rescorr)
 - [The sum of residuals is zero in simple linear regression](/P/slr-ressum)
+- [The variational free energy is a lower bound on the log model evidence](/P/fren-lme)
 - [Transformation matrices for ordinary least squares](/P/mlr-mat)
 - [Transformation matrices for simple linear regression](/P/slr-mat)
 - [Transitivity of Bayes Factors](/P/bf-trans)

P/entcross-conv.md

@@ -33,13 +33,13 @@ username: "JoramSoch"
 ---
 
 
-**Theorem:** The [cross-entropy](/D/ent-cross) is convex in the [probability distribution](/D/dist) $q$, i.e.
+**Theorem:** The [cross-entropy](/D/ent-cross) is convex in the second [probability distribution](/D/dist), i.e.
 
 $$ \label{eq:ent-cross-conv}
 \mathrm{H}[p,\lambda q_1 + (1-\lambda) q_2] \leq \lambda \mathrm{H}[p,q_1] + (1-\lambda) \mathrm{H}[p,q_2]
 $$
 
-where $p$ is a fixed and $q_1$ and $q_2$ are any two probability distributions and $0 \leq \lambda \leq 1$.
+where $p$ is a fixed and $q_1$ and $q_2$ are any two [probability mass functions](/D/pmf) and $0 \leq \lambda \leq 1$.
 
 
 **Proof:** The [relationship between Kullback-Leibler divergence, entropy and cross-entropy](/P/kl-ent) is:
@@ -51,7 +51,7 @@ $$
 Note that the [KL divergence is convex](/P/kl-conv) in the pair of [probability distributions](/D/dist) $(p,q)$:
 
 $$ \label{eq:kl-conv}
-\mathrm{KL}[\lambda p_1 + (1-\lambda) p_2||\lambda q_1 + (1-\lambda) q_2] \leq \lambda \mathrm{KL}[p_1||q_1] + (1-\lambda) \mathrm{KL}[p_2||q_2]
+\mathrm{KL}[\lambda p_1 + (1-\lambda) p_2||\lambda q_1 + (1-\lambda) q_2] \leq \lambda \mathrm{KL}[p_1||q_1] + (1-\lambda) \mathrm{KL}[p_2||q_2] \; .
 $$
 
 A special case of this is given by

P/fren-dec.md

@@ -48,7 +48,7 @@ $$ \label{eq:vb-fe3}
 \mathrm{F}[q(\theta)] = \left\langle \log p(y,\theta) \right\rangle_{q(\theta)} - \mathrm{h}[q(\theta)]
 $$
 
-where $p(y \vert m) = p(y)$ is the [marginal likelihood](/D/ml), $\left\langle \cdot \right\rangle_{p(x)}$ denotes an [expectation](/D/mean) with respect to the []density](/D/pdf) $p(x)$, $\mathrm{KL}[\cdot \vert\vert \cdot]$ denotes the [Kullback-Leibler divergence](/D/kl) and $\mathrm{h}[\cdot]$ denotes the [differential entropy](/D/dent).
+where $p(y \vert m) = p(y)$ is the [marginal likelihood](/D/ml), $\left\langle \cdot \right\rangle_{p(x)}$ denotes an [expectation](/D/mean) with respect to the [density](/D/pdf) $p(x)$, $\mathrm{KL}[\cdot \vert\vert \cdot]$ denotes the [Kullback-Leibler divergence](/D/kl) and $\mathrm{h}[\cdot]$ denotes the [differential entropy](/D/dent).
 
 
 **Proof:** The [log model evidence](/D/lme) is defined as

P/glm-llrmi.md

@@ -40,7 +40,7 @@ $$ \label{eq:m0}
 m_0: \; Y = E_0, \; E_0 \sim \mathcal{MN}(0, I_n, \Sigma_0) \; .
 $$
 
-Then, the [log-likelihood ratio](/D/llr) of $m_1$ vs. $m_0$ is equal to the estimated [mutual information](/D/mi) of $X$ and $Y$:
+Then, the [log-likelihood ratio](/D/llr) of $m_1$ vs. $m_0$ is equal to the [estimated](/D/est) [mutual information](/D/mi) of $X$ and $Y$:
 
 $$ \label{eq:glm-llrmi}
 \ln \Lambda_{10} = \hat{I}(X,Y) \; .

P/glm-mi.md

@@ -63,7 +63,7 @@ Since $X$ is [constant](/D/const) and thus only has [one possible value](/D/samp
 $$ \label{eq:dent-cond-const}
 \begin{split}
 \mathrm{h}(Y|X)
-&= \int_{z \in \mathcal{X}} p(z) \cdot \mathrm{h}(Y|z) \, \mathrm{d}z \\
+&= \int_{x \in \mathcal{X}} p(x) \cdot \mathrm{h}(Y|x) \, \mathrm{d}x \\
 &= p(X) \cdot \mathrm{h}(Y|X) \\
 &= \mathrm{h}\left[ p(Y|X,B,\Sigma_1) \right] \; .
 \end{split}